Rate Scanner — Project Status
Section titled “Rate Scanner — Project Status”Last Updated: 2026-04-03
Current Phase: Phase 4 Complete → Phase B Next
Section titled “Current Phase: Phase 4 Complete → Phase B Next”The Rate Intelligence Engine is live and running. Daily pipeline scrapes 8 sources, stores to SQLite, exports rates.json, rebuilds and deploys the Benefit Snapshot widget to Cloudflare Pages.
Completed Phases
Section titled “Completed Phases”Phase 1 — Foundation
Section titled “Phase 1 — Foundation”- YAML manifest (
rates.yaml) — institutions + products + rates - Pydantic models with validation
- SQLite schema — insert-only rates, change detection
- CLI:
rate-scanner list,rate-scanner run,rate-scanner export - BoC Valet API scraper — prime rate + overnight target
Phase 2 — Scrapers + Change Detection
Section titled “Phase 2 — Scrapers + Change Detection”-
highinterestsavings.caaggregator scraper (16 products, flaggedneeds_verification) - TD Bank scraper (Playwright) — 0.010% / 0.450% confirmed
- EQ Bank scraper (curl) — 10 products, 2.35%–3.65%
- Change detection (
diff.py) — logs when rates move
Phase 3 — Export + Widget
Section titled “Phase 3 — Export + Widget”-
rate-scanner export→data/rates.json - Benefit Snapshot widget (
benefit-snapshots/) — self-contained HTML/JS/CSS - Deployed to
https://benefit-snapshots.pages.dev
Phase 4 — Big 6 Bank Scrapers
Section titled “Phase 4 — Big 6 Bank Scrapers”- RBC scraper (curl_cffi) — 0.550%
- BMO scraper (curl_cffi) — 0.010%
- Scotiabank scraper (curl_cffi) — 0.400%
- Tangerine scraper (curl_cffi) — 0.700%
- CIBC — Akamai-blocked;
needs_verification: true, accepted as reference_only -
reference_only: trueflag — Big 6 + Tangerine tagged; excluded from recommendations, used only for inertia baseline - Widget updated — recommendation candidates filter out reference_only products
- Daily cron —
0 10 * * *= 06:00 ET;scripts/daily-run.sh→ scrape → export → build → deploy - Committed through
794775e
Active Work
Section titled “Active Work”None — Phase 4 complete. Daily pipeline running autonomously.
Key metrics (2026-04-03):
- 8 scrapers in daily pipeline
- 16 products exported (10 recommendable, 6 reference-only)
- Inertia baseline: 0.010% (TD Bank)
- Best HISA: 2.75% (EQ Bank) → delta $2,740/yr per $100k
Next (Phase B)
Section titled “Next (Phase B)”In priority order:
- Hassle-engine scaffold — core algorithm: read
rate_changesfrom DB, evaluate user threshold (e.g. “alert me if delta > 0.5%”), return alert payload. No delivery yet. - Cancer50Pledge.ca Phase B — custom domain (
cancer50pledge.ca), pledge form with email capture, analytics - Benefit Snapshot custom domain —
snapshot.mybetterrates.comorrate.mybetterrates.com - MBR consumer site — the actual mybetterrates.com web app (SvelteKit)
- National Bank scraper — last of the Big 6 not yet scraped
Architecture Decisions (Locked)
Section titled “Architecture Decisions (Locked)”| Decision | Choice | Reason |
|---|---|---|
| Language | Python + uv | Clean deps, fast install, matches GlobalDevRules |
| Storage | SQLite (insert-only) | Audit trail, no deletes, perfect for rate history |
| Scraping | curl_cffi first, Playwright fallback | curl_cffi beats Playwright on TLS fingerprinting for most banks |
| Deploy | Cloudflare Pages | Free tier, edge CDN, wrangler CLI |
reference_only flag | In YAML manifest only (not DB) | Static property — doesn’t change over time |
| Inertia baseline | TD Bank 0.010% (Big 6 worst) | Confirmed systemic: BMO also 0.010% |
Feedback
Section titled “Feedback”(leave blank — AI acts on feedback here before proceeding)