my_backup: full Utility-Reliability-Standards compliance audit

Trigger: Spun off from my_backup-more-critical-issues — Talbot confirmed the remaining Utility-Reliability-Standards.md items are “warranted — need world-class robustness.” my_backup is the reference implementation the standard was derived from, so a shallow pass wasn’t acceptable; every item was verified against real code and real runs, not assumed.

Findings and fixes

7 real gaps found and fixed; 2 items confirmed already compliant.

1. Structured Logging — backup.log had backup_count: 3 (spec = 5, bumped). virus_scan.log/system_image.log/check_backups.log were raw print() redirected by cron shell — no timestamps, levels, or rotation, unbounded growth. Converted all 3 to utils.setup_logging (same pattern pipeline.py uses). Verified live: ran check_backups and create_system_image --dry-run, confirmed [TS] LEVEL msg + RUN STARTED/RESULT format.

2. Alerts on Failure — PASS, no change. notify_manager tiering (email+Telegram on CRITICAL) confirmed working via a live check_backups run (email delivered). Subject format differs cosmetically from the spec’s literal template but conveys severity correctly — editing the shared notify_manager dependency for a cosmetic match would ripple into every other utility for no functional gain.

3. Weekly Status Report — send_status_report.py only parsed backup.log. Generalized to loop over all 4 utility logs, roll up worst status into one subject (Calm/Loud/Critical). Verified against real log data: correctly reported mixed per-utility statuses in one summary.

4. Independent Verification — already done in the parent task (check_backups.py scheduled + job_monitor-registered).

5. Dry-Run Mode — most significant finding — --dry-run set ctx.config['dry_run'] = True and logged a warning banner, but no call site in tasks.py ever read the flag. Every kopia snapshot/policy/maintenance command, robocopy/rclone mirror, and 7z archive operation ran for real regardless of --dry-run. The flag had existed long enough to be trusted without ever being verified end-to-end. Added ctx.dry_run property, gated every mutating call site (pre-hooks, file syncs, kopia policy/snapshot/maintenance, robocopy/rclone mirrors, 7z create/verify/cleanup/upload). Verified with a full uv run my_backup --dry-run run — every mutating step logged [DRY RUN] Would..., nothing executed.

6. Test Suite — tests/test_backups.py has real 20-check coverage (incl. a restore test) but wasn’t pytest-discoverable — uv run pytest tests/ silently collected 0 tests. Deeper cause: pytest was never declared as a project dependency, so uv run pytest was falling through to a global pytest on ~/.local/bin PATH. Added pytest via uv add --dev, added tests/test_pytest_wrapper.py (reflects over BackupTests.test_*, auto-syncs with future checks). Verified: 18 real checks now collected, 10 passed / 8 failed (failures are genuine — this WSL sandbox has no Windows/D: drive to test against, not a bug).

7. Config as SSOT — update_static_zips.py and debug_kopia.py hardcoded Windows paths. Added static_zips: config block; both scripts now load from config.yaml. Verified loaded values match and grep for hardcoded paths is clean.

8. Graceful Failure — PASS, no change. Confirmed each tasks.py function catches its own errors into ctx.failures/ctx.warnings independently; the pipeline’s outer try/except is only a last-resort catch for truly uncaught exceptions. Tri-state SUCCESS/WARNING/FAILED confirmed correct, not accidental.

9. Human Escalation Thresholds — nothing existed. Added escalation_thresholds: to config.yaml (snapshot_size_drop_pct: 15, zero_byte_snapshot: true). Extended check_backups.py::check_snapshots() to compare latest vs previous snapshot size (rootEntry.summ.size — already present in the JSON it was querying but discarding) and fire CRITICAL on breach. Verified with monkeypatched kopia output: 50% drop and 0-byte both correctly fired, a 5% drop correctly did not.

10. Annual Fire Drill — manual_reminders.start_of_year was a generic yearly checklist with no actual recovery-path test. Added an explicit fire-drill task naming the real kopia snapshot restore command plus spot-check and cleanup steps. Verified rendered output via send_manual_reminders.tasks_to_markdown().

Commit: my_backup 633383b (14 files changed).

Lessons captured

2 lessons added to my_backup/LESSONS.md:

• A CLI flag that sets a config value proves nothing about whether anything reads it — verify the read side, not just the write side.
• A passing uv run pytest can mean “0 tests found,” not “tests passed” — check the collected count, and confirm the test runner itself is actually a declared project dependency.