My_Backup System

Project: D:\FSS\Software\Utils\PythonUtils\my_backup\README.md | WSL: /mnt/d/FSS/Software/Utils/PythonUtils/my_backup/

1. Overview

This is a “World Class” backup solution using Kopia, a modern, enterprise-grade tool wrapped in a custom automation script. It is superior in every way to my previous cycling of ZIP files, for efficiency, risk management, and speed.

Key Benefits:

• Split-Brain Architecture: It intelligently separates “Critical/Active” data (Documents, Code, Databases) from “Bulk/Static” data (Photos, Videos, PDFs) that don’t change much.
• Free Cloud Backup: By isolating critical data, we can fit the most important business files (~5GB) onto free cloud storage (Google Drive), while keeping massive files (~1TB) on local drives (NAS/USB).
• Encrypted & Deduplicated: All data is military-grade encrypted.

---

2. The Strategy: Efficient Folder Structure

The secret to a fast, free, and robust backup system is how you organize your files. We separate data into two categories:

1. Active (Critical): Small files, text documents, code, financial data. Things that change often and are vital to life/business.
2. Static (Bulk): Large files, photos, videos, software installers, finished project archives. Things that rarely change and take up massive space.

The Recommended Structure

To make this automation work, create a folder named Static inside any main category. The system is trained to automatically treat that folder differently.

D:\ (Data Drive)
├── FSS (Business)
│   ├── Accounting           [ACTIVE] -> Goes to Cloud + Local
│   ├── Documents            [ACTIVE] -> Goes to Cloud + Local
│   └── Static               [STATIC] -> Local Only (NAS/USB)
│       ├── Software_ISOs        (Huge files)
│       └── Old_Project_Archives
│
└── Ta (Personal)
    ├── Financials           [ACTIVE] -> Goes to Cloud + Local
    ├── Writings             [ACTIVE] -> Goes to Cloud + Local
    └── Static               [STATIC] -> Local Only (NAS/USB)
        ├── Photos               (200GB+)
        └── Videos               (500GB+)

The Rule: If a folder grows larger than 500MB and contains content that doesn’t change (like vacation photos), move it into Static.

If you don’t have lots of static content (photos, videos, PDFs, etc.), you can implement a simpler single repo solution, for Active/Critical data.

---

3. Quick Start: Setup Guide

Follow these steps to set up the backup system on a new computer.

Step 1: Install Software

1. Install Python (Version 3.14 or newer):
   • Download: python.org/downloads
   • IMPORTANT: Check the box “Add Python to PATH” during install.
2. Install ‘uv’ (Script Runner):
   • Open Command Prompt (cmd) and type: pip install uv
3. Install Kopia CLI (The Engine):
   • Download Windows 64-bit CLI (zip) from: github.com/kopia/kopia/releases
   • Extract and move kopia.exe to: C:\Program Files\KopiaCLI\kopia.exe
   • Add to System PATH: Search Windows for “Environment Variables” -> Path -> New -> Paste C:\Program Files\KopiaCLI\.

Step 2: Prepare the Script Folder

1. Create a folder (e.g., D:\Backups\Scripts).
2. Place my_backup.py and my_backup.yaml here.
3. Create a file named .env and add your password/paths:

   KOPIA_PASSWORD=MySecretPassword123!
   GDRIVE_PATH=G:/My Drive

   Warning: If you lose this password, your data is lost forever.

Step 3: Initialize Repositories (One Time Only)

Because we use the “Split-Brain” strategy, we initialize two separate databases. Run these commands in Command Prompt:

1. Create the Critical (Cloud) Repo:

kopia repository create filesystem --path=D:\bak\Kopia_Active --config-file=D:\bak\Kopia_Active\kopia.config

2. Create the Static (Local) Repo:

kopia repository create filesystem --path=D:\bak\Kopia_Static --config-file=D:\bak\Kopia_Static\kopia.config

Step 4: Set Retention Policies

Tell Kopia how long to keep history.

• Active Data (Keep 10 years):

  kopia policy set --global --keep-daily 10 --keep-weekly 10 --keep-monthly 12 --keep-annual 10 --config-file=D:\bak\Kopia_Active\kopia.config

• Static Data (Keep 3 years - saves space):

  kopia policy set --global --keep-daily 10 --keep-weekly 4 --keep-monthly 12 --keep-annual 3 --config-file=D:\bak\Kopia_Static\kopia.config

Step 5: Automate

Two schedulers, deliberately — so no single automation layer can fail silently.

Job	Trigger	Runs
Daily Backup	Windows Task Scheduler, 08:00 (\TaTasks\Daily Backup)	D:\FSS\Software\Utils\Windows\my_backup-daily-task.bat
Backup verification	WSL cron, 11:00, via cmd.exe	D:\FSS\Software\Utils\Windows\my_backup-check-task.bat
Full maintenance	WSL cron, 1st of month 05:00, via cmd.exe	D:\FSS\Software\Utils\Windows\my_backup-maintenance-task.bat
System image	Windows Task Scheduler, monthly (elevated)	run_system_image.bat
Virus scan / status report	WSL cron (WSL-native, no wrapper needed)	uv run virus_scan / uv run send_status_report

Every job reports its exit code to job_monitor, so one that stops running is detected within its max_age_hours window.

The wrapper .bat files are not optional. Anything touching Windows paths (D:, F:, J:, UNC) or the Windows kopia/robocopy/7z binaries must execute under Windows Python, and each wrapper pins an isolated UV_PROJECT_ENVIRONMENT off the shared drive. Scheduling one as a bare WSL uv run breaks it — see the repo’s README.md and LESSONS.md.

---

4. User Responsibilities

Daily

• Nothing. Runs automatically.
• Monitor: If the script detects missing critical files (e.g., QuickBooks hasn’t been saved in 7 days), the script will alert you.

Periodic

• External Drives: Ensure your backup drives (NAS/USB) are connected. The script automatically mirrors data to them when found.
• Housekeeping: If you dump a large amount of new media (photos/videos) onto your computer, ensure you put them inside a Static folder so they don’t clog up the Cloud backup.

---

5. How to Restore Files (KopiaUI)

Use the visual interface to browse and restore files.

1. Open KopiaUI (Download from Kopia.io).
2. Connect:
   • Choose Filesystem.
   • Path: Browse to either D:\bak\Kopia_Active (Docs) or D:\bak\Kopia_Static (Media).
   • Config File: You must point to the specific config file inside that folder (e.g., D:\bak\Kopia_Active\kopia.config).
   • Enter Password.
3. Mount: Click “Mount” next to a snapshot. It appears as a drive letter (Z:).
4. Restore: Copy files out using Windows Explorer.

---

6. Troubleshooting

• KopiaUI Won’t Open: Check the System Tray (near the clock). It starts minimized. Kill “KopiaUI” in Task Manager if stuck.
• Backup “Stalled” or “Huge”: If the Active backup is taking hours or is larger than 10GB, you likely put large video/photo files in a folder not named Static. Move them to Static or update the .kopiaignore file in that directory.
• File Locked Errors: If Kopia complains about open files (like .eco files), ensure the .kopiaignore file in the source folder includes **/*.eco or similar exclusions for temp files.
• “System cannot find the file specified” error was caused by broken symbolic links for .kopiaignore files in source directories. Robust self-repair steps were added to my_backup.py that automatically detects and fixes these links by replacing them with the correct files.