Z2 Mini Server — Context for AI Assistants

Purpose: This document provides technical context about Gabriel's existing self-hosted infrastructure on his HP Z2 Mini G5 server. Future AI assistants should read this before suggesting changes or new projects to avoid conflicting with established configuration.

⚠️ MAJOR UPDATE — May 2026: Caddy reverse proxy added. Read 17-caddy.md. A single Caddy container (locally built = stock caddy + the caddy-dns/cloudflare plugin, network_mode: host, bound 100.67.235.68:443+:80) is now the only HTTPS ingress. Every web app — Immich, Navidrome, Homepage, Beszel, Radicale, Vaultwarden — was re-bound to 127.0.0.1:<port> only and is reached at https://<service>.z2mini.gabrielgabrie.com with an auto-renewing Let's Encrypt cert (ACME DNS-01 via the Cloudflare API — the box has no public inbound, so HTTP-01/TLS-ALPN-01 are unusable). The two old tailscale serve listeners (:443 Radicale, :8443 Vaultwarden) are retired. DNS for gabrielgabrie.com moved to Cloudflare (Hostinger still the registrar); *.z2mini wildcard A → 100.67.235.68; the Cloudflare API token lives in /data/docker/caddy/.env (mode 600). Tailscale --operator=gabriel is set; Tailscale global resolvers are 1.1.1.1+8.8.8.8 with "Override local DNS" (so tailnet devices resolve *.z2mini.gabrielgabrie.com — the housing router's DNS chokes on it). Throughout this document, any reference to tailscale serve, to apps being "bound to the Tailscale interface IP", or to http://z2mini:<port> / http://100.67.235.68:<port> URLs reflects the pre-Caddy state — translate to: app on 127.0.0.1:<port>, reach it via https://<svc>.z2mini.gabrielgabrie.com (or http://127.0.0.1:<port> on the box). 17-caddy.md and 10-system-reference.md are authoritative. Caddy's whole config is under /data/docker/caddy/ (rsync-able, folds into nightly backup) — unlike the old tailscale serve config which lived in tailscaled state.

Last updated: May 2026 (Radicale CalDAV/CardDAV server installed 2026-05-10 — replacing iCloud Calendar as the source of truth for personal scheduling. Single tomsquest/docker-radicale:3.7.2.0 container on port 5232, files-as-truth storage at /data/docker/radicale/data/collections/. Five iCloud calendars migrated via publish-as-iCal: 255 events across Events and Reminders, Deadlines, Conestoga Timetable, Bills, Work Schedule (alarms intentionally accepted-as-lost — iCloud's public-calendar feed strips VALARMs by design; Gabriel will re-add the ~10 important recurring alarms manually rather than use a CalDAV-to-CalDAV migration tool). iOS access requires HTTPS — iOS 18+ Calendar forces TLS regardless of in-app SSL toggle, so tailscale serve --bg http://127.0.0.1:5232 fronts Radicale with a Let's Encrypt cert at https://radicale.z2mini.gabrielgabrie.com/. Thunderbird on the laptop continues to use direct HTTP https://radicale.z2mini.gabrielgabrie.com. Two-week parallel-run window with iCloud Calendar still subscribed read-only; iCloud cut planned ~2026-05-24. iCloud subscription downgraded from 2 TB → 200 GB tier ($12.99 → $3.99/mo) after iCloud Photos cleared into Immich and iMessage attachments aggressively purged on-device. Post-purge iCloud usage ~150 GB: ~60 GB iMessage, ~85 GB Family Sharing pool, ~20 GB iPhone Backup. 50 GB tier was deliberately rejected — see Decisions table. Beszel system-metrics dashboard installed — hub+agent on port 8090, Tailscale-only, unix-socket communication, direct-to-Gmail SMTP alerts independent of msmtp's smartd channel. Homepage launcher dashboard installed — port 3000, Tailscale-only, native widgets for Immich + Navidrome + Tailscale + Docker socket integration for live container status. iCloud → Immich bulk migration complete as of 2026-05-05; 29,331 assets in Immich (~470 GB), /data/icloud-import deleted, icloudpd cookie still valid for incremental sync if desired. iCloud Photos was mass-deleted 2026-05-06 keeping only the ~100 most recent items via pyicloud-based ~/scripts/icloud-trim.py; ~31,288 of 31,298 deletions succeeded (10 transient failures), now sitting in iCloud's Recently Deleted (30-day soft-restore until ~2026-06-07). iOS Immich auto-backup enabled 2026-05-08 — Camera Roll → Immich is the daily-driver loop; new iPhone captures push automatically. GPU acceleration enabled on the Quadro T2000 — Immich ML container now uses CUDA via pre-signed linux-modules-nvidia-595-generic kernel modules + nvidia-container-toolkit. Earlier this month: Navidrome music streaming installed (port 4533, Samba [music] share, Subsonic clients play); Immich install with icloudpd + immich-go pipeline; system-state backup added; USB resilience improvements.)

---

How to Use This Document

If you're an AI assistant helping Gabriel with a project that involves his Z2 Mini server, read this entire document first. Pay particular attention to:

1. Reserved paths — directories already in use that should not be repurposed
2. Reserved ports — services already bound that should not be conflicted with
3. Existing services — what's running and how it's configured
4. Design decisions — choices already made and the reasoning, so you don't suggest reverting them
5. What's intentionally NOT installed — Gabriel evaluated and rejected several common tools

Before suggesting infrastructure changes, ask Gabriel to confirm rather than assume.

---

Hardware

• Model: HP Z2 Mini G5 Workstation
• CPU: Intel Core i9-10900K (10 cores, 20 threads, up to 5.3 GHz)
• RAM: 64GB DDR4-3200 SODIMM (2x32GB), upgraded from original 4GB
• GPU: NVIDIA Quadro T2000 (4GB GDDR5) — modest, only 4GB VRAM, prefer CPU-based workloads for ML
• Storage:
• Drive 1: Samsung MZVLB1T0HBLR-000H1 (1TB NVMe) — OS drive, mounted at /
• Drive 2: SK hynix PC601 HFS001TD9TNG-L2A0A (1TB NVMe) — data drive, mounted at /data
• Drive 3: Samsung 990 PRO 1TB NVMe SSD in an ASMedia ASM2462 USB enclosure — on-site backup drive, mounted at /mnt/backup (~916 GB usable; replaced the old Samsung T5 in May 2026; refurb verified genuine before deploy — see below)
• Drive 4 (normally NOT connected — lives at parents' house): Samsung Portable SSD T5 (500GB USB) — off-site rotation drive, reformatted as backup-offsite, mounted manually at /mnt/offsite for an off-site sync during family visits
• Network: Wi-Fi only (currently student housing network, ethernet planned for September 2026)

Operating System

• Distribution: Ubuntu Server 24.04 LTS
• Kernel: Linux 7.0.0-15-generic (as of last update)
• Hostname: z2mini
• Primary user: gabriel (in sudo and docker groups)
• Ubuntu Pro: Not yet activated (Gabriel intends to enable later)
• Boot: UEFI, GPT partition table

---

Storage Layout

/dev/nvme1n1 (Samsung 1TB) — OS drive
├── /dev/nvme1n1p1 → /boot/efi (1G, vfat)
└── /dev/nvme1n1p2 → /         (953G, ext4)   ← shows as "nvme1n1p2" in the Beszel UI (root fs can't be renamed there)

/dev/nvme0n1 (SK hynix 1TB) — Data drive
└── /dev/nvme0n1p1 → /data     (953G, ext4)   ← shows as "Data" in the Beszel UI

/dev/sdX (Samsung 990 PRO 1TB NVMe in an ASMedia ASM2462 USB enclosure) — on-site backup drive
└── /dev/sdX1 → /mnt/backup    (~916G usable, ext4, label=backup, reserved-blocks 0 via `tune2fs -m 0`)
                                UUID=8795cb2e-fe34-4543-b93b-dd45b642846f
                                fstab: UUID=8795cb2e-... /mnt/backup ext4 defaults,nofail 0 2
                                smartd / smartctl: /dev/disk/by-id/usb-ASMT_2462_NVME_2504178506CB-0:0 -d sntasmedia
                                ← shows as "Backup" in the Beszel UI
                                (replaced the old Samsung T5 in May 2026; rollback files /etc/fstab.pre-990pro,
                                 /etc/smartd.conf.pre-990pro, ~/scripts/backup-files.sh.pre-990pro)

/dev/sdY (Samsung Portable SSD T5 500GB USB) — OFF-SITE rotation drive — normally NOT connected
└── /dev/sdY1 → /mnt/offsite   (mounted MANUALLY only, ext4, label=backup-offsite, reserved-blocks 0)
                                UUID=2c8e8e38-f129-4823-a1db-1529d3296b44
                                NO fstab entry — `sudo mount /dev/disk/by-label/backup-offsite /mnt/offsite`
                                holds ~3 GB: files, music, db-dumps, service-config, system-state — NOT the Immich library

Reserved paths — do not use for new projects without consultation:

• /data/files/ — Samba share root, user files. Touched by SMB clients (Windows, iOS).
• /mnt/backup/current/ — nightly mirror tree: files/ (= /data/files/), immich/ (= Immich's library/ UPLOAD_LOCATION, incl. encoded-video/ and Immich's own backups/ DB dumps), music/ (= /data/music/), db-dumps/ (vaultwarden-db.sqlite3, beszel-data.db, beszel-auxiliary.db, navidrome.db — refreshed every run via host sqlite3 ".backup"), service-config/<svc>/ (each /data/docker/<svc>/ config — .env + compose + config, with bulk data / live DBs / container-owned-unreadable subdirs excluded).
• /mnt/backup/daily/YYYY-MM-DD/ — 7 daily hard-linked snapshots of current/ (dated by the run's START time).
• /mnt/backup/weekly/YYYY-MM-DD/ — 4 weekly hard-linked snapshots, Sunday only.
• /mnt/backup/system-state/ — System configuration backups (package lists incl. sqlite3 in manual-packages.txt, the /etc config tarballs, crontab).
• /mnt/backup/backup.log — Backup log file, rotated weekly.
• /mnt/backup/RECOVERY-README.md — Documented recovery procedure for full system rebuild (covers the service-config/ + db-dumps/ + Immich restore, the package-reinstall list with sqlite3, the Caddy docker compose build && up -d step).
• /mnt/offsite/ — mountpoint for the off-site T5 (backup-offsite). Created on the server; NO fstab entry — mounted manually only when the drive is plugged in. ~/scripts/backup-offsite.sh (MANUAL, not in cron) copies current/{files,music,db-dumps,service-config}/ + system-state/ onto it (~3 GB; the ~520 GB Immich library does NOT fit). Writes /mnt/offsite/offsite-backup.log.
• /data/.beszel/ and /mnt/backup/.beszel/ — empty placeholder dirs bind-mounted into the Beszel agent at /extra-filesystems/data__Data and /extra-filesystems/backup__Backup so the hub UI shows the disks as "Data" and "Backup". If the backup drive is swapped, recreate /mnt/backup/.beszel/ and docker restart beszel-agent.
• /data/docker/immich/ — Immich photo-storage service. Subdirs: library/ (= UPLOAD_LOCATION: library/ ~254 GB photos + upload/ ~188 GB + encoded-video/ ~68 GB + thumbs/ ~7.6 GB + backups/ ~979 MB = Immich's own nightly .sql.gz DB dumps + profile/; bind-mounted to container; rsync-mirrored to /mnt/backup/current/immich/ nightly with --no-owner --no-group), postgres/ (Postgres data dir, mode 700, owned by the container uid, MUST NOT be filesystem-rsync'd — instead Immich's built-in nightly DB auto-backup at 02:00 writes immich-db-backup-*.sql.gz into library/backups/ which the 03:00 rsync picks up; that setting must stay ENABLED), model-cache/ (ML model weights — NOT backed up, regenerable), plus docker-compose.yml, docker-compose.yml.pre-cuda, hwaccel.ml.yml, .env (mode 600, contains DB password) — these config files go to /mnt/backup/current/service-config/immich/.
• /data/docker/icloudpd/ — icloudpd one-shot iCloud Photos download tool. Subdirs: cookies/ (Apple session cookies, root-owned, ~2 month validity from last successful auth) and docker-compose.yml.
• /data/docker/navidrome/ — Navidrome music server. Subdirs: data/ (SQLite DB navidrome.db + WAL, cache, thumbnails, plugins; uid 1000 — navidrome.db is captured nightly via host sqlite3 ".backup" into /mnt/backup/current/db-dumps/navidrome.db, NEVER filesystem-rsync the live file; data/artwork/ + data/cache/ are NOT backed up — regenerable) plus docker-compose.yml and .env (these two go to service-config/navidrome/; the whole data/ dir is excluded from that rsync). Music files are NOT stored here — they live separately at /data/music/ so Samba can write into the music folder without exposing Navidrome's runtime data.
• /data/music/ — Music library; bind-mounted into Navidrome read-only as /music. Writable from \\z2mini\music Samba share. Owned gabriel:gabriel, mostly AAC 256 (.m4a) but FLAC also accepted (mixed-format library is fine — Navidrome and modern Subsonic clients decode both natively). rsync-mirrored to /mnt/backup/current/music/ nightly; also on the off-site T5.
• /data/docker/homepage/ — Homepage launcher dashboard. Subdirs: config/ (six YAML files: services.yaml, bookmarks.yaml, widgets.yaml, settings.yaml, docker.yaml, kubernetes.yaml) plus docker-compose.yml and .env (mode 600 — holds widget API tokens for Immich, Navidrome, Tailscale). Homepage 1.x only auto-creates settings.yaml and kubernetes.yaml on first boot — the rest are hand-written from the doc. .env + config/ are rsync'd nightly into /mnt/backup/current/service-config/homepage/ (and to the off-site T5).
• /data/docker/beszel/ — Beszel system-metrics hub + agent. Subdirs: data/ (hub SQLite DBs data.db + auxiliary.db — both captured nightly via host sqlite3 ".backup" into /mnt/backup/current/db-dumps/beszel-data.db + beszel-auxiliary.db, NEVER filesystem-rsync live; also holds data/id_ed25519 = the hub's SSH key, root-owned so the backup user can't read it — silently excluded, regenerable but regenerating it means re-pairing the one agent), socket/ (transient unix socket shared between hub and agent containers — never backed up), agent-data/ (agent fingerprint + buffer; rebuildable — not backed up) plus docker-compose.yml and .env (mode 600 — holds the agent SSH KEY and websocket TOKEN issued by the hub on system registration; these two go to service-config/beszel/, the whole data/ dir is excluded from that rsync). The agent's compose bind-mounts /data/.beszel:/extra-filesystems/data__Data:ro and /mnt/backup/.beszel:/extra-filesystems/backup__Backup:ro — the __<Label> suffix sets the displayed disk name in the hub UI ("Data" / "Backup"; root fs always shows as the device name nvme1n1p2). Rollback for that change: docker-compose.yml.pre-disklabels.
• /data/docker/radicale/ — Radicale CalDAV/CardDAV server. Subdirs: data/collections/ (calendars + contacts as a tree of .ics and .vcf files — files-as-truth, owned 2999:2999 after container's first-boot auto-chown via CHOWN capability, rsync-safe — load-bearing: this dir alone restores every calendar), config/ (Radicale config + htpasswd users file at mode 644 — bcrypt hashes are designed to be safe at the filesystem level) plus docker-compose.yml and .env (no secrets in .env, just version pin). All of .env + compose + config/ + data/collections/ are rsync'd nightly into /mnt/backup/current/service-config/radicale/ — no SQLite/pg_dump ceremony, files-as-truth; verified byte-for-byte; the calendars are also on the off-site T5. (The rsync also pulls Radicale's .Radicale.cache/ — harmless.) The HTTPS front-end is now the Caddy stack under /data/docker/caddy/ (in the backup), not tailscale serve config in /var/lib/tailscale/ (that's retired).
• /data/docker/vaultwarden/ — Vaultwarden (Bitwarden-compatible password server). Subdir: data/ (owned 1000:1000 — the container runs as user: 1000:1000): db.sqlite3 (+ -shm/-wal, WAL mode — the vault; NEVER filesystem-rsync the live file — captured nightly via host sqlite3 ".backup" into /mnt/backup/current/db-dumps/vaultwarden-db.sqlite3, which also goes to the off-site T5), rsa_key.pem (JWT signing key — load-bearing, rsync'd into service-config/vaultwarden/ with the DB dump; losing it logs every client out), attachments/ + sends/ (appear once any exist — rsync'd into service-config/vaultwarden/), icon_cache/ + tmp/ (rebuildable/transient — NOT backed up), config.json (only if the admin panel was used — and then it overrides .env; rsync'd along). Plus docker-compose.yml and .env (mode 600 — version pin + DOMAIN which is the WebAuthn/passkey relying-party ID, load-bearing; would also hold ADMIN_TOKEN / PUSH_* if those get added) — these go to service-config/vaultwarden/; the live data/db.sqlite3 and data/tmp/ are excluded from that rsync. The HTTPS front-end is the Caddy stack under /data/docker/caddy/ (in the backup), not tailscale serve config in /var/lib/tailscale/ (retired).
• /data/docker/caddy/ — Caddy HTTPS reverse proxy. Dockerfile, docker-compose.yml, Caddyfile, .env (mode 600 — the Cloudflare API token, load-bearing) — all rsync'd nightly into /mnt/backup/current/service-config/caddy/ (and the off-site T5). config/caddy/ (autosave) and data/caddy/ (ACME cert/key store) are NOT backed up — container-owned mode-700, unreadable by the backup user, and fully regenerable (Caddy re-issues every cert via the Cloudflare DNS-01 challenge on restore, given a valid .env token).
• /data/icloud-import/ — used as transient staging during the 2026-05 iCloud → Immich migration; deleted after immich-go confirmed ingest. Re-created if a future migration runs.
• /home/gabriel/scripts/backup-files.sh — the nightly 3 AM backup script (0 3 * * * in crontab -l). Rsync mirrors (/data/files/, Immich library/, /data/music/) + sqlite3 ".backup" of the 4 SQLite DBs into current/db-dumps/ + per-service config rsync into current/service-config/<svc>/ + hard-linked daily/weekly snapshots + prune + system-state. Depends on the host sqlite3 package. Old pre-990-PRO version preserved as backup-files.sh.pre-990pro.
• /home/gabriel/scripts/backup-offsite.sh — MANUAL, not in cron — the off-site rotation. Ritual (also in the script's header comment): plug the T5 into z2mini → sudo mount /dev/disk/by-label/backup-offsite /mnt/offsite → ~/scripts/backup-offsite.sh → sudo umount /mnt/offsite → unplug → carry off-site. Copies current/{files,music,db-dumps,service-config}/ + system-state/ onto the T5 (~3 GB). Does NOT copy current/immich/ (the ~520 GB photo library won't fit on the 500 GB T5 — accepted gap until a >1 TB off-site drive). Writes /mnt/offsite/offsite-backup.log.
• /home/gabriel/scripts/check-backup-mount.sh — hourly cron — emails an alert if /mnt/backup isn't mounted (catches USB disconnect between the daily 3 AM runs). Silent when fine.
• /home/gabriel/scripts/immich-go — immich-go v0.31.0 binary, used for bulk-importing files into Immich via its API (preserves storage-template behavior).
• /home/gabriel/scripts/icloudpd-bulk-download.sh — Wrapper script that runs the full icloudpd download command. Designed to run inside tmux new -s icloud-migration ....
• /home/gabriel/scripts/icloud-trim.py — pyicloud-based "keep newest N, delete the rest" tool. Used 2026-05-06 to mass-delete iCloud Photos down to ~100 items after the bulk migration verified. Requires the venv at /home/gabriel/scripts/icloud-trim-venv/. Idempotent (re-running picks up wherever iCloud's current state is). Invoke from a tmux session — long runs (hours) and ssh disconnect kills the script otherwise.
• /home/gabriel/scripts/icloud-trim-venv/ — Python venv with pyicloud for the trim script.
• /home/gabriel/.pyicloud/ — pyicloud's session cookie cache (separate from /data/docker/icloudpd/cookies/ which is icloudpd's). ~2-month validity.
• /home/gabriel/scripts/tailscale-token-age-check.sh — Weekly cron (Mondays 09:00) — emails the rotation procedure when Homepage's Tailscale API token is older than 75 days (90-day Tailscale cap; no permanent tokens exist; OAuth was evaluated and rejected — see decisions table). Reads /home/gabriel/.tailscale-token-rotated (rotation timestamp file, mode 600). Silent unless past threshold.
• /home/gabriel/.tailscale-token-rotated — Rotation timestamp file (mode 600) consumed by the age-check script.
• /home/gabriel/.immich-api-key — Immich admin API key (mode 600). Used by immich-go.
• /tmp/icloudpd-bulk.log — Live log of the running bulk download.

Available paths for new projects:

• /data/docker/<new-service>/ — Pattern for new Docker services. /data/docker/immich/ and /data/docker/icloudpd/ already follow this.
• /data/<new-project>/ — Use new subdirectories for new projects.
• /home/gabriel/scripts/ — Existing home for shell scripts and small one-off binaries (e.g., immich-go).
• /home/gabriel/<new-project>/ — Acceptable for non-Docker development work.

Checklist: bringing up a new service

Run through all of this when a new self-hosted service goes onto the Z2. Step 6 (the Homepage tile) is the one that's been missed — Radicale and Vaultwarden both went live and stayed invisible on the Homepage dashboard for days because nobody edited services.yaml. Homepage does not auto-discover containers; the tile is a manual edit, every time.

1. Pick a free port and bind it 127.0.0.1:PORT:PORT — not 0.0.0.0, not the tailnet IP. Check the "Reserved ports" table below — never reuse a bound one. (Pre-Caddy services bound the tailnet IP 100.67.235.68:PORT; that pattern is retired — everything is 127.0.0.1 behind Caddy now.)
2. Add a Caddy site block in /data/docker/caddy/Caddyfile: <svc>.z2mini.gabrielgabrie.com { log; reverse_proxy 127.0.0.1:PORT }, then docker compose -f /data/docker/caddy/docker-compose.yml exec caddy caddy reload --config /etc/caddy/Caddyfile. Caddy gets the Let's Encrypt cert via DNS-01 within ~30-60 s; the *.z2mini wildcard A record already resolves the name. See 17-caddy.md. (This replaces the old "front it with a tailscale serve listener" step.)
3. Follow the /data/docker/<service>/ layout. docker-compose.yml + .env (mode 600 if it holds a secret) + the data subdir. Mirror the upstream compose template verbatim; record the customizations in a numbered NN-<service>.md doc page and add that page to mkdocs.yml's nav:.
4. Update the reference tables (this doc and 10-system-reference.md): add the port to "Reserved ports", the container(s) to the Docker-managed services table, data paths to "Reserved paths", a row to the decisions table, and a "## N. " subsection to "Services Running" in this doc.
5. Plan the backup story in the service's doc page — pure rsync for files-as-truth (Radicale), pg_dump for Postgres (Immich), SQLite online .backup for SQLite (Navidrome, Beszel, Vaultwarden). Never filesystem-rsync a live database. (The Caddy site block + cert are under /data/docker/caddy/ like every other stack — nothing special to re-run after a rebuild beyond docker compose build && up -d for the Caddy stack itself.)
6. Add a Homepage tile. Edit /data/docker/homepage/config/services.yaml: icon + href: https://<svc>.z2mini.gabrielgabrie.com + server: my-docker + container: <name>. Add a widget: block only if Homepage ships one for the service (check https://gethomepage.dev/widgets/services/) with the token in .env as HOMEPAGE_VAR_*; widget.url must be the Caddy hostname (https://<svc>.z2mini.gabrielgabrie.com) — the Homepage container can't reach the app's 127.0.0.1:<port> on the host. Bump the group's columns: in settings.yaml if it now has 5+ tiles. Full procedure: 13-homepage.md → Adding a new service tile. Beszel needs nothing — its agent auto-discovers containers via the docker socket.

---

Networking

Tailscale (mesh VPN)

• Tailnet name: elk-kanyu.ts.net
• Z2 Mini hostname on tailnet: z2mini.elk-kanyu.ts.net
• Tailscale SSH: Enabled (tailscale up --ssh). Authentication uses tailnet identity, not password.
• MagicDNS: Enabled. Devices addressable by short hostname (e.g., ssh gabriel@z2mini).
• Operator: gabriel (sudo tailscale set --operator=gabriel) — tailscale serve/set etc. work without sudo.
• Global DNS resolvers: 1.1.1.1 + 8.8.8.8, with "Override local DNS" ON. So tailnet devices resolve *.z2mini.gabrielgabrie.com reliably (the student-housing router's DNS chokes on it — stale negative cache and/or it strips 100.64.0.0/10 CGNAT answers).
• HTTPS: All web services are fronted by the Caddy reverse proxy (17-caddy.md) on 100.67.235.68:443 — https://<svc>.z2mini.gabrielgabrie.com, auto-renewing Let's Encrypt certs via ACME DNS-01 through the Cloudflare API. Tailscale's own cert features (tailscale cert, tailscale serve) are not used anymore; the two old tailscale serve listeners (:443 Radicale, :8443 Vaultwarden) were retired in May 2026 when Caddy took over.
• Devices on tailnet: Z2 Mini, Gabriel's Windows laptop, Gabriel's iPhone.
• Plan: Free Personal plan. Headscale (self-hosted alternative) was discussed but rejected for now due to complexity.

Network policy

• NO public internet exposure. Every web app binds 127.0.0.1:<port> only; Caddy (network_mode: host) binds 100.67.235.68:443 + :80 (the Tailscale interface only). Samba binds all interfaces (LAN + tailnet) on 137-139/445.
• No port forwarding configured. Services are reachable only via the tailnet (Samba also via the local Wi-Fi LAN).
• The Z2 Mini sits behind student housing NAT — public-facing services are not feasible until Gabriel moves in September 2026.

Reserved ports

Services currently bound on the Z2:

Port	Service	Interface
22	SSH (handled by Tailscale SSH)	tailnet only
80	Caddy — HTTP→HTTPS redirect	100.67.235.68 only
137-139	Samba (NetBIOS)	all interfaces
443	Caddy — HTTPS reverse proxy for all web services (<svc>.z2mini.gabrielgabrie.com); Let's Encrypt via ACME DNS-01/Cloudflare	100.67.235.68 only
445	Samba (SMB)	all interfaces
2019	Caddy admin API (config/reload)	127.0.0.1 only (host loopback — Caddy is network_mode: host)
2283	Immich web UI + mobile API	127.0.0.1 only (fronted by Caddy → immich.z2mini.gabrielgabrie.com)
3000	Homepage launcher dashboard	127.0.0.1 only (fronted by Caddy → home.z2mini.gabrielgabrie.com)
4533	Navidrome web UI + Subsonic API	127.0.0.1 only (fronted by Caddy → navidrome.z2mini.gabrielgabrie.com)
5232	Radicale CalDAV/CardDAV (HTTP)	127.0.0.1 only (fronted by Caddy → radicale.z2mini.gabrielgabrie.com)
8080	Vaultwarden web vault + API (HTTP, ROCKET_PORT=8080)	127.0.0.1 only (fronted by Caddy → vault.z2mini.gabrielgabrie.com)
8082	OpenProject internal proxy (publishes the stack)	127.0.0.1 only (fronted by Caddy → openproject.z2mini.gabrielgabrie.com)
8090	Beszel hub web UI + agent ingest	127.0.0.1 only (fronted by Caddy → beszel.z2mini.gabrielgabrie.com)

When suggesting new services, avoid these ports unless specifically replacing one of them. New web services go behind Caddy: bind 127.0.0.1:<new-port>, add a Caddyfile block.

---

Services Running

1. Samba (file sharing)

• Config: /etc/samba/smb.conf (backup at /etc/samba/smb.conf.backup)
• Service: smbd (managed by systemd)
• Defined shares:
• [files] — /data/files, read-write, user gabriel only
• [backup] — /mnt/backup, read-only (intentional safety), user gabriel only, lost+found hidden via veto files
• [music] — /data/music, read-write, user gabriel only. Verbatim mirror of [files] (same fruit/macOS settings, same gabriel-only access) just pointed at /data/music. Consumed by Navidrome (read-only mount inside the container).
• macOS/iOS compatibility: vfs objects = catia fruit streams_xattr plus full fruit:* configuration block. Do not remove these without testing iOS Files behavior.
• Authentication: Samba's own password database (separate from Linux passwords). Set via smbpasswd.

2. Tailscale (always-on)

• Service: tailscaled (managed by systemd)
• Persists across reboots: Yes (systemctl enabled)
• Configuration: Via Tailscale admin panel (https://login.tailscale.com/admin), not local files

3. Backup automation

• On-site backup drive: Samsung 990 PRO 1TB NVMe in an ASMedia ASM2462 USB enclosure, mounted /mnt/backup, ~916 GB usable, ext4 label backup, UUID 8795cb2e-fe34-4543-b93b-dd45b642846f, reserved-blocks 0 (tune2fs -m 0). fstab line: UUID=8795cb2e-... /mnt/backup ext4 defaults,nofail 0 2. Replaced the old 500 GB Samsung T5 in May 2026 — a refurb verified genuine before deploy (Samsung vendor ID 0x144d, model "Samsung SSD 990 PRO 1TB", firmware 7B2QJXD7, SMART PASSED, 0% used, ~12 power-on hr; ~1056/1009 MB/s = the USB 3.2 Gen 2 link saturating, not the drive's ceiling). Hot-swap (both drives USB, no shutdown). Rollback files: /etc/fstab.pre-990pro, /etc/smartd.conf.pre-990pro, ~/scripts/backup-files.sh.pre-990pro. After the first full run (~520 GB, ~9 min over USB) + tune2fs -m 0: ~396 GB free.
• Off-site backup drive: the demoted 500 GB Samsung T5 — reformatted as ext4 label backup-offsite, UUID 2c8e8e38-f129-4823-a1db-1529d3296b44, reserved-blocks 0. Lives at parents' house; normally NOT connected. Mounted manually at /mnt/offsite for a sync during family visits (no fstab entry).
• Script: /home/gabriel/scripts/backup-files.sh (run as user gabriel via cron — 0 3 * * *; crontab unchanged. Old version backup-files.sh.pre-990pro). Steps: (1) auto-remount /mnt/backup if needed; (2) abort if still not mounted; (3) rsync mirrors with --delete --no-owner --no-group: /data/files/→current/files/, /data/docker/immich/library/→current/immich/, /data/music/→current/music/; (4) sqlite3 "<live db>" ".backup '<dest>'" (host sqlite3) for vaultwarden-db.sqlite3, beszel-data.db, beszel-auxiliary.db, navidrome.db → current/db-dumps/; (5) per-service config rsync (--delete-excluded) → current/service-config/<svc>/ for immich, vaultwarden, navidrome, radicale, beszel, homepage, caddy (excludes bulk data / live DBs / container-owned-unreadable subdirs — see the per-service "Reserved paths" entries above); (6) hard-linked cp -al snapshot → daily/<YYYY-MM-DD>/ (dated by run START time); (7) Sundays only → weekly/<YYYY-MM-DD>/; (8) prune (7 daily, 4 weekly); (9) system-state capture.
• New host dependency: sqlite3 (sudo apt install sqlite3) — needed for step 4. Now in system-state/manual-packages.txt; add to any "packages to reinstall" list.
• Off-site script: /home/gabriel/scripts/backup-offsite.sh — MANUAL, not in cron. Ritual (also in the script's header): plug the T5 into z2mini → sudo mount /dev/disk/by-label/backup-offsite /mnt/offsite → ~/scripts/backup-offsite.sh → sudo umount /mnt/offsite → unplug → carry off-site. Copies current/{files,music,db-dumps,service-config}/ + system-state/ onto the T5 (~3 GB). Does NOT copy current/immich/ — the ~520 GB photo library won't fit on the 500 GB T5; that's a known, accepted gap (the photos have copies only at the apartment: /data live + /mnt/backup backup) until a >1 TB off-site drive. (Unexercised option: fit just the ~442 GB of irreplaceable photo originals — library/library/ + library/upload/ — onto the T5 by skipping thumbs/transcodes/DB-dumps; deferred, Gabriel's call.) Writes /mnt/offsite/offsite-backup.log.
• Method: rsync mirror + sqlite3 .backup for live DBs + hard-linked snapshots (Time Machine-style)
• Retention: 7 daily, 4 weekly snapshots; 7 system-state tarballs
• Safety: Aborts if /mnt/backup not mounted (prevents writing backups to OS drive)
• Logs: /mnt/backup/backup.log, rotated weekly via /etc/logrotate.d/backup-files
• Immich's Postgres DB: NOT filesystem-rsync'd. Immich's own nightly DB auto-backup (Admin → Settings → Backup Settings — default ON, runs 02:00, keeps last 14) writes immich-db-backup-*.sql.gz (~137 MB each) into library/backups/, which the 03:00 current/immich/ rsync picks up. That Immich setting must stay enabled or the DB stops being captured silently.
• The SQLite DBs (Vaultwarden, Beszel ×2, Navidrome): captured with sqlite3 ".backup" from the host binary — a proper online backup, safe while the container holds the WAL-mode DB open. Never raw-rsync a live *.sqlite3/*.db (or its -wal/-shm) — corrupt snapshot.
• System state capture: Same script also captures system configuration daily:
• Package lists (dpkg --get-selections and apt-mark showmanual — manual-packages.txt now includes sqlite3)
• User crontab
• Critical config files: /etc/samba/smb.conf, /etc/smartd.conf, /etc/msmtprc, /etc/apparmor.d/local/usr.bin.msmtp, /etc/fstab, /etc/hostname, /etc/hosts, /etc/logrotate.d/*, /home/gabriel/scripts/
• Bundled into timestamped system-config-YYYY-MM-DD.tar.gz archives in /mnt/backup/system-state/
• Last 7 archives retained
• Sudo permissions for backup: /etc/sudoers.d/gabriel-backup grants gabriel passwordless sudo for THREE specific commands — unchanged by the May 2026 overhaul; the expanded script (Immich, the SQLite dumps, the per-service config rsync) all runs as gabriel in the docker group with no new grant:
• /usr/local/sbin/backup-system-state.sh — wrapper for tar archiving of /etc configs (modern sudo rejects wildcards in path arguments, so wrapper-script pattern is required)
• /usr/bin/mount /mnt/backup — used by backup script's auto-remount safety net if drive transiently disconnects
• /usr/bin/msmtp * — used by hourly mount-check script to send alert emails (msmtp config is root-readable only). Wildcard for recipient email argument; sudo permits wildcards in non-path args. Do not modify these grants or the wrapper script's permissions (must be root:root and 755) or the backup workflow will silently fail.
• Wrapper script: /usr/local/sbin/backup-system-state.sh (root-owned, contains the privileged operations: tar archiving of /etc configs and chown of resulting archive)
• Mount-check script: /home/gabriel/scripts/check-backup-mount.sh runs hourly via cron. Silently exits if /mnt/backup is mounted; sends an email alert if not. Catches USB disconnect events between daily backups so they're noticed within an hour.
• Auto-remount in backup script: before its safety check, backup-files.sh attempts sudo mount /mnt/backup if the drive isn't mounted. Handles transient disconnects without backup failure. Logs a NOTICE entry to backup.log when this fires (so frequency can be monitored).
• Restoring a deleted file: the path inside current//daily//weekly/ now has a files/ subdir — e.g. /mnt/backup/daily/<date>/files/path/to/file (and …/music/… for music, …/immich/… for a raw photo file). \\z2mini\backup SMB browsing still works.
• Recovery procedure: Documented at /mnt/backup/RECOVERY-README.md — covers the package-reinstall list (incl. sqlite3), the /etc config extract, the service-config/ + db-dumps/ + Immich-.sql.gz restore, and the Caddy docker compose build && up -d step.

4. Drive health monitoring (smartd)

• Config: /etc/smartd.conf (rollback: /etc/smartd.conf.pre-990pro)
• Service: smartd (managed by systemd)
• Devices monitored: /dev/nvme0n1, /dev/nvme1n1 (the two internal NVMe drives, unchanged), and the USB backup drive — now the 990 PRO behind the ASM2462 enclosure: /dev/disk/by-id/usb-ASMT_2462_NVME_2504178506CB-0:0 -d sntasmedia -a -m gabrielgabrie99@gmail.com. The off-site T5 is NOT monitored (normally disconnected).
• Why -d sntasmedia: the backup drive is an NVMe SSD behind an ASMedia ASM2462 USB↔NVMe bridge — without this flag smartctl/smartd only sees a generic USB mass-storage device and can't read the drive's real NVMe SMART data. -d sntasmedia uses the ASMedia bridge's SCSI-translation to talk NVMe through the enclosure (works on the box's smartmontools 7.5). Other bridge chips: -d sntjmicron, -d sntrealtek.
• Why the by-id path (not /dev/sdX): USB device names change after disconnect/reconnect cycles. The /dev/disk/by-id/usb-...-0:0 path is stable, and the -0:0 whole-disk form is mandatory because -d sntasmedia needs the whole disk, not a partition (so a /dev/disk/by-uuid/ path wouldn't work). Do not change it to /dev/sdX.
• No -s self-test schedule on the USB backup drive: USB drives don't get scheduled self-tests anyway, and the ASM2462 bridge doesn't pass the NVMe self-test admin command (0x14) through to the drive — so a -s schedule would just error.
• Self-tests: Daily short tests at 02:00 on the two internal NVMe drives only (configured via -s S/../.././02)
• Manual check on the backup drive: sudo smartctl -a -d sntasmedia /dev/sdb (or whatever lsblk shows the enclosure as).
• Alert mechanism: Email via msmtp on any pre-failure indicator

5. Email relay (msmtp)

• Config: /etc/msmtprc (mode 600, owned by root — contains Gmail app password)
• SMTP: Gmail via smtp.gmail.com:587 with STARTTLS and app password authentication
• Send-from address: gabrielgabrie99@gmail.com
• AppArmor profile: Custom local override at /etc/apparmor.d/local/usr.bin.msmtp allowing rwk on /var/log/msmtp.log. Do not remove this file or msmtp will silently fail to log.
• Log: /var/log/msmtp.log (rotated weekly via /etc/logrotate.d/msmtp)
• sendmail symlink: /usr/sbin/sendmail → /usr/bin/msmtp (provided by msmtp-mta package)

6. Immich (photo storage)

• Compose: /data/docker/immich/docker-compose.yml. .env at the same path holds the DB password (mode 600). Compose mirrors upstream Immich v2.7.5 template verbatim except for two intentional customizations: (a) port published only on the Tailscale interface IP 100.67.235.68:2283:2283, (b) the ML model cache is a bind mount to /data/docker/immich/model-cache/ instead of a Docker named volume so it lives under /data rather than /var/lib/docker/.
• Containers (managed via docker compose): immich-server (web UI + API + microservices), immich-machine-learning (face recognition, smart search, OCR — GPU-accelerated via the :cuda image variant + nvidia-container-toolkit since 2026-05-05), immich-redis (Valkey 9 — queue/cache), immich-postgres (vectorchord 0.4.3 — main DB + vector embeddings).
• Image pinning: IMMICH_VERSION lives in .env; postgres + redis pinned by tag + SHA256 digest in compose. Updates are deliberate (docker compose pull + restart), never automatic.
• Storage Template Engine: ON. Files organized at library/admin/{:%Y/%Y-%m-%d}/IMG_*.HEIC. Original filenames preserved.
• Bound interface: Tailscale only — https://immich.z2mini.gabrielgabrie.com (or http://127.0.0.1:2283). NOT reachable on localhost from the server itself; immich-go and other on-server tools must use the tailnet IP.
• First-run admin: Set via web UI on first launch.
• API keys: Account Settings → API Keys. The bulk-import key is saved at /home/gabriel/.immich-api-key (mode 600) for use by immich-go.
• Backup considerations: In the nightly backup since May 2026. Two pieces: (1) the photo files — the entire UPLOAD_LOCATION (/data/docker/immich/library/, incl. encoded-video/ deliberately) is rsync-mirrored to /mnt/backup/current/immich/ at 03:00 with --no-owner --no-group (the files are owned by Immich's container uid; on restore re-apply ownership or let Immich fix it); (2) the Postgres DB — captured via Immich's OWN built-in nightly DB auto-backup (Admin → Settings → Backup Settings — default ON, runs 02:00, keeps last 14) which writes immich-db-backup-*.sql.gz (~137 MB) into library/backups/, and the 03:00 rsync picks those up → current/immich/backups/. The postgres/ data dir is NEVER filesystem-rsync'd (live Postgres = corrupt snapshot) and Immich's "Database Backups" setting must stay enabled or the DB stops being captured silently. model-cache/ is NOT backed up (regenerable). Config (.env, docker-compose.yml, docker-compose.yml.pre-cuda, hwaccel.ml.yml) → service-config/immich/. Off-site: the photo library is NOT on the off-site T5 — ~520 GB doesn't fit on the 500 GB T5, so the photos have copies only at the apartment (/data live + /mnt/backup backup) — accepted gap until a >1 TB off-site drive. (iCloud is no longer the safety net either — iCloud Photos was cleared after the migration, iCloud downgraded to the 200 GB tier.) Restore = follow Immich's official backup/restore docs (recreate the DB + extensions, load the .sql.gz), then rsync library/ back. See 08-recovery.md Step 6b.
• GPU acceleration: Enabled 2026-05-05. Quadro T2000 with NVIDIA driver 595 (Canonical's pre-signed linux-modules-nvidia-595-generic — bypasses DKMS so Secure Boot stays on, no MOK enrollment needed). nvidia-container-toolkit exposes the GPU to the ML container via extends: hwaccel.ml.yml service: cuda in the compose. ML container uses the :cuda-suffixed image variant. CLIP + face recognition (buffalo_l) + OCR all run on GPU; ~3.5 GB of the 4 GB VRAM is used at peak when all three models are loaded with active inference. Models auto-unload after 300 s of idle.
• iOS Immich app: Installed, authenticated, and auto-backup ENABLED 2026-05-08 (Camera Roll album, both Background and Foreground backup on). The migration is complete and iCloud was mass-deleted to ~100 items first, so auto-backup has minimal historical work — it just picks up new captures from this point forward. This is the daily-driver loop.
• What's not yet done: External-library mode is not used (Immich manages all files via storage template). (Immich IS in the nightly backup now — done May 2026; the only remaining backup gap is off-site, which needs a >1 TB drive.)

7. icloudpd (one-shot bulk migration tool)

• Compose: /data/docker/icloudpd/docker-compose.yml, image pinned at icloudpd/icloudpd:1.32.2. Not a long-running service — invoked with docker compose run --rm (no up -d). Cookies persist between invocations via the cookies/ bind mount.
• Authentication: Apple ID gabrielgabrie@hotmail.com. First run was interactive (Apple ID password + 2FA code from trusted device); cookie now valid for ~2 months. Advanced Data Protection must remain OFF on the iCloud account during the migration window — icloudpd will hard-fail with ACCESS_DENIED if ADP is on. Gabriel has never had ADP enabled and does not plan to, so the post-migration "re-enable ADP" step that's in the public docs page does not apply to him.
• Bulk download (historical): Started 2026-05-04 02:13 UTC, ran in icloud-migration tmux session via /home/gabriel/scripts/icloudpd-bulk-download.sh with --watch-with-interval 60. Completed 2026-05-05 ~12:54 UTC after ~35 hr; ~36,750 files / 458 GB downloaded into /data/icloud-import/.
• immich-go ingest (historical): Three immich-go runs on 2026-05-05 — first bulk pass uploaded ~31k files before bailing on the 23^rd transient 500 (default --on-errors stop); resume run with --on-errors continue --concurrent-tasks 4 --pause-immich-jobs=false got further; final pickup pass after disk-recovery cleanup uploaded the remaining 1,030 files cleanly. Final state: 29,331 assets in Immich (Live Photo pairs merged into single assets vs. file count).
• Status now: Bulk migration complete. /data/icloud-import/ deleted post-verify. /data/docker/icloudpd/ and the cookie are kept around — useful for incremental sync via --until-found N if Gabriel ever wants Immich to one-way-mirror iCloud.
• Apple session cookie validity: ~2 months from last successful auth (~2026-07-04 ish before re-auth needed for any future incremental run).
• Lessons captured (see feedback_double_storage_migration.md in memory): during the migration, both /data/icloud-import/ and /data/docker/immich/library/ lived on the same 938 GB /data filesystem. Combined size hit 100% at the 86% upload mark, which surfaced as Postgres No space left on device errors → 500 Internal Server on every immich-go upload → cascading container restarts. Recovery: parsed immich-go's ~/.cache/immich-go/*.log for uploaded successfully|server has duplicate events to identify safely-deletable staging files (35,424 of them), sudo rm -rf those (root-owned because icloudpd runs as root in container), freed 441 GB, queue resumed draining. Future migrations should either stage on a different filesystem or delete-as-you-go.

8. Navidrome (music streaming)

• Compose: /data/docker/navidrome/docker-compose.yml. .env at the same path holds the version pin (NAVIDROME_VERSION=0.61.2). Compose mirrors the upstream Navidrome template verbatim except for: (a) port published only on the Tailscale interface IP 100.67.235.68:4533:4533, (b) music bind mount at /data/music:/music:ro — the :ro is the container's view, the host filesystem is unaffected and Samba writes work fine, © version pinned via ${NAVIDROME_VERSION} from .env, (d) TZ and ND_LOGLEVEL set explicitly.
• Container (managed via docker compose): navidrome (deluan/navidrome:0.61.2 as of install). Single container — Navidrome ships its own SQLite + ffmpeg, no external dependencies. ~50 MB RAM at idle, near-zero CPU.
• Image pinning: NAVIDROME_VERSION in .env. Updates are deliberate (docker compose pull + up -d), never automatic.
• Bound interface: Tailscale only — https://navidrome.z2mini.gabrielgabrie.com. NOT reachable on localhost from the server itself; on-server scripts must use http://127.0.0.1:4533 or https://navidrome.z2mini.gabrielgabrie.com. Same gotcha as Immich.
• First-run admin: Set via web UI on first launch. Separate user database from any other service.
• Music writes: [music] Samba share (added to /etc/samba/smb.conf) exposes /data/music/ for laptop / iOS Files / macOS Finder writes. Built-in file watcher detects new files and triggers a partial scan within ~10 seconds.
• Library format: Mostly AAC 256 (.m4a) — iTunes Store native, Apple Music app CD rips set to AAC 256, native iOS decoding (no transcoding lag), audibly transparent on car/BT-speaker/loudspeaker gear, ~5 MB/track. FLAC also accepted (Bandcamp / library CD lossless rips); decoded natively by modern Subsonic clients. Mixed-format library is fine.
• Tagging: Navidrome reads embedded tags (ID3 / MP4 atoms / Vorbis comments) — folder structure and filenames are cosmetic. For sources with bad/missing tags (yt-dlp downloads, random rips), use MusicBrainz Picard on the laptop (NOT iOS, NOT the server) to acoustic-fingerprint and tag before drop. iTunes Store purchases come pre-tagged and can be dropped in directly.
• iOS clients: Arpeggio (free) + Narjo (free, TestFlight beta) currently. Subsonic-API-compatible — server URL https://navidrome.z2mini.gabrielgabrie.com, same Navidrome admin credentials. CarPlay support on both. Amperfy was the original pick but requires iOS 26 (not available on the iPhone's current iOS).
• Laptop client: Feishin (github.com/jeffvli/feishin) — free FOSS Subsonic desktop client, cross-platform, modern Electron UI. Daily-driver since the web UI's shuffle is limited (shuffles within current view, not smartly across library) and Feishin's queue management is meaningfully better. Connects via Subsonic API to https://navidrome.z2mini.gabrielgabrie.com with Navidrome admin creds. The web UI is still useful for first-time setup, library administration, and one-off play from any browser.
• Backup considerations: In the nightly backup since May 2026. /data/music/ rsync-mirrored to /mnt/backup/current/music/ (also on the off-site T5). navidrome.db (playlists, play counts, listening history) captured via host sqlite3 ".backup" → /mnt/backup/current/db-dumps/navidrome.db (also off-site) — never raw-rsync the live navidrome.db/-wal/-shm. .env + compose → service-config/navidrome/ (the whole data/ dir is excluded from that rsync). data/artwork/ + data/cache/ NOT backed up — regenerated on the next scan. Restore: drop db-dumps/navidrome.db into data/navidrome.db, restore config, rsync music back, docker compose up -d. See 08-recovery.md Step 6b.
• Acquisition list: Working list of music to acquire is at the repo root in music-acquisition-list.md on feat/music-install branch. Tiered by priority with source/cost annotations.
• Use-case split: At-home listening (laptop, BT speaker on phone) is streamed from Navidrome over home Wi-Fi. Commute listening (CarPlay) uses the iOS app's offline-download feature to pre-cache albums — eliminates cellular dead zone stutters.
• What's not yet done: Mass acquisition not yet performed — only a handful of FLAC files dropped to validate the pipeline end-to-end. (Navidrome IS in the nightly backup now — done May 2026.)

9. Homepage (launcher dashboard)

• Compose: /data/docker/homepage/docker-compose.yml. .env at the same path holds the version pin (HOMEPAGE_VERSION) and the widget API tokens (Immich, Navidrome Subsonic token+salt, Tailscale access token + device ID — all referenced from services.yaml via {{HOMEPAGE_VAR_*}} substitution; .env is mode 600). Compose mirrors the upstream Homepage template verbatim except for: (a) port published on 127.0.0.1:3000 only (fronted by Caddy → home.z2mini.gabrielgabrie.com), (b) HOMEPAGE_ALLOWED_HOSTS = home.z2mini.gabrielgabrie.com,localhost:3000,127.0.0.1:3000, © version pinned via ${HOMEPAGE_VERSION} from .env, (d) TZ set explicitly, (e) /var/run/docker.sock:/var/run/docker.sock:ro mounted for live container status display.
• Container (managed via docker compose): homepage (ghcr.io/gethomepage/homepage:v1.12.3 as of install). Single container — Homepage is self-contained Next.js, no external dependencies. ~80 MB RAM at idle, near-zero CPU.
• Image pinning: HOMEPAGE_VERSION in .env. Updates are deliberate (docker compose pull + up -d), never automatic.
• Bound interface: 127.0.0.1:3000 only, fronted by Caddy → https://home.z2mini.gabrielgabrie.com. On-box scripts use http://127.0.0.1:3000 (e.g. the Tailscale-token-age-check's curl http://127.0.0.1:3000/api/services). http://z2mini:3000 / http://100.67.235.68:3000 no longer work (removed in the Caddy migration).
• Config files: /data/docker/homepage/config/{services,bookmarks,widgets,settings,docker}.yaml — bind-mounted from host. Live-reloaded on save (no container restart needed). Both href: and widget.url: are the Caddy hostname (https://<svc>.z2mini.gabrielgabrie.com) — the bridged Homepage container can't reach the apps' 127.0.0.1:<port> on the host, so widget queries go through Caddy too; tile-only entries (Radicale, Vaultwarden) have only href.
• Tiles under "Self-hosted" (services.yaml): Immich, Navidrome, Radicale, Vaultwarden — all with server: my-docker + container: <name> for the status dot. Layout is 4 columns (settings.yaml). Tile under "Network": Tailscale.
• Widgets active: immich (asset count, video count, total storage — uses Immich API key with server.statistics scope), navidrome (track/album count — uses Subsonic-style md5(password+salt) token), tailscale (device IP, last-seen, expiry — uses access token from login.tailscale.com/admin/settings/keys, NOT an auth key; rotates on the 80-day cycle via tailscale-token-age-check.sh). Radicale and Vaultwarden are tile-only — Homepage ships no native widget for either, so they're just clickable launchers with a status dot. Vaultwarden's href is the tailscale serve HTTPS endpoint https://vault.z2mini.gabrielgabrie.com (the container is bound to 127.0.0.1:8080 only — no tailnet-IP form).
• Docker integration: Read-only socket mount lets docker.yaml (my-docker: socket: /var/run/docker.sock) drive per-tile container-status dots. services.yaml references containers by name (e.g., container: immich_server — note the underscore in Immich's compose-generated name; radicale and vaultwarden are plain).
• Backup considerations: In the nightly backup since May 2026. .env + config/ (the six YAML files) rsync'd into /mnt/backup/current/service-config/homepage/ (and to the off-site T5). Small and rsync-safe — no live database. .env permission (600 — widget API tokens) preserved by rsync -a. The config is also committed verbatim in 13-homepage.md, so total backup loss is recoverable by re-pasting + re-minting the tokens. Restore: rsync service-config/homepage/ back, docker compose up -d; re-mint any expired widget token (the Tailscale one rotates ~80-day). See 08-recovery.md Step 6b.
• What's not yet done: New self-hosted stacks get appended to services.yaml as tile-only first; a widget is added later only if Homepage ships one for that service. (Homepage IS in the nightly backup now — done May 2026.)

10. Beszel (system-metrics dashboard)

• Compose: /data/docker/beszel/docker-compose.yml. .env at the same path holds the version pin (BESZEL_VERSION) and the agent's KEY (SSH public key issued by the hub) + TOKEN (websocket registration token). .env is mode 600. Compose mirrors the upstream Beszel template verbatim except for: (a) hub port published only on the Tailscale interface IP 100.67.235.68:8090:8090, (b) APP_URL set to https://beszel.z2mini.gabrielgabrie.com so email-link rendering uses the tailnet hostname, © all bind-mounts anchored under /data/docker/beszel/, (d) agent uses unix socket via shared socket/ volume (LISTEN: /beszel_socket/beszel.sock, NETWORK: unix) — agent has no TCP listener at all, (e) version pinned via ${BESZEL_VERSION} from .env.
• Containers (managed via docker compose): beszel (hub — henrygd/beszel:0.18.7, web UI + SQLite-backed time-series DB + alert engine + SMTP sender, ~30 MB RAM idle) and beszel-agent (agent — henrygd/beszel-agent:0.18.7, network_mode: host for /proc and host network-stack access, reads Docker socket read-only for per-container metrics, ~20 MB RAM idle).
• Image pinning: Same BESZEL_VERSION for hub and agent — they have a wire protocol that can change between minor versions, version skew breaks metric ingest. Updates are deliberate (docker compose pull + up -d), never automatic.
• Bound interface: Tailscale only — https://beszel.z2mini.gabrielgabrie.com. NOT reachable on localhost from the server itself; on-server scripts must use http://127.0.0.1:8090 or https://beszel.z2mini.gabrielgabrie.com. Same gotcha as Immich/Navidrome/Homepage. Exception: the agent's HUB_URL: http://localhost:8090 works because the agent is network_mode: host — its localhost IS the host's localhost, where the hub's published port is reachable.
• First-boot flow (two-phase): (1) docker compose up -d beszel — hub only, agent can't start without KEY/TOKEN. (2) Hub UI at https://beszel.z2mini.gabrielgabrie.com → set up admin user → "+ Add System" with host=/beszel_socket/beszel.sock (unix socket path) → copy the displayed KEY (with quotes — contains spaces) and TOKEN into .env → chmod 600 .env. (3) docker compose up -d beszel-agent. System flips Pending → Up within ~30 s.
• Email alerts: SMTP configured in hub UI (Settings → Mail Settings) — NOT via env vars. Direct to smtp.gmail.com:587 with a separate Gmail app password (created at https://myaccount.google.com/apppasswords with name beszel-z2mini), deliberately distinct from msmtp's app password. Reason: revocation independence — if Beszel's password is compromised, smartd's email channel via msmtp keeps working. Per-system alerts (CPU > 80% / 10min, mem > 85% / 5min, disk /data > 90%, disk / > 80%, agent down > 2min, container down > 2min) are configured in the system detail view, not globally.
• Disk monitoring: the agent's compose bind-mounts marker dirs so the hub UI gives the extra disks display names: /data/.beszel:/extra-filesystems/data__Data:ro and /mnt/backup/.beszel:/extra-filesystems/backup__Backup:ro — the __<Label> suffix on the container path is how you set the displayed name. Result in the hub UI: "Data" (= /data, the SK hynix internal NVMe), "Backup" (= /mnt/backup, the 990 PRO over USB), and nvme1n1p2 for the OS root drive (Beszel doesn't support renaming the root/main filesystem — it always shows the device name there). Rollback: docker-compose.yml.pre-disklabels. After a backup-drive swap, recreate /mnt/backup/.beszel/ and docker restart beszel-agent. (After relabeling, the hub briefly shows stale "ghost" entries — nvme0n1p1, sdb1, sda1 — until they age out of retention; harmless.)
• Backup considerations: In the nightly backup since May 2026. Both hub SQLite DBs — data/data.db AND data/auxiliary.db (the metrics history is the operationally valuable part) — captured via host sqlite3 ".backup" → /mnt/backup/current/db-dumps/beszel-data.db + beszel-auxiliary.db (also off-site). Never raw-rsync an open SQLite DB. .env (agent KEY/TOKEN, mode 600) + compose → service-config/beszel/ (the whole data/ dir is excluded). NOT backed up: data/id_ed25519 (the hub's SSH key — root-owned, the backup user can't read it; regenerable, but regenerating it means re-pairing the one agent), agent-data/ (rebuildable), socket/ (transient). Restore: drop the two db-dumps/beszel-*.db into data/, restore .env + compose, docker compose up -d (hub regenerates id_ed25519), re-register the agent. See 08-recovery.md Step 6b.
• What's not yet done: SMTP and per-system alerts are configured during install. Future agents (laptop, off-site backup VPS, parents'-house drive station) can register against the same hub without restructuring. (Beszel IS in the nightly backup now — done May 2026.)

11. Radicale (CalDAV / CardDAV — calendar and contacts)

• Compose: /data/docker/radicale/docker-compose.yml. .env at the same path holds the version pin (RADICALE_VERSION=3.7.2.0) and nothing else (no secrets — htpasswd file is separate). Compose mirrors the upstream tomsquest/docker-radicale template verbatim except for: (a) two port bindings — 100.67.235.68:5232:5232 for direct HTTP access from Thunderbird/scripts, plus 127.0.0.1:5232:5232 so tailscale serve can proxy from outside; (b) TZ set to America/Toronto; © bind-mount paths anchored under /data/docker/radicale/; (d) version pinned via ${RADICALE_VERSION} from .env. Hardening features (init: true, read_only: true, security_opt: no-new-privileges, cap_drop: ALL + minimal cap_add, memory + pids limits) are unchanged from upstream.
• Container (managed via docker compose): radicale (tomsquest/docker-radicale:3.7.2.0 as of install). Single container — Radicale is a small Python daemon, no external dependencies. ~30 MB RAM at idle. Runs as non-root (UID 2999); the image's "Option 0" pattern auto-chowns the data dir on first boot via the CHOWN capability.
• Image pinning: RADICALE_VERSION in .env. tomsquest tags follow [Radicale version].[image revision] (e.g., 3.7.2.0); pin to a specific tag, never :latest (latest is rebuilt daily and bypasses the deliberate-update posture). The image's README claims htpasswd is included — as of 3.7.2.0 it isn't; use the venv Python form (docker run --rm --entrypoint /venv/bin/python3 ... -c "import bcrypt; ...") to generate hashes.
• Two endpoints, two purposes:
  • https://radicale.z2mini.gabrielgabrie.com (plain HTTP, tailnet IP) — used by Thunderbird on the Windows laptop, and by scripts/curl on z2mini itself. Tailscale already encrypts the wire so HTTP-in-container is fine.
  • https://radicale.z2mini.gabrielgabrie.com/ (TLS via tailscale serve) — used by iOS Calendar. iOS 18+ forces TLS on initial CalDAV verification regardless of the in-app "Use SSL" toggle (verified via Radicale logs showing ~12 TLS ClientHellos rejected before iOS gives up). sudo tailscale serve --bg http://127.0.0.1:5232 puts a Tailscale-managed Let's Encrypt cert in front of the container at port 443. Cert auto-renews on Tailscale's 60/90-day cycle; the serve config persists in tailscaled state and is restored automatically on reboot.
• Auth: htpasswd with bcrypt. Single line in /data/docker/radicale/config/users (mode 644 — bcrypt hashes are designed to be safe at the filesystem level, same posture as Apache's default for htpasswd files). Cost factor 12 (the image's older htpasswd default of 5 is too weak on modern hardware). Both $2b$ (Python bcrypt) and $2y$ (Apache htpasswd) prefixes are accepted.
• Storage layout: /data/docker/radicale/data/collections/collection-root/gabriel/<uuid>/ — each collection (calendar or contact book) is a folder with a .Radicale.props (JSON metadata: displayname, color, type) and one .ics (or .vcf) file per event/contact. Pure files-as-truth. Files are owned 2999:2999 (post-first-boot chown) but the container's umask 0022 keeps them mode 644, so gabriel can read them for inspection without elevation.
• Calendars (as of 2026-05-10 migration): Personal (test calendar), Events and Reminders (49 events / 39 recurring), Deadlines (72 / 7), Conestoga Timetable (12 / 12), Bills (22 / 20), Work Schedule (100 / 93). Total 255 events spanning 2021-04-30 → 2026-09-04. Migration via iCloud "Public Calendar" → download .ics → Thunderbird import-into-collection → CalDAV sync.
• Alarms intentionally NOT preserved across migration. iCloud's public-calendar feed strips all VALARM components by design (Apple treats alarms as personal preferences that shouldn't leak to subscribers). The decision was to accept this loss and re-add alarms manually on the ~10 important recurring events (bills, key class times) rather than take on vdirsyncer-or-similar CalDAV-to-CalDAV migration tooling. Future events created in Radicale carry their alarms normally — this is a one-time migration loss only.
• Phase 6 parallel-run window: 2026-05-10 → ~2026-05-24 (two weeks). During this window, all new events go into Radicale only; iCloud Calendar stays logged in but read-only as a safety net. Phase 7 (cut iCloud Calendar via Settings → iCloud → Calendars OFF + delete on iCloud.com) follows after the parallel run is clean.
• Thunderbird (Windows laptop) clients: Native CalDAV via https://radicale.z2mini.gabrielgabrie.com/gabriel/. Plus subscribed read-only feeds for Conestoga Outlook publish-as-iCal, Gmail, and the clinic appointment feed. Conestoga publish gives titles + locations only (tenant policy); Laurier was dropped after Gabriel quit the job.
• iOS clients: Native CalDAV via https://radicale.z2mini.gabrielgabrie.com/. Plus separate Exchange accounts for Conestoga (writable for school events; iOS sees full Exchange detail) and Google account for Gmail (side-account awareness). The Conestoga publish feed is hidden on iOS (Exchange already shows the same events with full body/attendees/RSVP) but visible on Thunderbird (where Exchange isn't easily accessible).
• Backup considerations: In the nightly backup since May 2026. .env + compose + config/ (config + htpasswd users file) + data/collections/ (all calendars/events as plain .ics files — "files-as-truth", load-bearing: this dir alone restores every calendar) rsync'd into /mnt/backup/current/service-config/radicale/ — no pg_dump, no SQLite ceremony, just rsync; verified byte-for-byte; the calendars are also on the off-site T5. (The rsync also pulls Radicale's .Radicale.cache/ — harmless.) Restore: rsync service-config/radicale/config/ + data/collections/ back, sudo chown -R 2999:2999 /data/docker/radicale/data /data/docker/radicale/config, docker compose up -d. See 08-recovery.md Step 6b. The HTTPS front-end is the Caddy stack under /data/docker/caddy/ (in the backup), not tailscale serve config in /var/lib/tailscale/ — that's retired (no more tailscale serve re-init after a rebuild; bring up the Caddy stack instead).
• CardDAV (contacts) — deferred. The same daemon and same auth handle CardDAV at the same port; standing up a contacts collection is "create a gabriel/contacts/ collection in the web UI and add the same server URL as a CardDAV account on iOS." Gabriel intends to follow up on contacts after the calendar parallel-run window completes.
• What's not yet done: CardDAV (contacts) not yet wired. Phase 7 (cut iCloud Calendar) pending the end of the parallel-run window. Manual alarm re-add on the important recurring events pending. (Radicale IS in the nightly backup now — done May 2026.)

12. Vaultwarden (password manager)

• What it is: a self-hosted, Bitwarden-compatible password manager replacing iCloud Passwords (formerly iCloud Keychain). Vaultwarden is only the server (Rust reimplementation of Bitwarden's server API); every client is the official Bitwarden software pointed at this server — Bitwarden desktop (Windows/macOS/Linux), browser extensions, iOS/Android apps, bw CLI. There is no "Vaultwarden client" (anything in an app store calling itself one is unofficial). The only Vaultwarden-served UI is the web vault at the HTTPS URL, used in a browser for account creation, bulk import, and password-health reports.
• Compose: /data/docker/vaultwarden/docker-compose.yml. .env at the same path (mode 600) holds the version pin, DOMAIN, and SIGNUPS_ALLOWED — plus commented-out ADMIN_TOKEN / PUSH_* / SMTP placeholders for later. Compose mirrors the upstream Vaultwarden example except for: (a) image pinned via ${VAULTWARDEN_VERSION} from .env, never :latest; (b) user: "1000:1000" instead of the image-default root — so data/ stays gabriel-owned (clean backup, no sudo) and a non-root, capability-less process can run, hence cap_drop: ALL works; © ROCKET_PORT=8080 instead of the image default 80 — a non-root process can't bind <1024 without CAP_NET_BIND_SERVICE, and dropping to 8080 removes that need; (d) port published on 127.0.0.1:8080:8080 only — nothing reaches it but tailscale serve (which runs on the host); (e) TZ America/Toronto; (f) no-new-privileges, memory (256M) + pids (100) limits, and a /alive curl healthcheck added; (g) bind-mount under /data/docker/vaultwarden/. The original root-default run crash-looped with Error creating private key 'data/rsa_key.pem' ... PermissionDenied precisely because cap_drop: ALL strips CAP_DAC_OVERRIDE, so root couldn't write into the gabriel-owned dir — user: 1000:1000 is the fix, not adding caps back or chowning to root.
• Container (managed via docker compose): vaultwarden (vaultwarden/server:1.36.0 as of install — the Debian variant, the upstream :latest default; the -alpine tag is smaller but the Debian one is what upstream docs assume). Single container — Vaultwarden bundles its own web server (Rocket) and uses SQLite, no external DB. ~10–15 MB RAM at idle. WebSocket live-sync (ENABLE_WEBSOCKET=true, on by default) is served on the same port; no separate websocket port.
• HTTPS via tailscale serve on port 8443: sudo tailscale serve --bg --https=8443 http://127.0.0.1:8080 puts a Tailscale-managed Let's Encrypt cert in front of Vaultwarden at https://vault.z2mini.gabrielgabrie.com/. Required because the Bitwarden mobile app flatly refuses a non-HTTPS self-hosted server and the web vault's PBKDF2 key derivation needs a secure context. Port 8443 (not 443) because Radicale already owns 443. Cert auto-renews on Tailscale's 60/90-day cycle; the serve config persists in tailscaled state and is restored on reboot. A subpath on 443 (/vault) was the alternative — rejected as the fragile option (Vaultwarden subpath support is finicky with some clients; Radicale's CalDAV discovery wants root). Everything uses this one endpoint — there is no plain-HTTP tailnet endpoint like Radicale's (the container is 127.0.0.1-only); on the server itself, scripts hit http://127.0.0.1:8080.
• Account: single account gabrielgabrie99@gmail.com, created 2026-05-10 via the web vault. SIGNUPS_ALLOWED was flipped true → register → false (registration open ~2 minutes on a private tailnet). To add another account: flip it back, register, flip off, docker compose up -d each time. The admin panel (/admin via ADMIN_TOKEN) and SMTP are left disabled — single-user box doesn't need invites/hints/email-2FA; admin panel can be enabled later by generating an Argon2 hash (docker run --rm -it vaultwarden/server /vaultwarden hash) and uncommenting .env.
• Storage layout: /data/docker/vaultwarden/data/ (owned 1000:1000): db.sqlite3 (+ -shm/-wal — WAL mode) is the vault (logins, TOTP secrets, passkeys, folders); rsa_key.pem is the JWT signing key (load-bearing — losing it invalidates every session); attachments/ and sends/ appear once any exist; icon_cache/ and tmp/ are rebuildable/transient; config.json exists only if the admin panel was used (and then overrides .env at runtime). Vault contents are end-to-end encrypted client-side with a key derived from the master password — the server (and root on the box) stores only ciphertext; sqlite3 queries can show that an item exists and its metadata (ciphers.atype: 1=login, 2=note, 3=card, 4=identity) but not its name/username/password/TOTP/passkey.
• Clients: server URL for all of them is https://vault.z2mini.gabrielgabrie.com (set only the base URL — Vaultwarden serves api/identity/icons/notifications from that one origin). All four clients live: Bitwarden iOS app (region → Self-hosted, Face ID unlock, set as the iOS primary password manager + passkey provider in iOS Settings → General → AutoFill & Passwords — iCloud Passwords toggled OFF for AutoFill); Bitwarden Desktop on the MacBook (added when Gabriel got the Mac, ~2026-05-22); Bitwarden Desktop + browser extension on the Windows laptop. The iPhone needs Tailscale connected to sync; the vault is cached encrypted on-device so AutoFill of existing entries works offline.
• TOTP + passkeys plan (decided May 2026, gradual execution): consolidate both into the Vaultwarden vault. Vaultwarden's TOTP-in-vault is free (a paid Bitwarden-cloud feature) — store a site's TOTP secret in its login entry, Bitwarden autofills the code with the password. Bitwarden is already set as the iOS passkey provider, so new passkeys land in the vault. Existing passkeys: 8 in iCloud Passwords; can't be exported by anyone, by design — re-create per-site in Vaultwarden over time, and only delete each iCloud passkey after the Vaultwarden replacement is registered on that site (lockout risk otherwise). Moving off Microsoft Authenticator means re-enrolling 2FA per-site (no clean seed export); the few accounts that mandate the Microsoft Authenticator app (school/work Conditional Access, number-matching push — e.g. Conestoga) stay on it. The standalone Bitwarden Authenticator app (TOTP-only, would separate codes from the vault) was considered and not used; Ente Auth is the better standalone pick if separation is ever wanted.
• iCloud Passwords migration — COMPLETE (~2026-05-22). Gabriel got his own MacBook → Passwords app → File → Export All Passwords → CSV → Bitwarden web vault → Tools → Import data → format Apple Passwords (csv) → verified all entries present (minus Wi-Fi passwords and passkeys, both expected — neither can be exported from iCloud) → CSV securely deleted. iOS Bitwarden is now the primary password manager; iCloud Passwords AutoFill toggled OFF. New entries go to Vaultwarden only. The earlier MacinCloud cloud-Mac attempt (2026-05-11) had failed Apple's datacenter-IP anti-fraud — own/family/friend's physical Mac is the only path that works for the export.
• iCloud Passwords cleanup — pending (low-pressure). The iCloud Passwords entries still exist (the migration only copied them); ~400 entries. Recommended bulk-delete from the MacBook: Passwords app → Cmd+A → Delete → authenticate; syncs to all Apple devices in ~a minute. Do NOT disable iCloud Keychain itself — keep that on for Wi-Fi password sync (which can't migrate to Vaultwarden by design). Recommended after a few days of confirmed Bitwarden-only use; do this opportunistically, no urgency.
• Backup considerations: In the nightly backup since May 2026. db.sqlite3 (WAL-mode, live) captured via host sqlite3 "<live db>" ".backup '<dest>'" → /mnt/backup/current/db-dumps/vaultwarden-db.sqlite3 (also off-site) — never raw-rsync the live file (same rule as Beszel's hub DB and Navidrome's navidrome.db). rsa_key.pem (load-bearing) + attachments/ + sends/ + config.json (if present) + .env (mode 600 — version pin + DOMAIN = WebAuthn relying-party ID, load-bearing; +ADMIN_TOKEN/PUSH_* if added) + compose → service-config/vaultwarden/; the live data/db.sqlite3 and data/tmp/ are excluded from that rsync; icon_cache/ not backed up (rebuildable). Restore: stop the container, drop db-dumps/vaultwarden-db.sqlite3 in as data/db.sqlite3 (remove stale -wal/-shm), restore rsa_key.pem + attachments/ + sends/ + config.json + .env + compose, chown -R 1000:1000 data/, docker compose up -d. See 08-recovery.md Step 6b. The HTTPS front-end is the Caddy stack under /data/docker/caddy/ (in the backup), not tailscale serve config in /var/lib/tailscale/ — that's retired.
• What's not yet done: delete the iCloud Passwords entries (~400, still synced in iCloud — bulk-delete from the MacBook when ready; keep iCloud Keychain service on for Wi-Fi). Re-create the 8 existing passkeys in Vaultwarden, one site at a time (only delete each iCloud passkey AFTER its Vaultwarden replacement is registered — lockout risk otherwise). TOTP per-site re-enroll off Microsoft Authenticator. Run the password-health report (reused/weak/HIBP-exposed — free on Vaultwarden) and rotate the genuinely-bad ones. Optional: SSH keys in vault (Bitwarden's SSH-agent feature — useful for the GitHub key, syncs across MacBook + Windows laptop; doesn't apply to Tailscale SSH / iOS / headless CI); mobile push (instant iOS sync via free bitwarden.com/host registration + PUSH_* in .env); admin panel (ADMIN_TOKEN Argon2 hash in .env).

13. OpenProject (project management)

• What it is: a self-hosted FOSS project-management tool — WBS, Gantt with dependencies + critical path, baselines, milestones, work-package hierarchy, agile boards + sprints, budgets/cost, time tracking. The point of running it here is learning: the WBS + Gantt + baseline + EVM workflow is the same mental model as Microsoft Project and Primavera P6, which are what utility / power / EPC job postings ask for; the agile boards cover the Jira side of PMP-Agile. Same single tool teaches both predictive and agile sides of the PMBOK.
• Install path: /data/docker/openproject/ — the upstream opf/openproject-deploy repo stable/17 branch cloned verbatim (git clone --depth=1 --branch=stable/17 ... /data/docker/openproject). The upstream docker-compose.yml is untouched — every customization is in .env (mode 600). git pull on stable/17 is the upgrade path.
• Compose (upstream, do not edit): nine containers — web (Rails Puma, 4–16 threads), worker (background jobs), cron (scheduled jobs), seeder (runs db:migrate + db:seed on every up, then exits — restart: on-failure), db (postgres:17), cache (memcached), proxy (locally-built openproject/proxy — internal nginx/Caddy publishing on 127.0.0.1:8082), hocuspocus (openproject/hocuspocus:17.4.0 — WebSocket for collaborative editing), autoheal (willfarrell/autoheal:1.2.0 — restarts web if its healthcheck fails). All restart: unless-stopped except seeder. ~1.5–2 GB RAM at idle (the heaviest single service on the box).
• .env customizations (the only file edited):
  • TAG=17-slim, POSTGRES_VERSION=17 (the upstream compose still defaults to PG13 for legacy installs; new installs override to 17 via .env)
  • OPENPROJECT_HTTPS=true, OPENPROJECT_HOST__NAME=openproject.z2mini.gabrielgabrie.com, PORT=127.0.0.1:8082 — TLS-terminating proxy out front, Rails emits https:// URLs
  • PGDATA=/data/docker/openproject/postgres, OPDATA=/data/docker/openproject/assets — override the upstream named volumes to bind mounts under /data/docker/openproject/ so the backup user can reach them. postgres/ is mode 700 uid 999 (matches Immich's Postgres pattern); assets/ is uid 1000 (the web container's run-as user).
  • Three randomly-generated secrets: SECRET_KEY_BASE (64 chars), COLLABORATIVE_SERVER_SECRET (32 chars), POSTGRES_PASSWORD (32 chars). POSTGRES_PASSWORD is also embedded in DATABASE_URL in .env — must match. Regenerating SECRET_KEY_BASE invalidates every session and every 2FA recovery code.
  • COLLABORATIVE_SERVER_URL=wss://openproject.z2mini.gabrielgabrie.com/hocuspocus — Caddy auto-upgrades WebSocket connections, no extra config in the Caddyfile block.
  • IMAP_ENABLED=false; SMTP unconfigured. Single-user setup — no inbound email-to-ticket flow, no outbound notifications yet. Add later via IMAP_* (inbound) or EMAIL_DELIVERY_METHOD=smtp + SMTP_* (outbound) — with a separate Gmail app password from msmtp's (revocation-independence, same as Beszel's).
• HostAuthorization gotcha: Rails rejects any request whose Host: header isn't OPENPROJECT_HOST__NAME. Direct curl http://127.0.0.1:8082/ from on-box returns HTTP 400 until you pass -H "Host: openproject.z2mini.gabrielgabrie.com" -H "X-Forwarded-Proto: https". Going through Caddy (https://openproject.z2mini.gabrielgabrie.com) works because Caddy preserves the original Host:. This is the gate working, not a bug — note it when smoke-testing.
• Bound interface: 127.0.0.1:8082 only, fronted by Caddy → https://openproject.z2mini.gabrielgabrie.com (auto-renewing Let's Encrypt cert via Cloudflare DNS-01; cert issued May 2026, valid until Aug 2026 then auto-renews). Not reachable on the tailnet directly; not on the public internet.
• autoheal with Docker-socket-RW (upstream verbatim, kept): the autoheal container mounts /var/run/docker.sock read-write to call the Docker API and restart unhealthy containers. A compromised autoheal container would be effectively root on the host. Accepted because: (a) single-user, tailnet-only, no public inbound; (b) OpenProject is the heaviest Rails app on the box and most likely to go unhealthy under memory pressure or a slow query — auto-restart is genuinely useful; © keeping the upstream compose unedited is the documented "use upstream config verbatim" stance. Reconsider if a security posture change demands it.
• First-boot behavior: seeder runs db:migrate + db:seed (~30 s on a fresh DB; longer on major upgrades), then web goes healthy. Default login is admin / admin; OpenProject forces a password change on first login. Set a strong password and store it in Vaultwarden.
• Caddy block (already deployed):

  openproject.z2mini.gabrielgabrie.com {
      log
      reverse_proxy 127.0.0.1:8082
  }
  

  Reload via the admin API: docker compose -f /data/docker/caddy/docker-compose.yml exec -T caddy caddy reload --config /etc/caddy/Caddyfile. Gotcha: at install (May 2026), caddy reload had to be re-issued once — the first reload validated cleanly but never showed the new host in /config/apps/http/servers/srv0/routes. The second identical reload took effect. If a new site block doesn't appear in the admin API's routes list within a few seconds, reload again.
• Backup considerations: In the nightly backup since install. Three pieces, mirroring Immich's pattern:
  1. Postgres dumped via pg_dump inside the db container → gzipped → /data/docker/openproject/backups/openproject-db-YYYY-MM-DD.sql.gz (last 14 retained). The backups/ dir is then included in the per-service rsync. Never raw-rsync the postgres/ data dir — live Postgres = corrupt snapshot.
  2. assets/ (uploaded attachments + exports, uid 1000, files-as-truth) — rsynced as part of the per-service config.
  3. .env + upstream docker-compose.yml + proxy/ build context + the rest of the cloned repo — rsynced to service-config/openproject/. The whole cloned tree is included (small — ~50 KB excluding the bind mounts). Off-site (T5): service-config/openproject/ is included in backup-offsite.sh. Restore: re-clone upstream, drop .env + assets/ back, restore Postgres from the latest .sql.gz (gunzip -c <dump> | docker compose exec -T db psql -U postgres -d openproject), docker compose up -d.
• What's not yet done: First-login password change + first project + first work-package WBS are user actions, not install steps. SMTP for notifications not configured. OpenProject Homepage tile present (added at install time). No connection to Vikunja / Trello data — Gabriel's chosen project-management home is OpenProject; data migration from any prior tool is manual.

---

Design Decisions Already Made

These are settled choices. Suggesting reversal requires good reason and explicit discussion with Gabriel.

Decision	Rationale	Don't suggest reverting
Linux server (no Windows)	Lower resource use, better Docker, career-relevant skills	Don't propose dual-boot or Windows-based solutions
File sharing via SMB (not Nextcloud or WebDAV)	Best iOS Files experience, cross-platform native support	Don't suggest installing Nextcloud "for files"
No Docker for Samba	Samba is infrastructure, runs better natively	Don't suggest moving to a Samba container
Tailscale (managed, free tier)	Working well, complexity of Headscale not justified	Don't suggest Headscale until Gabriel asks
ext4 filesystems	Simple, reliable, well-supported	Don't suggest ZFS/Btrfs migration unless data integrity becomes a stated goal
OS and data on separate drives	Failure isolation	Don't propose merging drives or RAID 0
No RAID on data drive	Backups protect against drive failure; complexity not warranted for 1 user	Don't suggest mirror/RAID 1 unless uptime becomes a stated requirement
/data/files as the user data root	Already configured in Samba, backups, scripts	Don't change this path
Tailscale-only services (no public internet)	Privacy + security; student housing network limits anyway	Don't suggest opening ports until after September 2026 move
Caddy as the single HTTPS reverse-proxy ingress + custom domain (May 2026)	Every web app is at https://<svc>.z2mini.gabrielgabrie.com (Let's Encrypt cert via ACME DNS-01 through the Cloudflare API; auto-renews). Apps were re-bound to 127.0.0.1:<port> and speak plain HTTP behind Caddy — Tailscale still encrypts that wire. Replaced the per-service tailscale serve listeners. Caddy image is built locally = stock caddy + caddy-dns/cloudflare. See 17-caddy.md.	Don't propose ripping out Caddy or reverting to per-service tailscale serve. Don't propose Traefik/nginx/NPM instead unless Caddy proves limiting. New services go behind Caddy (bind 127.0.0.1, add a Caddyfile block) — see the checklist.
Local backups + a manually-rotated off-site drive (no cloud)	Nightly on-site backup to /mnt/backup (the 990 PRO) covers everything; off-site is the demoted T5 (backup-offsite.sh) carried to parents' house and synced during visits — protects everything except the ~520 GB photo library, which doesn't fit on the 500 GB T5 (accepted gap until a >1 TB off-site drive).	Don't push for paid cloud backup. Don't suggest copying the Immich library to the 500 GB off-site T5 — it doesn't fit; flag a >1 TB off-site drive as the next want instead.
Samsung 990 PRO 1TB (refurb, verified genuine) as the on-site /mnt/backup; the old 500 GB T5 → off-site (May 2026)	The refurb was authenticated before deploy (Samsung vendor ID 0x144d, model "Samsung SSD 990 PRO 1TB", firmware 7B2QJXD7, SMART PASSED, 0% used, ~12 power-on hr, 0 errors; ~1056/1009 MB/s = the USB 3.2 Gen 2 link saturating, not the drive's PCIe 4.0 ceiling). 1 TB fits the full backup incl. the ~520 GB photo library; the 500 GB T5 couldn't — so the bigger drive took the nightly on-site role and the T5 was demoted to off-site rotation. ext4 reserved-blocks set to 0 (tune2fs -m 0) on both since they only ever hold backups. smartd reads it via -d sntasmedia through the ASM2462 USB↔NVMe bridge.	Don't suggest swapping the roles back. Don't suggest a /dev/sdX reference in fstab/smartd — UUID / /dev/disk/by-id/usb-...-0:0 are the stable ones. Don't re-enable ext4 reserved blocks on these drives.
Immich for photo storage	Best self-hosted Immich-vs-PhotoPrism-vs-Ente-vs-Photoview tradeoff for iPhone integration + on-server ML + project momentum	Don't propose alternative photo-management apps
Storage Template ON (files organized by capture date)	Recoverable / browsable backups; date-organized library has meaningful filesystem layout if Immich's DB is ever lost	Don't suggest turning it off — flipping it triggers a one-time job that physically moves every file
Immich port 2283 bound to Tailscale interface only	Same security posture as everything else; Immich must not be publicly reachable	Don't suggest binding to 0.0.0.0 or adding a reverse proxy without Gabriel asking
ML cache as bind mount under /data/docker/immich/model-cache/	Keeps everything Immich-related under one path; ~2-3 GB of model weights stay on /data (the data drive) instead of /var/lib/docker/ (OS drive)	Don't switch back to a named volume
GPU acceleration enabled (Quadro T2000, May 2026)	Used 1,030 file-pickup pass to validate, peak VRAM ~3.5/4 GB across CLIP + face + OCR with all loaded; ~3-5 hr of post-ingest ML on GPU vs ~12-25 hr on CPU	Don't propose disabling GPU; the install used pre-signed Canonical kernel modules to keep Secure Boot on
iCloud → Immich migration via icloudpd (not CopyTrans Cloudly)	Server-side execution, FOSS, no laptop dependency, no $20 license, no SMB hop. CopyTrans is a credible Plan B if icloudpd's auth fails mid-migration	Don't switch to CopyTrans unless icloudpd is genuinely blocked
immich-go for bulk import (not the web UI or iOS app)	Web UI fragile at 31k items; iOS app only sees photos locally on the device. immich-go is purpose-built for bulk via Immich's API and respects storage template	Don't suggest dragging files into library/ directly — Immich's DB won't know about them
iCloud subscription downgraded to 200 GB tier ($3.99/mo), not the 50 GB tier ($1/mo) — May 2026	Post-migration iCloud usage settled at ~150 GB across iMessage (~60 GB after attachment purge), Family Sharing pool (~85 GB used by family members), and iPhone Backup (~20 GB). Photos are now in Immich and effectively zero in iCloud. The 50 GB tier was rejected for three reasons: (1) iMessage alone is ~60 GB and won't shrink further without losing meaningful conversation history; (2) iPhone has 128 GB total / ~58 GB free, so the "Messages in iCloud OFF, hold the full 60 GB on-device" path doesn't fit, and turning Messages in iCloud OFF would just shift the bytes into iCloud Backup (not save quota); (3) Family Sharing pools storage — downgrading would force family members onto their own paid plans. Cost delta is ~$3/mo, judged not worth the architectural inconvenience.	Don't suggest dropping further to the 50 GB tier — was deliberately evaluated and rejected. Don't suggest "turn off Messages in iCloud to save space" — it just moves bytes between buckets in the same quota
iCloud Photos cleared before a real off-site backup exists (accepted risk window)	Originally, iCloud Photos was kept full as the off-site safety net for Immich until a 2 TB external arrived. Gabriel chose to clear iCloud Photos at the same time as the iCloud downgrade rather than maintain it as a safety net, accepting that Immich on /data is the only copy of photos until the 2 TB external arrives. Local nightly rsync to /mnt/backup doesn't survive theft, fire, or rm -rf.	Don't suggest re-enabling iCloud Photos to recreate the safety net — the explicit choice was to live with the risk window. Do flag the off-site drive as the next priority when it comes up.
OpenProject chosen for project management / PMP learning (May 2026) — not Planka / Vikunja / Kanboard / WeKan / Taiga / Redmine	The lighter Trello clones teach kanban only; OpenProject's WBS + Gantt + baselines + EVM is the same mental model as MS Project and Primavera P6, which are what utility / power / EPC job postings actually list. Agile boards + sprints are in the same tool, so the Jira-Scrum side is covered too. The 5–10× RAM cost vs. Planka is the price of teaching the right thing. Upstream opf/openproject-deploy stable/17 cloned verbatim into /data/docker/openproject/; customization is .env-only. autoheal with Docker-socket-RW kept upstream-verbatim (single-user tailnet box, auto-restart on Rails OOM is useful).	Don't suggest swapping to a lighter Trello clone — explicitly evaluated and rejected; the PMP / utility-industry transfer is the whole point. Don't edit docker-compose.yml directly — customize via .env; the upstream compose is git pull-managed. Don't suggest removing autoheal to "harden" the stack without weighing the auto-restart benefit.
iOS Immich auto-backup ON (since 2026-05-08), iCloud mass-deleted to ~100 items first	Avoids re-uploading photos Gabriel already pruned in Immich web UI; iOS Photos = "what should be in Immich" simplifies the mental model	Don't suggest reverting to manual share-sheet uploads — the daily-driver pattern is auto-backup
Navidrome for music streaming	Audio-only purpose-built; lightweight (~50 MB RAM); Subsonic-API ecosystem (many iOS clients with CarPlay); FOSS; no Plex Pass / subscription requirement	Don't suggest Plex (Plex Pass cost), Funkwhale (smaller community), or Jellyfin specifically for music (overkill for audio-only — Jellyfin can be added later for video without disturbing Navidrome)
/data/music/ separate from /data/docker/navidrome/	Lets Samba [music] share write into music without exposing Navidrome runtime data; future Jellyfin-for-video can point at the same folder if music+video unification is ever wanted	Don't suggest moving music inside /data/docker/navidrome/library/ or similar
Music dir mounted read-only inside Navidrome container (:ro)	Navidrome only reads files; the :ro is the container's view (host is unaffected, Samba writes work fine)	Don't suggest making it :rw — there's no reason to and it removes a safety property
Library format AAC 256 with FLAC accepted	iTunes Store delivers AAC 256 natively; native iOS decoding; audibly transparent on car/BT-speaker gear; ~5 MB/track. FLAC also accepted (decoded natively by modern Subsonic clients). Mixed-format library is fine.	Don't suggest converting between formats just for consistency — lossless-to-lossy is destructive, lossy-to-lossless is pointless
Navidrome port 4533 bound to Tailscale interface only	Same security posture as everything else; Navidrome must not be publicly reachable	Don't suggest binding to 0.0.0.0 or adding a reverse proxy without explicit ask
Self-host as primary at-home + offline-cache for commute	At-home (laptop, BT speaker) is streamed; commute (CarPlay, away from Wi-Fi) uses iOS app's offline downloads to avoid cellular dead-zone stutters. Self-hosting earns its keep on the streaming half.	Don't suggest dropping the server in favor of "just sync iTunes purchases manually" — multi-device + cache-everywhere + central library is the value
iOS clients Arpeggio + Narjo (not Amperfy)	Amperfy's recent versions require iOS 26, which isn't on the iPhone. Arpeggio (App Store) and Narjo (TestFlight) both support CarPlay and connect cleanly to Navidrome via the Subsonic API.	Don't suggest Amperfy as the iOS client until iOS 26 is in use
Homepage for the launcher dashboard	Native widgets for Immich, Navidrome, Tailscale — no scraping required; YAML config (versionable, copy-pasteable in docs); active development; ~80 MB RAM idle. Glance is lighter but lacks integrations; Dashy is heavier; Heimdall/Homer/Flame are static-tile-only without live data.	Don't suggest swapping for Glance/Dashy/Heimdall/Homer/Flame unless Homepage proves limiting
Homepage port 3000 bound to Tailscale interface only	Same security posture as everything else; never publicly reachable	Don't suggest binding to 0.0.0.0 or adding a reverse proxy without explicit ask
Homepage Docker-socket integration mounted read-only	Lets Homepage display live container status dots (running / stopped / unhealthy) without needing Portainer. :ro is the safety property — Homepage cannot start/stop/exec into containers via this mount.	Don't suggest making it :rw or adding write capabilities unless Gabriel asks for container management from the dashboard
Homepage widget secrets in .env (mode 600), referenced via {{HOMEPAGE_VAR_*}}	Keeps API tokens out of services.yaml (which would otherwise be committable as documentation); same pattern as Immich's .env for the DB password	Don't suggest hardcoding tokens directly into services.yaml
Manual Tailscale token rotation (~80-day cycle) with cron-based email reminder	Tailscale enforces a 90-day max on access tokens — no plan offers permanent. OAuth client credentials issue 1-hour tokens which would force ~48 Homepage container restarts per day (env_file is loaded at container start, not live). Widget data (IP / last_seen / key-expiry) is low-value enough that auto-renewal isn't worth that complexity. Weekly tailscale-token-age-check.sh cron at Monday 09:00 emails the rotation steps when the token is older than 75 days.	Don't propose OAuth auto-renewal again unless Homepage adds live env reload, or unless multiple Tailscale-token-consuming services arrive that would amortize the cost. Don't propose dropping the rotation check — once silently expired the widget breaks invisibly.
Beszel for system-metrics dashboard	Single Go binary per side, ~50 MB RAM total for hub + agent, modern UI, built-in alerting with email; Netdata is heavier (~150 MB) and nags toward its cloud; Glances has no time-series persistence; Grafana stack is 4+ containers and overkill for one server	Don't suggest Netdata / Glances / Grafana+Prometheus / Checkmk / Zabbix / Nagios unless Beszel proves limiting
Beszel hub + agent in same compose file via unix socket	Standard Beszel single-host pattern. Agent has zero TCP listener, eliminates an entire class of agent-exposure concerns. network_mode: host on the agent is still required for /proc and host network-stack access (metric reading), but no inbound port is opened.	Don't suggest splitting to TCP listener mode on a single-host setup
Beszel port 8090 bound to Tailscale interface only	Same security posture as everything else; never publicly reachable	Don't suggest binding to 0.0.0.0 or adding a reverse proxy without explicit ask
Beszel emails directly to Gmail (not piped through msmtp)	Revocation independence — Beszel uses a separate Gmail app password (beszel-z2mini) so revoking it doesn't affect smartd's email channel via msmtp. Beszel's hub has built-in SMTP support; piping through localhost msmtp would add complexity for no security gain.	Don't suggest reusing msmtp's app password for Beszel — defeats the revocation-independence reason for splitting
Hub + agent always pinned to the same Beszel version	Wire protocol can change between minor versions; skew breaks metric ingest until both align.	Don't suggest updating one independently
Docker socket mounted read-only on Beszel agent	Per-container metric collection; :ro is the safety property — agent cannot start/stop containers via this mount. Same pattern as Homepage.	Don't suggest making it :rw
Radicale for self-hosted CalDAV/CardDAV (not Nextcloud, not Baïkal)	Files-as-truth storage (each event is an .ics file on disk — same recoverability property as Immich's storage template), ~30 MB RAM, single container, pure rsync backup with no DB ceremony, AGPL FOSS. Nextcloud has a much prettier calendar UI but is a 5+ container PHP platform with quarterly major-version upgrade maintenance — explicitly rejected on the "fewer moving parts during exam crunch" rationale. Baïkal stores in SQLite (more backup ceremony) and adds an admin GUI a single user doesn't need.	Don't propose Nextcloud, Baïkal, AgenDAV, SOGo, or InfCloud as a Radicale replacement unless Radicale itself proves limiting. Don't propose Nextcloud "for the prettier calendar UI" — was deliberated and the moving-parts trade-off won.
Radicale dual-endpoint pattern: plain HTTP for Thunderbird/scripts + HTTPS via tailscale serve for iOS	iOS 18+ Calendar forces TLS on initial CalDAV verification regardless of the in-app "Use SSL" toggle (verified via Radicale logs showing ~12 TLS ClientHellos rejected before iOS gives up). tailscale serve --bg http://127.0.0.1:5232 puts a Tailscale-managed Let's Encrypt cert in front of Radicale — auto-renews, zero cert maintenance, real publicly-trusted cert (no mobile-config trust profile needed on iOS). Thunderbird and on-server scripts continue to use direct HTTP since they don't force TLS.	Don't propose self-signed certs + iOS profile install (was rejected — tailscale serve is cleaner). Don't propose binding Radicale on 0.0.0.0 or removing the 127.0.0.1 port binding — tailscale serve proxies to localhost specifically to avoid bouncing back through the tailnet interface.
Radicale auth via htpasswd + bcrypt cost factor 12 (mode 644 on the users file)	Single-user setup; one bcrypt line is sufficient. Mode 644 because bcrypt hashes are designed to be safe at the filesystem level (Apache's default for htpasswd files is also 644) and 644 lets the container at UID 2999 read the file without chowning the host directory away from gabriel. Cost factor 12 (bcrypt's gensalt(rounds=12)) is current best practice; the image's old htpasswd -B default of 5 is too weak on modern hardware.	Don't suggest LDAP or OAuth for Radicale auth — premature complexity for one user. Don't suggest mode 600 + chown to 2999 — adds a sudo step for no security gain (mode 644 + bcrypt is cryptographically safe).
Radicale rights = owner_only	Default-safe right policy: only the user who owns a collection can read/write it. Moot for one user but the right default if a second user is ever added (e.g., a shared family calendar).	Don't change to authenticated (any logged-in user could read all collections) without explicit ask.
iCloud Calendar migration via publish-as-iCal .ics download (not vdirsyncer or Mac export)	Universal path: works from any browser, uses Apple's official "Public Calendar" feature, no extra tooling. Trade-off: iCloud's public-cal feed strips all VALARM components by design — alarms are lost across the migration. Decision was to accept this and re-add alarms manually on the ~10 important recurring events (bills, key class times) rather than take on vdirsyncer (a Python tool with Apple app-specific-password dependency) for one-time migration tooling.	Don't suggest re-running the migration with vdirsyncer to recover alarms — was evaluated and the manual-re-add path was chosen. Don't suggest dragging events directly into the collection-root filesystem; use Thunderbird's Import-into-collection workflow which round-trips correctly via CalDAV.
Conestoga Outlook stays as a separate Exchange account on iOS (not bridged into Radicale)	Org-managed Microsoft 365 doesn't expose CalDAV. DavMail-style bridges are fragile under modern auth. Cleaner architecture: Conestoga Exchange = native invite acceptance + writing school events on iOS (full detail visible). Conestoga's "Publish a calendar" .ics URL is subscribed read-only in Thunderbird only (publishes titles + locations only — tenant policy).	Don't suggest a DavMail-style Outlook→CalDAV bridge. Don't suggest also subscribing the Conestoga publish feed on iOS — duplicates what Exchange already shows in fuller detail.
Laurier Outlook account dropped (not migrated)	Job ended; keeping a logged-in account against an org Gabriel no longer works for is attack-surface and credential-rotation hassle for zero ongoing benefit. Old events viewable via OWA in a browser if ever needed.	Don't suggest re-adding the Laurier account or migrating Laurier events into Radicale.
Vaultwarden for self-hosted passwords (not the official Bitwarden self-host, not KeePassXC)	Vaultwarden is a Rust reimplementation of Bitwarden's server — ~10–15 MB RAM, single container, SQLite, and it unlocks Bitwarden's "premium" features (TOTP-in-vault, attachments, Sends, password-health reports) for free. The official "unified" image is ~1 GB+ RAM with a bundled DB engine and license-gated features. KeePassXC + a .kdbx on the Samba share is the "files-as-truth" purist option (server-less, rsync-safe) but loses on iOS: third-party iOS clients, sync-conflict risk, clunkier autofill — and "integrates with iOS nicely" was the brief. Same lightweight-single-binary pattern as Beszel and Radicale; clients are the official Bitwarden apps/extensions pointed at the self-hosted URL.	Don't propose the official Bitwarden self-host image, KeePassXC, Passbolt, or Padloc as a replacement unless Vaultwarden proves limiting. Don't suggest a separate "Vaultwarden client" — there isn't one.
Vaultwarden runs as user: 1000:1000 (gabriel), not the image-default root, with ROCKET_PORT=8080 and cap_drop: ALL	Running as the directory owner means no capability is needed to write data/ or bind a high port — so the hardened cap_drop: ALL stands and the data stays human-owned (clean backup, no sudo). The image's start.sh just execs the binary (no privilege-drop / chown), so non-root "just works". A root-default container crash-loops with PermissionDenied creating rsa_key.pem because cap_drop: ALL strips CAP_DAC_OVERRIDE and root then can't write a gabriel-owned dir. ROCKET_PORT=8080 (vs the image default 80) is what lets a non-root, capability-less process bind the port.	Don't suggest reverting to root + adding CAP_DAC_OVERRIDE/CAP_CHOWN back, or chown -R 0:0 data/ — running as uid 1000 is the deliberate, cleaner fix. Don't suggest port 80 inside the container.
Vaultwarden via tailscale serve HTTPS on port 8443 (container 127.0.0.1-only, no plain-HTTP tailnet endpoint)	Bitwarden clients (mobile especially) require HTTPS for self-hosted; the web vault's PBKDF2 needs a secure context. Radicale already owns :443, so a second listener: tailscale serve --bg --https=8443 http://127.0.0.1:8080 — real auto-renewing Let's Encrypt cert, zero maintenance. Unlike Radicale (which exposes plain HTTP on the tailnet IP for Thunderbird/scripts), nothing here needs that — every client uses :8443, so the container binds 127.0.0.1 only (smaller surface).	Don't propose a subpath on :443 (/vault) — finicky with some clients, and Radicale's CalDAV discovery wants root. Don't propose binding the container to 0.0.0.0 or the tailnet IP — tailscale serve reaches 127.0.0.1 fine. Don't propose self-signed certs + an iOS trust profile — tailscale serve's publicly-trusted CA path is cleaner (same reasoning as Radicale).
Vaultwarden registration closed (SIGNUPS_ALLOWED=false); admin panel and SMTP left disabled	Single-user box. Registration was open ~2 minutes on a private tailnet to create the one account, then closed. Admin panel (/admin) and SMTP add moving parts a single user doesn't need yet; both documented-but-commented in .env for later.	Don't suggest leaving signups open. Don't suggest wiring SMTP unless invites/hints/email-2FA become needed — and if so, it gets its own Gmail app password (revocation independence, like Beszel).
TOTP + passkeys to be consolidated into the Vaultwarden vault (decided May 2026; not yet executed)	One app for passwords + 2FA codes (Vaultwarden's TOTP-in-vault is free); Bitwarden can also be the iOS passkey provider so new passkeys land in the vault. The "all eggs in one basket" trade is accepted given a strong master password (with the option to add 2FA to the vault login later). Migrating off Microsoft Authenticator = re-enroll 2FA per-site (no clean seed export); accounts that mandate the MS app (Conditional Access, number-matching push) stay on it.	Don't suggest the standalone Bitwarden Authenticator app (TOTP-only, would defeat the consolidation goal) — Ente Auth is the better pick only if separation is ever explicitly wanted. Don't suggest keeping TOTP solely in MS Authenticator long-term.

What Was Considered and Rejected

These are tools Gabriel evaluated but chose not to install. Each could be revisited later, but suggest them with awareness that they were already deliberated.

• Nextcloud — rejected because iOS Files integration is poor; SMB chosen instead
• Caddy + Tailscale plugin for HTTPS — abandoned mid-setup — superseded: Caddy is the reverse proxy as of May 2026, via a different path — a custom domain (*.z2mini.gabrielgabrie.com, DNS on Cloudflare) with Let's Encrypt certs from the ACME DNS-01 challenge through the Cloudflare API (not the caddy-tailscale plugin / tailscale cert path that was abandoned). See 17-caddy.md.
• Traefik / nginx / Nginx Proxy Manager — evaluated as the reverse proxy; Caddy chosen (smallest config, automatic HTTPS + renewal built in, single static binary)
• TrueNAS Scale / Unraid (NAS-specific OSes) — rejected because dedicated NAS OSes optimize for multi-drive setups Gabriel doesn't have
• Snap packages (Nextcloud snap, Wekan snap, etc.) — rejected in favor of Docker-based deployments for future services
• Off-site cloud backup (Backblaze B2, etc.) — rejected; off-site backup will be a manually rotated drive at parents' house
• PhotoPrism / Ente / Photoview — evaluated as Immich alternatives in May 2026; Immich won on iPhone integration, on-server AI/ML, and project momentum
• CopyTrans Cloudly ($20 Windows GUI) — evaluated for iCloud → server bulk download; rejected in favor of icloudpd for server-side execution and FOSS. Kept available as Plan B if icloudpd's auth fails
• iCloud for Windows (free) — evaluated for iCloud → server bulk download; rejected because it requires 458 GB+ of free local storage on the laptop (sync model, not download model) and a separate transfer step
• Apple Privacy Portal export — evaluated as last-resort fallback for iCloud migration; not used. Available if both icloudpd and CopyTrans fail (5-7 day wait, big zip files)
• Immich External Library mode — considered for ingesting /data/icloud-import/ directly without moving files; rejected because it bypasses storage template — files would stay in icloudpd's date layout instead of getting reorganized into Immich's managed library
• Plex (with Plex Pass for Plexamp) — evaluated for music streaming; rejected because Plex Pass ($120 lifetime) violates the no-subscription constraint
• Jellyfin specifically for music — evaluated; rejected because Navidrome's purpose-built music admin/scanning is cleaner and lighter. Jellyfin remains a candidate for future video — both can coexist at /data/docker/<service>/
• Funkwhale, Ampache, Mopidy — evaluated as Navidrome alternatives; rejected for various reasons (community size, dated UX, "building block not finished product")
• Amperfy as the iOS Subsonic client — chosen on paper; rejected on install because recent versions require iOS 26 (not available on the iPhone's current iOS). Arpeggio + Narjo (TestFlight) chosen instead
• Picard installed on the server / on iOS — evaluated location; not viable — Picard is a desktop GUI app, runs on the Windows laptop only
• All-FLAC music library — evaluated; rejected for the user's listening environments (car / BT speaker / loudspeaker — lossless not audible) and for storage/workflow simplicity. FLAC is accepted on a per-file basis (Bandcamp / lossless CD rips), just not pursued as the standard format
• Glance — evaluated as a Homepage alternative; lighter and faster (single Go binary) but lacks native Immich/Navidrome/Tailscale widgets — would require RSS-style scraping. Reconsider if Homepage proves heavy
• Dashy — evaluated as a Homepage alternative; heavier (Vue SPA), rebuild-on-config-change flow, more theme options. Worse fit for YAML-as-code config
• Heimdall / Homer / Flame — older static-tile launcher pages, no live data widgets. Mostly superseded by Homepage / Glance
• Portainer / Dockge — evaluated as a Docker-management UI complement to Homepage; declined because hand-written compose files in /data/docker/<service>/ are already the source of truth, and adding a GUI layer trades simplicity for a feature not currently needed (only ~3 stacks). Reconsider above ~5 stacks or if multiple operators need access
• Netdata — evaluated as a Beszel alternative; rejected for higher RAM footprint (~150 MB vs ~50 MB for Beszel hub+agent), opinionated cloud-signup nudges, and more metric depth than this scale needs. Self-host fully if revisited
• Glances — evaluated as a Beszel alternative; lightweight but no time-series persistence (graphs are real-time only, no history). Better as a souped-up htop than a dashboard
• Grafana + Prometheus + node_exporter + cAdvisor — evaluated as a Beszel alternative; the "industry-standard" stack but 4+ containers, hours of YAML, real learning curve. Right answer for a job, overkill for one homelab. Could be revisited if many machines + complex dashboards become the goal
• Checkmk / Zabbix / Nagios — enterprise-monitoring tools; wrong scale entirely
• Tailscale OAuth client auto-renewal for Homepage's widget token — evaluated 2026-05-07; rejected because OAuth tokens expire in 1 hour (Tailscale design constant) and Homepage's env_file is loaded only at container start, forcing ~48 daily container restarts for low-value widget data. Replaced with weekly cron + email reminder for manual 80-day rotation. Reconsider only if Homepage gains live env reload, or if multiple Tailscale-API-consuming services emerge to amortize the complexity.
• Nextcloud (Calendar app only) for self-hosted calendar — evaluated 2026-05-09 as a Radicale alternative; the calendar UI is the closest-to-iCloud.com FOSS web calendar that exists, includes a polished Tasks app, and federation/sharing features Radicale lacks. Rejected because Nextcloud is a 5+ container PHP platform requiring quarterly major-version upgrade care; Gabriel explicitly chose "fewer moving parts during exam crunch" robustness over UI polish. Reconsider only if (a) Radicale proves limiting in some specific way, or (b) Gabriel's tolerance for Nextcloud-style platform maintenance changes.
• Baïkal as a Radicale alternative — evaluated 2026-05-09; sabre/dav-based, same protocol guarantees, slightly more polished admin UI. Rejected because storage is SQLite (vs Radicale's flat files — more backup ceremony) and the admin GUI is unnecessary for a single user.
• vdirsyncer for CalDAV-to-CalDAV migration (preserving alarms) — evaluated 2026-05-10; would have preserved VALARM components that the iCloud public-cal feed strips. Rejected because (a) past-event alarms don't matter, (b) future alarms get re-added once in Radicale and persist forever via CalDAV, © it adds a Python tool with an Apple app-specific-password dependency for a one-time migration. Worth revisiting if a future migration involves alarms-as-data.
• Self-signed cert + iOS mobile-config trust profile — evaluated 2026-05-10 as the iOS-HTTPS path; would avoid tailscale serve. Rejected because Tailscale's Let's Encrypt path uses a publicly-trusted CA (no trust profile install needed, no perpetual "trust this device" footnote).
• DavMail / similar Outlook-CalDAV bridges — evaluated for Conestoga and Laurier Outlook accounts; rejected as fragile under modern auth, and solving the wrong problem (Conestoga Exchange already works fine on iOS natively; the publish-as-iCal feed handles Thunderbird).
• AgenDAV / SOGo / InfCloud (third-party CalDAV web frontends pairing with Radicale) — evaluated as a way to get a prettier calendar UI without taking on Nextcloud; rejected because all three look "2014" — if pretty UI matters enough to take on a separate frontend, Nextcloud's UI is better.
• Official Bitwarden self-hosted ("unified" container) — evaluated 2026-05-10 as the Vaultwarden alternative; same clients, but ~1 GB+ RAM, a bundled DB engine, and a license key for a few features (free for personal, still a moving part). Heavier than the rest of the stack for no gain at this scale.
• KeePassXC + KeePassium/Strongbox (iOS) + .kdbx over Samba/WebDAV — evaluated 2026-05-10 as the server-less, files-as-truth password option (matches Radicale's storage philosophy, trivially rsync-safe). Rejected on iOS integration: third-party iOS apps, sync-conflict risk if two devices edit at once, browser autofill needs the desktop app running. The brief was "integrates with iOS nicely".
• Passbolt / Padloc — Passbolt is team-oriented and PHP+MySQL-heavy; Padloc's iOS story is weak. Neither beats Vaultwarden for a single user.
• Standalone Bitwarden Authenticator app — a TOTP-only companion that keeps codes outside the password vault. Not used — Vaultwarden's built-in TOTP-in-vault is free and the goal is consolidation, not separation. Ente Auth (open-source, clean import/export) is the better choice if separation is ever wanted.
• Cloud-Mac rental (MacinCloud / Scaleway / AWS EC2 Mac) for the iCloud Passwords export — attempted 2026-05-11 (MacinCloud 48-hour managed-server trial, RDP). Apple's anti-fraud blocks iCloud sign-in from datacenter IPs — the device-passcode verification looped with "There was an error verifying the passcode on your iPhone" despite a correct passcode. Abandoned; the migration was eventually done on Gabriel's personal MacBook (~2026-05-22). The cloud-Mac route remains a dead end for anything iCloud-related — Apple's datacenter-IP rejection is a hard wall.

---

Future Plans (informed by Gabriel's stated priorities)

These are projects Gabriel has expressed interest in. If asked about them, you can build on this context.

• Drop iCloud Photos to 50 GB tier — Done in modified form (May 2026): iCloud Photos cleared, subscription downgraded to 200 GB tier ($3.99/mo) instead of 50 GB. 50 GB was rejected because iMessage post-purge (~60 GB) + Family Sharing pool (~85 GB) wouldn't fit, and the iPhone (128 GB / ~58 GB free) can't hold the full Messages history offline anyway. See Decisions table for full rationale.
• 2 TB external SSD for off-site backup — Resolved in modified form (May 2026): a refurbished Samsung 990 PRO 1TB was acquired and verified genuine, and became the on-site /mnt/backup (~916 GB usable — fits the full backup incl. the photo library); the demoted 500 GB T5 is now the off-site rotation drive (backup-offsite.sh — carried to parents' house, synced during visits). A >1 TB off-site drive is still wanted eventually so the photo library can be off-site too — right now the off-site T5 covers everything except the ~520 GB Immich library (it doesn't fit on 500 GB; accepted gap). Interim option: fit just the ~442 GB of irreplaceable photo originals onto the T5 by skipping thumbs/transcodes/DB-dumps — deferred, Gabriel's call.
• OS-drive shrink + new partition — Decided 2026-05-03, deferred to a physical-access session. Plan: shrink /dev/nvme1n1p2 from 953 GB → 200 GB, create /dev/nvme1n1p3 ~750 GB ext4 mounted at /mnt/scratch. Method: GParted Live USB. Prereqs: USB stick, monitor + keyboard physically attached, ~45-60 min uninterrupted, fresh backup + system-state tarball run immediately before.
• Energy Observatory — Continuous scraping/storage of Ontario IESO data into a database, building a dataset for power systems analysis. Expected to use Docker + PostgreSQL or SQLite.
• GridShift — Web tool modeling neighborhood-level transformer impacts from electrification adoption. Would benefit from being hosted on Z2 with public access via reverse proxy after September move.
• GridPulse Canada — Real-time visualization of Canadian provincial grids.
• Off-Diesel — Stretch project for remote community renewable feasibility analysis.
• Off-site backup at parents' house — Implemented (May 2026): the demoted 500 GB Samsung T5, reformatted as backup-offsite (UUID 2c8e8e38-f129-4823-a1db-1529d3296b44), lives at parents' house. During a family visit: plug it into z2mini → sudo mount /dev/disk/by-label/backup-offsite /mnt/offsite → ~/scripts/backup-offsite.sh → sudo umount /mnt/offsite → unplug → carry back. Copies current/{files,music,db-dumps,service-config}/ + system-state/ (~3 GB) — NOT the ~520 GB Immich library (doesn't fit; a >1 TB off-site drive would close that). Writes /mnt/offsite/offsite-backup.log. See 05-backups.md.
• Mac connection setup — Will mirror Windows setup using smb://z2mini from Finder.
• Migration to ethernet — Planned for September 2026 move; will improve reliability and bandwidth.
• HTTPS with custom subdomains — Deferred; will be tackled when there's a public-facing service to justify it.
• Ansible playbook for full system reproduction — Future iteration of recovery: instead of restoring configs, write infrastructure-as-code that recreates the system on any hardware. Replaces the current tar-based config backup with a more powerful, version-controlled approach.
• Wire Immich into nightly backup — Done (May 2026): the library/ UPLOAD_LOCATION (incl. encoded-video/) is rsync-mirrored to /mnt/backup/current/immich/; the Postgres DB is captured via Immich's own built-in nightly auto-backup writing .sql.gz dumps into library/backups/ (default ON — must stay on), which the 03:00 rsync picks up. postgres/ is never filesystem-rsync'd; model-cache/ not backed up. See 11-immich.md / 05-backups.md.
• Wire Navidrome into nightly backup — Done (May 2026): /data/music/ rsync-mirrored; navidrome.db captured via host sqlite3 ".backup" into current/db-dumps/; .env+compose in service-config/navidrome/; data/artwork/+data/cache/ not backed up (regenerable). Never raw-rsync the live navidrome.db.
• Music acquisition — working list at music-acquisition-list.md on feat/music-install branch. Tier 1 (Imagine Dragons, Linkin Park, Bastille, Maroon 5, Ed Sheeran), Tier 2 (Latin / hip-hop / pop top hits), Tier 3 (curated playlists — classical canon, hymns, Christian pop, sports/hype, World Cup tracks). Mostly iTunes Store; library borrows for classical and older Latin; Bandcamp for indie acoustic.
• Wire Beszel into nightly backup — Done (May 2026): both hub SQLite DBs (data.db AND auxiliary.db) captured via host sqlite3 ".backup" into current/db-dumps/; .env+compose in service-config/beszel/; the hub's data/id_ed25519 SSH key not backed up (root-owned, unreadable — regenerable, but means re-pairing the one agent); agent-data/+socket/ not backed up. Also: the agent's disk-monitoring labels — /extra-filesystems/data__Data + /extra-filesystems/backup__Backup show as "Data"/"Backup" in the hub UI; the OS root shows as nvme1n1p2 (Beszel can't rename the root fs).
• Cut iCloud Calendar (Phase 7 of the Radicale migration) — Planned ~2026-05-24, after the two-week parallel-run window completes cleanly. Steps: iOS Settings → [name] → iCloud → Calendars OFF → "Delete from iPhone"; then iCloud.com → Calendar → delete each calendar to actually purge from Apple's servers. Don't disable iCloud Calendar before the parallel-run window completes — recovery from a Radicale problem is much harder if iCloud is already gone.
• Re-add alarms on the ~10 important recurring events post-migration — iCloud's public-calendar feed stripped all VALARMs during the 2026-05-10 migration. Re-add alarms once in Thunderbird or iOS for: monthly bills, key class times, deadline-style recurring reminders. Each alarm is set on the recurring master event and persists forever via CalDAV.
• Wire Radicale into nightly backup — Done (May 2026): .env+compose+config/+data/collections/ rsync'd into service-config/radicale/ — no pg_dump, no SQLite ceremony, files-as-truth; verified byte-for-byte; the calendars are on the off-site T5 too.
• Self-host CardDAV (contacts) on the same Radicale server — Deferred follow-up after the calendar parallel-run window completes. Same daemon, same auth, same storage tree; just create a gabriel/contacts/ collection of type addressbook and add the same server URL as a CardDAV account on iOS. Contact sync is a one-shot to-Radicale move similar to the calendar migration.
• Update RECOVERY-README.md for the Caddy stack — After a Z2 rebuild: restore /data/docker/caddy/ (or recreate Dockerfile + docker-compose.yml + Caddyfile + .env with the Cloudflare token), docker compose build && docker compose up -d, and sudo tailscale set --operator=gabriel after Tailscale re-auth. The Caddyfile already has every service's block, so all the <svc>.z2mini.gabrielgabrie.com endpoints come back at once. (This replaces the old "re-run sudo tailscale serve ... for Radicale and Vaultwarden" step — those listeners are retired.) See 17-caddy.md.
• Migrate iCloud Passwords into Vaultwarden — Done (~2026-05-22) on Gabriel's personal MacBook: Passwords app → File → Export All Passwords → CSV → web vault → Tools → Import data → format Apple Passwords (csv) → verified all present (minus Wi-Fi passwords + passkeys — both expected) → CSV deleted. iOS AutoFill switched to Bitwarden, iCloud Passwords AutoFill off. The MacinCloud cloud-Mac attempt (2026-05-11) had failed Apple's datacenter-IP anti-fraud; a personal trusted Mac was the path that worked.
• Configure the Windows Bitwarden clients — Done: desktop app + browser extension on Windows, plus Bitwarden Desktop on the MacBook. All four clients (iOS, Mac, Windows, browser ext) pointed at https://vault.z2mini.gabrielgabrie.com.
• Delete the iCloud Passwords entries (post-migration cleanup) — the migration only copied; the entries still exist in iCloud. Recommended: from the MacBook, Passwords app → Cmd+A → Delete → authenticate; syncs the deletions to all Apple devices. Do NOT disable iCloud Keychain itself — keep that on for Wi-Fi password sync (can't migrate to Vaultwarden by design). Low-pressure; do after a few days of confirmed Bitwarden-only use.
• Re-create the 8 existing passkeys in Vaultwarden, per-site — Bitwarden already set as the iOS passkey provider; new passkeys go to the vault automatically. Existing passkeys can't be exported (by design), so per site: register a new Vaultwarden passkey → confirm it works → then delete the iCloud passkey for that site. Doing it in that order avoids lockout.
• Consolidate TOTP into the Vaultwarden vault — per-site re-enroll off Microsoft Authenticator — no clean seed export from MS Authenticator, so per site: security settings → remove old authenticator → add new → scan QR into the matching Bitwarden login entry. MS Authenticator stays only for accounts that mandate the MS app (Conestoga / Conditional Access / number-matching push). Gradual.
• Run the Vaultwarden password-health report and rotate bad passwords — web vault → reused / weak / HIBP-exposed report (a paid Bitwarden-cloud feature, free on Vaultwarden) → rotate the genuinely-bad ones. This (not deleting stale entries) is the high-value cleanup post-migration.
• Optional: SSH keys in Vaultwarden — Bitwarden has an SSH key item type + the desktop app can act as an SSH agent. Worth doing for the GitHub key (cross-device sync between MacBook + Windows laptop; vault unlock per use). Wire-up: Bitwarden Desktop → Settings → SSH agent → enable → point the OS's SSH client at the Bitwarden agent socket. Does NOT apply to ssh gabriel@z2mini (Tailscale SSH uses tailnet identity, not a key), to iOS (no usable agent integration), or to headless/CI SSH (still needs a bare key file).
• Wire Vaultwarden into nightly backup — Done (May 2026): db.sqlite3 captured via host sqlite3 ".backup" into current/db-dumps/vaultwarden-db.sqlite3; rsa_key.pem + attachments/ + sends/ + config.json + .env + compose in service-config/vaultwarden/ (live db.sqlite3 + tmp/ excluded from that rsync). Never raw-rsync the live WAL-mode db.sqlite3 (same rule as Beszel / Navidrome). Vault DB dump is on the off-site T5 too.
• Optional: enable mobile push for instant iOS sync — register (free) at https://bitwarden.com/host/, add PUSH_ENABLED=true + PUSH_INSTALLATION_ID + PUSH_INSTALLATION_KEY to .env, docker compose up -d. Without it the iOS app syncs on open / manual pull rather than instantly.
• Optional: enable the Vaultwarden admin panel — generate an Argon2 hash (docker run --rm -it vaultwarden/server /vaultwarden hash), set ADMIN_TOKEN='$argon2id$...' (quoted) in .env, docker compose up -d, then /admin for server config / user list.

---

Operational Notes

Common tasks

# SSH access (from Gabriel's laptop)
ssh gabriel@z2mini

# Manual backup trigger (nightly on-site)
~/scripts/backup-files.sh

# Off-site sync (MANUAL — T5 plugged in)
sudo mount /dev/disk/by-label/backup-offsite /mnt/offsite
~/scripts/backup-offsite.sh
sudo umount /mnt/offsite

# Check backup status
tail -50 /mnt/backup/backup.log
ls /mnt/backup/current/         # files/ immich/ music/ db-dumps/ service-config/

# Drive health spot-check
sudo smartctl -a /dev/nvme0n1
sudo smartctl -a /dev/nvme1n1
sudo smartctl -a -d sntasmedia /dev/sdb     # the 990 PRO over the ASM2462 USB enclosure (use whatever lsblk shows)

# System updates (recommended monthly)
sudo apt update && sudo apt upgrade -y

# Disk usage check
df -h

# Service status check
sudo systemctl status smbd tailscaled smartd

# Immich — current container state
cd /data/docker/immich && docker compose ps

# Immich — tail the server logs
cd /data/docker/immich && docker compose logs -f --tail 50 immich-server

# Immich — disk use
du -sh /data/docker/immich/library/ /data/docker/immich/postgres/ /data/docker/immich/model-cache/

# Immich — total asset count via API (requires API key in ~/.immich-api-key)
curl -sH "x-api-key: $(cat ~/.immich-api-key)" http://127.0.0.1:2283/api/server/statistics | python3 -m json.tool

# icloudpd — re-attach to the running bulk download
tmux attach -t icloud-migration

# icloudpd — tail the bulk-download log
tail -f /tmp/icloudpd-bulk.log

# icloudpd — check tmux session status
tmux ls

# icloudpd — count files downloaded so far
find /data/icloud-import -type f | wc -l && du -sh /data/icloud-import

# immich-go — bulk import staging into Immich (after icloudpd finishes)
/home/gabriel/scripts/immich-go upload from-folder \
  --server http://127.0.0.1:2283 \
  --api-key "$(cat ~/.immich-api-key)" \
  --manage-heic-jpeg StackCoverJPG \
  --no-ui \
  /data/icloud-import

# Navidrome — current container state
cd /data/docker/navidrome && docker compose ps

# Navidrome — tail server logs
cd /data/docker/navidrome && docker compose logs -f --tail 50 navidrome

# Navidrome — disk use (music library + Navidrome state)
du -sh /data/docker/navidrome/data/ /data/music/

# Navidrome — manual scan trigger via Subsonic API (replace PASS)
curl -sS "http://127.0.0.1:4533/rest/startScan.view?u=admin&p=PASS&v=1.16.1&c=cli&f=json"

# Navidrome — count of indexed tracks via API (replace PASS)
curl -sS "http://127.0.0.1:4533/rest/getSongs.view?u=admin&p=PASS&v=1.16.1&c=cli&f=json&size=1" | python3 -m json.tool

# Homepage — current container state
cd /data/docker/homepage && docker compose ps

# Homepage — tail server logs
cd /data/docker/homepage && docker compose logs -f --tail 50 homepage

# Homepage — show widget env vars resolved inside container (sanity-check token loading)
docker compose -f /data/docker/homepage/docker-compose.yml exec homepage env | grep HOMEPAGE_VAR_

# Homepage — verify Docker socket reachable from inside the container (live status dots)
docker compose -f /data/docker/homepage/docker-compose.yml exec homepage ls -la /var/run/docker.sock

# Beszel — current container state (hub + agent)
cd /data/docker/beszel && docker compose ps

# Beszel — tail hub or agent logs
cd /data/docker/beszel && docker compose logs -f --tail 50 beszel
cd /data/docker/beszel && docker compose logs -f --tail 50 beszel-agent

# Beszel — disk use of hub DB
du -sh /data/docker/beszel/data/

# Beszel — SQLite-online backup (use this, NEVER raw rsync of an open SQLite DB)
docker compose -f /data/docker/beszel/docker-compose.yml exec beszel \
  sqlite3 /beszel_data/data.db ".backup /beszel_data/data-backup.db"

# Radicale — current container state
cd /data/docker/radicale && docker compose ps

# Radicale — tail server logs (auth failures, PROPFIND requests visible)
cd /data/docker/radicale && docker compose logs -f --tail 50 radicale

# Radicale — disk use of collections tree
du -sh /data/docker/radicale/data/collections/

# Radicale — list collections (per user)
ls /data/docker/radicale/data/collections/collection-root/gabriel/

# Radicale — count events in a specific collection
ls /data/docker/radicale/data/collections/collection-root/gabriel/<uuid>/*.ics | wc -l

# Radicale — verify HTTPS endpoint via tailscale serve (should return HTTP 207 PROPFIND)
curl -s -o /dev/null -w 'HTTP %{http_code}\n' \
  -X PROPFIND -H 'Depth: 0' \
  -u 'gabriel:<password>' \
  https://radicale.z2mini.gabrielgabrie.com/gabriel/

# Radicale — show tailscale serve proxy config
sudo tailscale serve status

# Radicale — generate a new bcrypt hash (e.g. password rotation)
docker run --rm --entrypoint /venv/bin/python3 \
  tomsquest/docker-radicale:3.7.2.0 \
  -c "import bcrypt; print('gabriel:' + bcrypt.hashpw(b'<new-password>', bcrypt.gensalt(rounds=12)).decode())"

# Vaultwarden — current container state
cd /data/docker/vaultwarden && docker compose ps

# Vaultwarden — tail server logs (failed logins, sync, WebSocket connections visible)
cd /data/docker/vaultwarden && docker compose logs -f --tail 50 vaultwarden

# Vaultwarden — health + resource use
curl -fsS https://vault.z2mini.gabrielgabrie.com/alive       # returns a timestamp
docker stats --no-stream vaultwarden                       # idles ~10–15 MiB

# Vaultwarden — disk use
du -sh /data/docker/vaultwarden/data/

# Vaultwarden — inspect the vault at the DB level (metadata only — contents are encrypted)
docker run --rm -v /data/docker/vaultwarden/data:/data:ro --entrypoint sh keinos/sqlite3 -c '
  sqlite3 -header -column /data/db.sqlite3 "SELECT email, updated_at FROM users;"
  sqlite3 -header -column /data/db.sqlite3 "SELECT count(*) AS n_items FROM ciphers WHERE deleted_at IS NULL;"'

# Vaultwarden — show tailscale serve proxy config (both Radicale :443 and Vaultwarden :8443)
sudo tailscale serve status

# Vaultwarden — SQLite online backup (NEVER raw-rsync the live WAL-mode db.sqlite3)
docker run --rm -v /data/docker/vaultwarden/data:/data --entrypoint sh keinos/sqlite3 \
  -c 'sqlite3 /data/db.sqlite3 ".backup /data/db-backup.sqlite3"'

# Vaultwarden — generate an Argon2 hash for the admin panel (then put ADMIN_TOKEN='...' in .env, quoted)
docker run --rm -it vaultwarden/server /vaultwarden hash

Recovery procedures

• Accidental file deletion within 7 days: Browse to /mnt/backup/daily/<date>/files/ and copy back (note the files/ subdir — current//daily//weekly/ also hold immich/, music/, db-dumps/, service-config/).
• Accidental file deletion within 4 weeks: Browse to /mnt/backup/weekly/<date>/files/.
• Z2 hardware failure: Plug the 990 PRO (in its ASM2462 USB enclosure) into another Linux machine, mount, copy data from /mnt/backup/current/{files,immich,music,db-dumps,service-config}/. The off-site T5 (backup-offsite) is a second copy of everything except the Immich library.
• Full system rebuild: Follow /mnt/backup/RECOVERY-README.md (and 08-recovery.md) for step-by-step. Restores OS configuration, package list (incl. sqlite3), crontab, data, and the Docker stacks (service-config/<svc>/ + the db-dumps/ SQLite DBs + the Immich library/ + .sql.gz DB dump per Immich's docs + docker compose build && up -d for Caddy). Estimated time: ~1 hour for the OS/config baseline; the Immich library rsync-back is the long pole if /data was lost.
• After any full rebuild — bring up the Caddy stack: /data/docker/caddy/ (Dockerfile + compose + Caddyfile + .env with the Cloudflare token) is under /data/docker/, so it's in the backup. Restore it, then docker compose build && docker compose up -d and sudo tailscale set --operator=gabriel (after Tailscale re-auth). The Caddyfile already has every service's block — all the https://<svc>.z2mini.gabrielgabrie.com endpoints come back at once. (No more tailscale serve re-init — those listeners are retired. See 17-caddy.md.)
• Vaultwarden vault restore: stop the container; copy db-dumps/vaultwarden-db.sqlite3 in as data/db.sqlite3 (remove any stale -shm/-wal); restore rsa_key.pem + attachments/ + sends/ + config.json + .env + compose from service-config/vaultwarden/; chown -R 1000:1000 data/; docker compose up -d. Losing rsa_key.pem is survivable (every client just has to log in again) but it's backed up anyway.
• Other-service restore (Navidrome / Beszel / Radicale / Homepage / Caddy / Immich): service-config/<svc>/ → /data/docker/<svc>/; the SQLite ones get their db-dumps/<name>.db dropped in as the live DB; Immich = rsync current/immich/ → library/, then recreate Postgres from the most recent library/backups/*.sql.gz per Immich's docs; Caddy = docker compose build && up -d (re-issues certs). See 08-recovery.md Step 6b for the exact commands.
• SSH lockout: Physical access required. Monitor and keyboard live somewhere accessible, not in deep storage.

Known quirks

• Backup log lives on the backup drive itself — if the backup drive is unmounted, errors silently fail to log. The script's mount check prevents this becoming dangerous but does mean diagnostic info is lost in that scenario.
• lost+found directory is hidden from the SMB share via veto files (Windows clients only) — it may still appear visible from iOS Files.
• AppArmor enforces strict permissions on msmtp; modifying its config requires updating the local override file.
• The backup drive's NVMe SMART data is only readable through the ASM2462 USB enclosure with smartctl -d sntasmedia; without that flag you see a generic USB mass-storage device. NVMe scheduled self-tests don't pass through the bridge (admin command 0x14 unsupported), so the smartd line for it has no -s schedule.
• After the backup drive was swapped (T5 → 990 PRO) the Beszel agent kept reporting the old sda1 until docker restart beszel-agent; the hub also shows stale "ghost" disk entries (nvme0n1p1, sdb1, sda1) for a while after a relabel until they age out of retention — harmless.
• A live SQLite DB (Vaultwarden's db.sqlite3, Beszel's data.db/auxiliary.db, Navidrome's navidrome.db) must never be raw-rsync'd — the backup script uses sqlite3 ".backup" for these. Likewise Immich's postgres/ data dir is never rsync'd — the .sql.gz dumps in library/backups/ (written by Immich's own nightly auto-backup, default ON) are the DB copy.

---

Things to Avoid Suggesting Without Explicit Request

• Reformatting drives or changing partition layouts (the OS-drive shrink is decided but stays deferred until Gabriel chooses a physical-access session)
• Removing or replacing core services (Samba, Tailscale, smartd)
• Changing the backup script logic or retention policy
• Installing alternative file-sharing systems (Nextcloud, Seafile, etc.) or alternative photo apps (PhotoPrism, Ente, Photoview — already evaluated)
• Moving to dedicated NAS hardware or NAS-focused operating systems
• Adding services that bind to common ports already in use
• Cloud-based backup services or paid SaaS replacements
• VPN alternatives to Tailscale unless Tailscale specifically fails
• Disabling GPU acceleration on Immich's ML container (it's been enabled on the Quadro T2000 with pre-signed kernel modules; rolling back loses 4-5× speedup on face/CLIP/OCR jobs)
• Dropping the iCloud subscription further to the 50 GB tier (deliberately rejected May 2026 — iMessage ~60 GB and Family Sharing pool ~85 GB don't fit; iPhone storage can't hold full Messages history offline; see Decisions table)
• Re-enabling iCloud Photos to recreate the off-site safety net for Immich (the explicit choice was to accept the risk window until the 2 TB external arrives, not to keep paying for iCloud as redundant storage)
• "Turn off Messages in iCloud to save iCloud space" — it doesn't save quota, it just moves bytes from the Messages bucket into the iCloud Backup bucket in the same plan
• Filesystem rsync of /data/docker/immich/postgres/ (live Postgres data dir is corruption-unsafe under rsync — use pg_dump)
• Dropping files into Immich's library/ directly (the DB won't know about them; storage template won't apply; faces and search won't work)
• Plex / Plex Pass alternatives to Navidrome (Plex Pass cost violates the no-subscription rule)
• Converting the AAC music library to FLAC for "quality" reasons (audibly transparent on the user's listening gear; cost is 7× storage and a destructive workflow)
• Treating /data/music/ as part of /data/docker/navidrome/ — they're deliberately separate so Samba can write to one without exposing the other
• Filesystem-rsync of /data/docker/navidrome/data/navidrome.db while the container is running (use SQLite's online .backup API or stop the container first)
• Suggesting Amperfy as the iOS Subsonic client until iOS 26 is in use (its recent versions require iOS 26)
• Replacing Homepage with Glance/Dashy/Heimdall/Homer/Flame (already evaluated — Homepage's native widgets for Immich/Navidrome/Tailscale are the deciding factor)
• Hardcoding Homepage widget API tokens directly into services.yaml (the .env + {{HOMEPAGE_VAR_*}} substitution pattern is intentional — keeps secrets out of a file that's otherwise safe to share as documentation)
• Mounting the Docker socket read-write into Homepage (the :ro mount is a safety property; Homepage only needs read access to enumerate container status)
• Replacing Beszel with Netdata / Glances / Grafana+Prometheus / enterprise monitoring tools (already evaluated — Beszel's RAM footprint, modern UI, and built-in alerting are the deciding factors at this scale)
• Updating the Beszel hub or agent independently (wire protocol can skew between minor versions — pin and update both together)
• Reusing msmtp's Gmail app password for Beszel SMTP (defeats the revocation-independence reason for using a separate one)
• Filesystem-rsync of /data/docker/beszel/data/data.db while the hub container is running (use SQLite's online .backup API or stop the container first — same corruption risk as Navidrome's SQLite)
• Splitting Beszel hub and agent into separate compose files on a single-host setup (the unix-socket pattern in one compose is the documented norm and avoids exposing the agent on TCP)
• Replacing Radicale with Nextcloud / Baïkal / AgenDAV / SOGo / InfCloud (already evaluated — files-as-truth storage, ~30 MB RAM, and "fewer moving parts during exam crunch" are the deciding factors)
• Suggesting Nextcloud "for the prettier calendar UI" — was deliberated and the moving-parts trade-off won
• Replacing Vaultwarden with the official Bitwarden self-host image / KeePassXC / Passbolt / Padloc (already evaluated — Vaultwarden's RAM footprint, free "premium" features, and best-in-class iOS integration via the official Bitwarden clients are the deciding factors)
• Looking for a "Vaultwarden client" — there isn't one; the clients are the official Bitwarden desktop app / browser extensions / mobile apps pointed at https://vault.z2mini.gabrielgabrie.com
• Reverting Vaultwarden to run as root (with CAP_DAC_OVERRIDE/CAP_CHOWN re-added) or chown -R 0:0 of its data/ — running as user: 1000:1000 is the deliberate fix that keeps cap_drop: ALL viable and the data human-owned; a root container crash-loops on rsa_key.pem under cap_drop: ALL
• Putting Vaultwarden on port 80 inside the container, or binding it to 0.0.0.0/the tailnet IP — ROCKET_PORT=8080 + 127.0.0.1-only is intentional (lets a non-root capability-less process bind it; Caddy reaches localhost fine via network_mode: host)
• Proposing a /vault subpath, self-signed certs + an iOS trust profile, or per-service tailscale serve listeners for Vaultwarden — all evaluated/superseded; the chosen path is Caddy fronting https://vault.z2mini.gabrielgabrie.com on :443 with a Let's Encrypt cert (ACME DNS-01 via Cloudflare). DOMAIN in .env must match that URL exactly (it's the WebAuthn/passkey RP-ID).
• Filesystem-rsync of /data/docker/vaultwarden/data/db.sqlite3 while the container is running (use SQLite's online .backup API or stop the container first — same corruption risk as Navidrome's and Beszel's SQLite); and always back up rsa_key.pem alongside it
• Re-running the cloud-Mac route for the iCloud Passwords export (MacinCloud was attempted 2026-05-11; Apple's anti-fraud blocks iCloud sign-in from datacenter IPs — the export needs a trusted physical Mac)
• Suggesting the standalone Bitwarden Authenticator app, or telling Gabriel to keep TOTP in Microsoft Authenticator long-term (the plan is TOTP-in-the-Vaultwarden-vault; MS Authenticator stays only for accounts that mandate the MS app)
• Suggesting Gabriel pre-clean / delete stale entries from his ~400 iCloud Passwords before the bulk import (same one-shot import either way; clean up opportunistically in Bitwarden afterward, and the password-health report is the high-value cleanup)
• Changing Radicale's 127.0.0.1:5232:5232 binding to 0.0.0.0 or the tailnet IP — 127.0.0.1-only is intentional; Caddy reaches it there (network_mode: host) and is the only thing exposed on the tailnet
• Removing Radicale's Caddyfile block / its HTTPS front-end (iOS 18+ Calendar will refuse to add the account — TLS is forced regardless of the in-app SSL toggle; Caddy supplies the real cert)
• Suggesting self-signed cert + iOS mobile-config trust profile for Radicale's HTTPS, or reverting to tailscale serve (both superseded — Caddy's Let's Encrypt-via-DNS-01 path is cleaner, no profile install, single ingress)
• Re-running the iCloud → Radicale migration with vdirsyncer to recover alarms (was evaluated and the manual-re-add path was deliberately chosen for the ~10 important recurring events)
• Dropping .ics files directly into /data/docker/radicale/data/collections/<user>/<uuid>/ (use Thunderbird's Import-into-collection workflow; direct file drops bypass CalDAV's etag/sync-collection bookkeeping and clients can desync)
• Filesystem permission changes on Radicale's config/users to mode 600 + chown to UID 2999 (mode 644 is intentional — bcrypt hashes are designed to be safe at the filesystem level, and the current perms keep gabriel as the file owner without needing sudo for routine ops)
• Disabling iCloud Calendar before the Phase 6 parallel-run window completes (planned ~2026-05-24; recovery from a Radicale problem is much harder once iCloud is gone)
• Adding a DavMail-style Outlook-CalDAV bridge for Conestoga or Laurier (was rejected as fragile under modern auth; Conestoga Exchange + the publish-as-iCal feed cover the use case)
• Re-adding the Laurier Outlook account (Gabriel quit; the account was dropped intentionally)

When in doubt: ask Gabriel before changing existing infrastructure.

---

This document is intended to be living. If significant infrastructure changes are made, this file should be updated to reflect the new state.