Skip to content

Power Resilience

What happens to the Z2 Mini when the building loses power, and what has been done in software to shorten the outage, detect it, and prove nothing broke.

This page exists because of a real outage. On 2026-09-09 at 12:52 EDT the apartment lost power. The Z2 died instantly. It came back at 19:35 EDT — not because power returned, but because Gabriel walked in the door and pressed the button. 6 hours 43 minutes of downtime from an outage that lasted seconds.

What actually failed, and what didn't

The outage itself was the smaller problem. Three separate weaknesses turned it into most of a day:

Weakness Consequence Fixed by
Machine set to stay off after AC loss 6h43m of the 6h43m BIOS After Power Loss (below)
No external monitoring Nobody knew until someone physically looked Dead-man's switch (below)
cron silently skips missed jobs A backup during an outage window would never run and never warn systemd timers with Persistent=true

What held up well: every service came back on its own. All 19 containers restarted because each stack uses restart: unless-stopped and — critically — nothing had been docker stopped beforehand. See 09-maintenance.md for why an explicit docker stop breaks that.

No data was lost. PostgreSQL replayed its WAL, and all four SQLite databases passed PRAGMA integrity_check.

The hard limit

Software cannot keep a computer running without electricity, and it cannot press a power button.

Everything on this page is about the three things software can do: come back faster, tell you it happened, and prove nothing broke. Riding through an outage, or shutting down cleanly before a battery dies, needs a UPS — the software half of that is NUT, but it needs hardware to talk to. Not currently installed.

1. Automatic power-on after AC returns

This is firmware, not software, and it recovers almost the entire outage window.

The Z2 Mini ships set to stay powered off when AC is restored. Change it:

  1. Power on and press F10 repeatedly to enter BIOS setup
  2. Advanced → Power Management Options
  3. Set After Power Loss to Power On
  4. F10 to save and exit

Why not do this from Linux? The setting lives in the chipset's AfterG3_En bit, reachable with setpci -s 00:1f.0 A4.B. Don't bother — the BIOS rewrites that register at every POST, so the change does not survive a reboot. The firmware menu is the only durable place to set it.

Getting into the BIOS without catching F10

The firmware advertises EFI_OS_INDICATIONS_BOOT_TO_FW_UI, so Linux can request a reboot straight into setup — no key timing:

sudo systemctl reboot --firmware-setup

Run it at the machine, or over SSH with ssh -t to force a TTY. Without a TTY, sudo cannot prompt and fails with "a terminal is required to read the password" — and dropping the sudo just moves the failure to polkit.

Verifying the setting from Linux

You cannot durably set After Power Loss outside the firmware, but you can read it. The BIOS writes the chipset's AfterG3_En bit at every POST, so it reflects the current setting:

sudo setpci -s 00:1f.0 A4.B

Bit 0 of the returned byte: 0 = powers on after AC loss (what you want), 1 = stays off. Confirmed 0x00 on 2026-09-09 after the setting was changed.

Also check uptime -s after the next power event — it should match when power returned, not when someone pressed the button.

2. Dead-man's switch — outbound heartbeat

Nothing self-hosted can tell you the Z2 is down. Beszel runs on the Z2, so it dies with the Z2. And nothing outside can poll in: every service is bound to the tailnet IP 100.67.235.68 with no inbound path from the public internet.

The only architecture that works is an outbound heartbeat where the absence of a ping is the alarm. The Z2 checks in every 5 minutes; when it stops, the external service notices and emails.

Script: /home/gabriel/scripts/heartbeat.sh, driven by z2mini-heartbeat.timer.

Setup

The timer is already installed and running, but stays a no-op until it has a URL:

  1. Sign up free at healthchecks.io
  2. Create a check named z2mini — period 5 minutes, grace 10 minutes
  3. Put its ping URL in the config file:
ssh gabriel@z2mini
nano ~/.config/z2mini/healthchecks-url
sudo systemctl start z2mini-heartbeat.service

sudo is required — without it systemctl start goes through polkit, which has no way to prompt you over a non-interactive SSH command and fails with "Interactive authentication required." The same trap applies to systemctl reboot --firmware-setup below.

  1. Confirm the check flips to green in the healthchecks.io dashboard

You'll get an email roughly 15 minutes after the Z2 goes dark — 5 for the missed ping, 10 for the grace period.

The URL file is mode 0600 and lives outside the repo. A ping URL is a write-only token — anyone holding it can only mark your check as alive — but there is no reason to publish it.

3. Post-unclean-boot integrity check

After a hard power cut you want to know, without doing it by hand, whether the databases survived. z2mini-post-crash-check.service runs at every boot and answers that question by email.

Script: /home/gabriel/scripts/post-crash-check.sh

How it decides a crash happened: systemd writes a shutdown sequence to the journal on the way down. A power cut leaves none of it, so the script greps the tail of the previous boot for systemd-shutdown / Reached target Power-Off. Absent means unclean. On a clean boot the script exits silently and sends nothing.

What it checks when the previous boot was unclean:

Check Detail
Filesystems ext4 orphan-cleanup / recovery / error lines from this boot
SQLite x4 PRAGMA integrity_check on Vaultwarden, Beszel, Beszel-aux, Navidrome
PostgreSQL x2 pg_isready on Immich and OpenProject, plus a log scan for corrupt / invalid page / checksum failed / PANIC
Containers running count against the expected 19
Backup /mnt/backup still mounted, and the last line of backup.log

The email subject states the verdict outright — all checks passed, or N PROBLEM(S) FOUND.

It waits 90 seconds after docker.service before judging anything, so Postgres has time to finish WAL replay and the stacks have time to come up.

Run it by hand any time:

ssh gabriel@z2mini '~/scripts/post-crash-check.sh --force'

ext4 "orphan cleanup" is not damage. It is the journal repairing an unclean mount, which is exactly what a journalling filesystem is for. The script reports it as context, not as a fault.

4. Scheduled jobs that survive an outage

cron does not run jobs it missed while the machine was off. An outage from 02:00 to 08:00 would mean that night's backup simply never happened — no error, no log line, no alert. A silent gap in the one system whose entire job is not having gaps.

All three jobs moved from the user crontab to systemd timers with Persistent=true, which records each run and fires a missed one shortly after the next boot.

Timer Schedule Persistent Runs
z2mini-backup.timer daily 03:00 yes backup-files.sh
z2mini-backup-mount-check.timer hourly yes check-backup-mount.sh
z2mini-token-age-check.timer Mondays 09:00 yes tailscale-token-age-check.sh
z2mini-heartbeat.timer every 5 min no heartbeat.sh

The heartbeat is deliberately not persistent. A missed heartbeat is the alarm itself — replaying it after the fact would be worse than useless, since it would tell the monitor everything was fine during an outage.

The old cron lines are commented out in place rather than deleted, prefixed # migrated to systemd timer 2026-09-09:, so crontab -l still shows the history.

Checking on them

systemctl list-timers 'z2mini-*'
systemctl status z2mini-backup.service
journalctl -u z2mini-backup.service -n 50
sudo systemctl start z2mini-backup.service

Unit files live in /etc/systemd/system/z2mini-*.service and z2mini-*.timer.

Verifying the whole chain

After any power event, these four commands tell you everything:

uptime -s
systemctl list-timers 'z2mini-*'
~/scripts/post-crash-check.sh --force
docker ps -q | wc -l

Expected: a boot time matching when power returned, no skipped timers, an all-clear email, and 19 containers.

What's still open

  • No UPS. A short brownout still takes the machine down hard. Every unclean shutdown is a roll of the dice on database corruption — so far the dice have come up clean.
  • The BIOS setting is manual. It cannot be scripted durably, so if the board is ever reset to defaults or replaced, it silently reverts. Re-check it after any firmware update.
  • The heartbeat depends on the home internet. An ISP outage with the Z2 still running will page you as though the machine died. That is the correct failure direction — a false alarm beats silence — but expect it.