There's a rule I've used for most of my working life, long before any of this: try bold things, but know how you'd take them back. If a change can be undone cheaply, you can afford to try it. If it can't, you slow right down, check twice, and ask someone first.

It turns out to be the single most important rule for letting an AI run a server. Not because the AI is careless. Because it's fast. It will happily make twenty changes before I've finished my coffee, and some of them will be wrong. It has made 24 mistakes that I know of, and they're all written down. What makes that fine is that nearly all of them cost minutes, not data.

The title of this post is a goal, not a promise. Configuration and data on this server can almost always be rolled back. Some things can't, and I'll get to those, because they're the more interesting half.

Every config change is a commit

Every time the AI changes something on the server (a config file, a script, a scheduled job), it commits the change to a private GitHub repository with a message saying what changed and why. In the first four weeks that came to 199 commits. Before it edits an important file, it also keeps a copy of the original on the server, so the old version is one command away even without GitHub. There are 151 of those so far.

A nightly job sweeps up anything that changed without a commit, such as a package update rewriting a file. That's the safety net, not the plan: the plan is one commit per deliberate change, so the history reads like a change log and any single change can be reverted on its own.

Putting a server's configuration on GitHub has an obvious risk, so nothing gets pushed until a secret scanner has been through it, along with a second check that looks for the actual passwords and keys used on this server. If either finds anything, the push stops. That check exists because of the first item on the "can't be undone" list below.

The data has an undo button too

I covered this in the backups post, so briefly: the important data is snapshotted every hour, and there are about 1,170 snapshots on the pool right now. Rolling a dataset back takes seconds. It's the undo button for "the AI broke the mail folder", and also for "I broke the mail folder", which is statistically more likely.

Updates that test themselves

Software updates are where most home servers quietly rot, because updating is a small risk every week and not updating is a big risk eventually. So the apps here update themselves, and every update can take itself back.

Before an app updates, its database is dumped and its data is snapshotted. Then the new version is pulled and started, and a health check asks the app whether it's actually working, not just whether the container is running. If the answer is no, the app is stopped, the data goes back to the snapshot, and the old version is started again. I get a message whenever anything was updated or rolled back.

Before I trusted that with anything real, we tested the rollback on a throwaway app that existed only to be broken.

The rollback has run for real once since then, and in the most annoying way possible: the update was fine and the health check was wrong. The torrent client's web interface answers "forbidden" when you're not logged in, which is exactly what it should do. The check was meant to treat that as healthy, but it never did, so it decided a perfectly good update had failed and rolled it back. The rollback itself worked perfectly. The lesson was about the check, and so is the fix: a test of the check that also covers the answers it's meant to accept.

The one app that used to be left out was Immich, my photo library, because its updates change the database and its database doesn't live on ZFS. This week a serious security flaw in it was published at 2:45 in the morning. By 7:30 it was patched, after a database dump and a snapshot of the photos. That took me waking up and saying "update it". It's now part of the automatic updates too, with a database dump, a full copy of the database files and a snapshot first, and its rollback puts back only the database, so photos uploaded in the meantime stay put.

The things outside the box

Some of what the AI can change doesn't live on the server at all: the DNS records for my domains, and my router's configuration. Those are the changes most likely to take everything offline at once, and they live in other companies' systems.

So every hour, the server saves a copy of both, if anything has changed, and keeps 30 days of them, plus a full router backup every day. Any DNS change can be put back from that history. We tested it for real the day it went in: we added a test record, let the history capture it, and rolled the zone back from that history.

The firewall gets a different kind of undo, one anyone who has managed network gear will recognise. When a firewall change is made, it's tested first, then applied with a 60-second timer. If nobody confirms it in that time, it undoes itself. A rule that locks us out of the server can't stay in place, because the lock-out also stops anyone confirming it.

What can't be undone

This is the honest half. Everything above is about state: files, data, settings. Undo works on state. It doesn't work on things that have already left the building.

  • A secret that's been published. Early on, installing a package wrote a login token into a config file, and the config backup pushed it to GitHub before anyone noticed. We rewrote the repository history so the file was gone, but that doesn't make it un-happen. Once a secret has been pushed, you have to assume someone has it. The only real undo was to make it worthless: the token was replaced. That's why the two checks above now stand between every commit and GitHub.
  • An email that's been sent. A test message meant for my own inbox landed in a family member's real one, because a temporary forwarding change failed without saying so. No snapshot recalls an email. The fix was a rule: test mail only ever goes to my own addresses, and a temporary redirect gets checked before anything is sent through it.
  • Something a system has learned. A "dry run" scan of some imported mail turned out not to be dry: the spam filter learned from it, and learned about 90 perfectly good old emails as spam. Most of that could be corrected, but about 30 of those lessons couldn't be unlearned. Small, but it's a one-way door I didn't know was there.
  • A database that has moved forward. When an app updates, it often upgrades its database to match, and the old version can't read the new format. Going back means restoring the snapshot from before the update, and anything written since is lost. That happened, almost. The AI ran an update script with an option asking for its help text. The script ignored options it didn't recognise and did a full, real update instead, a day early, including moving the blog up a version. Everything came back healthy and the snapshots were there, but nobody had decided to do it.
  • Anything outside the server. Someone who read a page, a sender who got a bounce, an email someone else received. Those are in other people's hands.

Making the one-way doors rarer

You can't make everything undoable, so the other half of the rule is to treat the irreversible things differently.

The AI works behind a guard that checks each command before it runs. Some things it simply refuses, like printing a password or touching the systems I've ruled off-limits. Things that are destructive, or would disrupt something live, stop and wait for me to say yes. After the help-text incident, every one of the server's own scripts now refuses any option it doesn't recognise and does nothing, so a typo or a guess can't turn into a real run. And every mistake gets an entry in a log with a fix built into the setup, not a promise to be more careful. That log is how I know the number is 24.

A good life rule, too

None of this is really about servers. Most of the decisions that feel big are easy to reverse: a new job you can leave, a tool you can stop using, a change you can roll back. Treat those as cheap, and try them. A few really are one-way doors: what you say, what you publish, a secret you share. Those are the ones to slow down for.

The trick is knowing which is which before you walk through. That's what most of the work on this server has been about, and it's why I'm comfortable letting an AI make changes while I'm asleep.

Next up: the box itself. Six drives, two fans, and keeping a small case cool.

Full disclosure

Claude drafted this one too, from its records of what it built and the mistakes it has logged. Several of those mistakes are in here. I asked for them to be.