Backups | Bas Nijholt

Friend-to-friend ZFS backups, version two

Mon, 28 Sep 2026 00:00:00 +0000

Two years ago I wrote about friend-to-friend backups with TrueNAS. A friend and I store each other’s ZFS snapshots, which is the cheapest off-site backup there is. TrueNAS’s replication tasks wanted root on the receiving system, and neither of us wanted to give the other root on the machine that holds our family photos. So each of us ran a TrueNAS VM on a hypervisor, backed by an iSCSI zvol on the real NAS, and gave the other person root inside that VM instead. Sounds complicated? It was. It worked, but it meant running ZFS on a zvol on ZFS, inside a VM that each of us had to keep updated.

Since then I replaced TrueNAS with NixOS, and my friend plans to move his machines to NixOS too. This time we looked for the simplest solution, built from standard Linux and ZFS features wherever possible. He wanted plain zfs send and zfs receive, with his data still encrypted and his keys never on my server. I agreed, on one condition: neither of us should be able to destroy the other’s datasets.

The result is zfs-tenant, which is now on PyPI and has documentation.

Table of contents

Why ZFS #

OpenZFS is a filesystem and a volume manager in one. You add disks to a pool, and it behaves like one large partition: every dataset you create draws from the same free space, and each can still have its own quota and encryption. ZFS checksums every block it writes, so it notices when a disk hands back something else, and with a mirror or RAIDZ it repairs the block from a good copy. That protection costs disk space: a mirror of two disks holds one disk’s worth of data.

For me, snapshots are one of the coolest parts. ZFS never overwrites data in place, so a snapshot only has to hold on to the blocks that existed at that moment. Taking one is instant, and it costs no space until the data changes. zfs send turns a snapshot, or the difference between two snapshots, into a stream that zfs receive writes into a pool on another machine, usually piped through SSH. It moves blocks, not files: an incremental send contains only the blocks that changed, and ZFS knows which ones those are without visiting a single file. A bajillion one-kilobyte files do not slow it down, because nothing has to look at them one by one. For comparison, restic on my PC needed an hour and a half per run just to check close to a hundred million files for changes.

Two tools from the same project automate this. sanoid takes snapshots on a schedule and prunes them by policy, for example hourly ones for a day and daily ones for a month. syncoid wraps zfs send and zfs receive over SSH: it finds the newest snapshot both sides share and sends everything after it.

What we wanted #

We wrote down the rules before anything else:

He cannot see any of my data, including metadata like dataset names and sizes.
He cannot change or destroy anything of mine.
I cannot read his data, because his keys stay at his house.
He gets one dataset where he can create and remove nested datasets as he likes.
That dataset has a size limit.
No new replication tool on either side: plain zfs send or syncoid.

Until then his NAS runs TrueNAS, so whatever I built also had to work on a machine without NixOS.

ZFS already had most of it #

OpenZFS has a permission system that most people never touch: zfs allow. It delegates individual operations like receive, create, and destroy on one dataset to an unprivileged user. Put a root-owned quota on that dataset and the user cannot store more than you agreed. Raw sends (zfs send -w) ship the encrypted blocks as they are, so the receiving side never needs the key.

On paper, that covers five of the six rules. The gap is visibility. Any local user can run zfs list and see every dataset on the machine, with names, sizes, and properties. And an SSH account for replication is, by default, also a shell.

There are tools that would sidestep some of this. zrepl has a sink mode with a subtree per client, but it replaces sanoid and syncoid on both sides and runs as root on the receiver. restic or borg to a friend’s box would also work, but everything else on my network is ZFS: my machines replicate to the NAS with syncoid, and a restore is zfs send and zfs receive. Adding another backup format means one more thing to understand when something breaks. My current off-site copy goes to Backblaze B2 with rclone. Ideally that would be ZFS as well, and some hosting services accept zfs send, but they cost far more than object storage. A friend with a ZFS box costs nothing.

Rehearsing the permissions in a VM #

Before writing any code, I had an agent build a throwaway NixOS VM test with a real pool, a delegated user, and a list of probes. Each probe tried one thing the friend should not be able to do and recorded what ZFS answered.

The first surprise came from the most obvious setup. With a plain zfs allow -u joe create,receive,destroy,... tank/friends/joe, the delegated user could run zfs destroy -r tank/friends/joe and delete the dataset I had created for him, quota included. Delegated rights apply to the dataset itself as well as everything below it, unless you say otherwise. The fix is to split them: on the root itself only create,mount,receive with zfs allow -l, and the full set only for descendants with zfs allow -d. After that change, destroying, snapshotting, re-delegating, or changing the quota of the root all failed with permission denied.

A send stream can carry properties too, such as a hostile mountpoint=/etc. ZFS applies them with the receiving user’s rights, so in the VM that property failed with permission denied.

The probes also confirmed the gap: the delegated user saw my tank/host dataset in zfs list, and zfs get used tank/host returned its size.

A gate that speaks syncoid #

The shell part has a standard answer. An authorized_keys line with restrict,command="..." runs one fixed program for every login, and passes whatever the client asked for in SSH_ORIGINAL_COMMAND. zfs-tenant gate is that program. It tokenizes the request, accepts only the handful of command shapes a backup needs, checks that every dataset name is inside the friend’s dataset, and runs zfs with an argument list it builds itself. It never starts a shell, so there is nothing to inject into.

The hard part was knowing exactly which commands syncoid sends, so an agent read syncoid’s Perl and logged every remote command it ran. Most are the zfs commands you would expect. One surprise: before each receive, syncoid runs ps -Ao args= on the receiving machine, to check that no other transfer is already writing to the same dataset. That command prints every running process with its full command line. On my NAS, it would show my friend everything I run, so the gate answers with an empty list.

Reading syncoid that closely also turned up a remote code execution (RCE) bug in version 2.3.0, outside zfs-tenant itself. In push mode, syncoid asks the receiving host for a ZFS resume token. It passed that answer unescaped into shell commands for estimating and resuming the send, which run on the source machine. A malicious or compromised backup host could therefore return shell syntax instead of a token and execute arbitrary commands on the sender with the privileges of the syncoid process.

I reproduced this in two isolated NixOS VMs using real ZFS and SSH: a harmless command embedded in the receiver’s answer created a marker file on the sender. I reported it to the maintainer and opened sanoid PR #1114 with a small fix that shell-quotes the token at both uses. Even with that fix, the sending side should run syncoid as an unprivileged user that may only send and hold. nixpkgs’ services.syncoid already works that way, so my friend needs nothing from zfs-tenant at all.

Zones: a second wall in the kernel #

At that point the gate was the only thing hiding my datasets. Then my friend sent me a message: ZFS has a feature called zones, which restricts a dataset tree to a Linux user namespace.

zfs zone attaches a dataset to one user namespace, and inside that namespace the ZFS kernel module answers dataset does not exist for everything that is not attached. Even if my gate had a bug that let arbitrary zfs commands through, tank/host would stay invisible. The zone does not replace the gate, though: it only filters ZFS, and it only applies once the gate has put the SSH session inside it. Who stops what shows how the two divide the work.

A user namespace can give its processes different user IDs than they have outside. Containers usually use that to make the user root inside the namespace, while it stays an ordinary user outside. ZFS treats root inside a zone as the zone’s administrator, allowed to do anything to the attached datasets, whatever zfs allow says. In the VM test, that let the friend destroy the dataset I had created for him. So zfs-tenant keeps him as his own ordinary user inside the namespace: he has no special rights there, zfs allow decides what he may do, and the kernel still hides everything else.

Root inside a user namespace also turned up in a real OpenZFS bug in August: before 2.4.4, 2.3.9, and 2.2.11, several pool operations, such as destroying a pool, accepted it as host root (CVE-2026-79619). Joe cannot reach that through the gate, but zfs-tenant needs unprivileged user namespaces, so a host running it should load a patched module: cat /sys/module/zfs/version shows the loaded one, which only changes after a reboot.

A zone needs a running namespace to attach to, so zfs-tenant runs a small service per friend that holds one. The gate joins that namespace before it looks at the command, drops the capabilities that joining grants, and refuses to run if the service is down.

Who stops what #

When Joe and I talked it through, I found it hard to say clearly which part protects against what, so I made this diagram.

Each column answers one question: who can connect, what he can run, what ZFS shows him, and what he can change. For the network, our tailnet only lets his router reach port 22 on my NAS, and from= in authorized_keys only accepts his key from that address. Raw sends then keep whatever reaches my pool unreadable to me.

The gate and the zone overlap in one job: hiding my other datasets. The gate does it first, by refusing any name outside his root before zfs runs. The zone repeats that check in the kernel, so hiding my datasets would take a bug in both. Everything else the gate does, the zone cannot:

Without the gate, his key would be a normal account on my NAS. The zone only filters ZFS; it would not stop him from reading /etc or running programs.
An SSH login does not start inside a user namespace. The gate joins the zone before it runs anything, so without the gate the zone would never apply.
The zone leaves some things visible: zpool status still shows my disks and how full the pool is, and the datasets above his root still show their sizes. User namespaces do not hide processes either, hence the empty answer to ps.
It deletes any unencrypted dataset he pushes by mistake, so I never keep his plaintext.
It logs every command, which is how I saw Joe try bash below.

The security model in the documentation lists every promise with the mechanism behind it.

What it cannot hide #

ZFS encryption protects file contents, not structure. In the words of zfs-load-key(8), ZFS “will not encrypt metadata related to the pool structure, including dataset and snapshot names, dataset hierarchy, properties, file size, file holes, and deduplication tables.” So I can see the names, sizes, and snapshot times of whatever my friend sends me. Boring dataset names help, and sending without -p keeps properties out of the stream.

I can also always delete his copy, because I am root on my own machine. What I can never do is read it. My monthly scrubs still verify his data without his key, because ZFS checksums the encrypted blocks.

What we protect against is a dead machine: a failed pool, a fire, a flood. His key may destroy anything below his root, which is what lets syncoid mirror his snapshot retention, so someone who steals that key can also delete his backups on my NAS. Covering that would take holds that I place as root and release on a schedule. We left them out on purpose, although I am reconsidering.

Setting it up #

On a NixOS host, giving a friend space takes this:

services.zfs-tenant = {
 enable = true;
 tenants.joe = {
 dataset = "tank/friends/joe";
 quota = "2T";
 authorizedKeys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... joe-nas" ];
 allowedFrom = [ "100.64.0.12" ]; # his tailnet address
 };
};

That creates the user, pins the key to the gate, applies the dataset, properties, and delegation on every boot, and runs the zone service. My own setup is in friend-backups.nix, with his real key and address. If your sanoid snapshots the whole pool, exclude the tenant tree: your sanoid must neither snapshot nor prune it, or his next incremental push finds snapshots on the target that he never sent. Once my friend is on NixOS, he can push with nixpkgs’ own module:

programs.ssh.knownHosts.bas-nas.publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA...";

services.syncoid = {
 enable = true;
 sshKey = "/var/lib/syncoid/id_ed25519";
 localSourceAllow = [ "send" "hold" ];
 commonArgs = [
 "--no-sync-snap"
 "--compress=none"
 "--delete-target-snapshots"
 "--sshoption=StrictHostKeyChecking=yes"
 ];
 commands."tank/offsite" = {
 target = "zfs-tenant-joe@bas-nas:tank/friends/joe/offsite";
 recursive = true;
 sendOptions = "w";
 };
};

My NAS will push to his with the same configuration once he hosts a root for me.

For his TrueNAS box, every release has a single zfs-tenant.pyz that runs with the Python TrueNAS already ships, and the getting started guide covers both setups. Hosting on TrueNAS has only run in VMs so far (issue #4). Sending from TrueNAS is where its built-in tools stop working: replication tasks wrap every remote command in sh -c, which the gate refuses, so he pushes with zfs send or syncoid instead.

Knowing when it stops working #

The last btrfs machine taught me a fifth question for any backup: how do I find out when it stops working? A failed push shows up in systemctl status on the sending side, which nobody reads. The check belongs on the receiving side, and it has to look at every pushed dataset separately, because one healthy dataset can hide another that stopped replicating. That is how I watch my own machines: the NAS checks the newest snapshot of every dataset it replicates every hour and alerts my phone when one is too old. zfs-tenant does not alert yet; issue #3 tracks it.

Restoring #

The gate also allows zfs send of his own snapshots, so getting data back is one pipe, run at his house:

ssh zfs-tenant-joe@bas-nas zfs send -w tank/friends/joe/offsite/photos@autosnap_2026-09-25_00:00:01_daily \
 | zfs receive -u tank/photos
zfs load-key tank/photos
zfs mount tank/photos

My photos are a child dataset that inherits its key from an encryption root, and I only push the child. If my NAS dies and I pull the child back, can I still unlock it without the root? A throwaway VM says yes: a raw send carries the dataset’s own wrapped key, so the child arrives as its own encryption root and opens with the same passphrase. The one catch: after a zfs change-key, the next push carries the new wrapping, so the passphrase that matters is the one at the time of the last push, and it has to live somewhere other than the NAS. Mine does: zfs-unlock keeps the passphrases on a separate device and unlocks the NAS after every boot.

The first real push #

Once my side was deployed, Joe tried it from his TrueNAS box. His first move was to see what else his key could do. The gate logged every attempt on my NAS:

root=tank/friends/joe denied interactive session
root=tank/friends/joe denied 'ls': only zfs commands and syncoid's probes are allowed
root=tank/friends/joe denied '/usr/bin/bash': only zfs commands and syncoid's probes are allowed
root=tank/friends/joe denied '/bin/bash': only zfs commands and syncoid's probes are allowed
root=tank/friends/joe allowed 'zfs receive -s -u tank/friends/joe/zt-test'

The last line is a small encrypted test dataset, sent with zfs send -w piped into ssh, without syncoid. It arrived with keystatus set to unavailable. Then I tried to read it as root on my own machine. zfs mount answered cannot mount 'tank/friends/joe/zt-test': dataset is exported to a local zone, and zfs load-key with a guessed passphrase gave Key load error: Incorrect key provided. The only readable text in the raw stream was ZFS property names and the snapshot name, which is the metadata ZFS leaves unencrypted, as described above.

Tested like everything else #

Every security claim in this post is also checked by a VM test: two NixOS machines back each other up, the way my friend and I do, with real sanoid and syncoid 2.3.0, through real sshd and the gate, into real OpenZFS 2.4.4. It also checks what happens over time, such as snapshot retention and both machines rebooting, and it runs on every push in GitHub Actions.

The route to the remote code execution bug in syncoid was slightly absurd. I built the initial version of zfs-tenant through coding agents, literally from the bathtub, and then asked several frontier models to review the security boundary independently. One of them followed receiver-controlled data beyond my new code and into syncoid. A model’s suspicion is not evidence, so I turned it into the two-VM reproduction above: it failed on unpatched syncoid 2.3.0 and passed after the seven-line fix. The most consequential result of reviewing my small new project was a command-injection bug in the mature tool next to it.

The whole thing is about 930 lines of Python, not counting comments and docstrings, with no dependencies. The kernel does the actual enforcing: delegation, the quota, and the zone. The gate only removes the shell, and it is small enough to read in one sitting.

AI scope creep or useful? #

The first version had grace holds: every day, the host placed a hold on the newest snapshot of each of my friend’s datasets and released it 14 days later. ZFS will not destroy a held snapshot until the hold is released, and only the host can release it. The agent built it without me asking for it, so I removed it right away as scope creep.

Then every model I asked to review the project suggested them again, each in its own context window: Opus 5.5, GPT-6 Astra, and even the one that reviewed a draft of this post. Joe and I had set out to protect against a machine dying (a fire or a flood), not against one of our own machines being hacked. But I am tempted to add the holds back. They cover the case where malware or an attacker gets onto my machine and uses my key to delete my backups on his NAS, or to replace them with garbage. Adding the feature without asking was still wrong; the feature itself might not be.

References #

The last btrfs machine: migrating my PC to ZFS

Thu, 23 Jul 2026 00:00:00 +0000

My GPU machine was the last machine in my fleet still running btrfs. I bought it for gaming (which never happened), and these days its dual RTX 3090s almost exclusively run local AI models.

btrfs itself never failed me. When I set up NixOS on this machine for the first time, I had read enough forum threads to become convinced that ZFS and NixOS were a troublesome combination, so I picked btrfs to be safe. That belief was simply wrong. As I went full Nix and eventually moved even my TrueNAS to NixOS, I learned there was no issue at all. I just had to figure out the right incantation to make ZFS work with boot drives, disko, and nixos-anywhere. My HP got ZFS first, that config grew into a shared disko module, and soon the whole fleet ran one pattern: NixOS on ZFS, snapshots and replication declared in code, and backups I can reason about.

Only the machine I sit at every day kept running on my old misconception, because a daily driver is exactly the one you do not casually reinstall.

Table of contents

Why touch a working machine #

The operating system was already uniform. The filesystem was not, and that turned out to matter more than I expected.

The real trigger was that I got increasingly paranoid about an AI agent messing up my system. I run agentic AI in YOLO mode, exclusively. All day, every day, agents execute commands on this machine without asking me first. YOLO does not mean root, though: the agents run as my regular user, and the backup credentials live where only root can read them. An agent that could destroy the backup itself would make everything else in this post pointless. Meanwhile the stories keep coming in about even the most frontier models wiping someone’s home directory:

GPT-5.6-Sol just accidentally deleted almost ALL of my Mac’s files. And this is why I trust Fable 1000x more.

— @mattshumer_

The reaction is always about which model deserves more trust. I read these tweets differently. Give any agent enough sessions and it will eventually delete something important, the same way I do every few years with a tired rm -rf. Whether that moment becomes a catastrophe or an inconvenience depends entirely on the restore path.

So I felt I needed to properly audit the way my backups were done. I want to be able to answer four questions about any machine I own:

What exactly is in the backup?
What is not in the backup, and did I choose that deliberately?
How big is a restore?
How long will it take?

The backup that wasn’t #

Asking question 1 delivered an unpleasant answer within the hour: the backups had quietly stopped four months earlier. Two boring causes had stacked up: an interrupted prune left a stale repository lock, which the job could never clear because its own unlock step only ran after a successful backup, and the repository URL still pointed at truenas.local, a name that had stopped resolving along the way. The last successful backup ran at 07:00 on March 22; the next one at 07:16 on July 22, after the fix. Four months, almost to the minute. systemd logged the failure on every single run, nobody read it, and nothing escalated. I found out because I asked the question, not because anything told me.

What the audit found #

On the btrfs box, question 3 alone had three different answers depending on which tool I asked. du counted reflinked copies at full size, so directories reported terabytes that physically did not exist. df counted physical extents, which is honest about disk usage but tells you nothing about what a file-based restore will write. And restic reported the apparent size of sparse VM images, which inflated the restore estimate to several times the disk.

When your storage layer gives you three sizes for the same data, you do not really understand your backup. You have a backup, and you have hope.

Question 4 was no better. Because the PC was my only btrfs machine, it could not join the ZFS snapshot replication the rest of the fleet uses, so its backups relied entirely on restic pushing to the NAS. Off-machine, note, but not off-site; question 2 turned out to apply to locations too, and closing that gap is its own project. restic is file-based: every run visits the metadata of every single file to find what changed. On this machine that meant close to a hundred million files per scan, which turned out to include some 600 GB of git worktrees stuffed with virtual environments and node_modules. Each run took about an hour and a half, even when almost nothing had changed.

That scan time dictated my backup cadence. The timer ran hourly, which in practice meant back-to-back scanning all day to move a couple hundred megabytes of real changes, and I recently dropped it to every six hours to stop the madness. Either way, my most recent work existed only on the machine an agent was busy poking at.

The scanning is also not free for the hardware. Reads are gentle on an SSD compared to writes, but hourly scans of an hour and a half each kept the drive at full duty cycle and elevated temperature for most of every day. And drives stopped being cheap: I paid $227 for a 4 TB NVMe drive in 2024, and the same drive lists for around $800 today. I would rather not run the experiment of what years of continuous scanning do to a drive I can no longer afford to replace.

btrfs has perfectly good snapshots of its own, but sending them off-site was never an option for me: snapshot replication needs the same filesystem on the receiving end, and my NAS speaks ZFS. With ZFS I can snapshot every few minutes and zfs send the actual block storage: an incremental send streams exactly the blocks that changed since the last snapshot, without walking a single file. A hundred million files cost nothing when nobody has to visit them.

ZFS does not magically collapse those numbers into one. But it gives me the same semantics everywhere, including on the NAS pools that hold the backups themselves. Eight of my machines already share a single ZFS disk layout, defined in one shared disko module. The PC was the odd one out. This migration makes it nine machines with the exact same strategy, and one backup story to audit instead of one per machine.

The plan: one shared disko module #

The Nix side was the easy part. The PC now imports the same shared disko module as the other eight machines, with one addition: a swapSize parameter, because ZFS has no reliable swapfile story and the PC actually needs its 96 GB of swap.

The disk is pinned by ID, not by device name:

device = "/dev/disk/by-id/nvme-Samsung_SSD_990_EVO_Plus_4TB_...";

This matters more than usual here. The PC dual-boots Windows from a second NVMe drive, and I want zero ambiguity about which of the two disks disko is allowed to erase.

The scary property of this migration is different from the NAS one. For the NAS, the whole point was that disko must not touch the data pools. For the PC, the new ZFS pool goes on the same disk the data currently lives on. No second disk, no local rollback, no btrfs to boot back into. From the moment disko runs, the restic repository on the NAS is the only road back.

That sentence is the reason most of this post is about verification.

The road back also has to survive the same agents that motivated all of this. The PC pushes its backups over sftp as a user that owns every file in the repository, which means a compromised PC could delete or encrypt its own safety net. So the NAS snapshots the repository dataset on its own side, where the PC’s credentials cannot reach: two days of hourly ZFS snapshots and two weeks of dailies that only root on the NAS can destroy.

Rehearsing in a VM, again #

The NAS migration taught me to rehearse the destructive step in a VM, so the flake grew a pc-vmtest target and I ran the whole install through nixos-anywhere --vm-test. Disko formatted a virtual disk, created the pool, installed the system, and booted it with swap active.

Passing that test proves the generated logic. It does not prove the backup, which is where the real risk lives.

The recovery kit #

A restore that starts with “first, log in to the dead machine and fetch the credentials” is not a restore plan. So everything a bare reinstall needs lives in a small recovery kit on two other machines in the house, permissions locked down: the restic password and SSH key that reach the repository, the staged restore script, a couple of verification scripts, and the PC’s SSH host keys. The host keys matter more than they look: the reinstalled machine comes back with its old identity, so no client complains and nothing in the fleet needs to be taught to trust a new machine. A munge key and a README round it out.

Trusting the backup enough to wipe the disk #

My recovery story has two layers. NixOS makes the system disposable: every service, driver fix, and firewall rule is in git, so no agent can leave the machine in a state I cannot rebuild. Verified backups make the data recoverable. This migration is the voluntary version of the tweets above: I am about to delete all of my PC’s files on purpose.

I did three verifications, each one converting an assumption into a fact. An AI agent did the tedious sweeps; I reviewed the evidence.

The first check was completeness. We walked everything on the machine outside the backup paths, until every byte was accounted for. It was all re-downloadable caches, and the largest unexplained thing turned out to be restic’s own local cache. Everything outside the backup set is now known disposable instead of assumed disposable.

The second check was the restore path. A backup you have never restored from is a hypothesis. On a different machine, using only the recovery kit and no access to the PC at all, we restored a sample from the latest snapshot and compared hashes against the live system. The hashes matched byte for byte, and even the symlinks came back pointing at the right targets. A sample proves the path works, from credentials to symlink handling; reading back every byte is the third check’s job. The machine running the rehearsal had the kit and nothing else, so recovery provably does not depend on the machine being recovered.

The third check is the data itself. restic check validates the repository structure without reading back the actual data blobs. Before the wipe, a --read-data pass ran on the NAS, where the repository sits on local disk instead of behind the network. When the backup is the only rollback, I want its bits read back at least once.

The fifth question #

The outage taught me a fifth question: how do I find out when any of this stops working? All three checks above are point-in-time, and verification rots. So the NAS now re-verifies every hour, from its own side of the sftp connection, that the newest snapshot in the repository is less than a day old, and pushes an alert to my phone when it is not. That catches every failure mode on the PC side, including “the timer is simply disabled.” The watcher itself is watched: the NAS pings an external dead-man’s switch every five minutes, so if the machine holding my only road back goes dark, the alert comes from outside the house.

While wiring this up we discovered that months of earlier NAS alerts had never arrived, because ntfy silently rejects message bodies over 4 KB. A notification you have never received is also a hypothesis.

Restoring in stages #

The measured restore set came out to roughly 1 TB, and 99.9% of it is /home. But two directories are 64% of that terabyte: Steam (430 GB of re-downloadable games) and my local-AI model collection (217 GB of GGUF files).

I do not need either of those to have a working machine. So the restore script now runs in stages:

sudo ./restore_from_backup.sh stage1 # everything else → a fully working machine
sudo ./restore_from_backup.sh stage2 # Steam + models, whenever

Stage 1 restores everything except the deferred paths, plus the system identity (host keys). Stage 2 can run days later, from the rebooted system, while I am already using the machine.

One detail I care about: stage 1 resolves latest to a concrete snapshot ID and pins it to a file. Stage 2 reads the pin. My backups run every six hours, so without this, a timer firing between the stages would silently split the restore across two different snapshots.

The backup itself is untouched: Steam and the models are still in every snapshot. Only the restore order changed.

Slimming down first #

The PC had also quietly become an Incus host for three LXC containers, including the MindRoom bots my family talks to every day. Before the migration, those containers moved to the NAS. That shrinks both the backup and the restore, and the family bots keep running while their old host is being wiped.

The cutover #

The install followed the same phased nixos-anywhere pattern as the NAS cutover, driven from another machine on the LAN: kexec into an installer running in RAM, stop, re-verify the target disk from inside the installer, and only then run the destructive disko/install phase. That checkpoint is the whole point: right up to the destructive phase, aborting was still cheap. The destructive part turned out to be the trivial part: from wiping the disk to booting the new system took less than twenty minutes.

The one surprise of the day was in the restore, and it was exactly the kind rehearsal is supposed to catch. Stage 1 would not start: the restore script runs restic as root, and a freshly installed root has no SSH identity and no known_hosts. The key it needed sat in the recovery kit inside my home directory. The rehearsal never hit this, because the rehearsal machine’s root had already talked to the NAS. A verified restore path can still depend on state you forgot you had.

Stage 1 restored 534 GiB across 4.5 million files in just over an hour and a half, at wire speed. The machine rebooted with its old host keys and its old /home, and stage 2 started immediately. The plan said stage 2 could wait for days; it started the minute the machine was back, because once the machine works there is no reason left to wait. The deferred 648 GiB of games and models took six more hours in the background.

With that, the PC finally joins the fleet pattern it was locked out of: sanoid snapshots the pool every hour, so a tired rm -rf, mine or an agent’s, now costs at most an hour of work and a rollback, and a daily syncoid run sends the changed blocks to the NAS. restic is on borrowed time. Its six-hour timer keeps running until I trust the new path, and then it stops: the repository stays on the NAS as the last file-level snapshots of the machine, but nothing will be added to it.

Aftercare was the same short list, written down instead of remembered: revert the temporary root-SSH commit the installer needed, and put the restic password back where the backup service expects it.

The slow work, as always, was not the cutover. The slow work was making the cutover boring.

References #

rsync-time-machine.py 🕒

Tue, 03 Dec 2024 00:00:00 +0000

(See my intro post)

Sometimes the best open source projects start with “I can do better than this.” That’s exactly what happened when I looked at laurent22/rsync-time-backup, a popular 600-line Bash script for Time Machine-style backups. While it worked, it had known bugs and was tricky to maintain due to Bash’s limitations. It also lacked tests, type hints (bash duh…), and proper error handling. So, I did what any reasonable developer would do: rewrote it in Python: rsync-time-machine.py! 🐍

📖 Origin Story #

I needed reliable backups that worked across platforms and filesystems, unlike Apple’s Time Machine. While many excellent backup tools exist, I was drawn to the simplicity of this approach: it just creates folders named YYYY-MM-DD-HH-MM-SS containing your files. No special tools needed for restoring or browsing - just your regular file explorer! I created a fully tested, typed, and documented Python port that maintains compatibility with the original while adding modern development practices.

🔧 Technical Highlights #

Creates incremental backups using hard links (saving space while keeping full snapshots)
Works on Linux, macOS, and Windows (via WSL or Cygwin)
Supports local and remote (SSH) backups
Smart backup expiration strategy
Built-in safety checks and locking mechanism
Pretty terminal output
Absolutely zero dependencies beyond Python itself - it just works!
Usable as a standalone .py script or pip installable package
100% test coverage

📊 Impact #

The project reached the front page of Y Combinator’s Hacker News, causing its popularity to explode:

381 GitHub stars
16 forks
6 contributors
Featured on Real Python podcast
Active discussion on Reddit’s r/commandline

🎯 Challenges and Solutions #

Maintaining compatibility with the original Bash script while improving the codebase
Making it work on Windows through WSL/Cygwin (surprisingly tricky!)
Handling various edge cases in filesystem operations
Implementing a robust backup expiration strategy
Adding comprehensive test coverage for file operations
Keeping the codebase dependency-free while maintaining functionality

💡 Lessons Learned #

Sometimes a rewrite is worth it, especially when adding modern development practices
Good documentation and test coverage are crucial for backup tools
The Python community appreciates well-structured ports of useful tools
Zero dependencies means zero headaches for users
Cross-platform compatibility (especially Windows) requires careful consideration
Simple, transparent solutions (like regular folders) can be better than complex ones

Want reliable, cross-platform Time Machine-style backups? Check out the project on GitHub!

#OpenSource #Python #Backups #DevTools #Programming

Friend‑to‑friend TrueNAS backups without the cloud

Sat, 24 Aug 2024 00:00:00 +0000

The Challenge: Escaping the Cloud #

I recently wrote about my homelab and how I use TrueNAS on my NAS for data storage. Most self-hosters are looking for ways to “de-cloud” themselves, taking control of their data away from big tech companies, and reducing monthly subscription fees. One subscription cost that many of us still rely on is cloud storage for backups. Personally, I use Rclone to back up my most essential data to Backblaze B2, but now that my data is growing, I’m looking for a more cost-effective solution.

Luckily, I have a friend who also uses TrueNAS and is looking for a similar solution.

TrueNAS offers built-in replication tasks that allow us to efficiently replicate our data to another TrueNAS system. Unfortunately, this requires root access to the remote system, which is a security risk neither of us are willing to take

So, how can we set up remote ZFS replication with a friend, maintaining our de-clouded status, without compromising the security of our TrueNAS systems?

The Requirements #

Use TrueNAS’s built-in replication tasks (uses zfs send and zfs receive for efficient replication, which is significantly faster than rsync)
Replicate to a friend’s system with ZFS capabilities
Avoid granting root access to each other’s TrueNAS systems
Maintain control over the allocated backup space
Keep our data private and out of corporate clouds

The Solution: TrueNAS VM on hypervisor and iSCSI Backup Storage on TrueNAS #

After exploring various options, we’ve found a secure and efficient solution using a hybrid VM setup that keeps us firmly in control of our data. Here’s the high-level overview:

Deploy a TrueNAS VM on a local hypervisor (e.g., Proxmox), running independently of your main TrueNAS system
Set up an iSCSI target on your TrueNAS for backup storage
Connect the iSCSI target as an additional disk to the VM and format it with ZFS
Give your friend secure access to this VM
Your friend sets up replication tasks to this VM’s iSCSI-connected storage

Why This Works #

Security: No need for root access on each other’s TrueNAS systems and data can be encrypted on the iSCSI storage with friend’s keys
Performance: VM OS runs from fast local storage, while backups use dedicated iSCSI storage
Flexibility: Easy to manage and update the VM independently of backup storage
Efficiency: Minimizes data transfer over iSCSI for system operations
De-Clouded: Your data stays between you and your friend, no cloud services involved

Step-by-Step Setup #

On your main TrueNAS #

Create a new zvol for backup storage
- Navigate to Storage > Pools > [Your Pool] > zvols
- Click “ADD zvol”
- Name it (e.g., “friend_backup”)
- Set an appropriate size
Set up an iSCSI target
- Go to Sharing > Block Shares (iSCSI)
- Click “Add” under Targets
- Give it a name and create
- Under Extents, click “Add”, select the zvol you created
- Under Associated Targets, add your new target and extent

On Proxmox #

Mount the iSCSI in Proxmox
- Go to Datacenter > Storage > Add > iSCSI
- Enter your TrueNAS IP and the target name
Create a TrueNAS VM
- Download TrueNAS ISO to /var/lib/vz/template/iso
- Click “Create VM”
- Select the TrueNAS ISO
- In the “Hard Disk” step, add two SCSI disks:
  - One on local storage (for OS)
  - One on the iSCSI storage (for backups)
Install TrueNAS
- Start the VM and open console
- Install TrueNAS on the local disk
Configure TrueNAS VM
- Access the TrueNAS web interface
- Create a ZFS pool on the iSCSI disk

Set up secure access #

Install and configure Tailscale (or preferred VPN)
- On both your system and the TrueNAS VM
- Use a self-hosted Headscale server for more control

For your friend #

Provide VPN access to the TrueNAS VM
They can now set up replication tasks to the VM’s iSCSI storage using the TrueNAS web interface and their own encryption keys

Repeat the process #

Follow the same steps on your friend’s system, reversing roles

Considerations #

Network: Ensure your internet connection can handle the data transfer
Initial Seeding: For large datasets, consider initial seeding in person
Encryption: Use encryption for the replicated datasets and VPN access for added security
Testing: Thoroughly test the setup before relying on it for critical backups
Trust: Choose your backup partner wisely - the other person will have access to the TrueNAS VM but not its data (if encrypted)
VM Maintenance: Regularly update and maintain the VM’s operating system

Conclusion #

This friend-to-friend setup offers TrueNAS users an effective approach to remote backups. By creating mutual TrueNAS VMs, we’ve established a secure, decentralized backup solution with several advantages:

Privacy and Security: Each friend has their own remote TrueNAS VM, with data remaining private when properly encrypted. Importantly, while you grant your friend access to your network, it’s limited to only the backup VM. You can firewall off the rest of your network, maintaining overall system security.
Full ZFS Feature Set: This setup allows us to use ZFS features like snapshot-based backups, incremental replication, and data integrity checks.
Flexible Backup Strategies: We can back up entire filesystems and maintain multiple snapshot versions.
De-clouded Solution: By partnering with a trusted friend, we’ve created a backup system that keeps our data out of corporate clouds, reducing costs and increasing control.
Scalability: As data needs grow, we can add additional iSCSI targets in Proxmox and incorporate these new virtual disks into our ZFS pool.
Network Isolation: Despite allowing access for backups, your main network remains protected. You control exactly what your backup partner can access, limiting potential security risks.

This solution represents a step towards data sovereignty and community-based storage. It’s secure, efficient, and independent of large tech companies, while still maintaining tight control over your network access.

Remember to regularly test your backup and restore processes, and periodically review your firewall rules. With this setup, you have a comprehensive, friend-to-friend backup solution that doesn’t compromise your overall network security.

Happy backing up!