ZFS | Bas Nijholt

Friend-to-friend ZFS backups, version two

Mon, 28 Sep 2026 00:00:00 +0000

Two years ago I wrote about friend-to-friend backups with TrueNAS. A friend and I store each other’s ZFS snapshots, which is the cheapest off-site backup there is. TrueNAS’s replication tasks wanted root on the receiving system, and neither of us wanted to give the other root on the machine that holds our family photos. So each of us ran a TrueNAS VM on a hypervisor, backed by an iSCSI zvol on the real NAS, and gave the other person root inside that VM instead. Sounds complicated? It was. It worked, but it meant running ZFS on a zvol on ZFS, inside a VM that each of us had to keep updated.

Since then I replaced TrueNAS with NixOS, and my friend plans to move his machines to NixOS too. This time we looked for the simplest solution, built from standard Linux and ZFS features wherever possible. He wanted plain zfs send and zfs receive, with his data still encrypted and his keys never on my server. I agreed, on one condition: neither of us should be able to destroy the other’s datasets.

The result is zfs-tenant, which is now on PyPI and has documentation.

Table of contents

Why ZFS #

OpenZFS is a filesystem and a volume manager in one. You add disks to a pool, and it behaves like one large partition: every dataset you create draws from the same free space, and each can still have its own quota and encryption. ZFS checksums every block it writes, so it notices when a disk hands back something else, and with a mirror or RAIDZ it repairs the block from a good copy. That protection costs disk space: a mirror of two disks holds one disk’s worth of data.

For me, snapshots are one of the coolest parts. ZFS never overwrites data in place, so a snapshot only has to hold on to the blocks that existed at that moment. Taking one is instant, and it costs no space until the data changes. zfs send turns a snapshot, or the difference between two snapshots, into a stream that zfs receive writes into a pool on another machine, usually piped through SSH. It moves blocks, not files: an incremental send contains only the blocks that changed, and ZFS knows which ones those are without visiting a single file. A bajillion one-kilobyte files do not slow it down, because nothing has to look at them one by one. For comparison, restic on my PC needed an hour and a half per run just to check close to a hundred million files for changes.

Two tools from the same project automate this. sanoid takes snapshots on a schedule and prunes them by policy, for example hourly ones for a day and daily ones for a month. syncoid wraps zfs send and zfs receive over SSH: it finds the newest snapshot both sides share and sends everything after it.

What we wanted #

We wrote down the rules before anything else:

He cannot see any of my data, including metadata like dataset names and sizes.
He cannot change or destroy anything of mine.
I cannot read his data, because his keys stay at his house.
He gets one dataset where he can create and remove nested datasets as he likes.
That dataset has a size limit.
No new replication tool on either side: plain zfs send or syncoid.

Until then his NAS runs TrueNAS, so whatever I built also had to work on a machine without NixOS.

ZFS already had most of it #

OpenZFS has a permission system that most people never touch: zfs allow. It delegates individual operations like receive, create, and destroy on one dataset to an unprivileged user. Put a root-owned quota on that dataset and the user cannot store more than you agreed. Raw sends (zfs send -w) ship the encrypted blocks as they are, so the receiving side never needs the key.

On paper, that covers five of the six rules. The gap is visibility. Any local user can run zfs list and see every dataset on the machine, with names, sizes, and properties. And an SSH account for replication is, by default, also a shell.

There are tools that would sidestep some of this. zrepl has a sink mode with a subtree per client, but it replaces sanoid and syncoid on both sides and runs as root on the receiver. restic or borg to a friend’s box would also work, but everything else on my network is ZFS: my machines replicate to the NAS with syncoid, and a restore is zfs send and zfs receive. Adding another backup format means one more thing to understand when something breaks. My current off-site copy goes to Backblaze B2 with rclone. Ideally that would be ZFS as well, and some hosting services accept zfs send, but they cost far more than object storage. A friend with a ZFS box costs nothing.

Rehearsing the permissions in a VM #

Before writing any code, I had an agent build a throwaway NixOS VM test with a real pool, a delegated user, and a list of probes. Each probe tried one thing the friend should not be able to do and recorded what ZFS answered.

The first surprise came from the most obvious setup. With a plain zfs allow -u joe create,receive,destroy,... tank/friends/joe, the delegated user could run zfs destroy -r tank/friends/joe and delete the dataset I had created for him, quota included. Delegated rights apply to the dataset itself as well as everything below it, unless you say otherwise. The fix is to split them: on the root itself only create,mount,receive with zfs allow -l, and the full set only for descendants with zfs allow -d. After that change, destroying, snapshotting, re-delegating, or changing the quota of the root all failed with permission denied.

A send stream can carry properties too, such as a hostile mountpoint=/etc. ZFS applies them with the receiving user’s rights, so in the VM that property failed with permission denied.

The probes also confirmed the gap: the delegated user saw my tank/host dataset in zfs list, and zfs get used tank/host returned its size.

A gate that speaks syncoid #

The shell part has a standard answer. An authorized_keys line with restrict,command="..." runs one fixed program for every login, and passes whatever the client asked for in SSH_ORIGINAL_COMMAND. zfs-tenant gate is that program. It tokenizes the request, accepts only the handful of command shapes a backup needs, checks that every dataset name is inside the friend’s dataset, and runs zfs with an argument list it builds itself. It never starts a shell, so there is nothing to inject into.

The hard part was knowing exactly which commands syncoid sends, so an agent read syncoid’s Perl and logged every remote command it ran. Most are the zfs commands you would expect. One surprise: before each receive, syncoid runs ps -Ao args= on the receiving machine, to check that no other transfer is already writing to the same dataset. That command prints every running process with its full command line. On my NAS, it would show my friend everything I run, so the gate answers with an empty list.

Reading syncoid that closely also turned up a remote code execution (RCE) bug in version 2.3.0, outside zfs-tenant itself. In push mode, syncoid asks the receiving host for a ZFS resume token. It passed that answer unescaped into shell commands for estimating and resuming the send, which run on the source machine. A malicious or compromised backup host could therefore return shell syntax instead of a token and execute arbitrary commands on the sender with the privileges of the syncoid process.

I reproduced this in two isolated NixOS VMs using real ZFS and SSH: a harmless command embedded in the receiver’s answer created a marker file on the sender. I reported it to the maintainer and opened sanoid PR #1114 with a small fix that shell-quotes the token at both uses. Even with that fix, the sending side should run syncoid as an unprivileged user that may only send and hold. nixpkgs’ services.syncoid already works that way, so my friend needs nothing from zfs-tenant at all.

Zones: a second wall in the kernel #

At that point the gate was the only thing hiding my datasets. Then my friend sent me a message: ZFS has a feature called zones, which restricts a dataset tree to a Linux user namespace.

zfs zone attaches a dataset to one user namespace, and inside that namespace the ZFS kernel module answers dataset does not exist for everything that is not attached. Even if my gate had a bug that let arbitrary zfs commands through, tank/host would stay invisible. The zone does not replace the gate, though: it only filters ZFS, and it only applies once the gate has put the SSH session inside it. Who stops what shows how the two divide the work.

A user namespace can give its processes different user IDs than they have outside. Containers usually use that to make the user root inside the namespace, while it stays an ordinary user outside. ZFS treats root inside a zone as the zone’s administrator, allowed to do anything to the attached datasets, whatever zfs allow says. In the VM test, that let the friend destroy the dataset I had created for him. So zfs-tenant keeps him as his own ordinary user inside the namespace: he has no special rights there, zfs allow decides what he may do, and the kernel still hides everything else.

Root inside a user namespace also turned up in a real OpenZFS bug in August: before 2.4.4, 2.3.9, and 2.2.11, several pool operations, such as destroying a pool, accepted it as host root (CVE-2026-79619). Joe cannot reach that through the gate, but zfs-tenant needs unprivileged user namespaces, so a host running it should load a patched module: cat /sys/module/zfs/version shows the loaded one, which only changes after a reboot.

A zone needs a running namespace to attach to, so zfs-tenant runs a small service per friend that holds one. The gate joins that namespace before it looks at the command, drops the capabilities that joining grants, and refuses to run if the service is down.

Who stops what #

When Joe and I talked it through, I found it hard to say clearly which part protects against what, so I made this diagram.

Each column answers one question: who can connect, what he can run, what ZFS shows him, and what he can change. For the network, our tailnet only lets his router reach port 22 on my NAS, and from= in authorized_keys only accepts his key from that address. Raw sends then keep whatever reaches my pool unreadable to me.

The gate and the zone overlap in one job: hiding my other datasets. The gate does it first, by refusing any name outside his root before zfs runs. The zone repeats that check in the kernel, so hiding my datasets would take a bug in both. Everything else the gate does, the zone cannot:

Without the gate, his key would be a normal account on my NAS. The zone only filters ZFS; it would not stop him from reading /etc or running programs.
An SSH login does not start inside a user namespace. The gate joins the zone before it runs anything, so without the gate the zone would never apply.
The zone leaves some things visible: zpool status still shows my disks and how full the pool is, and the datasets above his root still show their sizes. User namespaces do not hide processes either, hence the empty answer to ps.
It deletes any unencrypted dataset he pushes by mistake, so I never keep his plaintext.
It logs every command, which is how I saw Joe try bash below.

The security model in the documentation lists every promise with the mechanism behind it.

What it cannot hide #

ZFS encryption protects file contents, not structure. In the words of zfs-load-key(8), ZFS “will not encrypt metadata related to the pool structure, including dataset and snapshot names, dataset hierarchy, properties, file size, file holes, and deduplication tables.” So I can see the names, sizes, and snapshot times of whatever my friend sends me. Boring dataset names help, and sending without -p keeps properties out of the stream.

I can also always delete his copy, because I am root on my own machine. What I can never do is read it. My monthly scrubs still verify his data without his key, because ZFS checksums the encrypted blocks.

What we protect against is a dead machine: a failed pool, a fire, a flood. His key may destroy anything below his root, which is what lets syncoid mirror his snapshot retention, so someone who steals that key can also delete his backups on my NAS. Covering that would take holds that I place as root and release on a schedule. We left them out on purpose, although I am reconsidering.

Setting it up #

On a NixOS host, giving a friend space takes this:

services.zfs-tenant = {
 enable = true;
 tenants.joe = {
 dataset = "tank/friends/joe";
 quota = "2T";
 authorizedKeys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... joe-nas" ];
 allowedFrom = [ "100.64.0.12" ]; # his tailnet address
 };
};

That creates the user, pins the key to the gate, applies the dataset, properties, and delegation on every boot, and runs the zone service. My own setup is in friend-backups.nix, with his real key and address. If your sanoid snapshots the whole pool, exclude the tenant tree: your sanoid must neither snapshot nor prune it, or his next incremental push finds snapshots on the target that he never sent. Once my friend is on NixOS, he can push with nixpkgs’ own module:

programs.ssh.knownHosts.bas-nas.publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA...";

services.syncoid = {
 enable = true;
 sshKey = "/var/lib/syncoid/id_ed25519";
 localSourceAllow = [ "send" "hold" ];
 commonArgs = [
 "--no-sync-snap"
 "--compress=none"
 "--delete-target-snapshots"
 "--sshoption=StrictHostKeyChecking=yes"
 ];
 commands."tank/offsite" = {
 target = "zfs-tenant-joe@bas-nas:tank/friends/joe/offsite";
 recursive = true;
 sendOptions = "w";
 };
};

My NAS will push to his with the same configuration once he hosts a root for me.

For his TrueNAS box, every release has a single zfs-tenant.pyz that runs with the Python TrueNAS already ships, and the getting started guide covers both setups. Hosting on TrueNAS has only run in VMs so far (issue #4). Sending from TrueNAS is where its built-in tools stop working: replication tasks wrap every remote command in sh -c, which the gate refuses, so he pushes with zfs send or syncoid instead.

Knowing when it stops working #

The last btrfs machine taught me a fifth question for any backup: how do I find out when it stops working? A failed push shows up in systemctl status on the sending side, which nobody reads. The check belongs on the receiving side, and it has to look at every pushed dataset separately, because one healthy dataset can hide another that stopped replicating. That is how I watch my own machines: the NAS checks the newest snapshot of every dataset it replicates every hour and alerts my phone when one is too old. zfs-tenant does not alert yet; issue #3 tracks it.

Restoring #

The gate also allows zfs send of his own snapshots, so getting data back is one pipe, run at his house:

ssh zfs-tenant-joe@bas-nas zfs send -w tank/friends/joe/offsite/photos@autosnap_2026-09-25_00:00:01_daily \
 | zfs receive -u tank/photos
zfs load-key tank/photos
zfs mount tank/photos

My photos are a child dataset that inherits its key from an encryption root, and I only push the child. If my NAS dies and I pull the child back, can I still unlock it without the root? A throwaway VM says yes: a raw send carries the dataset’s own wrapped key, so the child arrives as its own encryption root and opens with the same passphrase. The one catch: after a zfs change-key, the next push carries the new wrapping, so the passphrase that matters is the one at the time of the last push, and it has to live somewhere other than the NAS. Mine does: zfs-unlock keeps the passphrases on a separate device and unlocks the NAS after every boot.

The first real push #

Once my side was deployed, Joe tried it from his TrueNAS box. His first move was to see what else his key could do. The gate logged every attempt on my NAS:

root=tank/friends/joe denied interactive session
root=tank/friends/joe denied 'ls': only zfs commands and syncoid's probes are allowed
root=tank/friends/joe denied '/usr/bin/bash': only zfs commands and syncoid's probes are allowed
root=tank/friends/joe denied '/bin/bash': only zfs commands and syncoid's probes are allowed
root=tank/friends/joe allowed 'zfs receive -s -u tank/friends/joe/zt-test'

The last line is a small encrypted test dataset, sent with zfs send -w piped into ssh, without syncoid. It arrived with keystatus set to unavailable. Then I tried to read it as root on my own machine. zfs mount answered cannot mount 'tank/friends/joe/zt-test': dataset is exported to a local zone, and zfs load-key with a guessed passphrase gave Key load error: Incorrect key provided. The only readable text in the raw stream was ZFS property names and the snapshot name, which is the metadata ZFS leaves unencrypted, as described above.

Tested like everything else #

Every security claim in this post is also checked by a VM test: two NixOS machines back each other up, the way my friend and I do, with real sanoid and syncoid 2.3.0, through real sshd and the gate, into real OpenZFS 2.4.4. It also checks what happens over time, such as snapshot retention and both machines rebooting, and it runs on every push in GitHub Actions.

The route to the remote code execution bug in syncoid was slightly absurd. I built the initial version of zfs-tenant through coding agents, literally from the bathtub, and then asked several frontier models to review the security boundary independently. One of them followed receiver-controlled data beyond my new code and into syncoid. A model’s suspicion is not evidence, so I turned it into the two-VM reproduction above: it failed on unpatched syncoid 2.3.0 and passed after the seven-line fix. The most consequential result of reviewing my small new project was a command-injection bug in the mature tool next to it.

The whole thing is about 930 lines of Python, not counting comments and docstrings, with no dependencies. The kernel does the actual enforcing: delegation, the quota, and the zone. The gate only removes the shell, and it is small enough to read in one sitting.

AI scope creep or useful? #

The first version had grace holds: every day, the host placed a hold on the newest snapshot of each of my friend’s datasets and released it 14 days later. ZFS will not destroy a held snapshot until the hold is released, and only the host can release it. The agent built it without me asking for it, so I removed it right away as scope creep.

Then every model I asked to review the project suggested them again, each in its own context window: Opus 5.5, GPT-6 Astra, and even the one that reviewed a draft of this post. Joe and I had set out to protect against a machine dying (a fire or a flood), not against one of our own machines being hacked. But I am tempted to add the holds back. They cover the case where malware or an attacker gets onto my machine and uses my key to delete my backups on his NAS, or to replace them with garbage. Adding the feature without asking was still wrong; the feature itself might not be.

References #

zfs-tenant

Sat, 26 Sep 2026 00:00:00 +0000

The last btrfs machine: migrating my PC to ZFS

Thu, 23 Jul 2026 00:00:00 +0000

My GPU machine was the last machine in my fleet still running btrfs. I bought it for gaming (which never happened), and these days its dual RTX 3090s almost exclusively run local AI models.

btrfs itself never failed me. When I set up NixOS on this machine for the first time, I had read enough forum threads to become convinced that ZFS and NixOS were a troublesome combination, so I picked btrfs to be safe. That belief was simply wrong. As I went full Nix and eventually moved even my TrueNAS to NixOS, I learned there was no issue at all. I just had to figure out the right incantation to make ZFS work with boot drives, disko, and nixos-anywhere. My HP got ZFS first, that config grew into a shared disko module, and soon the whole fleet ran one pattern: NixOS on ZFS, snapshots and replication declared in code, and backups I can reason about.

Only the machine I sit at every day kept running on my old misconception, because a daily driver is exactly the one you do not casually reinstall.

Table of contents

Why touch a working machine #

The operating system was already uniform. The filesystem was not, and that turned out to matter more than I expected.

The real trigger was that I got increasingly paranoid about an AI agent messing up my system. I run agentic AI in YOLO mode, exclusively. All day, every day, agents execute commands on this machine without asking me first. YOLO does not mean root, though: the agents run as my regular user, and the backup credentials live where only root can read them. An agent that could destroy the backup itself would make everything else in this post pointless. Meanwhile the stories keep coming in about even the most frontier models wiping someone’s home directory:

GPT-5.6-Sol just accidentally deleted almost ALL of my Mac’s files. And this is why I trust Fable 1000x more.

— @mattshumer_

The reaction is always about which model deserves more trust. I read these tweets differently. Give any agent enough sessions and it will eventually delete something important, the same way I do every few years with a tired rm -rf. Whether that moment becomes a catastrophe or an inconvenience depends entirely on the restore path.

So I felt I needed to properly audit the way my backups were done. I want to be able to answer four questions about any machine I own:

What exactly is in the backup?
What is not in the backup, and did I choose that deliberately?
How big is a restore?
How long will it take?

The backup that wasn’t #

Asking question 1 delivered an unpleasant answer within the hour: the backups had quietly stopped four months earlier. Two boring causes had stacked up: an interrupted prune left a stale repository lock, which the job could never clear because its own unlock step only ran after a successful backup, and the repository URL still pointed at truenas.local, a name that had stopped resolving along the way. The last successful backup ran at 07:00 on March 22; the next one at 07:16 on July 22, after the fix. Four months, almost to the minute. systemd logged the failure on every single run, nobody read it, and nothing escalated. I found out because I asked the question, not because anything told me.

What the audit found #

On the btrfs box, question 3 alone had three different answers depending on which tool I asked. du counted reflinked copies at full size, so directories reported terabytes that physically did not exist. df counted physical extents, which is honest about disk usage but tells you nothing about what a file-based restore will write. And restic reported the apparent size of sparse VM images, which inflated the restore estimate to several times the disk.

When your storage layer gives you three sizes for the same data, you do not really understand your backup. You have a backup, and you have hope.

Question 4 was no better. Because the PC was my only btrfs machine, it could not join the ZFS snapshot replication the rest of the fleet uses, so its backups relied entirely on restic pushing to the NAS. Off-machine, note, but not off-site; question 2 turned out to apply to locations too, and closing that gap is its own project. restic is file-based: every run visits the metadata of every single file to find what changed. On this machine that meant close to a hundred million files per scan, which turned out to include some 600 GB of git worktrees stuffed with virtual environments and node_modules. Each run took about an hour and a half, even when almost nothing had changed.

That scan time dictated my backup cadence. The timer ran hourly, which in practice meant back-to-back scanning all day to move a couple hundred megabytes of real changes, and I recently dropped it to every six hours to stop the madness. Either way, my most recent work existed only on the machine an agent was busy poking at.

The scanning is also not free for the hardware. Reads are gentle on an SSD compared to writes, but hourly scans of an hour and a half each kept the drive at full duty cycle and elevated temperature for most of every day. And drives stopped being cheap: I paid $227 for a 4 TB NVMe drive in 2024, and the same drive lists for around $800 today. I would rather not run the experiment of what years of continuous scanning do to a drive I can no longer afford to replace.

btrfs has perfectly good snapshots of its own, but sending them off-site was never an option for me: snapshot replication needs the same filesystem on the receiving end, and my NAS speaks ZFS. With ZFS I can snapshot every few minutes and zfs send the actual block storage: an incremental send streams exactly the blocks that changed since the last snapshot, without walking a single file. A hundred million files cost nothing when nobody has to visit them.

ZFS does not magically collapse those numbers into one. But it gives me the same semantics everywhere, including on the NAS pools that hold the backups themselves. Eight of my machines already share a single ZFS disk layout, defined in one shared disko module. The PC was the odd one out. This migration makes it nine machines with the exact same strategy, and one backup story to audit instead of one per machine.

The plan: one shared disko module #

The Nix side was the easy part. The PC now imports the same shared disko module as the other eight machines, with one addition: a swapSize parameter, because ZFS has no reliable swapfile story and the PC actually needs its 96 GB of swap.

The disk is pinned by ID, not by device name:

device = "/dev/disk/by-id/nvme-Samsung_SSD_990_EVO_Plus_4TB_...";

This matters more than usual here. The PC dual-boots Windows from a second NVMe drive, and I want zero ambiguity about which of the two disks disko is allowed to erase.

The scary property of this migration is different from the NAS one. For the NAS, the whole point was that disko must not touch the data pools. For the PC, the new ZFS pool goes on the same disk the data currently lives on. No second disk, no local rollback, no btrfs to boot back into. From the moment disko runs, the restic repository on the NAS is the only road back.

That sentence is the reason most of this post is about verification.

The road back also has to survive the same agents that motivated all of this. The PC pushes its backups over sftp as a user that owns every file in the repository, which means a compromised PC could delete or encrypt its own safety net. So the NAS snapshots the repository dataset on its own side, where the PC’s credentials cannot reach: two days of hourly ZFS snapshots and two weeks of dailies that only root on the NAS can destroy.

Rehearsing in a VM, again #

The NAS migration taught me to rehearse the destructive step in a VM, so the flake grew a pc-vmtest target and I ran the whole install through nixos-anywhere --vm-test. Disko formatted a virtual disk, created the pool, installed the system, and booted it with swap active.

Passing that test proves the generated logic. It does not prove the backup, which is where the real risk lives.

The recovery kit #

A restore that starts with “first, log in to the dead machine and fetch the credentials” is not a restore plan. So everything a bare reinstall needs lives in a small recovery kit on two other machines in the house, permissions locked down: the restic password and SSH key that reach the repository, the staged restore script, a couple of verification scripts, and the PC’s SSH host keys. The host keys matter more than they look: the reinstalled machine comes back with its old identity, so no client complains and nothing in the fleet needs to be taught to trust a new machine. A munge key and a README round it out.

Trusting the backup enough to wipe the disk #

My recovery story has two layers. NixOS makes the system disposable: every service, driver fix, and firewall rule is in git, so no agent can leave the machine in a state I cannot rebuild. Verified backups make the data recoverable. This migration is the voluntary version of the tweets above: I am about to delete all of my PC’s files on purpose.

I did three verifications, each one converting an assumption into a fact. An AI agent did the tedious sweeps; I reviewed the evidence.

The first check was completeness. We walked everything on the machine outside the backup paths, until every byte was accounted for. It was all re-downloadable caches, and the largest unexplained thing turned out to be restic’s own local cache. Everything outside the backup set is now known disposable instead of assumed disposable.

The second check was the restore path. A backup you have never restored from is a hypothesis. On a different machine, using only the recovery kit and no access to the PC at all, we restored a sample from the latest snapshot and compared hashes against the live system. The hashes matched byte for byte, and even the symlinks came back pointing at the right targets. A sample proves the path works, from credentials to symlink handling; reading back every byte is the third check’s job. The machine running the rehearsal had the kit and nothing else, so recovery provably does not depend on the machine being recovered.

The third check is the data itself. restic check validates the repository structure without reading back the actual data blobs. Before the wipe, a --read-data pass ran on the NAS, where the repository sits on local disk instead of behind the network. When the backup is the only rollback, I want its bits read back at least once.

The fifth question #

The outage taught me a fifth question: how do I find out when any of this stops working? All three checks above are point-in-time, and verification rots. So the NAS now re-verifies every hour, from its own side of the sftp connection, that the newest snapshot in the repository is less than a day old, and pushes an alert to my phone when it is not. That catches every failure mode on the PC side, including “the timer is simply disabled.” The watcher itself is watched: the NAS pings an external dead-man’s switch every five minutes, so if the machine holding my only road back goes dark, the alert comes from outside the house.

While wiring this up we discovered that months of earlier NAS alerts had never arrived, because ntfy silently rejects message bodies over 4 KB. A notification you have never received is also a hypothesis.

Restoring in stages #

The measured restore set came out to roughly 1 TB, and 99.9% of it is /home. But two directories are 64% of that terabyte: Steam (430 GB of re-downloadable games) and my local-AI model collection (217 GB of GGUF files).

I do not need either of those to have a working machine. So the restore script now runs in stages:

sudo ./restore_from_backup.sh stage1 # everything else → a fully working machine
sudo ./restore_from_backup.sh stage2 # Steam + models, whenever

Stage 1 restores everything except the deferred paths, plus the system identity (host keys). Stage 2 can run days later, from the rebooted system, while I am already using the machine.

One detail I care about: stage 1 resolves latest to a concrete snapshot ID and pins it to a file. Stage 2 reads the pin. My backups run every six hours, so without this, a timer firing between the stages would silently split the restore across two different snapshots.

The backup itself is untouched: Steam and the models are still in every snapshot. Only the restore order changed.

Slimming down first #

The PC had also quietly become an Incus host for three LXC containers, including the MindRoom bots my family talks to every day. Before the migration, those containers moved to the NAS. That shrinks both the backup and the restore, and the family bots keep running while their old host is being wiped.

The cutover #

The install followed the same phased nixos-anywhere pattern as the NAS cutover, driven from another machine on the LAN: kexec into an installer running in RAM, stop, re-verify the target disk from inside the installer, and only then run the destructive disko/install phase. That checkpoint is the whole point: right up to the destructive phase, aborting was still cheap. The destructive part turned out to be the trivial part: from wiping the disk to booting the new system took less than twenty minutes.

The one surprise of the day was in the restore, and it was exactly the kind rehearsal is supposed to catch. Stage 1 would not start: the restore script runs restic as root, and a freshly installed root has no SSH identity and no known_hosts. The key it needed sat in the recovery kit inside my home directory. The rehearsal never hit this, because the rehearsal machine’s root had already talked to the NAS. A verified restore path can still depend on state you forgot you had.

Stage 1 restored 534 GiB across 4.5 million files in just over an hour and a half, at wire speed. The machine rebooted with its old host keys and its old /home, and stage 2 started immediately. The plan said stage 2 could wait for days; it started the minute the machine was back, because once the machine works there is no reason left to wait. The deferred 648 GiB of games and models took six more hours in the background.

With that, the PC finally joins the fleet pattern it was locked out of: sanoid snapshots the pool every hour, so a tired rm -rf, mine or an agent’s, now costs at most an hour of work and a rollback, and a daily syncoid run sends the changed blocks to the NAS. restic is on borrowed time. Its six-hour timer keeps running until I trust the new path, and then it stops: the repository stays on the NAS as the last file-level snapshots of the machine, but nothing will be added to it.

Aftercare was the same short list, written down instead of remembered: revert the temporary root-SSH commit the installer needed, and put the restic password back where the backup service expects it.

The slow work, as always, was not the cutover. The slow work was making the cutover boring.

References #

ZFS Unlock

Mon, 29 Jun 2026 00:00:00 +0000

Migrating from TrueNAS to NixOS without losing data after the build-system rug pull

Sun, 28 Jun 2026 00:00:00 +0000

TrueNAS was the last non-NixOS machine in my home lab.

I kept postponing it for a simpler reason: moving something with ~100 TiB of data feels scary. My NAS holds the big ZFS pools. It has the SMB shares, NFS exports, snapshots, replication tasks, encrypted datasets, Incus state, Docker workloads, and all the boring-but-important health checks that are easy to take for granted until something fails.

I had already migrated most of my machines from Proxmox to NixOS with Incus, as I wrote about in I’ve gone full Nix: Proxmox to NixOS + Incus. That migration made me increasingly intolerant of appliance-style infrastructure. If the important state lives in a database behind a web UI, I cannot review it in a pull request. If I change something while debugging, I cannot guarantee the change made it back into the source of truth. And if I let an AI agent help me, imperative systems become even more uncomfortable: an agent can run a hundred commands, succeed, and leave behind a system that nobody can reproduce. For the NAS specifically, that basically meant I did not dare touch the machine with an AI agent at all.

NixOS is different. The diff is the state. If an agent changes my infrastructure, I can review the patch. If the change is wrong, I can reject it. If it works, I can commit it.

TrueNAS was the last holdout. It had been itching for a while. Then the build-system rug pull made the decision easy.

Table of contents

The itch before the last straw #

The build-system change was not the first thing that bothered me. It was just the last straw.

The deeper problem was that my TrueNAS setup had slowly become complicated in exactly the way I dislike. I had clicked many buttons in the UI over time:

SMB shares
NFS exports
snapshot tasks
replication tasks
users and groups
encrypted datasets
apps and containers
Incus storage
old Jailmaker/systemd-nspawn state
alerting
UPS settings

TrueNAS can export and import its configuration. That is genuinely useful. But a database export is not the same thing as a configuration I can comfortably review. When I look at a TrueNAS backup, it is not obvious to me which settings I intentionally changed, which settings are defaults, which settings are historical leftovers, and which settings only exist because I clicked something while debugging two years ago.

One concrete example was NFS. I needed the crossmnt export option so child datasets would show up correctly inside the Docker container host, but TrueNAS did not expose that option in the UI. So I had a periodic root cron job patching /etc/exports after TrueNAS generated it. It worked, but it was exactly the kind of small appliance hack that made me uneasy: partly outside the UI, partly outside the exported configuration, and very easy to forget during a migration. In NixOS, that behavior is just part of the export line.

That is the exact discomfort that pushed me away from Proxmox. I do not want to remember what I clicked. I want the system to remember it as code.

The second itch was virtualization. My TrueNAS container story had already gone through one migration. For a while I used Jailmaker, which is a clever script around systemd-nspawn for running Linux “jails” on TrueNAS SCALE. It worked well enough for what I needed at the time: a Debian environment where I could run Docker workloads close to the data. But it also reinforced the feeling that I was building a fairly custom server on top of an appliance OS.

TrueNAS 25.04 “Fangtooth” introduced experimental Containers, named Instances in the early 25.04 releases, and the 25.04.1 announcement explicitly described Instances as being based on Incus, supporting both VMs and LXCs. That was interesting to me because the older TrueNAS VM system was libvirt-based, while Incus felt like a more coherent container/VM story. Then the TrueNAS virtualization plan for 25.04.2 explained that Fangtooth’s Instances feature caused challenges for some users and that 25.04.2 would re-enable the older Virtualization VM path alongside Instances. The 25.04.2 release announcement made that split official: Virtual Machines came back, while VMs created through the earlier Instances feature continued to appear under Containers.

For many people that churn was probably annoying. For me it was especially ironic. I had migrated from Jailmaker’s systemd-nspawn setup to Incus, read up on Incus, got excited about it, started using it, and then became genuinely happy with it when I migrated from Proxmox to NixOS with Incus. Incus fit my mental model perfectly: a clean CLI, containers and VMs in one place, easy scripting, and a workflow that agents can inspect.

Then TrueNAS started moving away from that direction again. Forum discussion around the 25.04.2 and later virtualization plans made it clear that Incus was not the stable long-term abstraction I hoped it would be inside TrueNAS. In the long-running Linux Jails with Incus thread, Kris Moore is quoted saying that Incus is being removed from the base system and that libvirt will handle the backend for VMs and LXC. That probably makes sense for TrueNAS. They have enterprise users, compatibility concerns, and a support matrix that is very different from mine.

But for me it meant the NAS was again becoming an appliance whose internal direction I had to follow. I had just escaped that with Proxmox. I did not want to do it again for the most important machine in my house.

The last straw #

Then I saw Jeff Geerling mention TrueNAS’s recent “shady behavior” in a video, and I started looking into what had actually changed.

The short version is that iXsystems made part of the TrueNAS ecosystem closed source. Specifically, iXsystems moved the TrueNAS build scripts and related build infrastructure internal for the TrueNAS 27 line. Their forum post, Clearing the Air on Build Scripts, frames this as a practical build-system decision and says normal installation and updates remain unaffected. The old truenas/scale-build repository now says it is preserved historically and is no longer the maintained build system. Community reactions, for example this TrueNAS forum thread, understandably read this as a meaningful step away from fully open development.

Would I ever have built TrueNAS from source myself? Almost certainly not.

That misses the point. I also rarely compile my Linux kernel manually, yet I care deeply that the ecosystem is open enough that I could understand, rebuild, inspect, and fork the pieces if necessary. For a storage appliance, trust matters more than convenience. The NAS is not a toy VM. It is the machine that owns my data.

That was the final push. The UI state bothered me. The virtualization churn bothered me. The build-system change made the decision obvious.

Why this was scarier than Proxmox #

Migrating Proxmox to NixOS was scary in the normal infrastructure sense. I had containers and VMs to move, but the blast radius was understandable. I could export with vzdump, import into Incus, and test a lot of it in virtual machines.

Migrating TrueNAS is a different kind of scary. The data is not a VM disk that I can casually copy around. The important thing is not to recreate the data pools. The important thing is to not touch them.

The target architecture is simple in principle:

# Conceptually:
# - disko owns only the boot disk
# - existing data pools are imported by name
boot.zfs.extraPools = [
 "ssd"
 "tank"
];

The NixOS boot disk can be destroyed and recreated. The existing tank and ssd pools cannot.

This is the part where many migration guides become hand-wavy:

Install the new OS.
Import your ZFS pools.
Good luck.

That is not enough for me. I could physically access the machine, but unplugging the relevant NVMe hardware would mean taking apart more of the box than I wanted to touch. So I treated the cutover as if the disks had to stay connected. And I absolutely did not want one typo in a disk path to become yet another meme in AI agent failures.

Giving an agent the keys, carefully #

Without AI this would probably have become a week-long project that I would forever postpone. With AI, it felt plausible to get the first serious version done in a single sitting. My approach was to give GPT 5.5 in xhigh mode SSH access to the live TrueNAS box and tell it, repeatedly and very explicitly, not to change anything. To be clear, it had root access. At this point these models have become powerful enough that I trust them to follow explicit instructions. The safety boundary was instruction-following, narrow scope, and review.

Not “please be careful.” More like:

You can inspect.
You cannot change anything.
Do not start or stop services.
Do not import or export pools.
Do not load keys.
Do not mount datasets.
Read only.

This is the part where people either nod along or think I have completely lost my mind. I think the distinction matters. I did not ask an agent to “migrate my NAS.” I asked it to inspect the current system, summarize the live state, compare that against my NixOS repository, and propose declarative config.

Then I reviewed the diffs. Then I asked for another review. Then another. The process was messier than one neat two-agent pass: build a scaffold, ask a model to attack it, compare that against the live TrueNAS state, refine the scaffold, and repeat. GPT 5.5 did most of the initial construction. Claude Opus 4.8 on max effort did one of the later independent cross-checks against the live system. By the end, I trusted the result because multiple independent reviews had converged on the same assumptions, and I understood the remaining differences. The loop felt like code review: inspect live state, turn the desired parts into Nix, review the diff, and ask another model to attack it. The scariest assumption got a VM test, and the destructive step stayed small.

Byte-for-byte TrueNAS cloning was never the target. I wanted every important behavior explicit: storage import, shares, snapshots, replication, Incus recovery, monitoring, and the few places where NixOS intentionally differs from the old appliance. At that point, the NixOS config reproduced the parts of TrueNAS I actually depended on, and the remaining differences were deliberate.

The agent was excellent at inventory:

Which pools exist
Which datasets are encrypted
Which NFS exports are live
Which SMB shares exist
Which users and numeric IDs matter
What Incus storage pool is in use
What replication tasks exist
What SMART/NUT/health monitoring exists
What host ID ZFS expects
Which disk is the current boot pool

Preserving everything blindly would have been wrong. The useful parts became code, and the dangerous parts became explicit.

The NixOS scaffold #

The resulting NixOS host is split into small files:

hosts/nas/
├── default.nix
├── disko.nix
├── hardware-configuration.nix
├── health.nix
├── identity.nix
├── networking.nix
├── nfs.nix
├── replication.nix
├── samba.nix
├── storage.nix
├── virtualization.nix
├── CUTOVER.md
└── PLANNING.md

This is exactly why I like NixOS. The migration stopped being a list of steps I needed to follow and became a pull request.

The important storage boundary is:

disko.nix describes only the boot disk.
The data pools are imported by name.
ZFS force-import is disabled.
Boot-time encrypted credential prompts are disabled.
Encrypted datasets are unlocked manually after boot.
Sanoid/Syncoid replace TrueNAS snapshot and replication tasks.
SMB, NFS, Docker, Incus, SMART, ZED, NUT, Netdata, and exporters are all declared in code.

The data pools are intentionally not in disko. That is the entire point.

The disko problem #

disko is a fantastic tool. It lets you describe disks declaratively and then formats/partitions/mounts them. For a fresh NixOS install, it is exactly what I want.

This also fits how I install NixOS machines. I build a custom installer ISO, put it on a USB stick, and boot from it. The image already has SSH enabled with my keys, flakes enabled, and the tooling needed to run the repo-defined disko and nixos-install flow. That turns the installer environment into part of the reviewed configuration, instead of another place where I have to remember which choices to make.

It is also exactly the tool that can ruin your day if pointed at the wrong disk.

My requirement was:

Use disko to destroy and recreate the boot disk, but prove it cannot touch the existing ZFS data pools.

The static config already looked right. The generated disko script only referenced the boot device and generated partition labels. The data pools were only present under boot.zfs.extraPools.

But “looks right” is not enough when the downside is wiping a NAS.

So we built a VM rehearsal.

Rehearsing the dangerous part in a VM #

The VM test creates a miniature version of the dangerous situation:

Attach several throwaway disks to a disposable NixOS VM.
Create a fake boot-pool on one disk.
Create fake tank and ssd ZFS pools on other disks.
Write sentinel files into the fake data pools.
Export the fake data pools.
Run the generated nas disko script against the fake boot disk.
Confirm the boot disk now contains the new zroot.
Confirm fake tank and ssd still have their ZFS labels.
Re-import fake tank and ssd.
Read the sentinel files.

The check lives in the flake now:

cd configs/nixos
nix build .#checks.x86_64-linux.nas-disko-safety

It passed.

Passing that VM test gives me confidence in the generated Nix/disko logic. The read-only check on the live TrueNAS box gives me confidence in the configured disk target: the by-id path resolves to the 500 GB boot-pool disk, and tank and ssd live on different devices. The installer preflight repeats that same check immediately before the destructive step.

This is the level of paranoia I want around storage migrations.

The parts that are not just storage #

Once the “do not wipe the pools” boundary was in place, the rest of the migration became more normal infrastructure work. Still annoying, but normal.

SMB was not a byte-for-byte port. TrueNAS ships private Samba VFS modules and has its own ACL/middleware behavior. Stock NixOS Samba covers the parts I need with fruit, streams_xattr, and sensible share definitions, but it is not literally the same stack. So the plan calls out what must be tested from real clients: Time Machine, guest access, and photo/media access.

Snapshots and replication also changed. TrueNAS snapshot tasks become Sanoid policy. TrueNAS replication tasks become Syncoid systemd timers. Existing snapshots are preserved, but Sanoid will not magically prune old TrueNAS-named snapshots. That is a feature, not a bug. I would rather manually decide when old snapshots age out than have a new tool silently “clean up” history.

Incus is another subtle one. The storage pool lives on a data pool, so disko will not touch it. But a fresh NixOS install has a fresh Incus database. The plan is to run incus admin recover, then apply a small declarative reconciler for the known instances. The state boundary has to be clear: preserve the storage dataset, recover the Incus database, then reconcile the instance config.

Health monitoring is also declared:

SMARTD
ZFS Event Daemon
NUT/UPS client
Netdata bound locally
Prometheus exporters
A shared alert hook that can log, wall, and optionally send ntfy alerts

TrueNAS gives you many of these things in the UI. NixOS gives them to me as code.

I know using your NAS for other services is a bit of an anti-pattern. In an ideal world, the storage box just stores data and stays boring. But when I originally bought and configured this machine, it became the most powerful machine in my home lab. So it naturally started attracting the Docker workloads too. Most of those workloads are not resource intensive, and keeping them close to the data is genuinely convenient. At one point I was running more than a hundred containers on that machine and it barely made a dent. They ran inside LXC containers on Incus rather than directly on the TrueNAS host, which made it feel like a reasonable compromise for a long time.

The migration also became a chance to fix a real operational problem. My worst TrueNAS incident was a long OOM death spiral: no swap, too many unbounded containers, and services repeatedly getting killed and restarted. The philosophy came from Nix and open infrastructure. The OOM incident was the practical reminder that the NAS had become a general-purpose compute host. In the NixOS version, container memory limits, zram, earlyoom, and an explicit ZFS ARC cap are part of the configuration instead of being tribal knowledge or post-incident notes.

Secrets stay out of the repo #

My dotfiles are public. That makes the migration more interesting.

The Nix config necessarily contains some operational facts: hostnames, share names, service definitions, and so on. But the plan deliberately keeps secrets and sensitive cutover material out of git.

For this migration, that means:

ZFS dataset passphrases and unlock material stay off-box and outside the repo.
Replication private keys are installed manually under /etc/ssh.
Inbound authorized keys for backup senders are installed during cutover.
Alerting secrets live in a runtime env file.
A local ~/nas-cutover/ staging directory can exist, but it is explicitly not part of the repo and should not be casually read by agents.

One important detail: the encrypted datasets use passphrase keys. In the TrueNAS setup I handled that with truenas-unlock, a small tool I wrote for this exact problem. The unlock material lives on another device; when that device is on the same network, it can unlock the datasets through the TrueNAS API. That gave me a lightweight hardware/network-presence factor: the NAS could boot, and the encrypted datasets only unlocked when the separate unlock device was present too.

That changes the migration footgun. The TrueNAS API path does not survive the cutover as-is, so I split the same idea out into zfs-unlock: the NixOS/OpenZFS version that talks to a restricted SSH receiver instead. During the migration I still kept an independent manual passphrase recovery path, because I do not want the automatic unlock mechanism to be the only way back into my data.

Where this leaves TrueNAS #

I still like TrueNAS. It got me very far. It made ZFS approachable. It gave me a good NAS UI when I wanted a NAS UI. And if you want an appliance that mostly owns storage for you, it is still a very compelling project.

I like NixOS more for my own infrastructure. The most important machine in my house should live in reviewable code instead of mutable appliance state spread across a UI, a database, and old debugging decisions. NixOS and AI are a particularly good match for that. I can describe messy infrastructure intent in imprecise English, let the agent turn that into exact Nix code, and review the diff before it becomes system state. The prompt stays fuzzy. The machine state does not.

After the Proxmox migration, TrueNAS was the last major exception. This migration removes that exception.

The cutover #

I ended up using a phased nixos-anywhere cutover instead of the USB installer path. The important safety property stayed the same: first kexec into a temporary NixOS installer, stop there, run the disk preflight from inside that installer, and only then run the destructive disko,install,reboot phase. Before the destructive phase, the box was running the installer in RAM and the disks had not been intentionally modified yet. That gave me a real checkpoint where aborting was still cheap.

I kept my hands on the few commands that actually changed the machine. I ran the nixos-anywhere phases and the later nixos-rebuild commands myself. The agent stayed next to it, reading the outputs, checking the assumptions, SSHing in to verify pool health, services, mounts, Incus state, NFS clients, and then turning any findings back into code or documentation. That split felt right to me: I pressed enter on the small number of irreversible commands, and the AI did the tedious verification work that I would otherwise be tempted to rush.

The actual migration was almost anticlimactic compared to the preparation. From the moment TrueNAS stopped being the running OS to having the Incus containers back with my hundred-plus services running again took less than 20 minutes. The slow work was not the cutover. The slow work was making the cutover boring.

The small things that still needed fixing during cutover

There were a few small first-boot reconciliations. I am writing them down mostly for future readers who want to follow a similar path:

imported data-pool mountpoints needed to be reconciled to the /mnt/... paths my clients and services already used;
encrypted datasets needed a better manual post-boot unlock helper, and the old TrueNAS API unlock flow needed a NixOS/OpenZFS successor;
Incus needed incus admin recover because the storage pool survived but the new OS had a fresh Incus database;
one unprivileged recovered container needed the matching subordinate UID/GID ranges declared for its passthrough idmap;
nas.local needed an explicit DNS record so my wildcard did not send it to a workload container;
the old PC job that backed up the TrueNAS API config needed to be removed;
the PC NFS mounts, Nix cache endpoints, and recovered Docker workload container all needed client-side validation.

Those findings went back into the cutover runbooks and the Nix config in a follow-up PR, so the notes are available for anyone who might benefit from them later.

At the end of the cutover, the machine was nas, the ZFS pools were healthy, the NFS mounts worked from my PC, the Incus containers were recovered, and the Docker workloads were running again. SMB authentication, guest access, Time Machine, and normal media browsing from macOS Finder worked after one final share-root permission fix. At that point the migration was working end to end: the pools were healthy and the services I actually use were back.

References #

Friend‑to‑friend TrueNAS backups without the cloud

Sat, 24 Aug 2024 00:00:00 +0000

The Challenge: Escaping the Cloud #

I recently wrote about my homelab and how I use TrueNAS on my NAS for data storage. Most self-hosters are looking for ways to “de-cloud” themselves, taking control of their data away from big tech companies, and reducing monthly subscription fees. One subscription cost that many of us still rely on is cloud storage for backups. Personally, I use Rclone to back up my most essential data to Backblaze B2, but now that my data is growing, I’m looking for a more cost-effective solution.

Luckily, I have a friend who also uses TrueNAS and is looking for a similar solution.

TrueNAS offers built-in replication tasks that allow us to efficiently replicate our data to another TrueNAS system. Unfortunately, this requires root access to the remote system, which is a security risk neither of us are willing to take

So, how can we set up remote ZFS replication with a friend, maintaining our de-clouded status, without compromising the security of our TrueNAS systems?

The Requirements #

Use TrueNAS’s built-in replication tasks (uses zfs send and zfs receive for efficient replication, which is significantly faster than rsync)
Replicate to a friend’s system with ZFS capabilities
Avoid granting root access to each other’s TrueNAS systems
Maintain control over the allocated backup space
Keep our data private and out of corporate clouds

The Solution: TrueNAS VM on hypervisor and iSCSI Backup Storage on TrueNAS #

After exploring various options, we’ve found a secure and efficient solution using a hybrid VM setup that keeps us firmly in control of our data. Here’s the high-level overview:

Deploy a TrueNAS VM on a local hypervisor (e.g., Proxmox), running independently of your main TrueNAS system
Set up an iSCSI target on your TrueNAS for backup storage
Connect the iSCSI target as an additional disk to the VM and format it with ZFS
Give your friend secure access to this VM
Your friend sets up replication tasks to this VM’s iSCSI-connected storage

Why This Works #

Security: No need for root access on each other’s TrueNAS systems and data can be encrypted on the iSCSI storage with friend’s keys
Performance: VM OS runs from fast local storage, while backups use dedicated iSCSI storage
Flexibility: Easy to manage and update the VM independently of backup storage
Efficiency: Minimizes data transfer over iSCSI for system operations
De-Clouded: Your data stays between you and your friend, no cloud services involved

Step-by-Step Setup #

On your main TrueNAS #

Create a new zvol for backup storage
- Navigate to Storage > Pools > [Your Pool] > zvols
- Click “ADD zvol”
- Name it (e.g., “friend_backup”)
- Set an appropriate size
Set up an iSCSI target
- Go to Sharing > Block Shares (iSCSI)
- Click “Add” under Targets
- Give it a name and create
- Under Extents, click “Add”, select the zvol you created
- Under Associated Targets, add your new target and extent

On Proxmox #

Mount the iSCSI in Proxmox
- Go to Datacenter > Storage > Add > iSCSI
- Enter your TrueNAS IP and the target name
Create a TrueNAS VM
- Download TrueNAS ISO to /var/lib/vz/template/iso
- Click “Create VM”
- Select the TrueNAS ISO
- In the “Hard Disk” step, add two SCSI disks:
  - One on local storage (for OS)
  - One on the iSCSI storage (for backups)
Install TrueNAS
- Start the VM and open console
- Install TrueNAS on the local disk
Configure TrueNAS VM
- Access the TrueNAS web interface
- Create a ZFS pool on the iSCSI disk

Set up secure access #

Install and configure Tailscale (or preferred VPN)
- On both your system and the TrueNAS VM
- Use a self-hosted Headscale server for more control

For your friend #

Provide VPN access to the TrueNAS VM
They can now set up replication tasks to the VM’s iSCSI storage using the TrueNAS web interface and their own encryption keys

Repeat the process #

Follow the same steps on your friend’s system, reversing roles

Considerations #

Network: Ensure your internet connection can handle the data transfer
Initial Seeding: For large datasets, consider initial seeding in person
Encryption: Use encryption for the replicated datasets and VPN access for added security
Testing: Thoroughly test the setup before relying on it for critical backups
Trust: Choose your backup partner wisely - the other person will have access to the TrueNAS VM but not its data (if encrypted)
VM Maintenance: Regularly update and maintain the VM’s operating system

Conclusion #

This friend-to-friend setup offers TrueNAS users an effective approach to remote backups. By creating mutual TrueNAS VMs, we’ve established a secure, decentralized backup solution with several advantages:

Privacy and Security: Each friend has their own remote TrueNAS VM, with data remaining private when properly encrypted. Importantly, while you grant your friend access to your network, it’s limited to only the backup VM. You can firewall off the rest of your network, maintaining overall system security.
Full ZFS Feature Set: This setup allows us to use ZFS features like snapshot-based backups, incremental replication, and data integrity checks.
Flexible Backup Strategies: We can back up entire filesystems and maintain multiple snapshot versions.
De-clouded Solution: By partnering with a trusted friend, we’ve created a backup system that keeps our data out of corporate clouds, reducing costs and increasing control.
Scalability: As data needs grow, we can add additional iSCSI targets in Proxmox and incorporate these new virtual disks into our ZFS pool.
Network Isolation: Despite allowing access for backups, your main network remains protected. You control exactly what your backup partner can access, limiting potential security risks.

This solution represents a step towards data sovereignty and community-based storage. It’s secure, efficient, and independent of large tech companies, while still maintaining tight control over your network access.

Remember to regularly test your backup and restore processes, and periodically review your firewall rules. With this setup, you have a comprehensive, friend-to-friend backup solution that doesn’t compromise your overall network security.

Happy backing up!

My homelab: from a single Raspberry Pi to a Proxmox cluster and a dedicated NAS 🔧

Sat, 06 Jul 2024 00:00:00 +0000

In 2020, I began my homelab journey (even though I never heard of the term “homelab” at that point) with a simple yet powerful setup: Home Assistant running on Proxmox on a NUC8i3BEH. My Home Assistant instance, with over 100 automations and about 20 addons (docker containers), had outgrown the capabilities of my Raspberry Pi 4, leading me to upgrade to the NUC8i3BEH. You can check out my Home Assistant configuration.

This system was more than sufficient for running Home Assistant—arguably overkill. However, because I passionately develop addons and custom components, I needed the ability to quickly restart and reload everything while debugging. For a long time, I ran Proxmox, mainly because the Home Assistant community recommended it, but I only used it to host one extra Ubuntu VM and Home Assistant OS.

In 2022, I stumbled upon the Proxmox VE Helper-Scripts by tteck, which got me interested in the whole homelabber/self-hosting community. These scripts opened a new world of possibilities. With them, I expanded my Proxmox environment to run 25 different VMs and LXC containers, each performing a variety of tasks, from DNS servers and reverse proxies to local dashboards and backup software.

Initially, I used an external 2 TB drive for storage, but it quickly became insufficient as my data needs grew. After some research, I found the HP EliteDesk 800 G4 SFF, a renewed enterprise-grade desktop with slots for two 3.5-inch drives. At approximately $160 on Amazon, it offered excellent specs for a budget-friendly price. I decided to create a TrueNAS VM on this machine using Proxmox and passed through the two 16TB renewed enterprise-level HDDs (via the PCIe SATA controller), which I had purchased from serverpartdeals.com. I chose these drives based on their reliability as reported in the BackBlaze drive stats.

When I got the HP, I created a Proxmox cluster and was very excited about how easy it was to migrate containers and VMs from one machine to the other. However, I soon encountered a significant drawback: enterprise drives are notably loud. Since I had my Proxmox “cluster” (just the two machines) running behind the TV, the loudness was noticeable and disruptive. This noise was just the beginning of my troubles. The NFS server on the TrueNAS VM often crashed, causing the entire Proxmox system to lock up. Initially, I mounted the NFS drives directly on the Proxmox host and used bind-mounts to pass through the drives. Due to the locking behavior of NFS, any problem would cause the entire system to lock up. To prevent this, I started to use autofs to mount the drives in the LXC containers and VMs instead. This worked fine until Proxmox started its periodic backups. Any issue with the NFS during backup caused the whole process to hang, locking up the VMs and LXC containers. This could be partially avoided by using the “stop” backup mode, but it still resulted in almost daily debugging sessions where I had to restart Proxmox-related services.

After running all services as LXC containers, I realized that updating them and keeping up with all security updates was labor-intensive. So, I started to look into migrating everything to Docker Compose. This transition turned out great because I could manage my config files in a git repo and keep the relevant data separate, streamlining updates and maintenance.

To resolve these issues and because I was having a lot of fun, I decided to build a dedicated NAS running TrueNAS Scale, not that I really needed one, but mostly because I could. This new setup included high-performance components, ensuring both stability and quiet operation. I used “Manufacturer Recertified” Western Digital Ultrastar DC HC550 HDDs, known for their reliability and performance. I have 6 x 16TB = 96TB now with the option to add 2 more 3.5" drives, 2 nvme M.2 drives, and 1 SATA 2.5" SSD. Now, I run all services that previously relied on NFS locally on TrueNAS via a Debian systemd-nspawn container managed with a new script called Jailmaker, which leverages the systemd-nspawn program in TrueNAS Scale (not to be confused with FreeBSD jails). Inside this systemd-nspawn container, I run Debian with Docker. For managing Docker Compose files, I use Dockge, which provides a nice web UI for easy management.

Along the way, I got familiar with ZFS and grew to love it for its amazing snapshotting and data integrity features. ZFS has been a game-changer, providing robust data protection and ease of management. Initially, I used mirrored vdevs, but I recently switched to RAIDZ2 for the added storage capacity and the promise of OpenZFS 2.3 supporting expansion of RAIDZ vdevs. I also use ZFS for my Proxmox VMs and LXC containers, whose backups I store on the NAS via rsync in a cronjob to another jail running plain Debian (no more NFS on the Proxmox host for me…)

This journey from a simple NUC running Home Assistant to a Proxmox cluster and dedicated NAS has been very time-consuming but a lot of fun. The lessons learned and the improvements in stability and performance have made the effort worthwhile. If you’re considering building your own homelab, investing in reliable hardware and being prepared for a bit of trial and error can make all the difference.

Lessons learned #

Run your services in Docker containers, not LXC containers, for easier management and updates.
Do not have virtualized NAS (TrueNAS VM) on Proxmox, it’s a bad idea if you are also sharing the drives with Proxmox via NFS. There are many folks on the internet who run their NAS in a VM on Proxmox, seemingly without issues, but I was not one of them.
Do not encrypt the root ZFS dataset! Encrypt its children instead.
Use passphrase encryption instead of a generated key. It allows you to use the UI and rotating keys.
Do not remove the physical drive that is passed through in a Proxmox VM, because Proxmox will not boot properly afterwards.
Do not use random 100 character passwords for your Proxmox, you have only 60 seconds to enter it during boot.
When making Proxmox backups, send them to a different physical machine, not only the same machine that is being backed up.
When building a NAS or home server, consider noise levels, especially if the equipment will be in a living area. Enterprise-grade drives can be surprisingly loud.
When using Intel I226-V 2.5GbE controllers, be prepared for potential issues.
When planning your NAS storage size needs, remember to account for the extra space needed for redundancy (e.g., in RAIDZ2). Additionally, keep in mind that ZFS performs optimally when drives aren’t filled to capacity. It’s wise to plan for more storage than you think you’ll immediately need.
Buy as many used enterprise-grade hardware components as you can, they are often cheaper and more reliable than consumer-grade components.

Components #

For completeness, here are the components for each machine in my homelab:

NUC Components #

Name	Full Name	Server	Price	Quantity	Cost	Date
NUC8i3BEH	Intel NUC Kit NUC8i3BEH	NUC	278.3	1	278.3	2020-02-22
500GB M2 SSD	Samsung 970 EVO M.2 500GB	NUC	94.99	1	94.99	2020-02-21
32 GB RAM	TEAMGROUP 32GB DDR4 3200MHz	NUC	58.99	1	58.99	2024-05-01
4TB SSD	Crucial P3 Plus 4TB NVMe	NUC	226.99	1	226.99	2024-04-27

HP Components #

Name	Full Name	Server	Price	Quantity	Cost	Date
64 GB RAM	TEAMGROUP 32GB DDR4 3200MHz	HP	57.99	2	115.98	2024-05-02
HP EliteDesk 800 G4 SFF	HP EliteDesk 800 G4 SFF	HP	164	1	164	2024-04-27

Notes on the HP components:

Initially, I planned to use this machine as a NAS because it has two 3.5" drive bays, 2 M.2 slots, and the disc drive bay can be converted to a 2.5" drive bay. However, the 2 x 16TB drives I bought quickly filled up because I used a mirrored vdev, so I decided to build a dedicated NAS instead.

NAS Components #

Name	Full Name	Server	Price	Quantity	Cost	Date
16 TB HDD	Western Digital Ultrastar DC HC550 16TB	NAS	159.99	2	319.98	2024-05-01
16 TB HDD	Western Digital Ultrastar DC HC550 16TB	NAS	187.5	4	750	2024-06-21
Motherboard	Q670 8-bay NAS motherboard	NAS	203.21	1	203.21	2024-06-22
CPU i5 13500T	Intel Core i5-13500T	NAS	180.88	1	180.88	2024-06-22
PSU SFX	FSP Dagger Pro 650W	NAS	87.13	1	87.13	2024-06-21
Jonsbo N3 case	JONSBO N3 Mini-ITX NAS Case	NAS	187.51	1	187.51	2024-06-16
CPU cooler	Thermalright AXP90 X36 CPU cooler	NAS	21.83	1	21.83	2024-06-25
64 GB DDR5 RAM	CORSAIR VENGEANCE 64GB DDR5	NAS	162.13	1	162.13	2024-06-29

Notes on the NAS components:

The 13500T is a 35W TDP CPU I bought on eBay, which will help keep the system cool and quiet.
I bought the motherboard from AliExpress because it seems to be the only mini-ITX motherboard that supports 8 SATA drives, and also comes with 2x2.5GbE ports, 3xM.2 slots, and a full sized PCIe slot.
The motherboard uses Intel I225-V 2.5GbE controllers, which are notoriously bad, initially iperf3 maxed out at 100Mbps. The internet is full with people complaining about these controllers auto-negotiating to 100Mbps, however, for me it showed 1Gbps. After trying all suggestions I could find (there are few), as a last resort I connecting the NAS directly to my ASUS XT8 router instead of the NETGEAR GS305 switch, and now I got 1Gbps speeds. So I bought a new managed TL-SG108E switch, which now also gives me 1Gbps speeds. I’m not sure what the issue was, but I’m happy it’s resolved.