Migration | Bas Nijholt

The last btrfs machine: migrating my PC to ZFS

Thu, 23 Jul 2026 00:00:00 +0000

My GPU machine was the last machine in my fleet still running btrfs. I bought it for gaming (which never happened), and these days its dual RTX 3090s almost exclusively run local AI models.

btrfs itself never failed me. When I set up NixOS on this machine for the first time, I had read enough forum threads to become convinced that ZFS and NixOS were a troublesome combination, so I picked btrfs to be safe. That belief was simply wrong. As I went full Nix and eventually moved even my TrueNAS to NixOS, I learned there was no issue at all. I just had to figure out the right incantation to make ZFS work with boot drives, disko, and nixos-anywhere. My HP got ZFS first, that config grew into a shared disko module, and soon the whole fleet ran one pattern: NixOS on ZFS, snapshots and replication declared in code, and backups I can reason about.

Only the machine I sit at every day kept running on my old misconception, because a daily driver is exactly the one you do not casually reinstall.

Table of contents

Why touch a working machine #

The operating system was already uniform. The filesystem was not, and that turned out to matter more than I expected.

The real trigger was that I got increasingly paranoid about an AI agent messing up my system. I run agentic AI in YOLO mode, exclusively. All day, every day, agents execute commands on this machine without asking me first. YOLO does not mean root, though: the agents run as my regular user, and the backup credentials live where only root can read them. An agent that could destroy the backup itself would make everything else in this post pointless. Meanwhile the stories keep coming in about even the most frontier models wiping someone’s home directory:

GPT-5.6-Sol just accidentally deleted almost ALL of my Mac’s files. And this is why I trust Fable 1000x more.

— @mattshumer_

The reaction is always about which model deserves more trust. I read these tweets differently. Give any agent enough sessions and it will eventually delete something important, the same way I do every few years with a tired rm -rf. Whether that moment becomes a catastrophe or an inconvenience depends entirely on the restore path.

So I felt I needed to properly audit the way my backups were done. I want to be able to answer four questions about any machine I own:

What exactly is in the backup?
What is not in the backup, and did I choose that deliberately?
How big is a restore?
How long will it take?

The backup that wasn’t #

Asking question 1 delivered an unpleasant answer within the hour: the backups had quietly stopped four months earlier. Two boring causes had stacked up: an interrupted prune left a stale repository lock, which the job could never clear because its own unlock step only ran after a successful backup, and the repository URL still pointed at truenas.local, a name that had stopped resolving along the way. The last successful backup ran at 07:00 on March 22; the next one at 07:16 on July 22, after the fix. Four months, almost to the minute. systemd logged the failure on every single run, nobody read it, and nothing escalated. I found out because I asked the question, not because anything told me.

What the audit found #

On the btrfs box, question 3 alone had three different answers depending on which tool I asked. du counted reflinked copies at full size, so directories reported terabytes that physically did not exist. df counted physical extents, which is honest about disk usage but tells you nothing about what a file-based restore will write. And restic reported the apparent size of sparse VM images, which inflated the restore estimate to several times the disk.

When your storage layer gives you three sizes for the same data, you do not really understand your backup. You have a backup, and you have hope.

Question 4 was no better. Because the PC was my only btrfs machine, it could not join the ZFS snapshot replication the rest of the fleet uses, so its backups relied entirely on restic pushing to the NAS. Off-machine, note, but not off-site; question 2 turned out to apply to locations too, and closing that gap is its own project. restic is file-based: every run visits the metadata of every single file to find what changed. On this machine that meant close to a hundred million files per scan, which turned out to include some 600 GB of git worktrees stuffed with virtual environments and node_modules. Each run took about an hour and a half, even when almost nothing had changed.

That scan time dictated my backup cadence. The timer ran hourly, which in practice meant back-to-back scanning all day to move a couple hundred megabytes of real changes, and I recently dropped it to every six hours to stop the madness. Either way, my most recent work existed only on the machine an agent was busy poking at.

The scanning is also not free for the hardware. Reads are gentle on an SSD compared to writes, but hourly scans of an hour and a half each kept the drive at full duty cycle and elevated temperature for most of every day. And drives stopped being cheap: I paid $227 for a 4 TB NVMe drive in 2024, and the same drive lists for around $800 today. I would rather not run the experiment of what years of continuous scanning do to a drive I can no longer afford to replace.

btrfs has perfectly good snapshots of its own, but sending them off-site was never an option for me: snapshot replication needs the same filesystem on the receiving end, and my NAS speaks ZFS. With ZFS I can snapshot every few minutes and zfs send the actual block storage: an incremental send streams exactly the blocks that changed since the last snapshot, without walking a single file. A hundred million files cost nothing when nobody has to visit them.

ZFS does not magically collapse those numbers into one. But it gives me the same semantics everywhere, including on the NAS pools that hold the backups themselves. Eight of my machines already share a single ZFS disk layout, defined in one shared disko module. The PC was the odd one out. This migration makes it nine machines with the exact same strategy, and one backup story to audit instead of one per machine.

The plan: one shared disko module #

The Nix side was the easy part. The PC now imports the same shared disko module as the other eight machines, with one addition: a swapSize parameter, because ZFS has no reliable swapfile story and the PC actually needs its 96 GB of swap.

The disk is pinned by ID, not by device name:

device = "/dev/disk/by-id/nvme-Samsung_SSD_990_EVO_Plus_4TB_...";

This matters more than usual here. The PC dual-boots Windows from a second NVMe drive, and I want zero ambiguity about which of the two disks disko is allowed to erase.

The scary property of this migration is different from the NAS one. For the NAS, the whole point was that disko must not touch the data pools. For the PC, the new ZFS pool goes on the same disk the data currently lives on. No second disk, no local rollback, no btrfs to boot back into. From the moment disko runs, the restic repository on the NAS is the only road back.

That sentence is the reason most of this post is about verification.

The road back also has to survive the same agents that motivated all of this. The PC pushes its backups over sftp as a user that owns every file in the repository, which means a compromised PC could delete or encrypt its own safety net. So the NAS snapshots the repository dataset on its own side, where the PC’s credentials cannot reach: two days of hourly ZFS snapshots and two weeks of dailies that only root on the NAS can destroy.

Rehearsing in a VM, again #

The NAS migration taught me to rehearse the destructive step in a VM, so the flake grew a pc-vmtest target and I ran the whole install through nixos-anywhere --vm-test. Disko formatted a virtual disk, created the pool, installed the system, and booted it with swap active.

Passing that test proves the generated logic. It does not prove the backup, which is where the real risk lives.

The recovery kit #

A restore that starts with “first, log in to the dead machine and fetch the credentials” is not a restore plan. So everything a bare reinstall needs lives in a small recovery kit on two other machines in the house, permissions locked down: the restic password and SSH key that reach the repository, the staged restore script, a couple of verification scripts, and the PC’s SSH host keys. The host keys matter more than they look: the reinstalled machine comes back with its old identity, so no client complains and nothing in the fleet needs to be taught to trust a new machine. A munge key and a README round it out.

Trusting the backup enough to wipe the disk #

My recovery story has two layers. NixOS makes the system disposable: every service, driver fix, and firewall rule is in git, so no agent can leave the machine in a state I cannot rebuild. Verified backups make the data recoverable. This migration is the voluntary version of the tweets above: I am about to delete all of my PC’s files on purpose.

I did three verifications, each one converting an assumption into a fact. An AI agent did the tedious sweeps; I reviewed the evidence.

The first check was completeness. We walked everything on the machine outside the backup paths, until every byte was accounted for. It was all re-downloadable caches, and the largest unexplained thing turned out to be restic’s own local cache. Everything outside the backup set is now known disposable instead of assumed disposable.

The second check was the restore path. A backup you have never restored from is a hypothesis. On a different machine, using only the recovery kit and no access to the PC at all, we restored a sample from the latest snapshot and compared hashes against the live system. The hashes matched byte for byte, and even the symlinks came back pointing at the right targets. A sample proves the path works, from credentials to symlink handling; reading back every byte is the third check’s job. The machine running the rehearsal had the kit and nothing else, so recovery provably does not depend on the machine being recovered.

The third check is the data itself. restic check validates the repository structure without reading back the actual data blobs. Before the wipe, a --read-data pass ran on the NAS, where the repository sits on local disk instead of behind the network. When the backup is the only rollback, I want its bits read back at least once.

The fifth question #

The outage taught me a fifth question: how do I find out when any of this stops working? All three checks above are point-in-time, and verification rots. So the NAS now re-verifies every hour, from its own side of the sftp connection, that the newest snapshot in the repository is less than a day old, and pushes an alert to my phone when it is not. That catches every failure mode on the PC side, including “the timer is simply disabled.” The watcher itself is watched: the NAS pings an external dead-man’s switch every five minutes, so if the machine holding my only road back goes dark, the alert comes from outside the house.

While wiring this up we discovered that months of earlier NAS alerts had never arrived, because ntfy silently rejects message bodies over 4 KB. A notification you have never received is also a hypothesis.

Restoring in stages #

The measured restore set came out to roughly 1 TB, and 99.9% of it is /home. But two directories are 64% of that terabyte: Steam (430 GB of re-downloadable games) and my local-AI model collection (217 GB of GGUF files).

I do not need either of those to have a working machine. So the restore script now runs in stages:

sudo ./restore_from_backup.sh stage1 # everything else → a fully working machine
sudo ./restore_from_backup.sh stage2 # Steam + models, whenever

Stage 1 restores everything except the deferred paths, plus the system identity (host keys). Stage 2 can run days later, from the rebooted system, while I am already using the machine.

One detail I care about: stage 1 resolves latest to a concrete snapshot ID and pins it to a file. Stage 2 reads the pin. My backups run every six hours, so without this, a timer firing between the stages would silently split the restore across two different snapshots.

The backup itself is untouched: Steam and the models are still in every snapshot. Only the restore order changed.

Slimming down first #

The PC had also quietly become an Incus host for three LXC containers, including the MindRoom bots my family talks to every day. Before the migration, those containers moved to the NAS. That shrinks both the backup and the restore, and the family bots keep running while their old host is being wiped.

The cutover #

The install followed the same phased nixos-anywhere pattern as the NAS cutover, driven from another machine on the LAN: kexec into an installer running in RAM, stop, re-verify the target disk from inside the installer, and only then run the destructive disko/install phase. That checkpoint is the whole point: right up to the destructive phase, aborting was still cheap. The destructive part turned out to be the trivial part: from wiping the disk to booting the new system took less than twenty minutes.

The one surprise of the day was in the restore, and it was exactly the kind rehearsal is supposed to catch. Stage 1 would not start: the restore script runs restic as root, and a freshly installed root has no SSH identity and no known_hosts. The key it needed sat in the recovery kit inside my home directory. The rehearsal never hit this, because the rehearsal machine’s root had already talked to the NAS. A verified restore path can still depend on state you forgot you had.

Stage 1 restored 534 GiB across 4.5 million files in just over an hour and a half, at wire speed. The machine rebooted with its old host keys and its old /home, and stage 2 started immediately. The plan said stage 2 could wait for days; it started the minute the machine was back, because once the machine works there is no reason left to wait. The deferred 648 GiB of games and models took six more hours in the background.

With that, the PC finally joins the fleet pattern it was locked out of: sanoid snapshots the pool every hour, so a tired rm -rf, mine or an agent’s, now costs at most an hour of work and a rollback, and a daily syncoid run sends the changed blocks to the NAS. restic is on borrowed time. Its six-hour timer keeps running until I trust the new path, and then it stops: the repository stays on the NAS as the last file-level snapshots of the machine, but nothing will be added to it.

Aftercare was the same short list, written down instead of remembered: revert the temporary root-SSH commit the installer needed, and put the restic password back where the backup service expects it.

The slow work, as always, was not the cutover. The slow work was making the cutover boring.

References #

Migrating from TrueNAS to NixOS without losing data after the build-system rug pull

Sun, 28 Jun 2026 00:00:00 +0000

TrueNAS was the last non-NixOS machine in my home lab.

I kept postponing it for a simpler reason: moving something with ~100 TiB of data feels scary. My NAS holds the big ZFS pools. It has the SMB shares, NFS exports, snapshots, replication tasks, encrypted datasets, Incus state, Docker workloads, and all the boring-but-important health checks that are easy to take for granted until something fails.

I had already migrated most of my machines from Proxmox to NixOS with Incus, as I wrote about in I’ve gone full Nix: Proxmox to NixOS + Incus. That migration made me increasingly intolerant of appliance-style infrastructure. If the important state lives in a database behind a web UI, I cannot review it in a pull request. If I change something while debugging, I cannot guarantee the change made it back into the source of truth. And if I let an AI agent help me, imperative systems become even more uncomfortable: an agent can run a hundred commands, succeed, and leave behind a system that nobody can reproduce. For the NAS specifically, that basically meant I did not dare touch the machine with an AI agent at all.

NixOS is different. The diff is the state. If an agent changes my infrastructure, I can review the patch. If the change is wrong, I can reject it. If it works, I can commit it.

TrueNAS was the last holdout. It had been itching for a while. Then the build-system rug pull made the decision easy.

Table of contents

The itch before the last straw #

The build-system change was not the first thing that bothered me. It was just the last straw.

The deeper problem was that my TrueNAS setup had slowly become complicated in exactly the way I dislike. I had clicked many buttons in the UI over time:

SMB shares
NFS exports
snapshot tasks
replication tasks
users and groups
encrypted datasets
apps and containers
Incus storage
old Jailmaker/systemd-nspawn state
alerting
UPS settings

TrueNAS can export and import its configuration. That is genuinely useful. But a database export is not the same thing as a configuration I can comfortably review. When I look at a TrueNAS backup, it is not obvious to me which settings I intentionally changed, which settings are defaults, which settings are historical leftovers, and which settings only exist because I clicked something while debugging two years ago.

One concrete example was NFS. I needed the crossmnt export option so child datasets would show up correctly inside the Docker container host, but TrueNAS did not expose that option in the UI. So I had a periodic root cron job patching /etc/exports after TrueNAS generated it. It worked, but it was exactly the kind of small appliance hack that made me uneasy: partly outside the UI, partly outside the exported configuration, and very easy to forget during a migration. In NixOS, that behavior is just part of the export line.

That is the exact discomfort that pushed me away from Proxmox. I do not want to remember what I clicked. I want the system to remember it as code.

The second itch was virtualization. My TrueNAS container story had already gone through one migration. For a while I used Jailmaker, which is a clever script around systemd-nspawn for running Linux “jails” on TrueNAS SCALE. It worked well enough for what I needed at the time: a Debian environment where I could run Docker workloads close to the data. But it also reinforced the feeling that I was building a fairly custom server on top of an appliance OS.

TrueNAS 25.04 “Fangtooth” introduced experimental Containers, named Instances in the early 25.04 releases, and the 25.04.1 announcement explicitly described Instances as being based on Incus, supporting both VMs and LXCs. That was interesting to me because the older TrueNAS VM system was libvirt-based, while Incus felt like a more coherent container/VM story. Then the TrueNAS virtualization plan for 25.04.2 explained that Fangtooth’s Instances feature caused challenges for some users and that 25.04.2 would re-enable the older Virtualization VM path alongside Instances. The 25.04.2 release announcement made that split official: Virtual Machines came back, while VMs created through the earlier Instances feature continued to appear under Containers.

For many people that churn was probably annoying. For me it was especially ironic. I had migrated from Jailmaker’s systemd-nspawn setup to Incus, read up on Incus, got excited about it, started using it, and then became genuinely happy with it when I migrated from Proxmox to NixOS with Incus. Incus fit my mental model perfectly: a clean CLI, containers and VMs in one place, easy scripting, and a workflow that agents can inspect.

Then TrueNAS started moving away from that direction again. Forum discussion around the 25.04.2 and later virtualization plans made it clear that Incus was not the stable long-term abstraction I hoped it would be inside TrueNAS. In the long-running Linux Jails with Incus thread, Kris Moore is quoted saying that Incus is being removed from the base system and that libvirt will handle the backend for VMs and LXC. That probably makes sense for TrueNAS. They have enterprise users, compatibility concerns, and a support matrix that is very different from mine.

But for me it meant the NAS was again becoming an appliance whose internal direction I had to follow. I had just escaped that with Proxmox. I did not want to do it again for the most important machine in my house.

The last straw #

Then I saw Jeff Geerling mention TrueNAS’s recent “shady behavior” in a video, and I started looking into what had actually changed.

The short version is that iXsystems made part of the TrueNAS ecosystem closed source. Specifically, iXsystems moved the TrueNAS build scripts and related build infrastructure internal for the TrueNAS 27 line. Their forum post, Clearing the Air on Build Scripts, frames this as a practical build-system decision and says normal installation and updates remain unaffected. The old truenas/scale-build repository now says it is preserved historically and is no longer the maintained build system. Community reactions, for example this TrueNAS forum thread, understandably read this as a meaningful step away from fully open development.

Would I ever have built TrueNAS from source myself? Almost certainly not.

That misses the point. I also rarely compile my Linux kernel manually, yet I care deeply that the ecosystem is open enough that I could understand, rebuild, inspect, and fork the pieces if necessary. For a storage appliance, trust matters more than convenience. The NAS is not a toy VM. It is the machine that owns my data.

That was the final push. The UI state bothered me. The virtualization churn bothered me. The build-system change made the decision obvious.

Why this was scarier than Proxmox #

Migrating Proxmox to NixOS was scary in the normal infrastructure sense. I had containers and VMs to move, but the blast radius was understandable. I could export with vzdump, import into Incus, and test a lot of it in virtual machines.

Migrating TrueNAS is a different kind of scary. The data is not a VM disk that I can casually copy around. The important thing is not to recreate the data pools. The important thing is to not touch them.

The target architecture is simple in principle:

# Conceptually:
# - disko owns only the boot disk
# - existing data pools are imported by name
boot.zfs.extraPools = [
 "ssd"
 "tank"
];

The NixOS boot disk can be destroyed and recreated. The existing tank and ssd pools cannot.

This is the part where many migration guides become hand-wavy:

Install the new OS.
Import your ZFS pools.
Good luck.

That is not enough for me. I could physically access the machine, but unplugging the relevant NVMe hardware would mean taking apart more of the box than I wanted to touch. So I treated the cutover as if the disks had to stay connected. And I absolutely did not want one typo in a disk path to become yet another meme in AI agent failures.

Giving an agent the keys, carefully #

Without AI this would probably have become a week-long project that I would forever postpone. With AI, it felt plausible to get the first serious version done in a single sitting. My approach was to give GPT 5.5 in xhigh mode SSH access to the live TrueNAS box and tell it, repeatedly and very explicitly, not to change anything. To be clear, it had root access. At this point these models have become powerful enough that I trust them to follow explicit instructions. The safety boundary was instruction-following, narrow scope, and review.

Not “please be careful.” More like:

You can inspect.
You cannot change anything.
Do not start or stop services.
Do not import or export pools.
Do not load keys.
Do not mount datasets.
Read only.

This is the part where people either nod along or think I have completely lost my mind. I think the distinction matters. I did not ask an agent to “migrate my NAS.” I asked it to inspect the current system, summarize the live state, compare that against my NixOS repository, and propose declarative config.

Then I reviewed the diffs. Then I asked for another review. Then another. The process was messier than one neat two-agent pass: build a scaffold, ask a model to attack it, compare that against the live TrueNAS state, refine the scaffold, and repeat. GPT 5.5 did most of the initial construction. Claude Opus 4.8 on max effort did one of the later independent cross-checks against the live system. By the end, I trusted the result because multiple independent reviews had converged on the same assumptions, and I understood the remaining differences. The loop felt like code review: inspect live state, turn the desired parts into Nix, review the diff, and ask another model to attack it. The scariest assumption got a VM test, and the destructive step stayed small.

Byte-for-byte TrueNAS cloning was never the target. I wanted every important behavior explicit: storage import, shares, snapshots, replication, Incus recovery, monitoring, and the few places where NixOS intentionally differs from the old appliance. At that point, the NixOS config reproduced the parts of TrueNAS I actually depended on, and the remaining differences were deliberate.

The agent was excellent at inventory:

Which pools exist
Which datasets are encrypted
Which NFS exports are live
Which SMB shares exist
Which users and numeric IDs matter
What Incus storage pool is in use
What replication tasks exist
What SMART/NUT/health monitoring exists
What host ID ZFS expects
Which disk is the current boot pool

Preserving everything blindly would have been wrong. The useful parts became code, and the dangerous parts became explicit.

The NixOS scaffold #

The resulting NixOS host is split into small files:

hosts/nas/
├── default.nix
├── disko.nix
├── hardware-configuration.nix
├── health.nix
├── identity.nix
├── networking.nix
├── nfs.nix
├── replication.nix
├── samba.nix
├── storage.nix
├── virtualization.nix
├── CUTOVER.md
└── PLANNING.md

This is exactly why I like NixOS. The migration stopped being a list of steps I needed to follow and became a pull request.

The important storage boundary is:

disko.nix describes only the boot disk.
The data pools are imported by name.
ZFS force-import is disabled.
Boot-time encrypted credential prompts are disabled.
Encrypted datasets are unlocked manually after boot.
Sanoid/Syncoid replace TrueNAS snapshot and replication tasks.
SMB, NFS, Docker, Incus, SMART, ZED, NUT, Netdata, and exporters are all declared in code.

The data pools are intentionally not in disko. That is the entire point.

The disko problem #

disko is a fantastic tool. It lets you describe disks declaratively and then formats/partitions/mounts them. For a fresh NixOS install, it is exactly what I want.

This also fits how I install NixOS machines. I build a custom installer ISO, put it on a USB stick, and boot from it. The image already has SSH enabled with my keys, flakes enabled, and the tooling needed to run the repo-defined disko and nixos-install flow. That turns the installer environment into part of the reviewed configuration, instead of another place where I have to remember which choices to make.

It is also exactly the tool that can ruin your day if pointed at the wrong disk.

My requirement was:

Use disko to destroy and recreate the boot disk, but prove it cannot touch the existing ZFS data pools.

The static config already looked right. The generated disko script only referenced the boot device and generated partition labels. The data pools were only present under boot.zfs.extraPools.

But “looks right” is not enough when the downside is wiping a NAS.

So we built a VM rehearsal.

Rehearsing the dangerous part in a VM #

The VM test creates a miniature version of the dangerous situation:

Attach several throwaway disks to a disposable NixOS VM.
Create a fake boot-pool on one disk.
Create fake tank and ssd ZFS pools on other disks.
Write sentinel files into the fake data pools.
Export the fake data pools.
Run the generated nas disko script against the fake boot disk.
Confirm the boot disk now contains the new zroot.
Confirm fake tank and ssd still have their ZFS labels.
Re-import fake tank and ssd.
Read the sentinel files.

The check lives in the flake now:

cd configs/nixos
nix build .#checks.x86_64-linux.nas-disko-safety

It passed.

Passing that VM test gives me confidence in the generated Nix/disko logic. The read-only check on the live TrueNAS box gives me confidence in the configured disk target: the by-id path resolves to the 500 GB boot-pool disk, and tank and ssd live on different devices. The installer preflight repeats that same check immediately before the destructive step.

This is the level of paranoia I want around storage migrations.

The parts that are not just storage #

Once the “do not wipe the pools” boundary was in place, the rest of the migration became more normal infrastructure work. Still annoying, but normal.

SMB was not a byte-for-byte port. TrueNAS ships private Samba VFS modules and has its own ACL/middleware behavior. Stock NixOS Samba covers the parts I need with fruit, streams_xattr, and sensible share definitions, but it is not literally the same stack. So the plan calls out what must be tested from real clients: Time Machine, guest access, and photo/media access.

Snapshots and replication also changed. TrueNAS snapshot tasks become Sanoid policy. TrueNAS replication tasks become Syncoid systemd timers. Existing snapshots are preserved, but Sanoid will not magically prune old TrueNAS-named snapshots. That is a feature, not a bug. I would rather manually decide when old snapshots age out than have a new tool silently “clean up” history.

Incus is another subtle one. The storage pool lives on a data pool, so disko will not touch it. But a fresh NixOS install has a fresh Incus database. The plan is to run incus admin recover, then apply a small declarative reconciler for the known instances. The state boundary has to be clear: preserve the storage dataset, recover the Incus database, then reconcile the instance config.

Health monitoring is also declared:

SMARTD
ZFS Event Daemon
NUT/UPS client
Netdata bound locally
Prometheus exporters
A shared alert hook that can log, wall, and optionally send ntfy alerts

TrueNAS gives you many of these things in the UI. NixOS gives them to me as code.

I know using your NAS for other services is a bit of an anti-pattern. In an ideal world, the storage box just stores data and stays boring. But when I originally bought and configured this machine, it became the most powerful machine in my home lab. So it naturally started attracting the Docker workloads too. Most of those workloads are not resource intensive, and keeping them close to the data is genuinely convenient. At one point I was running more than a hundred containers on that machine and it barely made a dent. They ran inside LXC containers on Incus rather than directly on the TrueNAS host, which made it feel like a reasonable compromise for a long time.

The migration also became a chance to fix a real operational problem. My worst TrueNAS incident was a long OOM death spiral: no swap, too many unbounded containers, and services repeatedly getting killed and restarted. The philosophy came from Nix and open infrastructure. The OOM incident was the practical reminder that the NAS had become a general-purpose compute host. In the NixOS version, container memory limits, zram, earlyoom, and an explicit ZFS ARC cap are part of the configuration instead of being tribal knowledge or post-incident notes.

Secrets stay out of the repo #

My dotfiles are public. That makes the migration more interesting.

The Nix config necessarily contains some operational facts: hostnames, share names, service definitions, and so on. But the plan deliberately keeps secrets and sensitive cutover material out of git.

For this migration, that means:

ZFS dataset passphrases and unlock material stay off-box and outside the repo.
Replication private keys are installed manually under /etc/ssh.
Inbound authorized keys for backup senders are installed during cutover.
Alerting secrets live in a runtime env file.
A local ~/nas-cutover/ staging directory can exist, but it is explicitly not part of the repo and should not be casually read by agents.

One important detail: the encrypted datasets use passphrase keys. In the TrueNAS setup I handled that with truenas-unlock, a small tool I wrote for this exact problem. The unlock material lives on another device; when that device is on the same network, it can unlock the datasets through the TrueNAS API. That gave me a lightweight hardware/network-presence factor: the NAS could boot, and the encrypted datasets only unlocked when the separate unlock device was present too.

That changes the migration footgun. The TrueNAS API path does not survive the cutover as-is, so I split the same idea out into zfs-unlock: the NixOS/OpenZFS version that talks to a restricted SSH receiver instead. During the migration I still kept an independent manual passphrase recovery path, because I do not want the automatic unlock mechanism to be the only way back into my data.

Where this leaves TrueNAS #

I still like TrueNAS. It got me very far. It made ZFS approachable. It gave me a good NAS UI when I wanted a NAS UI. And if you want an appliance that mostly owns storage for you, it is still a very compelling project.

I like NixOS more for my own infrastructure. The most important machine in my house should live in reviewable code instead of mutable appliance state spread across a UI, a database, and old debugging decisions. NixOS and AI are a particularly good match for that. I can describe messy infrastructure intent in imprecise English, let the agent turn that into exact Nix code, and review the diff before it becomes system state. The prompt stays fuzzy. The machine state does not.

After the Proxmox migration, TrueNAS was the last major exception. This migration removes that exception.

The cutover #

I ended up using a phased nixos-anywhere cutover instead of the USB installer path. The important safety property stayed the same: first kexec into a temporary NixOS installer, stop there, run the disk preflight from inside that installer, and only then run the destructive disko,install,reboot phase. Before the destructive phase, the box was running the installer in RAM and the disks had not been intentionally modified yet. That gave me a real checkpoint where aborting was still cheap.

I kept my hands on the few commands that actually changed the machine. I ran the nixos-anywhere phases and the later nixos-rebuild commands myself. The agent stayed next to it, reading the outputs, checking the assumptions, SSHing in to verify pool health, services, mounts, Incus state, NFS clients, and then turning any findings back into code or documentation. That split felt right to me: I pressed enter on the small number of irreversible commands, and the AI did the tedious verification work that I would otherwise be tempted to rush.

The actual migration was almost anticlimactic compared to the preparation. From the moment TrueNAS stopped being the running OS to having the Incus containers back with my hundred-plus services running again took less than 20 minutes. The slow work was not the cutover. The slow work was making the cutover boring.

The small things that still needed fixing during cutover

There were a few small first-boot reconciliations. I am writing them down mostly for future readers who want to follow a similar path:

imported data-pool mountpoints needed to be reconciled to the /mnt/... paths my clients and services already used;
encrypted datasets needed a better manual post-boot unlock helper, and the old TrueNAS API unlock flow needed a NixOS/OpenZFS successor;
Incus needed incus admin recover because the storage pool survived but the new OS had a fresh Incus database;
one unprivileged recovered container needed the matching subordinate UID/GID ranges declared for its passthrough idmap;
nas.local needed an explicit DNS record so my wildcard did not send it to a workload container;
the old PC job that backed up the TrueNAS API config needed to be removed;
the PC NFS mounts, Nix cache endpoints, and recovered Docker workload container all needed client-side validation.

Those findings went back into the cutover runbooks and the Nix config in a follow-up PR, so the notes are available for anyone who might benefit from them later.

At the end of the cutover, the machine was nas, the ZFS pools were healthy, the NFS mounts worked from my PC, the Incus containers were recovered, and the Docker workloads were running again. SMB authentication, guest access, Time Machine, and normal media browsing from macOS Finder worked after one final share-root permission fix. At that point the migration was working end to end: the pools were healthy and the services I actually use were back.

References #

I've gone full Nix: Proxmox to NixOS + Incus

Sat, 29 Nov 2025 13:00:00 +0100

I’ve gone full Nix: Proxmox to NixOS + Incus

I have officially decommissioned my Proxmox cluster. After years of running my homelab on Proxmox, starting with a single NUC and expanding to a multi-node cluster, I have migrated everything to NixOS running Incus.

Update 2026-06-26: This post reached the front page of Hacker News, and several readers correctly pointed out that Proxmox is not limited to clicking around in the web UI. It has qm, pct, config files, a REST API, Terraform providers, and Ansible workflows. My point is not that Proxmox cannot be automated. Even with that automation, state drift can still creep in when debugging means running one quick command, especially if an agent is allowed to execute imperative fixes that never make it back into the automation framework. It is that, for my setup, I wanted the reproducible configuration itself to be the source of truth. The thing I care about is not buttons versus commands, but whether I can rebuild the host from version-controlled text files and know that every important change is captured there.

From Skeptic to Believer #

I wasn’t always a Nix evangelist. In fact, I initially despised the language and its syntax. I couldn’t figure out how it worked, and I already had my own specific way of setting up my dotfiles. I used Dotbot for symlinking and a tool I wrote called dotbins for managing binaries. I didn’t feel like I required Nix for most of my tools. I used nix-darwin on my Mac for a long time, but only to specify Homebrew packages and application settings.

My true conversion happened when I bought my gaming PC, as described in my local LLM post. I initially installed Pop!_OS because I wanted to play games and absolutely wanted to avoid Windows. I got some games to work, but I constantly ran into NVIDIA driver issues that required running random, imperative commands to fix. I felt that was a bad solution because I could never reproduce those debugging steps later. Then I did the dumb thing of updating my NVIDIA drivers, not realizing that imperatively managing driver versions and repositories is a recipe for disaster, and got stuck in a GRUB boot loop. Frustrated, I installed NixOS, hoping its promise of atomic updates would solve this. The result was glorious. I never really believed that everything would be byte-for-byte equivalent until I migrated that system to a new disk. I didn’t clone the drive; I just applied my Nix configuration to a fresh install. It booted, I copied my data, and everything was identical. That was the moment it clicked.

The Friction of Imperative Systems #

Proxmox is fantastic software. It lowered the barrier to entry for me and taught me almost everything I know about virtualization, LXC containers, and ZFS. But fundamentally, Proxmox is built around clicking buttons. It is a GUI-first paradigm. While you can automate it with Terraform or Ansible, it often feels like fighting the tool. State drift is real. You change a setting in the UI to debug something, forget about it, and six months later your “infrastructure as code” is out of sync with reality. For a human maintaining a system manually, this is annoying. But when you introduce AI agents, this becomes a disaster for the operator. An agent running in “YOLO mode” might execute hundreds of imperative commands to fix a problem. It might succeed, but it leaves your system in an undefined, unreproducible state that no one—not even the agent—can fully understand or replicate later.

This friction manifests in hardware management too. On my HP EliteDesk, the Intel I219-LM network card has a known bug where it hangs with hardware offloading enabled. I vaguely remembered fixing this years ago on Proxmox, but I had forgotten the details. When I set up NixOS, I ran into the same issue: the network would randomly drop. This time, however, the fix isn’t a forgotten command run in a root shell history. It is a documented systemd service in my configuration. I added a comment explaining exactly why tso off gso off is needed, citing the forum threads. If I ever reinstall this machine, the fix applies automatically. On Proxmox, I would have had to rediscover this pain all over again.

Another example is my Intel NUC. Since my homelab lives behind my TV, I wondered if I could also use it as a Home Theater PC (HTPC). On Proxmox, this would have required passing the GPU through to a VM to get video output. But doing so would mean the Proxmox host loses access to the GPU entirely, meaning no local console if things go wrong. It was a strict trade-off: either a media player, or a debuggable hypervisor. I tried it, but it was so much trouble that I quickly reverted. With NixOS, I don’t have to choose. The host OS runs Kodi directly, giving me native hardware acceleration and video output. Simultaneously, incus runs in the background, hosting my containers. I get my HTPC and my server on the same metal, without the virtualization tax or the “headless host” limitation.

There is a deeper philosophical difference too. Systems like Proxmox or TrueNAS are designed as appliances. You aren’t supposed to run arbitrary commands on the host; installing packages or tweaking config files is discouraged because you might break the middleware or lose changes on upgrade. You are effectively locked out of your own hardware’s full potential. With NixOS, the host is fully mine. I can mess with it—installing Kodi, tweaking network drivers, running local LLMs—without fear. Because the state is declarative, it is 100% obvious and reproducible. I can break the host configuration and recover to a working state in seconds, even if the machine is running essential services.

The Agentic Multiplier #

I have written before about my shift towards agentic coding. In a world where AI agents execute tasks, CLI-first and declarative systems are king. An AI agent cannot reliably “click buttons” in a web interface to configure a VLAN tag or resize a disk. It needs text. It needs determinism.

By moving to NixOS, my entire infrastructure is defined in text files. This means my AI agents can read, understand, and even safely modify my infrastructure. Proxmox’s opaque database and UI-driven workflow were a black box to my agents. NixOS is an open book. If I want my agent to “deploy a Faster Whisper API server on port 9000 and expose it to the LAN”, it doesn’t need to navigate a “Services” menu, then a “Network” menu, and then a “Firewall” menu. It just writes a systemd service definition and adds networking.firewall.allowedTCPPorts = [ 9000 ]; in the same file. The agent can even verify the change was successful by checking the git diff or the active configuration. This is the infrastructure counterpart to the “agentic coding” revolution I’m living in.

A friend who has used NixOS for many years recently complimented me on how nicely my configuration was structured, especially for managing 8 different machines. The funny thing is, I haven’t written almost a single line of my configuration myself. I did all of it using agentic AI over many sessions. I have restructured and refactored it quite a few times, going from a single machine to 2 machines, and eventually to 9. The AI handles the heavy lifting of refactoring, ensuring that my PC, NUC, and HP all share common modules while keeping their unique personalities.

The Architecture: Incus & Simulation #

I still wish there was a purely “Nix” way to manage persistent, stateful LXC containers and VMs. There are projects like nixos-containers or microvm.nix, but they often lack the operational maturity or live-migration features of a robust hypervisor. Incus (the community fork of LXD) fills this gap perfectly. It gives me the “cattle” management of NixOS for the host, while allowing me to run “pet” legacy workloads (like my old Ubuntu containers or Home Assistant VM) in a stable, manageable environment. Crucially, Incus is entirely controllable via a clean CLI, making it a perfect citizen in my agentic workflow.

Over time, I had already migrated most of my legacy LXC containers—services like DNS or media managers—to declarative Docker Compose files running inside VMs. But I still had one major holdout: Home Assistant OS. I didn’t have a good alternative for it. I assumed I needed a dedicated appliance OS like Proxmox to run it reliably. I didn’t realize that with Incus on NixOS, running a full VM for Home Assistant is trivial. It’s just another QEMU process, but managed with the same ease as a container.

One neat thing I did was create Incus VMs that replicate the exact same configuration as my physical machines. Before I actually switched off my last Proxmox host, I was already confident that the full configuration worked because I had a virtual machine running the same setup. I just have a small file with overrides. Then I could validate that the virtual machine worked. This removed the fear of “nuking and paving” my physical servers.

The Migration #

Migrating was surprisingly straightforward, thanks to vzdump and qemu-img. Here is how I moved my 7+ years of digital history without losing data.

I also kept some detailed migration notes in my dotfiles. These notes should be sufficient for anyone trying to do the same. And if not, giving them to an AI (I used Gemini 3 Pro) will certainly allow you to figure it out.

1. Migrating LXC Containers #

For my LXC containers (running Docker, DNS, etc.), I essentially “teleported” them. I dumped the running container on Proxmox to a standardized tarball, copied it over, and streamed it directly into a fresh Incus container.

On Proxmox:

# Dump container 101 to a compressed archive
vzdump 101 --dumpdir /var/lib/vz/dump --mode suspend --compress zstd

On NixOS (using my migrate-lxc.sh script):

# Stream the backup into a new Incus container
./migrate-lxc.sh vzdump-lxc-101-*.tar.zst ubuntu-container

This script creates a fresh container, mounts its root filesystem, and overwrites it with the Proxmox dump. It handles the messy parts—like mapping UIDs and fixing machine-id—automatically.

2. Migrating Virtual Machines #

For VMs (like Home Assistant OS), it was a matter of converting the disk format. Proxmox uses LVM-thin or ZFS zvols; Incus is happy with QCOW2 or raw ZFS.

On Proxmox:

# Export the ZFS volume to a QCOW2 file
qemu-img convert -p -O qcow2 /dev/zvol/rpool/data/vm-100-disk-0 vm-100.qcow2

On NixOS (using migrate-vm.sh):

# Import the disk and create the VM
./migrate-vm.sh vm-100.qcow2 home-assistant

The Result #

I now manage my entire fleet—Host OS, networking, storage, and hypervisor configuration—from a single git repository. I get the best of both worlds: persistent, stateful services where I need them, and a reproducible, bulletproof host operating system. I can wipe my host machine, reinstall NixOS, run a restore script, and be back online in minutes. Best of all, I never have to remember which checkbox I clicked in a web UI three years ago. It is all in the code. And because it is code, my agents can help me manage it.

References & Configuration #

All configuration files mentioned in this post are available in my dotfiles repository.

flake.nix: The entry point defining my entire fleet (PC, NUC, HP).
networking.nix: The declarative fix for the Intel I219-LM network hang.
kodi.nix: NUC HTPC configuration running Kodi directly on the host.
ai.nix: Declarative service definition for Faster Whisper.
incus-overrides.nix: Overrides used to simulate physical machines inside Incus VMs.
migrate-lxc.sh: Script to import Proxmox vzdump archives into Incus containers.
migrate-vm.sh: Script to import QCOW2 disk images into Incus VMs.
PROXMOX_MIGRATION.md: Detailed raw notes and inventory of the migration.