Nixos | Bas Nijholt

Friend-to-friend ZFS backups, version two

Mon, 28 Sep 2026 00:00:00 +0000

Two years ago I wrote about friend-to-friend backups with TrueNAS. A friend and I store each other’s ZFS snapshots, which is the cheapest off-site backup there is. TrueNAS’s replication tasks wanted root on the receiving system, and neither of us wanted to give the other root on the machine that holds our family photos. So each of us ran a TrueNAS VM on a hypervisor, backed by an iSCSI zvol on the real NAS, and gave the other person root inside that VM instead. Sounds complicated? It was. It worked, but it meant running ZFS on a zvol on ZFS, inside a VM that each of us had to keep updated.

Since then I replaced TrueNAS with NixOS, and my friend plans to move his machines to NixOS too. This time we looked for the simplest solution, built from standard Linux and ZFS features wherever possible. He wanted plain zfs send and zfs receive, with his data still encrypted and his keys never on my server. I agreed, on one condition: neither of us should be able to destroy the other’s datasets.

The result is zfs-tenant, which is now on PyPI and has documentation.

Table of contents

Why ZFS #

OpenZFS is a filesystem and a volume manager in one. You add disks to a pool, and it behaves like one large partition: every dataset you create draws from the same free space, and each can still have its own quota and encryption. ZFS checksums every block it writes, so it notices when a disk hands back something else, and with a mirror or RAIDZ it repairs the block from a good copy. That protection costs disk space: a mirror of two disks holds one disk’s worth of data.

For me, snapshots are one of the coolest parts. ZFS never overwrites data in place, so a snapshot only has to hold on to the blocks that existed at that moment. Taking one is instant, and it costs no space until the data changes. zfs send turns a snapshot, or the difference between two snapshots, into a stream that zfs receive writes into a pool on another machine, usually piped through SSH. It moves blocks, not files: an incremental send contains only the blocks that changed, and ZFS knows which ones those are without visiting a single file. A bajillion one-kilobyte files do not slow it down, because nothing has to look at them one by one. For comparison, restic on my PC needed an hour and a half per run just to check close to a hundred million files for changes.

Two tools from the same project automate this. sanoid takes snapshots on a schedule and prunes them by policy, for example hourly ones for a day and daily ones for a month. syncoid wraps zfs send and zfs receive over SSH: it finds the newest snapshot both sides share and sends everything after it.

What we wanted #

We wrote down the rules before anything else:

He cannot see any of my data, including metadata like dataset names and sizes.
He cannot change or destroy anything of mine.
I cannot read his data, because his keys stay at his house.
He gets one dataset where he can create and remove nested datasets as he likes.
That dataset has a size limit.
No new replication tool on either side: plain zfs send or syncoid.

Until then his NAS runs TrueNAS, so whatever I built also had to work on a machine without NixOS.

ZFS already had most of it #

OpenZFS has a permission system that most people never touch: zfs allow. It delegates individual operations like receive, create, and destroy on one dataset to an unprivileged user. Put a root-owned quota on that dataset and the user cannot store more than you agreed. Raw sends (zfs send -w) ship the encrypted blocks as they are, so the receiving side never needs the key.

On paper, that covers five of the six rules. The gap is visibility. Any local user can run zfs list and see every dataset on the machine, with names, sizes, and properties. And an SSH account for replication is, by default, also a shell.

There are tools that would sidestep some of this. zrepl has a sink mode with a subtree per client, but it replaces sanoid and syncoid on both sides and runs as root on the receiver. restic or borg to a friend’s box would also work, but everything else on my network is ZFS: my machines replicate to the NAS with syncoid, and a restore is zfs send and zfs receive. Adding another backup format means one more thing to understand when something breaks. My current off-site copy goes to Backblaze B2 with rclone. Ideally that would be ZFS as well, and some hosting services accept zfs send, but they cost far more than object storage. A friend with a ZFS box costs nothing.

Rehearsing the permissions in a VM #

Before writing any code, I had an agent build a throwaway NixOS VM test with a real pool, a delegated user, and a list of probes. Each probe tried one thing the friend should not be able to do and recorded what ZFS answered.

The first surprise came from the most obvious setup. With a plain zfs allow -u joe create,receive,destroy,... tank/friends/joe, the delegated user could run zfs destroy -r tank/friends/joe and delete the dataset I had created for him, quota included. Delegated rights apply to the dataset itself as well as everything below it, unless you say otherwise. The fix is to split them: on the root itself only create,mount,receive with zfs allow -l, and the full set only for descendants with zfs allow -d. After that change, destroying, snapshotting, re-delegating, or changing the quota of the root all failed with permission denied.

A send stream can carry properties too, such as a hostile mountpoint=/etc. ZFS applies them with the receiving user’s rights, so in the VM that property failed with permission denied.

The probes also confirmed the gap: the delegated user saw my tank/host dataset in zfs list, and zfs get used tank/host returned its size.

A gate that speaks syncoid #

The shell part has a standard answer. An authorized_keys line with restrict,command="..." runs one fixed program for every login, and passes whatever the client asked for in SSH_ORIGINAL_COMMAND. zfs-tenant gate is that program. It tokenizes the request, accepts only the handful of command shapes a backup needs, checks that every dataset name is inside the friend’s dataset, and runs zfs with an argument list it builds itself. It never starts a shell, so there is nothing to inject into.

The hard part was knowing exactly which commands syncoid sends, so an agent read syncoid’s Perl and logged every remote command it ran. Most are the zfs commands you would expect. One surprise: before each receive, syncoid runs ps -Ao args= on the receiving machine, to check that no other transfer is already writing to the same dataset. That command prints every running process with its full command line. On my NAS, it would show my friend everything I run, so the gate answers with an empty list.

Reading syncoid that closely also turned up a remote code execution (RCE) bug in version 2.3.0, outside zfs-tenant itself. In push mode, syncoid asks the receiving host for a ZFS resume token. It passed that answer unescaped into shell commands for estimating and resuming the send, which run on the source machine. A malicious or compromised backup host could therefore return shell syntax instead of a token and execute arbitrary commands on the sender with the privileges of the syncoid process.

I reproduced this in two isolated NixOS VMs using real ZFS and SSH: a harmless command embedded in the receiver’s answer created a marker file on the sender. I reported it to the maintainer and opened sanoid PR #1114 with a small fix that shell-quotes the token at both uses. Even with that fix, the sending side should run syncoid as an unprivileged user that may only send and hold. nixpkgs’ services.syncoid already works that way, so my friend needs nothing from zfs-tenant at all.

Zones: a second wall in the kernel #

At that point the gate was the only thing hiding my datasets. Then my friend sent me a message: ZFS has a feature called zones, which restricts a dataset tree to a Linux user namespace.

zfs zone attaches a dataset to one user namespace, and inside that namespace the ZFS kernel module answers dataset does not exist for everything that is not attached. Even if my gate had a bug that let arbitrary zfs commands through, tank/host would stay invisible. The zone does not replace the gate, though: it only filters ZFS, and it only applies once the gate has put the SSH session inside it. Who stops what shows how the two divide the work.

A user namespace can give its processes different user IDs than they have outside. Containers usually use that to make the user root inside the namespace, while it stays an ordinary user outside. ZFS treats root inside a zone as the zone’s administrator, allowed to do anything to the attached datasets, whatever zfs allow says. In the VM test, that let the friend destroy the dataset I had created for him. So zfs-tenant keeps him as his own ordinary user inside the namespace: he has no special rights there, zfs allow decides what he may do, and the kernel still hides everything else.

Root inside a user namespace also turned up in a real OpenZFS bug in August: before 2.4.4, 2.3.9, and 2.2.11, several pool operations, such as destroying a pool, accepted it as host root (CVE-2026-79619). Joe cannot reach that through the gate, but zfs-tenant needs unprivileged user namespaces, so a host running it should load a patched module: cat /sys/module/zfs/version shows the loaded one, which only changes after a reboot.

A zone needs a running namespace to attach to, so zfs-tenant runs a small service per friend that holds one. The gate joins that namespace before it looks at the command, drops the capabilities that joining grants, and refuses to run if the service is down.

Who stops what #

When Joe and I talked it through, I found it hard to say clearly which part protects against what, so I made this diagram.

Each column answers one question: who can connect, what he can run, what ZFS shows him, and what he can change. For the network, our tailnet only lets his router reach port 22 on my NAS, and from= in authorized_keys only accepts his key from that address. Raw sends then keep whatever reaches my pool unreadable to me.

The gate and the zone overlap in one job: hiding my other datasets. The gate does it first, by refusing any name outside his root before zfs runs. The zone repeats that check in the kernel, so hiding my datasets would take a bug in both. Everything else the gate does, the zone cannot:

Without the gate, his key would be a normal account on my NAS. The zone only filters ZFS; it would not stop him from reading /etc or running programs.
An SSH login does not start inside a user namespace. The gate joins the zone before it runs anything, so without the gate the zone would never apply.
The zone leaves some things visible: zpool status still shows my disks and how full the pool is, and the datasets above his root still show their sizes. User namespaces do not hide processes either, hence the empty answer to ps.
It deletes any unencrypted dataset he pushes by mistake, so I never keep his plaintext.
It logs every command, which is how I saw Joe try bash below.

The security model in the documentation lists every promise with the mechanism behind it.

What it cannot hide #

ZFS encryption protects file contents, not structure. In the words of zfs-load-key(8), ZFS “will not encrypt metadata related to the pool structure, including dataset and snapshot names, dataset hierarchy, properties, file size, file holes, and deduplication tables.” So I can see the names, sizes, and snapshot times of whatever my friend sends me. Boring dataset names help, and sending without -p keeps properties out of the stream.

I can also always delete his copy, because I am root on my own machine. What I can never do is read it. My monthly scrubs still verify his data without his key, because ZFS checksums the encrypted blocks.

What we protect against is a dead machine: a failed pool, a fire, a flood. His key may destroy anything below his root, which is what lets syncoid mirror his snapshot retention, so someone who steals that key can also delete his backups on my NAS. Covering that would take holds that I place as root and release on a schedule. We left them out on purpose, although I am reconsidering.

Setting it up #

On a NixOS host, giving a friend space takes this:

services.zfs-tenant = {
 enable = true;
 tenants.joe = {
 dataset = "tank/friends/joe";
 quota = "2T";
 authorizedKeys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... joe-nas" ];
 allowedFrom = [ "100.64.0.12" ]; # his tailnet address
 };
};

That creates the user, pins the key to the gate, applies the dataset, properties, and delegation on every boot, and runs the zone service. My own setup is in friend-backups.nix, with his real key and address. If your sanoid snapshots the whole pool, exclude the tenant tree: your sanoid must neither snapshot nor prune it, or his next incremental push finds snapshots on the target that he never sent. Once my friend is on NixOS, he can push with nixpkgs’ own module:

programs.ssh.knownHosts.bas-nas.publicKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA...";

services.syncoid = {
 enable = true;
 sshKey = "/var/lib/syncoid/id_ed25519";
 localSourceAllow = [ "send" "hold" ];
 commonArgs = [
 "--no-sync-snap"
 "--compress=none"
 "--delete-target-snapshots"
 "--sshoption=StrictHostKeyChecking=yes"
 ];
 commands."tank/offsite" = {
 target = "zfs-tenant-joe@bas-nas:tank/friends/joe/offsite";
 recursive = true;
 sendOptions = "w";
 };
};

My NAS will push to his with the same configuration once he hosts a root for me.

For his TrueNAS box, every release has a single zfs-tenant.pyz that runs with the Python TrueNAS already ships, and the getting started guide covers both setups. Hosting on TrueNAS has only run in VMs so far (issue #4). Sending from TrueNAS is where its built-in tools stop working: replication tasks wrap every remote command in sh -c, which the gate refuses, so he pushes with zfs send or syncoid instead.

Knowing when it stops working #

The last btrfs machine taught me a fifth question for any backup: how do I find out when it stops working? A failed push shows up in systemctl status on the sending side, which nobody reads. The check belongs on the receiving side, and it has to look at every pushed dataset separately, because one healthy dataset can hide another that stopped replicating. That is how I watch my own machines: the NAS checks the newest snapshot of every dataset it replicates every hour and alerts my phone when one is too old. zfs-tenant does not alert yet; issue #3 tracks it.

Restoring #

The gate also allows zfs send of his own snapshots, so getting data back is one pipe, run at his house:

ssh zfs-tenant-joe@bas-nas zfs send -w tank/friends/joe/offsite/photos@autosnap_2026-09-25_00:00:01_daily \
 | zfs receive -u tank/photos
zfs load-key tank/photos
zfs mount tank/photos

My photos are a child dataset that inherits its key from an encryption root, and I only push the child. If my NAS dies and I pull the child back, can I still unlock it without the root? A throwaway VM says yes: a raw send carries the dataset’s own wrapped key, so the child arrives as its own encryption root and opens with the same passphrase. The one catch: after a zfs change-key, the next push carries the new wrapping, so the passphrase that matters is the one at the time of the last push, and it has to live somewhere other than the NAS. Mine does: zfs-unlock keeps the passphrases on a separate device and unlocks the NAS after every boot.

The first real push #

Once my side was deployed, Joe tried it from his TrueNAS box. His first move was to see what else his key could do. The gate logged every attempt on my NAS:

root=tank/friends/joe denied interactive session
root=tank/friends/joe denied 'ls': only zfs commands and syncoid's probes are allowed
root=tank/friends/joe denied '/usr/bin/bash': only zfs commands and syncoid's probes are allowed
root=tank/friends/joe denied '/bin/bash': only zfs commands and syncoid's probes are allowed
root=tank/friends/joe allowed 'zfs receive -s -u tank/friends/joe/zt-test'

The last line is a small encrypted test dataset, sent with zfs send -w piped into ssh, without syncoid. It arrived with keystatus set to unavailable. Then I tried to read it as root on my own machine. zfs mount answered cannot mount 'tank/friends/joe/zt-test': dataset is exported to a local zone, and zfs load-key with a guessed passphrase gave Key load error: Incorrect key provided. The only readable text in the raw stream was ZFS property names and the snapshot name, which is the metadata ZFS leaves unencrypted, as described above.

Tested like everything else #

Every security claim in this post is also checked by a VM test: two NixOS machines back each other up, the way my friend and I do, with real sanoid and syncoid 2.3.0, through real sshd and the gate, into real OpenZFS 2.4.4. It also checks what happens over time, such as snapshot retention and both machines rebooting, and it runs on every push in GitHub Actions.

The route to the remote code execution bug in syncoid was slightly absurd. I built the initial version of zfs-tenant through coding agents, literally from the bathtub, and then asked several frontier models to review the security boundary independently. One of them followed receiver-controlled data beyond my new code and into syncoid. A model’s suspicion is not evidence, so I turned it into the two-VM reproduction above: it failed on unpatched syncoid 2.3.0 and passed after the seven-line fix. The most consequential result of reviewing my small new project was a command-injection bug in the mature tool next to it.

The whole thing is about 930 lines of Python, not counting comments and docstrings, with no dependencies. The kernel does the actual enforcing: delegation, the quota, and the zone. The gate only removes the shell, and it is small enough to read in one sitting.

AI scope creep or useful? #

The first version had grace holds: every day, the host placed a hold on the newest snapshot of each of my friend’s datasets and released it 14 days later. ZFS will not destroy a held snapshot until the hold is released, and only the host can release it. The agent built it without me asking for it, so I removed it right away as scope creep.

Then every model I asked to review the project suggested them again, each in its own context window: Opus 5.5, GPT-6 Astra, and even the one that reviewed a draft of this post. Joe and I had set out to protect against a machine dying (a fire or a flood), not against one of our own machines being hacked. But I am tempted to add the holds back. They cover the case where malware or an attacker gets onto my machine and uses my key to delete my backups on his NAS, or to replace them with garbage. Adding the feature without asking was still wrong; the feature itself might not be.

References #

My declarative multi-machine homelab, explained

Mon, 28 Sep 2026 00:00:00 +0000

I open https://mealie.lab.nijho.lt on my phone to look up a recipe. It works at home, on a train, or on hotel Wi-Fi in another country, over an encrypted HTTPS connection. If a stranger on the internet tries the same address, they get nothing. Nearly everything I run works this way: about 150 containers on four machines, used every day by seven people besides me. How that works is the thing friends actually ask me about, and it is what my 2024 homelab post never explained.

That post was a story about hardware: a NUC, an HP EliteDesk, a TrueNAS box, and a lot of trial and error. Almost everything in it has since been replaced. Proxmox and TrueNAS are gone, and every machine runs NixOS, including the NAS.

The part I think is the coolest is the balance I found: everything is declarative, but with as little machinery as possible. Declarative means I describe the end result in text files, and a tool makes reality match it. One extreme is what I had before, clicking through web UIs and running one-off install scripts. The other is Kubernetes, which many self-hosted projects don’t support and which is a lot to babysit at home, or running every app as a NixOS module, which often lags behind upstream.¹ I landed in between: NixOS declares the machines, each project’s own Compose file declares its app, and compose-farm, a thin tool I wrote, decides which machine runs what, which is all the multi-host orchestration I need. Every layer is a text file in git, each uses the simplest tool that keeps it that way, and I still get new app releases as soon as upstream ships them. The price is that nothing fails over automatically, but because every machine sees the same files and data, moving services to another machine is a one-line change and one command.

It took me several years and many iterations to get here. This is the first version I would recommend to friends, because its declarative configuration makes changes auditable, reproducible, and reversible. That is what makes me comfortable managing it with AI: I can review the diff, rebuild the same configuration, and revert a change if needed.

This post tries to explain it all. I wrote it for people who have never set up a reverse proxy or a VPN, so every piece gets a short explanation before I show how I configured it. I deliberately made it comprehensive, with enough detail to build something similar yourself. That also makes it long, so read the parts you find interesting and skip the rest.

If it is too long, send it to your AI agent, discuss it, and figure out together which parts make sense for your own network.

If you have a single machine that already runs NixOS, you probably don’t need Docker or compose-farm at all: adding your services as NixOS modules is simpler, and that is what I recommend to friends in that situation. If you would rather use Docker, which is usually the path a project officially supports, I recommend compose-farm: it works just as well on a single host and lets you fan out later. The multi-machine part is where my setup really pays off, but the rest of this post, from the reverse proxy to the VPNs, is just as useful on one machine.

Throughout the post I use a few services as running examples: Mealie for recipes, ntfy for push notifications, and my git server at git.nijho.lt.

Table of contents

The big picture #

To reach my services away from home, I use a VPN (virtual private network): an encrypted connection that lets my devices communicate across the internet as part of a private network. Three projects fit together here:

WireGuard provides the encrypted tunnels. I use it directly between my devices and my home router.
Tailscale builds a private network using WireGuard tunnels. Its apps handle finding and connecting to the other devices.
Headscale replaces the coordination server that Tailscale normally hosts. I run that server myself, while my devices run the normal Tailscale apps. Headscale distributes device information and access rules; the apps carry the traffic.

That gives me two VPN options: WireGuard to my router for access to my home network, or Tailscale coordinated by Headscale for sharing selected services and connecting from networks that block direct WireGuard. Both use WireGuard encryption.

This is the whole setup in one diagram. Every box gets its own section below.

flowchart LR home["At home
(home network)"] wg["Away
(WireGuard)"] ts["Away
(Tailscale)"] friends["Friends and family
(Tailscale)"] net["Anyone
(internet)"] subgraph router["Home router"] wgs["WireGuard server"] fwd["Port 443 forward"] end home --> traefik wg --> wgs --> traefik ts -- "encrypted tunnel
straight to the NAS" --> traefik net --> fwd -- "wan entrypoint:
public routes only" --> traefik traefik{{"Traefik
reverse proxy
+ IP allowlist"}} traefik --> nas["Containers on nas"] traefik --> nuc["Containers on nuc"] traefik --> hp["Containers on hp"] traefik --> pc["Containers on pc"] friends -. "ACL: only the ports
shared with them" .-> nas friends -.-> pc

In short:

Four machines run all my services as containers.
compose-farm decides which service runs on which machine.
One reverse proxy, Traefik, is the front door for every service on every machine.
Let’s Encrypt certificates give every service real HTTPS, including the private ones.
DNS turns names like mealie.lab.nijho.lt into addresses, and gives a different answer depending on where I am.
There are four ways in: my home network, WireGuard on my router, Tailscale coordinated by my own Headscale server, and the open internet.
One allowlist in Traefik limits every service to the first three by default. A service reaches the internet only if its config asks for it.
Headscale ACLs decide which of my friends and family can reach which service.
NixOS, Compose files, and Terraform describe nearly all of it in git, so I rarely click a button, and AI agents can work on it the same way I do.

If that list looks overwhelming, I get it. A reverse proxy, Let’s Encrypt, DNS, an allowlist, ACLs, NixOS, compose-farm, Terraform: that is a lot of moving parts for something that serves recipes. What lets me sleep at night is the last point (9). Apart from a handful of router settings, every piece of configuration is declarative and lives in git, so the whole setup is reproducible. If I break something, or more likely, an AI agent breaks something, git log tells me what changed and git revert undoes it. And the NAS takes a ZFS snapshot of all app data every 10 minutes, so even if something goes badly wrong, I lose at most ten minutes of data. You don’t need to understand every piece at once, either; each one is a file you can read when you get to it. I come back to this in Declarative everything.

To make the list concrete, this is what happens when I open Mealie in three situations:

At home, my phone asks my home DNS server for mealie.lab.nijho.lt and gets 192.168.1.6, the NAS. Traefik sees a request from 192.168.1.x, which is on the allowlist, and passes it to the Mealie container.
On hotel Wi-Fi that blocks WireGuard, my laptop uses Tailscale instead. It asks Headscale’s DNS for the same name and gets 100.64.0.28, the address of the NAS inside my Tailscale network. The request travels through an encrypted tunnel straight to the NAS. Traefik sees a 100.64.0.x address, also on the allowlist, and passes it on.
A stranger on the internet gets 192.168.1.6 from public DNS, which is a private address that leads nowhere outside my home. If they find my home IP and connect to it directly, the router hands them to Traefik’s separate door for the internet, where Mealie doesn’t exist, and they get 404 Not Found.

Same name, HTTPS on all three paths, and only the stranger is refused.

The machines, and NixOS #

Four machines run my services:

Machine	What it is	What it does
`nas`	The physical NAS with all the disks	Traefik and most services run in an Incus system container, close to the data
`nuc`	A small, always-on Intel NUC	Home DNS server and a few lightweight services
`hp`	An HP EliteDesk	Second home DNS server and a mix of services
`pc`	My desktop with two RTX 3090s	Local AI, including my dictation server

Each name links to that machine’s NixOS configuration. The hardware of the NUC, the HP, and the NAS is in my original homelab post.

That system container is called docker-lxc. It holds my Docker stack files and runs my Docker containers. In compose-farm, I call it nas, so deployment commands using that name target the container on the physical NAS.

Why the NAS runs its containers inside another container (click to unfold)

Incus uses LXC to run system containers. This keeps Docker and its services separate from the NAS host that manages my disks. That container is a full NixOS system of its own, configured like the other machines.

What NixOS brings #

All four physical machines run NixOS, and so does the docker-lxc container. In NixOS, the entire operating system is described in text files: which packages are installed, which services run, which disks get mounted, which firewall ports are open. You don’t install things by typing commands and hoping you remember them later. You write down what the machine should look like, and NixOS makes it so. If a machine dies, I install NixOS on a new one, point it at the same files, and get the same machine back.

My NixOS configuration lives in my public dotfiles, in configs/nixos. A shared layer applies to every machine, and each machine adds its own specifics. Every machine joins my Tailscale network with one line in common/services.nix:

services.tailscale.enable = true;

The NUC, the HP, and the PC mount the same shared folders from the NAS over NFS, a protocol for sharing folders over the network. That shared folder, /opt/stacks, is what makes the next section work: every machine sees the same service definitions at the same path.

The NFS mount in NixOS (click to unfold)

From optional/nfs-docker.nix:

fileSystems."/opt/stacks" = {
 device = "truenas.local:/mnt/ssd/docker/stacks";
 fsType = "nfs";
 options = [ "nfsvers=4" "nofail" "bg" "soft" ];
};

docker-lxc doesn’t need NFS; the NAS hands it the same folders directly from its disks (hosts/nas/virtualization.nix).

Deploying NixOS with comin #

Changing a NixOS machine normally means logging in and running nixos-rebuild switch. I don’t do that anymore. Every machine runs comin, a small GitOps agent. It watches my dotfiles on GitHub, and when a new commit lands on main, each machine pulls it, builds its own configuration, and switches to it. The heavy builds come from my local build cache, so the machines themselves rarely compile anything. comin only deploys commits signed with my SSH key, so being able to push to the repo is not enough to take over my machines. comin originally only understood GPG signatures, so I added support for SSH-signed commits upstream.

The comin config (click to unfold)

The core of common/comin.nix:

services.comin = {
 enable = true;
 remotes = [{
 name = "origin";
 url = "https://github.com/basnijholt/dotfiles.git";
 branches.main.name = "main";
 }];
 repositorySubdir = "configs/nixos";
 sshAllowedSignersPath = "${cominAllowedSigners}";
};

The last line points comin at the list of keys whose signatures it accepts.

Changing a machine means pushing a signed commit. That comes back at the end of this post.

What runs on it #

As I write this, the four machines run 138 containers. A small selection:

Area	Services
Infrastructure	Traefik (reverse proxy), Headscale and Headplane (my tailnet), Forgejo (git), the compose-farm web UI, Homepage (dashboard), code-server (VS Code in the browser)
Monitoring	Uptime Kuma, Glances, Netdata, Prometheus and Grafana, Dozzle (container logs), Diun and What’s Up Docker (image updates), LibreSpeed
Tools	ntfy (push notifications), Syncthing (file sync), Atuin (shell history sync), Wakapi (coding stats)
AI	agent-cli and Diction (dictation), Speaches (speech-to-text), Kokoro (text-to-speech), Ollama, LiteLLM (one API in front of all models), Open WebUI, LibreChat, LobeChat, Khoj, SearXNG (private metasearch), Unsloth (fine-tuning)
Home and documents	Mealie (recipes), Grocy (household inventory), Paperless-ngx with paperless-ai and paperless-gpt, Immich (photos), Nextcloud, Hoarder (bookmarks), Home Assistant
Chat	Cinny (Matrix client for MindRoom), The Lounge (IRC)

The GPU-heavy AI services, like dictation, text-to-speech, and fine-tuning, run on pc; the local AI post covers that side. Home Assistant runs as a virtual machine in Incus on the HP; everything else in the table is a container.

Containers and compose-farm #

What a container is #

Almost everything I run is a Docker container. A container is an application packaged with everything it needs, isolated from the rest of the system. Mealie’s developers publish a Mealie image, I run it, it doesn’t care what else is installed on the machine, and removing it leaves nothing behind.

Docker Compose describes containers in a YAML file: which image, which folders it can see, which ports it uses, which settings. This is a trimmed version of my Mealie definition:

services:
 mealie:
 image: ghcr.io/mealie-recipes/mealie
 container_name: mealie
 ports:
 - 9925:9000
 volumes:
 - /mnt/data/mealie:/app/data/
 environment:
 ALLOW_SIGNUP: false
 BASE_URL: https://mealie.${DOMAIN}
 restart: unless-stopped

A port is a number that tells apart the network services on one machine, so one address can serve many apps. Under ports, 9925:9000 means that port 9925 on the machine leads to port 9000 inside the container, where Mealie listens.

Every service (“stack”) gets its own folder with a compose.yaml in one git repo, mounted at /opt/stacks on every machine.

Upstream’s Compose file, my folders #

Back when I ran Proxmox, I was a big fan of the Proxmox VE Helper-Scripts, started by tteck (RIP) and now maintained by the community. Each one is a one-liner that creates an LXC container with an app installed inside, and they are how I got into self-hosting and set up almost everything at first.

The catch is that they are a community effort, and the way they install an app is usually not a way the app’s own developers support. Immich shows the problem well: its docs say it requires Docker with the Docker Compose plugin, while today’s helper script builds Immich from source, installs PostgreSQL and Redis, and compiles six image-processing libraries directly in the container. That is effectively a mirror of the official setup, maintained by someone else; the script even pins the Immich version and only bumps it after testing each release. Every upgrade depends on a second set of maintainers keeping up with upstream, and for my containers that made upgrades painful and stressful.

Nowadays almost every project publishes a Docker Compose file, so that’s what I use, nearly unmodified. After the first setup, the Compose file almost never changes; the image does. Most of my services use the latest tag, and upgrading is cf update <stack> (or --all), which pulls the newest image and recreates the container.

That is YOLOing it a bit. In the beginning I pinned every image to an exact digest, because I was afraid a new version would break something. Bumping all those pins became its own chore, so I switched almost everything to latest. I only still pin the pieces where a surprise would really hurt: Traefik, Headscale, Forgejo, and databases, which stay on a major version like postgres:16 because moving to a new major version of PostgreSQL needs a manual migration. Since the switch I haven’t had real problems, and the rare breakage took a five-word prompt to an AI agent to fix.

The one thing I do change is where the data goes. Many Compose files keep data in named Docker volumes, which Docker manages somewhere under /var/lib/docker/volumes. I stopped using them: too many times I deleted a volume by accident (docker compose down -v and docker volume prune make that easy), and I never had a good view of what was inside. Instead, every stack gets plain folders, which Docker calls bind mounts, in two fixed places:

/opt/stacks/<stack>/ holds the compose.yaml and its .env, in git.
/mnt/data/<stack>/ holds the app’s data.

Each /mnt/data/<stack> is its own ZFS dataset on the NAS, with frequent snapshots and backups. The Mealie example above shows the swap: upstream’s file uses a volume called mealie-data, and mine says /mnt/data/mealie:/app/data/.

Why I wrote compose-farm #

Docker Compose manages containers on one machine, and I have four. The usual answers are Kubernetes or Docker Swarm.

Kubernetes feels like overkill for a homelab, and many projects don’t support it: they publish a Compose file, not a Helm chart (Kubernetes’ package format), so I would end up writing and maintaining my own charts. That’s the helper-script problem again.

I also experimented with Docker Swarm, writing a NixOS configuration for a three-manager cluster. Docker Swarm would meet my needs almost perfectly, but it is effectively in maintenance mode: it still ships with Docker, but Docker no longer invests in it.

So I wrote compose-farm. It is deliberately thin: one YAML file says which stack runs on which machine, and the cf command SSHes to that machine and runs plain docker compose there.

compose_dir: /opt/stacks

hosts:
 nas: { address: 192.168.1.6 }
 nuc: { address: 192.168.1.2 }
 hp: { address: 192.168.1.3 }
 pc: { address: 192.168.1.5 }

stacks:
 traefik: nas
 headscale: nas
 forgejo: nas
 mealie: nas
 ntfy: nuc
 agent-cli: pc
 glances: all  # runs on every machine

traefik_file: /opt/stacks/traefik/dynamic.d/compose-farm.yml

Day to day, that looks like this:

cf up mealie # start Mealie on whichever machine the config says
cf logs -f ntfy # follow logs, wherever ntfy runs
cf ps # what is running where
cf apply # make reality match the config: start, move, and stop stacks

Moving a service to another machine is a one-line change. I edit mealie: nas to mealie: hp and run cf up mealie, and compose-farm stops it on the old machine and starts it on the new one. Because every machine sees the same /opt/stacks and /mnt/data, the service finds its configuration and data waiting for it.

That is also my answer to “what if a machine dies?” There is no automatic failover: nothing notices and reschedules the services for me, the way Kubernetes would. But because no stack is tied to a machine, recovering by hand takes a minute. The NAS is the exception: it holds the data and runs Traefik, so if it dies, nothing else can take over.

Moving a dead machine’s services by hand (click to unfold)

If the NUC died, I would point its name at the HP:

hosts:
 nas: { address: 192.168.1.6 }
 nuc: { address: 192.168.1.3 } # was 192.168.1.2; the NUC is dead, so the HP takes over
 hp: { address: 192.168.1.3 }
 pc: { address: 192.168.1.5 }

Then cf up --host nuc starts every NUC stack on the HP and updates Traefik’s routes, and I can clean up the names later. (Not cf apply: while two names point at the same machine, it sees every stack there running twice and stops the “extra” copy.) This works for the NUC, the HP, and the PC, as long as the published ports don’t clash and a service doesn’t need the PC’s GPUs.

There is no cluster, no database, and no agent running on each machine. If compose-farm disappeared tomorrow, every stack would still be a normal Compose folder I could start by hand.

The last line of that config, traefik_file, is where compose-farm connects to the front door.

The front door: a reverse proxy #

What a reverse proxy brings #

Every service listens on its own port on its own machine: Mealie on port 9925 of the NAS, ntfy on port 8089 of the NUC, and so on. I could bookmark http://192.168.1.2:8089, but that gets annoying fast, it is unencrypted, and every service would need its own security.

A reverse proxy is a receptionist in front of all of them. Every request arrives there first. It looks at which name you asked for, mealie.lab.nijho.lt or ntfy.lab.nijho.lt, and passes the request to the right service on the right machine. Because everything goes through it, the reverse proxy is also the one place to:

handle HTTPS and certificates, so the services don’t have to;
decide who is allowed in;
give every service a clean name instead of an IP and a port.

I use Traefik. It runs on the NAS and is the only thing listening on the standard web ports: 80 for plain HTTP and 443 for HTTPS.

Services announce themselves #

What I like most about Traefik is that services describe their own routing with labels in their Compose file. These are Mealie’s:

 labels:
 - traefik.enable=true
 # https://mealie.lab.nijho.lt, only for trusted networks
 - traefik.http.routers.mealie.rule=Host(`mealie.${DOMAIN}`)
 # http://mealie.local, plain HTTP at home
 - traefik.http.routers.mealie-local.rule=Host(`mealie.local`)
 - traefik.http.routers.mealie-local.entrypoints=web
 - traefik.http.services.mealie.loadbalancer.server.port=9000

In words: if someone asks for mealie.lab.nijho.lt or mealie.local, send them to port 9000 of this container. Every service gets two names by convention: an HTTPS name under lab.nijho.lt that works everywhere, and a short .local name for quick access at home. The HTTPS route doesn’t mention HTTPS, a certificate, or who is allowed in; Traefik’s defaults cover all three.

Traefik reads those labels by watching Docker on its own machine, and ignores any container that doesn’t opt in with traefik.enable=true. Its own config sets the defaults:

command:
 - --providers.docker=true
 - --providers.docker.exposedbydefault=false
 - --providers.file.directory=/dynamic.d
 - --entrypoints.web.address=:80
 - --entrypoints.web.http.middlewares=local-ips-only@file
 - --entrypoints.websecure.address=:443
 - --entrypoints.websecure.asDefault=true
 - --entrypoints.websecure.http.middlewares=local-ips-only@file
 - --entrypoints.websecure.http.tls.certresolver=le
 - --entrypoints.wan.address=:4443
 - --entrypoints.wan.http.tls.certresolver=le

An entrypoint is a port Traefik listens on. web and websecure are for home, WireGuard, and Tailscale, and a route that doesn’t name an entrypoint lands on websecure. local-ips-only is the allowlist, more on that in Keeping it safe. wan is for the internet, more on that in The internet.

Services on other machines #

There is a catch: Traefik only sees Docker containers on its own machine. It has no idea that ntfy runs on the NUC.

compose-farm fills that gap. It reads the Traefik labels of every stack, looks up which machine the stack runs on and which port it publishes, and writes the result to a file that Traefik watches. After cf up ntfy, that file contains:

# /opt/stacks/traefik/dynamic.d/compose-farm.yml
# Auto-generated by compose-farm (only the ntfy entries shown)
http:
 routers:
 ntfy:
 rule: Host(`ntfy.lab.nijho.lt`)
 service: ntfy
 services:
 ntfy:
 loadbalancer:
 servers:
 - url: http://192.168.1.2:8089

Note the ports: on its own machine, Traefik talks to a container directly on its internal port (9000 for Mealie), but a route to another machine has to use the port that machine publishes (8089 for ntfy). I never edit that file. I write labels the same way no matter where a service runs, and when a service moves, its route moves with it.

Things that don’t run in Docker, like Home Assistant in its own virtual machine, get hand-written routes in a second file in the same folder.

HTTPS and certificates #

What a certificate is #

When you visit an https:// address, your browser checks the server’s certificate: a small file, signed by an authority your browser trusts, that proves the server really is mealie.lab.nijho.lt. The certificate is also what makes the encryption between you and the server possible. Without a valid one you get a scary warning, and many apps refuse to connect at all.

A private service with a real certificate

Let’s Encrypt hands out certificates for free, automatically, to anyone who can prove they control a domain.

Proving I own the domain without opening my network #

The usual proof is for Let’s Encrypt to connect to your server and fetch a file from it (the “HTTP-01 challenge”). That doesn’t work for private services, because the whole point is that the internet can’t connect to them.

So I use the DNS-01 challenge instead. Let’s Encrypt asks me to publish a specific text record in my domain’s DNS, the internet’s phone book that turns names into addresses (more on that below). My DNS is hosted at Cloudflare, so Traefik creates that record through the Cloudflare API, waits for Let’s Encrypt to see it, and removes it again. Let’s Encrypt never has to reach my network.

DNS-01 also lets me request a wildcard certificate: one certificate for *.lab.nijho.lt that covers every name under it. A new service has HTTPS the moment its route exists, without requesting anything.

 - --certificatesresolvers.le.acme.dnsChallenge.provider=cloudflare

 - traefik.http.routers.traefik-lab.tls.certresolver=le
 - traefik.http.routers.traefik-lab.tls.domains[0].main=lab.nijho.lt
 - traefik.http.routers.traefik-lab.tls.domains[0].sans=*.lab.nijho.lt

Traefik renews it before it expires, and I don’t think about certificates anymore.

The .local names can’t get a public certificate, because nobody owns .local. Those stay plain HTTP, and Traefik only answers them for devices on my home network. It is also why I don’t redirect HTTP to HTTPS everywhere: that would break every .local name.

DNS: the phone book #

What DNS does #

Computers talk to each other with IP addresses like 192.168.1.6. Humans prefer names like mealie.lab.nijho.lt. DNS, the Domain Name System, is the phone book that turns one into the other. Before your browser connects anywhere, it asks a DNS server for the address that belongs to a name.

Two details matter for the rest of this post.

First, some addresses are private. Ranges like 192.168.x.x and 10.x.x.x, and Tailscale’s 100.64.x.x, are not reachable on the public internet; they only mean something inside one particular network. My NAS is 192.168.1.6 inside my home, and so is some random device in thousands of other homes.

Second, the answer depends on which DNS server you ask. My setup relies on that: the same name gets a different answer depending on where I am.

Three answers for the same name #

Who is asking	Which DNS server	Answer for `mealie.lab.nijho.lt`
Anyone on the internet	Public DNS at Cloudflare	`192.168.1.6`
A device at home or on WireGuard	My own DNS servers on `nuc` and `hp`	`192.168.1.6`
A device on my Tailscale network	Headscale’s built-in DNS	`100.64.0.28`

Public DNS has one wildcard record: *.lab.nijho.lt points to 192.168.1.6. Yes, that is a private address in public DNS. It sounds odd, but it works nicely: at home, or on a tunnel that routes my home network, the public answer is simply correct. For everyone else it leads nowhere, and it reveals nothing beyond the fact that I have a machine at 192.168.1.6, like half the internet. That is convenience, not security; more on that below. (Some routers with “DNS rebinding protection” refuse to pass on answers like this. Mine doesn’t.)

I don’t click those records together in Cloudflare’s dashboard. All my public DNS is Terraform: a text file that lists every record, and a tool that makes Cloudflare match it. These are the lab records:

resource "cloudflare_dns_record" "lab_wildcard" {
 zone_id = var.cloudflare_zone_id
 name = "*.lab.nijho.lt"
 content = "192.168.1.6"
 type = "A"
 ttl = 10800
 proxied = false # Private IP: Cloudflare can't proxy to it
}

terraform plan shows exactly which records will be added, changed, or removed before anything happens, and terraform apply makes the change. The Terraform files live in a private repo, next to my other secrets, because they need a Cloudflare API token.

My home DNS servers run CoreDNS on the NUC and the HP, declared in optional/coredns.nix in the same NixOS repo as everything else. They answer .local and lab.nijho.lt names themselves and forward everything else to public DNS:

labNijholtZone = pkgs.writeText "lab.nijho.lt.zone" ''
 $ORIGIN lab.nijho.lt.
 @ 3600 IN A 192.168.1.6
 * 3600 IN A 192.168.1.6
'';

Without them, .local names wouldn’t work at all, since public DNS knows nothing about them.

Strictly speaking, .local is reserved for multicast DNS, the protocol printers and AirPlay speakers use to announce themselves, and Apple warns that its devices might not resolve .local names served by a regular DNS server. I know, and I have rarely run into it. The official name for home networks is home.arpa, but some browsers treat a name like example.home.arpa as a search term: Chrome on my phone searched Google instead of opening the site. I had to type the full address, https://example.home.arpa, every time. The .local names are a shortcut anyway; the lab.nijho.lt names are the ones that work everywhere. They also keep lab names working when my internet connection is down. There are two so that one can reboot without taking name resolution with it.

Headscale’s DNS is the odd one out, and it needs Tailscale explained first.

Four ways in #

There are four ways a request can reach Traefik:

Way in	When I use it	Address Traefik sees
Home network	At home, on Wi-Fi or a cable	`192.168.1.x`
WireGuard on the router	My phone, laptop, and travel router when I’m away	`10.6.0.x`
Tailscale, via Headscale	Friends and family, cloud VMs, and my own devices when WireGuard can’t get through	`100.64.0.x`
The internet	Anyone, for the few public services	Any public address

1. The home network #

At home my devices are on 192.168.1.x, the home DNS servers answer, and everything just works.

2. WireGuard on the router #

A VPN, a virtual private network, creates an encrypted tunnel from your device to another network, so your device behaves as if it were plugged in there. WireGuard is a modern VPN protocol: small, fast, and based on key pairs instead of usernames and passwords.

My router, an ASUS ZenWiFi XT8, has a WireGuard server built in. When I’m away, my phone connects to it and gets an address in 10.6.0.x, and from then on it is effectively at home. It can reach 192.168.1.6, and it uses my home DNS servers. The WireGuard iOS app has “On-Demand” rules, so the tunnel turns on by itself whenever I leave my home Wi-Fi. I wrote more about that in my post about coding from my phone.

To find my router, the phone needs my home’s public IP address. My internet provider can change that address whenever it likes, so the router also runs dynamic DNS (DDNS): it keeps a public hostname pointed at whatever my current IP is. The WireGuard client connects to that hostname.

3. Tailscale, via my own Headscale #

Tailscale is a VPN built on WireGuard, but organized differently. Instead of every device tunneling to one central point, Tailscale builds a mesh: every device can make a direct, encrypted WireGuard connection to every other device. It gets through firewalls and home routers without opening any ports, and when a direct connection is impossible, it falls back to encrypted relay servers. Every device gets a stable address in 100.64.x.x, no matter which network it is on. All those devices together are called a tailnet.

A mesh like that needs something that knows which devices exist, what their keys are, and who may talk to whom. That is the coordination server, which Tailscale normally runs for you. Headscale is an open-source coordination server that you run yourself. The official Tailscale apps for phones, laptops, and Linux all work with it; you point them at your own server instead of Tailscale’s.

Headscale runs as one of my stacks on the NAS, and Headplane next to it gives it a web UI. The coordination server only introduces devices to each other. My actual traffic flows directly between devices and never passes through it. Its configuration, the list of who may reach what, and its DNS records are all files in my stacks repo.

Why both? #

Two VPNs look redundant, but they are good at different things.

WireGuard on the router puts a device on my home network. Adding a client is one entry in the router’s settings. That device can then reach everything at home, from the NAS to the printer to the router itself, at the same addresses and with the same DNS answers as on my home Wi-Fi. Nothing needs to be installed on the machines at home, and nothing depends on my homelab: if the NAS is down, or I broke something, WireGuard still gets me home so I can fix it. So my own devices use WireGuard: my iPhone, my laptop, and my travel router, which brings every device connected to it along.

Tailscale puts a device on a separate, virtual network. A device on the tailnet can only reach other devices that run Tailscale themselves. My home network isn’t part of it, which is why the DNS trick below is needed. In return, it works almost everywhere. WireGuard to my router only works if the network I’m on lets UDP traffic out to my router’s port, and some hotel, airport, and office networks don’t. Tailscale doesn’t need an open port, and its relays speak HTTPS, which nearly every network allows. It also connects machines that are all away from home, like my laptop and a cloud VM, directly, without a detour through my house.

Tailscale also has fine-grained access control. On my router, every WireGuard client gets my whole home network. On the tailnet, Headscale’s ACL decides per person and per device what they can reach. So everything that isn’t one of my own devices goes on the tailnet: friends, family, and cloud VMs.

The Tailscale DNS problem #

The public answer, 192.168.1.6, works on WireGuard because the tunnel routes my home network. It doesn’t work on Tailscale, because I don’t route my home network through the tailnet. Worse, 192.168.1.x is one of the most common home network ranges in the world, so on a friend’s Wi-Fi 192.168.1.6 might be their printer.

The fix is Headscale’s built-in DNS, MagicDNS. Every device on the tailnet asks Headscale’s DNS first, and I tell it to answer every lab name with the NAS’s tailnet address, 100.64.0.28:

dns:
 magic_dns: true
 base_domain: ts.nijho.lt
 extra_records:
 # BEGIN MANAGED HEADSCALE LAB DNS
 - name: mealie.lab.nijho.lt
 type: A
 value: 100.64.0.28
 - name: ntfy.lab.nijho.lt
 type: A
 value: 100.64.0.28
 # ... one entry per name, about a hundred of them
 # END MANAGED HEADSCALE LAB DNS

ntfy points at the NAS too, even though it runs on the NUC. These records don’t point at services; they point at the front door, and Traefik forwards the request to the right machine as usual.

I don’t maintain that list by hand. A small script collects every Host(...) rule from my Traefik labels and route files and rewrites the block between the BEGIN and END markers:

python3 scripts/sync-headscale-lab-dns.py --dry-run # preview
python3 scripts/sync-headscale-lab-dns.py # write
cf restart headscale # Headscale doesn't hot-reload

Forgetting this step is sneaky: the new service works perfectly at home and silently fails over Tailscale. That’s why it is written down in AGENTS.md, the instruction file my AI agents read, with the exact commands.

The result is that one name works everywhere I am. https://mealie.lab.nijho.lt reaches the same Mealie, with the same certificate, whether my phone is at home, on WireGuard, or on Tailscale; only the path underneath differs.

4. The internet #

The last way in is the open internet. My router forwards TCP port 443, HTTPS, to Traefik on the NAS, but to its own entrypoint, wan on port 4443, not to the websecure one that everything else uses. Only routes that list wan answer there.

A few services are public on purpose. The clearest example is my git server, Forgejo, at git.nijho.lt. A git server is only useful if the people and machines I work with can reach it, and not all of them are on my tailnet.

Its public name is a Terraform record that points at the router’s DDNS hostname, so it follows my home IP around automatically:

resource "cloudflare_dns_record" "git" {
 zone_id = var.cloudflare_zone_id
 name = "git.nijho.lt"
 content = "<my router's DDNS hostname>"
 type = "CNAME"
 ttl = 10800
 proxied = false # SSH uses port 222, CF only proxies 80/443
}

In Traefik, Forgejo simply has two HTTPS routes: the usual git.lab.nijho.lt, private like everything else, and git.nijho.lt, which also listens on wan.

 - traefik.http.routers.forgejo.rule=Host(`git.${DOMAIN}`)
 - traefik.http.routers.forgejo-public.rule=Host(`git.nijho.lt`)
 - traefik.http.routers.forgejo-public.entrypoints=websecure,wan

Git over SSH doesn’t go through Traefik; the router forwards a second port, 222, straight to Forgejo on the NAS. That is also why this DNS record is “DNS-only”: Cloudflare’s proxy only handles web traffic. Without the allowlist, Forgejo has to guard its own door: sign-ups are off, you have to log in to see anything, and SSH only accepts keys.

The Headscale coordination server is public too, at headscale.nijho.lt, and it has to be: a laptop on hotel Wi-Fi needs to reach it to log in and find the other devices before any tunnel exists. That record goes through Cloudflare’s proxy, which hides my home IP behind Cloudflare’s. Only the part the Tailscale apps talk to is public; the Headplane admin UI and Headscale’s lab name stay behind the allowlist. Headscale doesn’t run its own relay server either; the tailnet uses Tailscale’s public relays when a direct connection isn’t possible.

Forwarding port 443 sounds scary if you assume it means everything is on the internet now. It doesn’t: the internet only reaches wan, and a route has to ask for that.

Keeping it safe: one allowlist #

DNS is not a lock #

Remember the public DNS record from earlier: *.lab.nijho.lt points to 192.168.1.6, a private IP address on my home network. I made this mistake myself: I thought pointing DNS at a private address was enough to keep my services off the internet. It wasn’t. DNS only tells a client where to go, and nothing forces a client to ask. Anyone can skip DNS and send a request for mealie.lab.nijho.lt straight to my home IP:

curl --resolve mealie.lab.nijho.lt:443:<my home IP> https://mealie.lab.nijho.lt

A home IP is not a secret: any public service at home points to it, and scanners sweep every address on the internet anyway. That request reaches the router, gets forwarded to Traefik like any other HTTPS request, and asks for Mealie by name. The names aren’t secret either, because they’re easy to guess. The wildcard certificate keeps the individual names out of the public certificate transparency logs, which is nice, but that is hiding, not locking.

So the DNS setup is there for convenience: one name that works everywhere. The security boundary is Traefik, which knows how every request actually arrived.

Who is knocking #

Every network request carries a source address: where it came from. When the router forwards a request from the internet, the source stays the real public address of whoever sent it. Unlike a name, a source address can’t be faked for a real connection: the reply goes back to that address, so a forged one never completes the connection. So Traefik can tell which of the four ways in a request took, just by looking at where it came from.

I use that in a single Traefik middleware, a check that runs before a request reaches the service, called local-ips-only:

http:
 middlewares:
 local-ips-only:
 ipAllowList:
 sourceRange:
 - "192.168.1.0/24" # home network
 - "100.64.0.0/24" # Tailscale devices
 - "10.6.0.0/24" # WireGuard clients
 - "172.20.0.1/32" # the NAS itself (see below)
 - "127.0.0.1/32" # localhost

/24 means the first three numbers are fixed and the last can be anything, so 192.168.1.0/24 covers 192.168.1.0 through 192.168.1.255. /32 means exactly one address.

The middleware sits on the whole websecure entrypoint, so every HTTPS route there gets the check without a word in its labels. A request from any other address gets 403 Forbidden before it gets anywhere near the service. For the .local names, the same middleware sits on the plain-HTTP entrypoint. Internet traffic doesn’t arrive on websecure at all, so this is the second lock: if I ever point the port forward at the wrong port, strangers get 403 instead of my services.

The odd one, 172.20.0.1, is the gateway of my Docker network. Requests that start on the NAS itself, like a container calling another service by its lab name, show up with that address.

Keeping real Tailscale addresses (click to unfold)

Tailscale requests keep their real 100.64.0.x address thanks to one line in the NixOS config of docker-lxc:

services.tailscale.extraSetFlags = [ "--snat-subnet-routes=false" ];

Without it, Tailscale replaces the source address of traffic it passes on to another network, like Docker’s internal one, and every Tailscale request would look like it came from 172.20.0.1.

To check all of this, I test from every way in.

How I test it (click to unfold)

# From home, WireGuard, or Tailscale: expect 200
curl -s -o /dev/null -w '%{http_code}\n' https://mealie.lab.nijho.lt
# Through the router's port forward, skipping DNS: expect 404
curl -s -o /dev/null -w '%{http_code}\n' --resolve mealie.lab.nijho.lt:443:<my home IP> https://mealie.lab.nijho.lt
# A public service, from anywhere: expect 200
curl -s -o /dev/null -w '%{http_code}\n' https://git.nijho.lt

The second one works from inside too: the router sends the request back in through the same port forward, so it lands on wan like a stranger’s would.

Private by default #

Out of roughly a hundred HTTPS routes, only a handful are public, and each of those is a deliberate decision.

A route is public only if it lists the wan entrypoint and has no allowlist of its own. A private service needs nothing in its labels: it lands on websecure, where the allowlist guards the whole entrypoint. Forget a label and a service ends up private, not public. The one change that publishes something is typing wan, which stands out in a diff and is easy to list:

grep -rnw wan /opt/stacks/*/compose.yaml /opt/stacks/traefik/dynamic.d/

One wrinkle: devices at home that use a public name, like git.nijho.lt, also come in through the port forward, because DNS points that name at my home IP. For public routes that changes nothing. A couple of private services that I reach by a public name list wan too, and keep the allowlist on the route itself.

Trusting my own network #

For most services, the allowlist is the only lock in front. Anything on my home network or on WireGuard, and any of my own devices on Tailscale, can reach them without an extra login.

I could add one. Authelia runs next to Traefik, and putting its login page in front of a service takes one label. For Traefik’s own dashboard, that looks like this:

 - traefik.http.routers.traefik-lab.middlewares=authelia@docker

I do that for some services, but not all, because a login page in front of an app sometimes breaks things, like mobile apps that talk to the app directly.

Most apps have their own login, and I trust that enough on my own network. I also trust the people on it. If your home network has guests or devices you don’t trust, the trade-off looks different.

The allowlist trusts the whole 100.64.0.x range, and some devices on my tailnet belong to friends and family. That doesn’t give them everything, because there is a second layer in front: Headscale’s access control list (ACL). It decides, at the network level, which device may reach which port on which machine, and everything not listed is blocked.

A few examples of what that looks like in practice:

A friend uses my Mealie. Their router runs Tailscale and shares their home network with the tailnet, so every device in their home can reach Mealie without installing anything.
A friend uses my self-hosted dictation on the GPU machine, and nothing else.
An AI agent I run for a family member, in a MindRoom container, can reach one Home Assistant instance and nothing else.
A small NixOS box at a family member’s home joins the tailnet and makes a few shared services available on their network under simple .local names, for devices that can’t run Tailscale themselves. It is defined in the same NixOS repo as my own machines.

A simplified version of those rules:

{
 "hosts": {
 "nas": "100.64.0.28/32",
 "pc": "100.64.0.3/32",
 "friend-a-home": "10.10.0.0/16"
 },
 "acls": [
 { "action": "accept", "src": ["friend-a@", "friend-a-home"], "dst": ["nas:9925"] },
 { "action": "accept", "src": ["friend-b@"], "dst": ["pc:61337", "pc:8080"] },
 { "action": "accept", "src": ["me@"], "dst": ["*:*"] }
 ]
}

Friends reach their service directly on its port, not through Traefik. They can’t even reach port 443 on the NAS, so Traefik never sees them. My own devices can reach everything.

That also means friends don’t use my lab names. I give them the machine’s tailnet address and the port, like http://100.64.0.28:9925, and that’s it. A device keeps its tailnet address, so it can go in a bookmark, and one friend points a name in their own homelab’s DNS at it. It is plain HTTP, but between their network and mine it travels inside the encrypted tunnel.

Because the ACL is a file in my stacks repo, sharing one more service with someone is a one-line diff and a cf restart headscale. The commit history doubles as a log of who got access to what, and when.

This is also how friend-to-friend setups like my remote TrueNAS backups work: a tailnet is a private network you can extend to exactly the people you trust.

Why not Cloudflare Tunnel or Tailscale’s own server? #

This is usually the next question. Cloudflare Tunnel publishes services without forwarding any ports: a small daemon at home keeps an outbound connection open to Cloudflare, and Cloudflare serves your sites to the world. Tailscale runs the coordination server for you, for free, so you don’t need Headscale at all. Both are great products, and both are easier than what I do.

The reason I don’t use them is simple: I don’t want to depend on a third-party company for any of this. The plumbing in this post is open source and runs on my own machines, so if one of these companies disappears or changes its terms tomorrow, everything keeps working. With Cloudflare Tunnel, every request to my services would pass through Cloudflare, which decrypts it on the way. With Tailscale’s hosted server, a company would decide which devices are part of my network.

There are a few places where I still lean on someone else, and each one is replaceable:

Cloudflare hosts my public DNS. The records are a Terraform file, so moving to another provider means rewriting that file for a different provider. The same goes for the certificate challenge, which Traefik supports for dozens of DNS providers, and for the few names that go through Cloudflare’s proxy.
Tailscale’s public relays carry traffic when two devices can’t connect directly. Headscale has a relay server built in, which I can turn on in its config.
Let’s Encrypt issues my certificates, but it speaks a standard protocol, ACME, that other certificate authorities speak too.
Terraform moved to a source-available license in 2023. OpenTofu is the open-source fork, and it reads the same files.

Declarative everything #

One idea runs through all of this: every piece of configuration is a text file in git, not a setting I clicked somewhere. The opposite is running commands and clicking buttons until things look right, and then hoping you remember what you did.

Piece	Where it’s defined	How it’s applied
Operating system, Docker, Tailscale, NFS mounts, home DNS, firewalls	NixOS config in my public dotfiles	comin pulls signed commits and switches
Which service runs where	`compose-farm.yaml`	`cf apply`
Each service and its routing	`compose.yaml` with Traefik labels	`cf up`
Routes to services on other machines	Generated by compose-farm	Automatically on `cf up`
Non-Docker routes and the allowlist	A route file next to Traefik	Traefik watches the folder
Public DNS records	Terraform in a private repo	`terraform apply`
Tailnet DNS records	Generated from Traefik rules into Headscale’s config	`cf restart headscale`
Who may reach what on the tailnet	Headscale ACL file	`cf restart headscale`
Router: WireGuard, port forward, DDNS	The router’s web UI	❌ Clicked
Which devices are on the tailnet	Headscale’s database	Enrolled once per device; state, not configuration

The only configuration I still click is the router, and those settings change maybe once a year. Everything that changes weekly is a file. The next step in project “overengineered homelab” is a router running NixOS, but that’s for another time. What isn’t configuration, like which devices are enrolled in my tailnet and the data inside each app, lives on ZFS and is backed up like any other data.

It wasn’t always like this. My previous setup was built from good products that are all driven by clicking through a web UI:

Job	Before: clicked in a web UI	Now: a text file in git
Virtualization	Proxmox	NixOS and Incus
NAS	TrueNAS	NixOS with ZFS
DNS	Technitium	CoreDNS in my NixOS config
Reverse proxy	Nginx Proxy Manager	Traefik labels in each `compose.yaml`
Containers	Dockge, one machine at a time	compose-farm, all machines from one file

Yes, most of them have APIs, but that is not how most people use them, me included. Most of the configuration lived in each product’s database, and the reasons behind each setting lived in my head. Now both live in git.

Why it matters #

The classic benefits are real: git history is my changelog, rebuilding a dead machine is boring, and nothing lives only in my head.

But the benefit that matters most to me now is a newer one: this setup is easy to work on with AI.

Coding agents are very good at reading and editing text files, running commands, and checking the result. They are much worse at clicking through a web dashboard. Because almost everything here is files and CLI commands, an agent can do exactly what I do: add a service, move it to another machine, find out why a route returns 403, add a DNS record, write the NixOS change, and show me the diff. Terraform is a good example of why this is safe: the agent edits the record, runs terraform plan, and I see exactly what will change at Cloudflare before anything does.

I run my agents in YOLO mode, and declarative config is a big part of why that doesn’t scare me. Every change is a reviewable diff, and every mistake is a git revert away. It’s the same reason I prefer plain files over databases for AI memory.

The stacks repo has an AGENTS.md that captures the conventions and the sharp edges (a symlink to CLAUDE.md, so Claude Code and other agents read the same file):

> **⚠️ IMPORTANT: NEVER run `docker compose` directly!**
>
> Always use `cf` (compose-farm) for ALL compose operations.
> Running `docker compose` directly bypasses host routing,
> state tracking, and Traefik config generation.

The Tailscale DNS step from earlier lives there too, with the exact commands. Whenever something bites me once, it goes into that file, so it doesn’t bite an agent twice.

The other half is generating everything I would otherwise keep in sync by hand. The cross-machine Traefik routes come from the Compose labels, the Headscale DNS records come from the Traefik rules, and even my Uptime Kuma monitors are synced from the Traefik labels. Hand-maintained lists drift; generated ones don’t.

Putting it together: adding a service #

This is everything it takes to add a new service, say ntfy on the NUC:

Create /opt/stacks/ntfy/compose.yaml with the container and its Traefik labels: an HTTPS name and a .local name, like Mealie.
Add ntfy: nuc to compose-farm.yaml.
Run cf up ntfy. compose-farm starts it on the NUC and writes its route into Traefik’s file, and Traefik picks it up immediately.
Run the Headscale DNS script and cf restart headscale, so the name also works over Tailscale.
Commit.

After that:

https://ntfy.lab.nijho.lt works at home, over WireGuard, and over Tailscale, with a valid certificate I never requested.
http://ntfy.local works at home.
Sent to my home IP from the internet, it answers 404 Not Found.

To make it public like git.nijho.lt, I would add a Terraform record and a second route that also listens on wan. To share it with a friend, one line in the ACL.

In practice, I describe the service to an agent and review what it did.

Lessons learned #

Use the DNS-01 challenge. You get real certificates for private services and one wildcard for everything, and the internet never needs to reach you.
One front door, private by default. Knowing that a route has to ask to be on the internet is what made forwarding port 443 feel fine.
A private address in public DNS is a great trick for home and WireGuard. A mesh VPN like Tailscale needs its own answer.
DNS is not a security boundary. Anyone can send any name to your IP. The check has to happen at the front door.
Point DNS at the front door, not at the service. Traefik knows where things run, so moving a service never touches DNS.
Keep one way in that doesn’t depend on the homelab. For me that’s WireGuard on the router. The NAS holds the data and runs Traefik, so when it is down, everything is, and I still need a way in to fix it.
Test from every way in. Some problems only show up on one path, like a new name that works at home but not over Tailscale.
Two layers for sharing. Headscale ACLs decide who reaches what; Traefik’s wan entrypoint decides what the internet sees.
Make private the default. Public should take an extra word in the config, so a forgotten label fails closed.
Generate what you can. A generated list can’t drift.
Make everything a file. It’s better for future me, and it’s the difference between an AI agent that can help and one that can only give advice.

References #

compose-farm: my tool for running Docker Compose stacks across multiple machines
My NixOS configuration
comin: GitOps for NixOS machines
Traefik, including the IP allowlist middleware
Let’s Encrypt challenge types: HTTP-01 vs DNS-01
Terraform Cloudflare provider
WireGuard
How Tailscale works
Headscale and Headplane
CoreDNS
Forgejo

Yes, nixpkgs is the largest and most up-to-date package repository there is. Even so, I follow nixos-unstable, and a new version only reaches me once it is merged, built, and tested, and the channel moves forward, which usually takes a couple of days. Updates that trigger large rebuilds go through a staging branch first and take longer, and not every package gets updated as quickly as the popular ones. With Docker, I can run a release the day upstream publishes it. ↩︎

The last btrfs machine: migrating my PC to ZFS

Thu, 23 Jul 2026 00:00:00 +0000

My GPU machine was the last machine in my fleet still running btrfs. I bought it for gaming (which never happened), and these days its dual RTX 3090s almost exclusively run local AI models.

btrfs itself never failed me. When I set up NixOS on this machine for the first time, I had read enough forum threads to become convinced that ZFS and NixOS were a troublesome combination, so I picked btrfs to be safe. That belief was simply wrong. As I went full Nix and eventually moved even my TrueNAS to NixOS, I learned there was no issue at all. I just had to figure out the right incantation to make ZFS work with boot drives, disko, and nixos-anywhere. My HP got ZFS first, that config grew into a shared disko module, and soon the whole fleet ran one pattern: NixOS on ZFS, snapshots and replication declared in code, and backups I can reason about.

Only the machine I sit at every day kept running on my old misconception, because a daily driver is exactly the one you do not casually reinstall.

Table of contents

Why touch a working machine #

The operating system was already uniform. The filesystem was not, and that turned out to matter more than I expected.

The real trigger was that I got increasingly paranoid about an AI agent messing up my system. I run agentic AI in YOLO mode, exclusively. All day, every day, agents execute commands on this machine without asking me first. YOLO does not mean root, though: the agents run as my regular user, and the backup credentials live where only root can read them. An agent that could destroy the backup itself would make everything else in this post pointless. Meanwhile the stories keep coming in about even the most frontier models wiping someone’s home directory:

GPT-5.6-Sol just accidentally deleted almost ALL of my Mac’s files. And this is why I trust Fable 1000x more.

— @mattshumer_

The reaction is always about which model deserves more trust. I read these tweets differently. Give any agent enough sessions and it will eventually delete something important, the same way I do every few years with a tired rm -rf. Whether that moment becomes a catastrophe or an inconvenience depends entirely on the restore path.

So I felt I needed to properly audit the way my backups were done. I want to be able to answer four questions about any machine I own:

What exactly is in the backup?
What is not in the backup, and did I choose that deliberately?
How big is a restore?
How long will it take?

The backup that wasn’t #

Asking question 1 delivered an unpleasant answer within the hour: the backups had quietly stopped four months earlier. Two boring causes had stacked up: an interrupted prune left a stale repository lock, which the job could never clear because its own unlock step only ran after a successful backup, and the repository URL still pointed at truenas.local, a name that had stopped resolving along the way. The last successful backup ran at 07:00 on March 22; the next one at 07:16 on July 22, after the fix. Four months, almost to the minute. systemd logged the failure on every single run, nobody read it, and nothing escalated. I found out because I asked the question, not because anything told me.

What the audit found #

On the btrfs box, question 3 alone had three different answers depending on which tool I asked. du counted reflinked copies at full size, so directories reported terabytes that physically did not exist. df counted physical extents, which is honest about disk usage but tells you nothing about what a file-based restore will write. And restic reported the apparent size of sparse VM images, which inflated the restore estimate to several times the disk.

When your storage layer gives you three sizes for the same data, you do not really understand your backup. You have a backup, and you have hope.

Question 4 was no better. Because the PC was my only btrfs machine, it could not join the ZFS snapshot replication the rest of the fleet uses, so its backups relied entirely on restic pushing to the NAS. Off-machine, note, but not off-site; question 2 turned out to apply to locations too, and closing that gap is its own project. restic is file-based: every run visits the metadata of every single file to find what changed. On this machine that meant close to a hundred million files per scan, which turned out to include some 600 GB of git worktrees stuffed with virtual environments and node_modules. Each run took about an hour and a half, even when almost nothing had changed.

That scan time dictated my backup cadence. The timer ran hourly, which in practice meant back-to-back scanning all day to move a couple hundred megabytes of real changes, and I recently dropped it to every six hours to stop the madness. Either way, my most recent work existed only on the machine an agent was busy poking at.

The scanning is also not free for the hardware. Reads are gentle on an SSD compared to writes, but hourly scans of an hour and a half each kept the drive at full duty cycle and elevated temperature for most of every day. And drives stopped being cheap: I paid $227 for a 4 TB NVMe drive in 2024, and the same drive lists for around $800 today. I would rather not run the experiment of what years of continuous scanning do to a drive I can no longer afford to replace.

btrfs has perfectly good snapshots of its own, but sending them off-site was never an option for me: snapshot replication needs the same filesystem on the receiving end, and my NAS speaks ZFS. With ZFS I can snapshot every few minutes and zfs send the actual block storage: an incremental send streams exactly the blocks that changed since the last snapshot, without walking a single file. A hundred million files cost nothing when nobody has to visit them.

ZFS does not magically collapse those numbers into one. But it gives me the same semantics everywhere, including on the NAS pools that hold the backups themselves. Eight of my machines already share a single ZFS disk layout, defined in one shared disko module. The PC was the odd one out. This migration makes it nine machines with the exact same strategy, and one backup story to audit instead of one per machine.

The plan: one shared disko module #

The Nix side was the easy part. The PC now imports the same shared disko module as the other eight machines, with one addition: a swapSize parameter, because ZFS has no reliable swapfile story and the PC actually needs its 96 GB of swap.

The disk is pinned by ID, not by device name:

device = "/dev/disk/by-id/nvme-Samsung_SSD_990_EVO_Plus_4TB_...";

This matters more than usual here. The PC dual-boots Windows from a second NVMe drive, and I want zero ambiguity about which of the two disks disko is allowed to erase.

The scary property of this migration is different from the NAS one. For the NAS, the whole point was that disko must not touch the data pools. For the PC, the new ZFS pool goes on the same disk the data currently lives on. No second disk, no local rollback, no btrfs to boot back into. From the moment disko runs, the restic repository on the NAS is the only road back.

That sentence is the reason most of this post is about verification.

The road back also has to survive the same agents that motivated all of this. The PC pushes its backups over sftp as a user that owns every file in the repository, which means a compromised PC could delete or encrypt its own safety net. So the NAS snapshots the repository dataset on its own side, where the PC’s credentials cannot reach: two days of hourly ZFS snapshots and two weeks of dailies that only root on the NAS can destroy.

Rehearsing in a VM, again #

The NAS migration taught me to rehearse the destructive step in a VM, so the flake grew a pc-vmtest target and I ran the whole install through nixos-anywhere --vm-test. Disko formatted a virtual disk, created the pool, installed the system, and booted it with swap active.

Passing that test proves the generated logic. It does not prove the backup, which is where the real risk lives.

The recovery kit #

A restore that starts with “first, log in to the dead machine and fetch the credentials” is not a restore plan. So everything a bare reinstall needs lives in a small recovery kit on two other machines in the house, permissions locked down: the restic password and SSH key that reach the repository, the staged restore script, a couple of verification scripts, and the PC’s SSH host keys. The host keys matter more than they look: the reinstalled machine comes back with its old identity, so no client complains and nothing in the fleet needs to be taught to trust a new machine. A munge key and a README round it out.

Trusting the backup enough to wipe the disk #

My recovery story has two layers. NixOS makes the system disposable: every service, driver fix, and firewall rule is in git, so no agent can leave the machine in a state I cannot rebuild. Verified backups make the data recoverable. This migration is the voluntary version of the tweets above: I am about to delete all of my PC’s files on purpose.

I did three verifications, each one converting an assumption into a fact. An AI agent did the tedious sweeps; I reviewed the evidence.

The first check was completeness. We walked everything on the machine outside the backup paths, until every byte was accounted for. It was all re-downloadable caches, and the largest unexplained thing turned out to be restic’s own local cache. Everything outside the backup set is now known disposable instead of assumed disposable.

The second check was the restore path. A backup you have never restored from is a hypothesis. On a different machine, using only the recovery kit and no access to the PC at all, we restored a sample from the latest snapshot and compared hashes against the live system. The hashes matched byte for byte, and even the symlinks came back pointing at the right targets. A sample proves the path works, from credentials to symlink handling; reading back every byte is the third check’s job. The machine running the rehearsal had the kit and nothing else, so recovery provably does not depend on the machine being recovered.

The third check is the data itself. restic check validates the repository structure without reading back the actual data blobs. Before the wipe, a --read-data pass ran on the NAS, where the repository sits on local disk instead of behind the network. When the backup is the only rollback, I want its bits read back at least once.

The fifth question #

The outage taught me a fifth question: how do I find out when any of this stops working? All three checks above are point-in-time, and verification rots. So the NAS now re-verifies every hour, from its own side of the sftp connection, that the newest snapshot in the repository is less than a day old, and pushes an alert to my phone when it is not. That catches every failure mode on the PC side, including “the timer is simply disabled.” The watcher itself is watched: the NAS pings an external dead-man’s switch every five minutes, so if the machine holding my only road back goes dark, the alert comes from outside the house.

While wiring this up we discovered that months of earlier NAS alerts had never arrived, because ntfy silently rejects message bodies over 4 KB. A notification you have never received is also a hypothesis.

Restoring in stages #

The measured restore set came out to roughly 1 TB, and 99.9% of it is /home. But two directories are 64% of that terabyte: Steam (430 GB of re-downloadable games) and my local-AI model collection (217 GB of GGUF files).

I do not need either of those to have a working machine. So the restore script now runs in stages:

sudo ./restore_from_backup.sh stage1 # everything else → a fully working machine
sudo ./restore_from_backup.sh stage2 # Steam + models, whenever

Stage 1 restores everything except the deferred paths, plus the system identity (host keys). Stage 2 can run days later, from the rebooted system, while I am already using the machine.

One detail I care about: stage 1 resolves latest to a concrete snapshot ID and pins it to a file. Stage 2 reads the pin. My backups run every six hours, so without this, a timer firing between the stages would silently split the restore across two different snapshots.

The backup itself is untouched: Steam and the models are still in every snapshot. Only the restore order changed.

Slimming down first #

The PC had also quietly become an Incus host for three LXC containers, including the MindRoom bots my family talks to every day. Before the migration, those containers moved to the NAS. That shrinks both the backup and the restore, and the family bots keep running while their old host is being wiped.

The cutover #

The install followed the same phased nixos-anywhere pattern as the NAS cutover, driven from another machine on the LAN: kexec into an installer running in RAM, stop, re-verify the target disk from inside the installer, and only then run the destructive disko/install phase. That checkpoint is the whole point: right up to the destructive phase, aborting was still cheap. The destructive part turned out to be the trivial part: from wiping the disk to booting the new system took less than twenty minutes.

The one surprise of the day was in the restore, and it was exactly the kind rehearsal is supposed to catch. Stage 1 would not start: the restore script runs restic as root, and a freshly installed root has no SSH identity and no known_hosts. The key it needed sat in the recovery kit inside my home directory. The rehearsal never hit this, because the rehearsal machine’s root had already talked to the NAS. A verified restore path can still depend on state you forgot you had.

Stage 1 restored 534 GiB across 4.5 million files in just over an hour and a half, at wire speed. The machine rebooted with its old host keys and its old /home, and stage 2 started immediately. The plan said stage 2 could wait for days; it started the minute the machine was back, because once the machine works there is no reason left to wait. The deferred 648 GiB of games and models took six more hours in the background.

With that, the PC finally joins the fleet pattern it was locked out of: sanoid snapshots the pool every hour, so a tired rm -rf, mine or an agent’s, now costs at most an hour of work and a rollback, and a daily syncoid run sends the changed blocks to the NAS. restic is on borrowed time. Its six-hour timer keeps running until I trust the new path, and then it stops: the repository stays on the NAS as the last file-level snapshots of the machine, but nothing will be added to it.

Aftercare was the same short list, written down instead of remembered: revert the temporary root-SSH commit the installer needed, and put the restic password back where the backup service expects it.

The slow work, as always, was not the cutover. The slow work was making the cutover boring.

References #

Migrating from TrueNAS to NixOS without losing data after the build-system rug pull

Sun, 28 Jun 2026 00:00:00 +0000

TrueNAS was the last non-NixOS machine in my home lab.

I kept postponing it for a simpler reason: moving something with ~100 TiB of data feels scary. My NAS holds the big ZFS pools. It has the SMB shares, NFS exports, snapshots, replication tasks, encrypted datasets, Incus state, Docker workloads, and all the boring-but-important health checks that are easy to take for granted until something fails.

I had already migrated most of my machines from Proxmox to NixOS with Incus, as I wrote about in I’ve gone full Nix: Proxmox to NixOS + Incus. That migration made me increasingly intolerant of appliance-style infrastructure. If the important state lives in a database behind a web UI, I cannot review it in a pull request. If I change something while debugging, I cannot guarantee the change made it back into the source of truth. And if I let an AI agent help me, imperative systems become even more uncomfortable: an agent can run a hundred commands, succeed, and leave behind a system that nobody can reproduce. For the NAS specifically, that basically meant I did not dare touch the machine with an AI agent at all.

NixOS is different. The diff is the state. If an agent changes my infrastructure, I can review the patch. If the change is wrong, I can reject it. If it works, I can commit it.

TrueNAS was the last holdout. It had been itching for a while. Then the build-system rug pull made the decision easy.

Table of contents

The itch before the last straw #

The build-system change was not the first thing that bothered me. It was just the last straw.

The deeper problem was that my TrueNAS setup had slowly become complicated in exactly the way I dislike. I had clicked many buttons in the UI over time:

SMB shares
NFS exports
snapshot tasks
replication tasks
users and groups
encrypted datasets
apps and containers
Incus storage
old Jailmaker/systemd-nspawn state
alerting
UPS settings

TrueNAS can export and import its configuration. That is genuinely useful. But a database export is not the same thing as a configuration I can comfortably review. When I look at a TrueNAS backup, it is not obvious to me which settings I intentionally changed, which settings are defaults, which settings are historical leftovers, and which settings only exist because I clicked something while debugging two years ago.

One concrete example was NFS. I needed the crossmnt export option so child datasets would show up correctly inside the Docker container host, but TrueNAS did not expose that option in the UI. So I had a periodic root cron job patching /etc/exports after TrueNAS generated it. It worked, but it was exactly the kind of small appliance hack that made me uneasy: partly outside the UI, partly outside the exported configuration, and very easy to forget during a migration. In NixOS, that behavior is just part of the export line.

That is the exact discomfort that pushed me away from Proxmox. I do not want to remember what I clicked. I want the system to remember it as code.

The second itch was virtualization. My TrueNAS container story had already gone through one migration. For a while I used Jailmaker, which is a clever script around systemd-nspawn for running Linux “jails” on TrueNAS SCALE. It worked well enough for what I needed at the time: a Debian environment where I could run Docker workloads close to the data. But it also reinforced the feeling that I was building a fairly custom server on top of an appliance OS.

TrueNAS 25.04 “Fangtooth” introduced experimental Containers, named Instances in the early 25.04 releases, and the 25.04.1 announcement explicitly described Instances as being based on Incus, supporting both VMs and LXCs. That was interesting to me because the older TrueNAS VM system was libvirt-based, while Incus felt like a more coherent container/VM story. Then the TrueNAS virtualization plan for 25.04.2 explained that Fangtooth’s Instances feature caused challenges for some users and that 25.04.2 would re-enable the older Virtualization VM path alongside Instances. The 25.04.2 release announcement made that split official: Virtual Machines came back, while VMs created through the earlier Instances feature continued to appear under Containers.

For many people that churn was probably annoying. For me it was especially ironic. I had migrated from Jailmaker’s systemd-nspawn setup to Incus, read up on Incus, got excited about it, started using it, and then became genuinely happy with it when I migrated from Proxmox to NixOS with Incus. Incus fit my mental model perfectly: a clean CLI, containers and VMs in one place, easy scripting, and a workflow that agents can inspect.

Then TrueNAS started moving away from that direction again. Forum discussion around the 25.04.2 and later virtualization plans made it clear that Incus was not the stable long-term abstraction I hoped it would be inside TrueNAS. In the long-running Linux Jails with Incus thread, Kris Moore is quoted saying that Incus is being removed from the base system and that libvirt will handle the backend for VMs and LXC. That probably makes sense for TrueNAS. They have enterprise users, compatibility concerns, and a support matrix that is very different from mine.

But for me it meant the NAS was again becoming an appliance whose internal direction I had to follow. I had just escaped that with Proxmox. I did not want to do it again for the most important machine in my house.

The last straw #

Then I saw Jeff Geerling mention TrueNAS’s recent “shady behavior” in a video, and I started looking into what had actually changed.

The short version is that iXsystems made part of the TrueNAS ecosystem closed source. Specifically, iXsystems moved the TrueNAS build scripts and related build infrastructure internal for the TrueNAS 27 line. Their forum post, Clearing the Air on Build Scripts, frames this as a practical build-system decision and says normal installation and updates remain unaffected. The old truenas/scale-build repository now says it is preserved historically and is no longer the maintained build system. Community reactions, for example this TrueNAS forum thread, understandably read this as a meaningful step away from fully open development.

Would I ever have built TrueNAS from source myself? Almost certainly not.

That misses the point. I also rarely compile my Linux kernel manually, yet I care deeply that the ecosystem is open enough that I could understand, rebuild, inspect, and fork the pieces if necessary. For a storage appliance, trust matters more than convenience. The NAS is not a toy VM. It is the machine that owns my data.

That was the final push. The UI state bothered me. The virtualization churn bothered me. The build-system change made the decision obvious.

Why this was scarier than Proxmox #

Migrating Proxmox to NixOS was scary in the normal infrastructure sense. I had containers and VMs to move, but the blast radius was understandable. I could export with vzdump, import into Incus, and test a lot of it in virtual machines.

Migrating TrueNAS is a different kind of scary. The data is not a VM disk that I can casually copy around. The important thing is not to recreate the data pools. The important thing is to not touch them.

The target architecture is simple in principle:

# Conceptually:
# - disko owns only the boot disk
# - existing data pools are imported by name
boot.zfs.extraPools = [
 "ssd"
 "tank"
];

The NixOS boot disk can be destroyed and recreated. The existing tank and ssd pools cannot.

This is the part where many migration guides become hand-wavy:

Install the new OS.
Import your ZFS pools.
Good luck.

That is not enough for me. I could physically access the machine, but unplugging the relevant NVMe hardware would mean taking apart more of the box than I wanted to touch. So I treated the cutover as if the disks had to stay connected. And I absolutely did not want one typo in a disk path to become yet another meme in AI agent failures.

Giving an agent the keys, carefully #

Without AI this would probably have become a week-long project that I would forever postpone. With AI, it felt plausible to get the first serious version done in a single sitting. My approach was to give GPT 5.5 in xhigh mode SSH access to the live TrueNAS box and tell it, repeatedly and very explicitly, not to change anything. To be clear, it had root access. At this point these models have become powerful enough that I trust them to follow explicit instructions. The safety boundary was instruction-following, narrow scope, and review.

Not “please be careful.” More like:

You can inspect.
You cannot change anything.
Do not start or stop services.
Do not import or export pools.
Do not load keys.
Do not mount datasets.
Read only.

This is the part where people either nod along or think I have completely lost my mind. I think the distinction matters. I did not ask an agent to “migrate my NAS.” I asked it to inspect the current system, summarize the live state, compare that against my NixOS repository, and propose declarative config.

Then I reviewed the diffs. Then I asked for another review. Then another. The process was messier than one neat two-agent pass: build a scaffold, ask a model to attack it, compare that against the live TrueNAS state, refine the scaffold, and repeat. GPT 5.5 did most of the initial construction. Claude Opus 4.8 on max effort did one of the later independent cross-checks against the live system. By the end, I trusted the result because multiple independent reviews had converged on the same assumptions, and I understood the remaining differences. The loop felt like code review: inspect live state, turn the desired parts into Nix, review the diff, and ask another model to attack it. The scariest assumption got a VM test, and the destructive step stayed small.

Byte-for-byte TrueNAS cloning was never the target. I wanted every important behavior explicit: storage import, shares, snapshots, replication, Incus recovery, monitoring, and the few places where NixOS intentionally differs from the old appliance. At that point, the NixOS config reproduced the parts of TrueNAS I actually depended on, and the remaining differences were deliberate.

The agent was excellent at inventory:

Which pools exist
Which datasets are encrypted
Which NFS exports are live
Which SMB shares exist
Which users and numeric IDs matter
What Incus storage pool is in use
What replication tasks exist
What SMART/NUT/health monitoring exists
What host ID ZFS expects
Which disk is the current boot pool

Preserving everything blindly would have been wrong. The useful parts became code, and the dangerous parts became explicit.

The NixOS scaffold #

The resulting NixOS host is split into small files:

hosts/nas/
├── default.nix
├── disko.nix
├── hardware-configuration.nix
├── health.nix
├── identity.nix
├── networking.nix
├── nfs.nix
├── replication.nix
├── samba.nix
├── storage.nix
├── virtualization.nix
├── CUTOVER.md
└── PLANNING.md

This is exactly why I like NixOS. The migration stopped being a list of steps I needed to follow and became a pull request.

The important storage boundary is:

disko.nix describes only the boot disk.
The data pools are imported by name.
ZFS force-import is disabled.
Boot-time encrypted credential prompts are disabled.
Encrypted datasets are unlocked manually after boot.
Sanoid/Syncoid replace TrueNAS snapshot and replication tasks.
SMB, NFS, Docker, Incus, SMART, ZED, NUT, Netdata, and exporters are all declared in code.

The data pools are intentionally not in disko. That is the entire point.

The disko problem #

disko is a fantastic tool. It lets you describe disks declaratively and then formats/partitions/mounts them. For a fresh NixOS install, it is exactly what I want.

This also fits how I install NixOS machines. I build a custom installer ISO, put it on a USB stick, and boot from it. The image already has SSH enabled with my keys, flakes enabled, and the tooling needed to run the repo-defined disko and nixos-install flow. That turns the installer environment into part of the reviewed configuration, instead of another place where I have to remember which choices to make.

It is also exactly the tool that can ruin your day if pointed at the wrong disk.

My requirement was:

Use disko to destroy and recreate the boot disk, but prove it cannot touch the existing ZFS data pools.

The static config already looked right. The generated disko script only referenced the boot device and generated partition labels. The data pools were only present under boot.zfs.extraPools.

But “looks right” is not enough when the downside is wiping a NAS.

So we built a VM rehearsal.

Rehearsing the dangerous part in a VM #

The VM test creates a miniature version of the dangerous situation:

Attach several throwaway disks to a disposable NixOS VM.
Create a fake boot-pool on one disk.
Create fake tank and ssd ZFS pools on other disks.
Write sentinel files into the fake data pools.
Export the fake data pools.
Run the generated nas disko script against the fake boot disk.
Confirm the boot disk now contains the new zroot.
Confirm fake tank and ssd still have their ZFS labels.
Re-import fake tank and ssd.
Read the sentinel files.

The check lives in the flake now:

cd configs/nixos
nix build .#checks.x86_64-linux.nas-disko-safety

It passed.

Passing that VM test gives me confidence in the generated Nix/disko logic. The read-only check on the live TrueNAS box gives me confidence in the configured disk target: the by-id path resolves to the 500 GB boot-pool disk, and tank and ssd live on different devices. The installer preflight repeats that same check immediately before the destructive step.

This is the level of paranoia I want around storage migrations.

The parts that are not just storage #

Once the “do not wipe the pools” boundary was in place, the rest of the migration became more normal infrastructure work. Still annoying, but normal.

SMB was not a byte-for-byte port. TrueNAS ships private Samba VFS modules and has its own ACL/middleware behavior. Stock NixOS Samba covers the parts I need with fruit, streams_xattr, and sensible share definitions, but it is not literally the same stack. So the plan calls out what must be tested from real clients: Time Machine, guest access, and photo/media access.

Snapshots and replication also changed. TrueNAS snapshot tasks become Sanoid policy. TrueNAS replication tasks become Syncoid systemd timers. Existing snapshots are preserved, but Sanoid will not magically prune old TrueNAS-named snapshots. That is a feature, not a bug. I would rather manually decide when old snapshots age out than have a new tool silently “clean up” history.

Incus is another subtle one. The storage pool lives on a data pool, so disko will not touch it. But a fresh NixOS install has a fresh Incus database. The plan is to run incus admin recover, then apply a small declarative reconciler for the known instances. The state boundary has to be clear: preserve the storage dataset, recover the Incus database, then reconcile the instance config.

Health monitoring is also declared:

SMARTD
ZFS Event Daemon
NUT/UPS client
Netdata bound locally
Prometheus exporters
A shared alert hook that can log, wall, and optionally send ntfy alerts

TrueNAS gives you many of these things in the UI. NixOS gives them to me as code.

I know using your NAS for other services is a bit of an anti-pattern. In an ideal world, the storage box just stores data and stays boring. But when I originally bought and configured this machine, it became the most powerful machine in my home lab. So it naturally started attracting the Docker workloads too. Most of those workloads are not resource intensive, and keeping them close to the data is genuinely convenient. At one point I was running more than a hundred containers on that machine and it barely made a dent. They ran inside LXC containers on Incus rather than directly on the TrueNAS host, which made it feel like a reasonable compromise for a long time.

The migration also became a chance to fix a real operational problem. My worst TrueNAS incident was a long OOM death spiral: no swap, too many unbounded containers, and services repeatedly getting killed and restarted. The philosophy came from Nix and open infrastructure. The OOM incident was the practical reminder that the NAS had become a general-purpose compute host. In the NixOS version, container memory limits, zram, earlyoom, and an explicit ZFS ARC cap are part of the configuration instead of being tribal knowledge or post-incident notes.

Secrets stay out of the repo #

My dotfiles are public. That makes the migration more interesting.

The Nix config necessarily contains some operational facts: hostnames, share names, service definitions, and so on. But the plan deliberately keeps secrets and sensitive cutover material out of git.

For this migration, that means:

ZFS dataset passphrases and unlock material stay off-box and outside the repo.
Replication private keys are installed manually under /etc/ssh.
Inbound authorized keys for backup senders are installed during cutover.
Alerting secrets live in a runtime env file.
A local ~/nas-cutover/ staging directory can exist, but it is explicitly not part of the repo and should not be casually read by agents.

One important detail: the encrypted datasets use passphrase keys. In the TrueNAS setup I handled that with truenas-unlock, a small tool I wrote for this exact problem. The unlock material lives on another device; when that device is on the same network, it can unlock the datasets through the TrueNAS API. That gave me a lightweight hardware/network-presence factor: the NAS could boot, and the encrypted datasets only unlocked when the separate unlock device was present too.

That changes the migration footgun. The TrueNAS API path does not survive the cutover as-is, so I split the same idea out into zfs-unlock: the NixOS/OpenZFS version that talks to a restricted SSH receiver instead. During the migration I still kept an independent manual passphrase recovery path, because I do not want the automatic unlock mechanism to be the only way back into my data.

Where this leaves TrueNAS #

I still like TrueNAS. It got me very far. It made ZFS approachable. It gave me a good NAS UI when I wanted a NAS UI. And if you want an appliance that mostly owns storage for you, it is still a very compelling project.

I like NixOS more for my own infrastructure. The most important machine in my house should live in reviewable code instead of mutable appliance state spread across a UI, a database, and old debugging decisions. NixOS and AI are a particularly good match for that. I can describe messy infrastructure intent in imprecise English, let the agent turn that into exact Nix code, and review the diff before it becomes system state. The prompt stays fuzzy. The machine state does not.

After the Proxmox migration, TrueNAS was the last major exception. This migration removes that exception.

The cutover #

I ended up using a phased nixos-anywhere cutover instead of the USB installer path. The important safety property stayed the same: first kexec into a temporary NixOS installer, stop there, run the disk preflight from inside that installer, and only then run the destructive disko,install,reboot phase. Before the destructive phase, the box was running the installer in RAM and the disks had not been intentionally modified yet. That gave me a real checkpoint where aborting was still cheap.

I kept my hands on the few commands that actually changed the machine. I ran the nixos-anywhere phases and the later nixos-rebuild commands myself. The agent stayed next to it, reading the outputs, checking the assumptions, SSHing in to verify pool health, services, mounts, Incus state, NFS clients, and then turning any findings back into code or documentation. That split felt right to me: I pressed enter on the small number of irreversible commands, and the AI did the tedious verification work that I would otherwise be tempted to rush.

The actual migration was almost anticlimactic compared to the preparation. From the moment TrueNAS stopped being the running OS to having the Incus containers back with my hundred-plus services running again took less than 20 minutes. The slow work was not the cutover. The slow work was making the cutover boring.

The small things that still needed fixing during cutover

There were a few small first-boot reconciliations. I am writing them down mostly for future readers who want to follow a similar path:

imported data-pool mountpoints needed to be reconciled to the /mnt/... paths my clients and services already used;
encrypted datasets needed a better manual post-boot unlock helper, and the old TrueNAS API unlock flow needed a NixOS/OpenZFS successor;
Incus needed incus admin recover because the storage pool survived but the new OS had a fresh Incus database;
one unprivileged recovered container needed the matching subordinate UID/GID ranges declared for its passthrough idmap;
nas.local needed an explicit DNS record so my wildcard did not send it to a workload container;
the old PC job that backed up the TrueNAS API config needed to be removed;
the PC NFS mounts, Nix cache endpoints, and recovered Docker workload container all needed client-side validation.

Those findings went back into the cutover runbooks and the Nix config in a follow-up PR, so the notes are available for anyone who might benefit from them later.

At the end of the cutover, the machine was nas, the ZFS pools were healthy, the NFS mounts worked from my PC, the Incus containers were recovered, and the Docker workloads were running again. SMB authentication, guest access, Time Machine, and normal media browsing from macOS Finder worked after one final share-root permission fix. At that point the migration was working end to end: the pools were healthy and the services I actually use were back.

References #

I've gone full Nix: Proxmox to NixOS + Incus

Sat, 29 Nov 2025 13:00:00 +0100

I’ve gone full Nix: Proxmox to NixOS + Incus

I have officially decommissioned my Proxmox cluster. After years of running my homelab on Proxmox, starting with a single NUC and expanding to a multi-node cluster, I have migrated everything to NixOS running Incus.

Update 2026-06-26: This post reached the front page of Hacker News, and several readers correctly pointed out that Proxmox is not limited to clicking around in the web UI. It has qm, pct, config files, a REST API, Terraform providers, and Ansible workflows. My point is not that Proxmox cannot be automated. Even with that automation, state drift can still creep in when debugging means running one quick command, especially if an agent is allowed to execute imperative fixes that never make it back into the automation framework. It is that, for my setup, I wanted the reproducible configuration itself to be the source of truth. The thing I care about is not buttons versus commands, but whether I can rebuild the host from version-controlled text files and know that every important change is captured there.

From Skeptic to Believer #

I wasn’t always a Nix evangelist. In fact, I initially despised the language and its syntax. I couldn’t figure out how it worked, and I already had my own specific way of setting up my dotfiles. I used Dotbot for symlinking and a tool I wrote called dotbins for managing binaries. I didn’t feel like I required Nix for most of my tools. I used nix-darwin on my Mac for a long time, but only to specify Homebrew packages and application settings.

My true conversion happened when I bought my gaming PC, as described in my local LLM post. I initially installed Pop!_OS because I wanted to play games and absolutely wanted to avoid Windows. I got some games to work, but I constantly ran into NVIDIA driver issues that required running random, imperative commands to fix. I felt that was a bad solution because I could never reproduce those debugging steps later. Then I did the dumb thing of updating my NVIDIA drivers, not realizing that imperatively managing driver versions and repositories is a recipe for disaster, and got stuck in a GRUB boot loop. Frustrated, I installed NixOS, hoping its promise of atomic updates would solve this. The result was glorious. I never really believed that everything would be byte-for-byte equivalent until I migrated that system to a new disk. I didn’t clone the drive; I just applied my Nix configuration to a fresh install. It booted, I copied my data, and everything was identical. That was the moment it clicked.

The Friction of Imperative Systems #

Proxmox is fantastic software. It lowered the barrier to entry for me and taught me almost everything I know about virtualization, LXC containers, and ZFS. But fundamentally, Proxmox is built around clicking buttons. It is a GUI-first paradigm. While you can automate it with Terraform or Ansible, it often feels like fighting the tool. State drift is real. You change a setting in the UI to debug something, forget about it, and six months later your “infrastructure as code” is out of sync with reality. For a human maintaining a system manually, this is annoying. But when you introduce AI agents, this becomes a disaster for the operator. An agent running in “YOLO mode” might execute hundreds of imperative commands to fix a problem. It might succeed, but it leaves your system in an undefined, unreproducible state that no one—not even the agent—can fully understand or replicate later.

This friction manifests in hardware management too. On my HP EliteDesk, the Intel I219-LM network card has a known bug where it hangs with hardware offloading enabled. I vaguely remembered fixing this years ago on Proxmox, but I had forgotten the details. When I set up NixOS, I ran into the same issue: the network would randomly drop. This time, however, the fix isn’t a forgotten command run in a root shell history. It is a documented systemd service in my configuration. I added a comment explaining exactly why tso off gso off is needed, citing the forum threads. If I ever reinstall this machine, the fix applies automatically. On Proxmox, I would have had to rediscover this pain all over again.

Another example is my Intel NUC. Since my homelab lives behind my TV, I wondered if I could also use it as a Home Theater PC (HTPC). On Proxmox, this would have required passing the GPU through to a VM to get video output. But doing so would mean the Proxmox host loses access to the GPU entirely, meaning no local console if things go wrong. It was a strict trade-off: either a media player, or a debuggable hypervisor. I tried it, but it was so much trouble that I quickly reverted. With NixOS, I don’t have to choose. The host OS runs Kodi directly, giving me native hardware acceleration and video output. Simultaneously, incus runs in the background, hosting my containers. I get my HTPC and my server on the same metal, without the virtualization tax or the “headless host” limitation.

There is a deeper philosophical difference too. Systems like Proxmox or TrueNAS are designed as appliances. You aren’t supposed to run arbitrary commands on the host; installing packages or tweaking config files is discouraged because you might break the middleware or lose changes on upgrade. You are effectively locked out of your own hardware’s full potential. With NixOS, the host is fully mine. I can mess with it—installing Kodi, tweaking network drivers, running local LLMs—without fear. Because the state is declarative, it is 100% obvious and reproducible. I can break the host configuration and recover to a working state in seconds, even if the machine is running essential services.

The Agentic Multiplier #

I have written before about my shift towards agentic coding. In a world where AI agents execute tasks, CLI-first and declarative systems are king. An AI agent cannot reliably “click buttons” in a web interface to configure a VLAN tag or resize a disk. It needs text. It needs determinism.

By moving to NixOS, my entire infrastructure is defined in text files. This means my AI agents can read, understand, and even safely modify my infrastructure. Proxmox’s opaque database and UI-driven workflow were a black box to my agents. NixOS is an open book. If I want my agent to “deploy a Faster Whisper API server on port 9000 and expose it to the LAN”, it doesn’t need to navigate a “Services” menu, then a “Network” menu, and then a “Firewall” menu. It just writes a systemd service definition and adds networking.firewall.allowedTCPPorts = [ 9000 ]; in the same file. The agent can even verify the change was successful by checking the git diff or the active configuration. This is the infrastructure counterpart to the “agentic coding” revolution I’m living in.

A friend who has used NixOS for many years recently complimented me on how nicely my configuration was structured, especially for managing 8 different machines. The funny thing is, I haven’t written almost a single line of my configuration myself. I did all of it using agentic AI over many sessions. I have restructured and refactored it quite a few times, going from a single machine to 2 machines, and eventually to 9. The AI handles the heavy lifting of refactoring, ensuring that my PC, NUC, and HP all share common modules while keeping their unique personalities.

The Architecture: Incus & Simulation #

I still wish there was a purely “Nix” way to manage persistent, stateful LXC containers and VMs. There are projects like nixos-containers or microvm.nix, but they often lack the operational maturity or live-migration features of a robust hypervisor. Incus (the community fork of LXD) fills this gap perfectly. It gives me the “cattle” management of NixOS for the host, while allowing me to run “pet” legacy workloads (like my old Ubuntu containers or Home Assistant VM) in a stable, manageable environment. Crucially, Incus is entirely controllable via a clean CLI, making it a perfect citizen in my agentic workflow.

Over time, I had already migrated most of my legacy LXC containers—services like DNS or media managers—to declarative Docker Compose files running inside VMs. But I still had one major holdout: Home Assistant OS. I didn’t have a good alternative for it. I assumed I needed a dedicated appliance OS like Proxmox to run it reliably. I didn’t realize that with Incus on NixOS, running a full VM for Home Assistant is trivial. It’s just another QEMU process, but managed with the same ease as a container.

One neat thing I did was create Incus VMs that replicate the exact same configuration as my physical machines. Before I actually switched off my last Proxmox host, I was already confident that the full configuration worked because I had a virtual machine running the same setup. I just have a small file with overrides. Then I could validate that the virtual machine worked. This removed the fear of “nuking and paving” my physical servers.

The Migration #

Migrating was surprisingly straightforward, thanks to vzdump and qemu-img. Here is how I moved my 7+ years of digital history without losing data.

I also kept some detailed migration notes in my dotfiles. These notes should be sufficient for anyone trying to do the same. And if not, giving them to an AI (I used Gemini 3 Pro) will certainly allow you to figure it out.

1. Migrating LXC Containers #

For my LXC containers (running Docker, DNS, etc.), I essentially “teleported” them. I dumped the running container on Proxmox to a standardized tarball, copied it over, and streamed it directly into a fresh Incus container.

On Proxmox:

# Dump container 101 to a compressed archive
vzdump 101 --dumpdir /var/lib/vz/dump --mode suspend --compress zstd

On NixOS (using my migrate-lxc.sh script):

# Stream the backup into a new Incus container
./migrate-lxc.sh vzdump-lxc-101-*.tar.zst ubuntu-container

This script creates a fresh container, mounts its root filesystem, and overwrites it with the Proxmox dump. It handles the messy parts—like mapping UIDs and fixing machine-id—automatically.

2. Migrating Virtual Machines #

For VMs (like Home Assistant OS), it was a matter of converting the disk format. Proxmox uses LVM-thin or ZFS zvols; Incus is happy with QCOW2 or raw ZFS.

On Proxmox:

# Export the ZFS volume to a QCOW2 file
qemu-img convert -p -O qcow2 /dev/zvol/rpool/data/vm-100-disk-0 vm-100.qcow2

On NixOS (using migrate-vm.sh):

# Import the disk and create the VM
./migrate-vm.sh vm-100.qcow2 home-assistant

The Result #

I now manage my entire fleet—Host OS, networking, storage, and hypervisor configuration—from a single git repository. I get the best of both worlds: persistent, stateful services where I need them, and a reproducible, bulletproof host operating system. I can wipe my host machine, reinstall NixOS, run a restore script, and be back online in minutes. Best of all, I never have to remember which checkbox I clicked in a web UI three years ago. It is all in the code. And because it is code, my agents can help me manage it.

References & Configuration #

All configuration files mentioned in this post are available in my dotfiles repository.

flake.nix: The entry point defining my entire fleet (PC, NUC, HP).
networking.nix: The declarative fix for the Intel I219-LM network hang.
kodi.nix: NUC HTPC configuration running Kodi directly on the host.
ai.nix: Declarative service definition for Faster Whisper.
incus-overrides.nix: Overrides used to simulate physical machines inside Incus VMs.
migrate-lxc.sh: Script to import Proxmox vzdump archives into Incus containers.
migrate-vm.sh: Script to import QCOW2 disk images into Incus VMs.
PROXMOX_MIGRATION.md: Detailed raw notes and inventory of the migration.

Automating nightly NixOS builds with a local cache server to avoid local compilation

Sat, 29 Nov 2025 09:30:00 +0100

The CUDA compilation nightmare: how I solved 18-hour builds with NixOS

I recently migrated my entire fleet to NixOS (read my full migration story). If you use NixOS on unstable and have heavy dependencies, you’ve probably faced the dreaded “compiling from source” wall.

It starts innocently enough. You run nixos-rebuild switch, expecting a quick update. Suddenly, Nix decides it needs to compile PyTorch with CUDA support, or even “standard” desktop apps like Firefox and Thunderbird, from scratch.

Your fans spin up to 100%. Your RAM usage creeps up… 16GB, 32GB, 64GB. The UI starts to stutter, the mouse freezes, and your music stops. OOM Killer might save you, but often the system locks up completely. There is no way to recover; you have to physically hit the reset button. I recently installed earlyoom, which helps avoid the total system lockup by killing the process before the kernel panics. But that is a hollow victory. It is incredibly frustrating to have a build run for 5 hours across 24 cores at 100% usage, only to be killed right at the linking stage. The result is the same: I didn’t get my update, and I wasted a lot of electricity.

To fix this, you try the safe route: --cores 1 --max-jobs 1. It works, but now your update takes 18 hours. During that time, you can’t easily iterate on your system configuration because the Nix lock is held, and your CPU is busy.

I solved this by building a dedicated, automated build cache that effectively “pins” my entire fleet to a nightly snapshot. Here is how it works.

The Inspiration #

This entire setup was born from a “nerd snipe” just 5 days ago. I was comparing NixOS configurations with a colleague (see his config), and he mentioned he ran his own cache server. It instantly clicked. This was the missing piece. A dedicated cache server wouldn’t just be “nice to have”—it was the definitive solution to my OOM crashes and 18-hour compilation marathons.

The Architecture #

The core idea is simple: Never compile heavy packages on your daily driver.

I manage 3 physical NixOS machines (pc, nuc, hp), and updating them individually was becoming a chore. So, I set up a dedicated build server (an LXC container named nix-cache) that does the heavy lifting overnight for the entire fleet. Here is the high-level workflow:

1. The Build Server (`nix-cache`) #

Every night at 2 AM, a systemd timer triggers my auto-build service. This service:

Pulls the latest dotfiles from git.
Updates the flake.lock.
Crucially: Extracts the exact commit hash of nixpkgs it just locked.
Builds the system configurations for all my machines (pc, nuc, hp) using conservative settings (--cores 1) to ensure stability.
Saves the successful nixpkgs commit hash to a simple text file: /var/lib/nix-auto-build/<hostname>.rev.

2. The Binary Cache (Harmonia) #

The build server runs Harmonia, a lightweight binary cache. Since the builds finished successfully on the server, all the resulting binaries (the “store paths”) are now waiting in the server’s nix store, ready to be served over HTTP. You can see my Harmonia configuration here.

3. The Client (`nixcacheupdate`) #

This is the key mechanism. On my workstation, I don’t just run nix flake update (which would likely grab a slightly newer version of nixpkgs than my server built, causing a cache miss).

Instead, I use a custom script, upgrade-from-cache.sh, which I aliased to nixcacheupdate. This script:

SSHs into the build server and asks: “What commit did you successfully build for me last night?”
Reads the hash from the .rev file.

Runs the rebuild with a forced override:

nixos-rebuild switch \
 --flake .#pc \
 --override-input nixpkgs github:NixOS/nixpkgs/<REMOTE_COMMIT_HASH>

The benefits #

Guaranteed Cache Hits: My workstation is mathematically forced to use the exact same package inputs as the server. Nix calculates the hash, sees the match in the Harmonia cache, and downloads the binary instead of compiling it.
No More OOMs: The heavy lifting happens on a machine where I don’t care if the build is slow or memory-intensive.
Iterate Faster: I can still tweak my local configuration files (configuration.nix, home.nix). Since changing a config file doesn’t alter the source code of PyTorch, I still get the heavy binaries from the cache, while only locally building the tiny activation scripts.

The Result #

What used to be a frightening, system-freezing event is now a boring, fast download. I get the stability of a frozen release with the freshness of nixos-unstable, minus the compilation heat.

If you are struggling with CUDA builds on NixOS, stop compiling locally. Let a server take the heat.

References & Configuration #

All configuration files mentioned in this post are available in my dotfiles repository.

auto-build.nix: The systemd service that orchestrates nightly builds.
harmonia.nix: Configuration for the Harmonia binary cache server.
upgrade-from-cache.sh: The client-side script aliased to nixcacheupdate.
aos/nix-config: My colleague’s configuration that inspired this setup.

Switching from Ollama to llama-swap + llama.cpp on NixOS: the power user's choice 🦙

Sat, 29 Nov 2025 00:00:00 +0000

Table of contents

The draft I never published #

I started writing this blog post four months ago. Back then, I was frustrated because a specific model I wanted to try—gpt-oss-20b—was completely broken in Ollama. I went down the rabbit hole, set up llama-swap and llama.cpp, got the model working perfectly, wrote half this post… and then I stopped.

Why? Laziness. Once the novelty of that specific model wore off, I drifted back to Ollama. The real convenience of Ollama was never the ollama run command—I almost exclusively interact with my models via the API using my own tools like agent-cli. The convenience was ollama pull. Being able to grab a model with one command and have it “just work” was great.

But as a NixOS user, this imperative convenience is actually a flaw. I want my system to be declarative. I want my model configuration committed to git, not hiding in a local database. So I let convenience win for a while, but it always felt wrong.

But last month, that changed. I upgraded my hobbyist AI rig by adding a second RTX 3090. Suddenly, I had 48GB of VRAM and a whole new set of problems—and possibilities. I wanted to run massive models (like gpt-oss-120b or Qwen-3-VL-32B with a huge context window). I needed to balance layers between two GPUs. I needed to spill specific parts of the model into system RAM without crashing.

When you start pushing hardware to its limits, “magic” abstractions like Ollama stop being helpful and start getting in the way. I realized that llama.cpp isn’t just a backend; it is the place where development happens. If you want to use split-mode inference, granular CPU offloading, or the latest quantization tricks immediately, there is no alternative.

So, I dusted off this draft, updated my Nix configuration, and I’ve been running llama-swap full-time for a month. I’m finally ready to vouch for it.

The original trigger: broken models #

Four months ago, my frustration stemmed from compatibility. I was trying to run gpt-oss-20b, a model that required specific chat templates and GGUF handling that Ollama simply broke.

The maintainer of llama.cpp, Georgi Gerganov, explained it perfectly regarding the gpt-oss debacle: Ollama forked the ggml inference engine to rush out “day-1 support” for gpt-oss without coordinating with upstream. The result? Their implementation was not only incompatible with standard GGUF files but also significantly slower and unoptimized.

Benchmarks at the time showed Ollama had significant performance overhead—often 20-30% slower inference speeds compared to running llama.cpp directly.

This pattern means Ollama will always be behind llama.cpp. Meanwhile, I’ve had a pull request sitting unreviewed for over a month that fixes their broken OpenAI API. Compare that to llama.cpp: I submitted a PR there and it was merged in less than an hour. If you want the latest features now (and a responsive community), you go to the source.

The catalyst: dual GPUs and control #

With a single GPU, you usually either fit the model or you don’t. With two GPUs (and 64GB of system RAM), things get interesting. You enter the territory of heterogeneous inference.

Ollama tries to handle splitting automatically, but I found myself fighting it. I wanted to say: “Put 40 layers on GPU 0, 40 layers on GPU 1, and keep the KV cache here.” Or: “This model is slightly too big for VRAM; I want to offload exactly the last 10% to system RAM.”

With llama.cpp directly, this is just a command line flag. With llama-swap, I can define this behavior per-model in a config file:

"gpt-oss:120b":
 cmd: |
 ${pkgs.llama-cpp}/bin/llama-server
 -hf ggml-org/gpt-oss-120b-GGUF
 --port ${PORT}
 --ctx-size 65536
 --split-mode layer
 --tensor-split 3,1.3 # Tuned balance between my GPUs
 --n-cpu-moe 15 # Offload 15 experts to CPU RAM
 --threads 8
 --chat-template-kwargs '{"reasoning_effort": "high"}'

This level of granularity turned my rig from a “black box” into a tunable instrument. The --tensor-split and --n-cpu-moe flags allow me to run a 120B parameter model that technically shouldn’t fit, by carefully balancing the load across both GPUs and system RAM.

API compatibility and the “switch” #

One of the reasons I hesitated to switch was the fear of breaking my existing workflows. I use agent-cli and other custom Python scripts daily. But it turned out to be a non-issue. Both Ollama and llama-swap (wrapping llama-server) expose an OpenAI-compatible API.

Once I adopted the pattern of setting a custom OPENAI_BASE_URL in my projects, the backend became irrelevant. My tools don’t care if they are talking to api.openai.com, a local Ollama instance, or llama-swap. They just send JSON and get JSON back. This standardization meant I could swap the entire inference engine underneath my applications without changing a single line of their code.

My NixOS setup #

Since I manage my entire system with Nix, setting up llama-swap and pinning llama-cpp to the absolute bleeding edge was surprisingly elegant. I even wrote an auto-update script to ensure I’m always on the latest commit.

This is where my NixOS build cache shines. Unlike Ollama, which releases every few weeks, llama.cpp can have multiple releases a day. My cache server runs this update script automatically, builds the new binaries, and serves them to my workstation. I get the absolute latest performance improvements without ever compiling source code locally.

Compiler flags matter #

One critical lesson I learned: default builds matter. Initially, I was getting a measly 8 tokens/sec on gpt-oss:120b. It turned out the default llama.cpp package wasn’t enabling BLAS or native CPU optimizations, crippling the layers I offloaded to system RAM.

By overriding the package to enable blasSupport and passing -DGGML_NATIVE=ON (as seen below), performance jumped to 50 tokens/sec. This is another reason why I prefer managing this via Nix: I can enforce these compile-time flags declaratively.

Here’s my complete configuration (see package-overrides.nix and ai.nix):

# Override llama-cpp to latest version with CUDA support
llama-cpp =
 (pkgs.llama-cpp.override {
 cudaSupport = true;
 rocmSupport = false;
 metalSupport = false;
 # Enable BLAS for optimized CPU layer performance (OpenBLAS)
 blasSupport = true;
 }).overrideAttrs
 (oldAttrs: rec {
 version = "7205";
 src = pkgs.fetchFromGitHub {
 owner = "ggml-org";
 repo = "llama.cpp";
 tag = "b${version}";
 hash = "sha256-1CcYbc8RWAPVz8hoxKEmbAgQesC1oGFZ3fhfuU5vmOc=";
 leaveDotGit = true;
 postFetch = ''
 git -C "$out" rev-parse --short HEAD > $out/COMMIT
 find "$out" -name .git -print0 | xargs -0 rm -rf
 '';
 };
 # Enable native CPU optimizations (AVX, AVX2, etc.)
 cmakeFlags = (oldAttrs.cmakeFlags or []) ++ [
 "-DGGML_NATIVE=ON"
 ];
 # Disable Nix's march=native stripping
 preConfigure = ''
 export NIX_ENFORCE_NO_NATIVE=0
 ${oldAttrs.preConfigure or ""}
 '';
 });

# llama-swap from GitHub releases
llama-swap = pkgs.runCommand "llama-swap" { } ''
 mkdir -p $out/bin
 tar -xzf ${
 pkgs.fetchurl {
 url = "https://github.com/mostlygeek/llama-swap/releases/download/v175/llama-swap_175_linux_amd64.tar.gz";
 hash = "sha256-zeyVz0ldMxV4HKK+u5TtAozfRI6IJmeBo92IJTgkGrQ=";
 }
 } -C $out/bin
 chmod +x $out/bin/llama-swap
'';

# Configure llama-swap as a systemd service
systemd.services.llama-swap = {
 description = "llama-swap - OpenAI compatible proxy with automatic model swapping";
 after = [ "network.target" ];
 wantedBy = [ "multi-user.target" ];
 
 serviceConfig = {
 Type = "simple";
 User = "basnijholt";
 Group = "users";
 # Point to your declarative config file
 ExecStart = "${pkgs.llama-swap}/bin/llama-swap --config /etc/llama-swap/config.yaml --listen 0.0.0.0:9292 --watch-config";
 Restart = "always";
 RestartSec = 10;
 
 # Environment for CUDA support
 Environment = [
 "PATH=/run/current-system/sw/bin"
 "LD_LIBRARY_PATH=/run/opengl-driver/lib:/run/opengl-driver-32/lib"
 ];
 };
};

What I gained (and lost) #

Advantages of llama-swap + llama.cpp #

Heterogeneous compute: Perfectly balance loads between GPUs and CPU RAM.
Bleeding edge: I use features (like new quantization types) the day they land in llama.cpp.
Zero idle VRAM: Models unload completely when not in use.
Transparent operation: No hidden prompts or “helpful” formatting that breaks complex tasks.

The trade-off #

The main trade-off is “laziness.” You can’t just ollama pull new-model and have it appear. You have to find the GGUF on HuggingFace, decide on the quantization (Q4_K_M? Q6_K?), and add 5 lines of YAML to your config.

But honestly? That’s a feature, not a bug. It forces you to understand what you are running. It stops you from running a quantized model that is too stupid for your task just because it was the default. And for a NixOS user, putting that configuration into code instead of a hidden database is exactly how it should be.

Conclusion: maturing as an AI hobbyist #

My switch back to llama-swap wasn’t about fixing a bug; it was about outgrowing a tool. Ollama is the “starter bike” of local AI—fantastic for getting rolling. But when you add a second GPU, start caring about tokens per second, and want to maximize every gigabyte of your 48GB VRAM buffer, you need a manual transmission.

For anyone else building a hobbyist multi-GPU rig: stop fighting the abstraction. Go to the source. The control is worth it.

Check out my NixOS configuration for the complete setup!

Coding from my phone 📱

Mon, 20 Oct 2025 00:00:00 +0000

I am dictating this post at ten kilometers altitude on a flight home from Mexico, Blink Shell open on my iPhone, while an agentic assistant on my NixOS machine writes an initial draft for this post. Admittedly, I’m a little addicted to agentic coding: when an idea pops up, I want to jump on it right away—even if that means doing it from a cramped seat in the sky. This is the workflow I’ve arrived at after trying many alternatives, it extends my agentic coding write-up and the self-hosted AI obsession in my local AI journey. It’s my backup plan for those moments when a computer simply isn’t nearby—the rest of the time I’m still at a keyboard like any other developer. This is in no way a replacement for a proper computer/laptop workflow but gets the job done when needed. It’s a personal, mostly open-source stack built around agent-cli—my local-first voice layer. Here’s how it works: I speak into my phone, the audio goes to FasterWhisper for transcription, Ollama cleans up the text, and it lands on my clipboard ready to paste. My iOS Shortcut handles this through a small HTTP interface that agent-cli exposes. The coding agent itself still uses the best proprietary model I can access.

Table of contents

1. Why Phone Coding Works Now #

For years I used iSH (a full Alpine Linux emulator) with SSH to hop into servers from my phone, but dropped connections were a constant frustration (Mosh fixes this, more on that later). Because coding on a phone keyboard is terrible, I kept it to tiny configuration tweaks or one-off fixes of a few characters. Agentic tools changed that: with a CLI coding agent, I don’t need to type the code—I describe the change, review the patch, and run it. That made meaningful work on the phone possible for the first time, for those moments when a computer isn’t around.

I tried VS Code in the browser, bounced between iSH and Terminus for SSH, and even lived inside a handful of in-browser terminal clients. I also spent time with mobile companions like Happy and Omnara, both designed to mirror Claude Code sessions on the phone, but they still felt like extra steps between me and my actual terminal. That friction pushed me toward a phone‑ready, self‑hosted workflow that still gives me raw SSH access to my own machine when the computer is out of reach.

This post is based on the way I develop software today. Your mileage may vary, but if you also care about privacy, open tooling, and reproducible environments, I think there are useful pieces here.

2. Constraints and Trade-offs #

This setup grew organically over time. What emerged are less strict requirements and more happy accidents that turned into core benefits:

Single trust boundary: Only the model provider (OpenAI) sees code context; audio and automation stay local, and I avoid any additional third‑party relays.
Resilient sessions: Connections should survive sleep and spotty networks.
Voice-friendly: Dictation should be accurate enough that I can trust it.
Reproducible config: The entire stack must live in my dotfiles and NixOS configuration.

To give you a sense of what I tried, here is the short comparison that convinced me to roll my own:

Approach	Pros	Why I moved on
VS Code in the browser	Familiar editor UI	Needs a steady connection and still lives outside my dotfiles comfort zone
iSH / Terminus SSH	Works without extra infrastructure	Laggy, no Mosh, and awkward keybindings
In-browser terminals	Instant access from anywhere	Poor copy/paste ergonomics and flaky mobile keyboards
Happy (Claude Code companion)	Push notifications, encrypted mobile UI for Claude Code	Requires wrapping every session with a separate CLI and still abstracts away my shell, and was quite buggy for me
Omnara (agent command center)	Centralizes Claude Code/Codex sessions with terminal replay	Proxies via their servers; I already trust OpenAI for the model and don’t want another third‑party handling my code

The stack below gives me the resilience of Mosh, the ergonomics of Zellij, and full control over the AI layer.

3. Layer 1: WireGuard From the Router #

WireGuard is a modern VPN that lets me securely access my home network from anywhere. I terminate WireGuard on my router so every device in the house (and on the road) can dial home with the same config. (If you want something easier to set up without opening ports, Tailscale is a more user-friendly option.)

Server: ASUS XT8 router with WireGuard enabled via the router UI.
Client: The WireGuard iOS app with On-Demand rules so the tunnel flips on whenever I am off trusted Wi-Fi.
DNS: All mobile sessions resolve through my home DNS, so git.nijho.lt and internal services resolve instantly.

This gives Blink a local-LAN address for my desktop nixos.

4. Layer 2: Blink Shell + Mosh for Durable Sessions #

What is Mosh? It’s like SSH, but it stays connected when the network changes or the phone sleeps, and it feels faster on bad connections.

Blink Shell is my daily driver on iOS because it pairs beautifully with Mosh and has great keyboard ergonomics.

I launch sessions using mosh bas@nixos -- zellij attach --create phone (Zellij is my terminal multiplexer—more on that in the next section).
Mosh smooths over spotty LTE and keeps my session alive when the phone sleeps.

Mosh keeps the session responsive across flaky networks and sleep.

5. Layer 3: Zellij Layouts #

I use Zellij as my terminal multiplexer for mobile work.

I stick to the defaults and use a couple of predefined layouts (e.g., a “phone” layout with editor + shell panes).

Ergonomics matter even more on a glass keyboard, so I lean on a few shell helpers:

zoxide means I can jump between repos with z foo instead of pecking long paths.
Single-character aliases like p for pytest and gs for git status live in configs/shell/10_aliases.sh, keeping command entry to a minimum.
Autocompletion plugins—zsh-autosuggestions and zsh-syntax-highlighting—are wired up in configs/shell/70_zsh_plugins.sh, so I get inline hints and colour cues while typing.

I break these tricks down in more depth in Terminal Ninja, explain how I sync them with my Dotfiles, and even package binaries like zoxide with Dotbins.

6. Layer 4: `agent-cli` Server #

agent-cli runs as a small server on my NixOS machine. In this workflow I only use transcribe from iOS Shortcuts: the phone records audio, the server transcribes and cleans it up, and I paste the text. (Agent-CLI has other features like autocorrect and chat, but I don’t use them for this workflow.) On this machine I run a long‑lived agent-cli user service (systemd) so Shortcuts can POST audio to it and get cleaned text back—ready to paste into Codex CLI. The service is defined in my dotfiles—see: configs/nixos/modules/user.nix (agent-cli service).

For more background on why I built it and how I use it in practice, see Local AI Journey (§3) and Agentic Coding.

The models and services run on the same box:

FasterWhisper via faster-whisper-server for high‑accuracy streaming transcription.
Ollama for on‑device rewrite/cleanup before sending prompts to the coding agent.

My iOS Shortcut dictation workflow is slower than Apple’s built-in dictation, but the accuracy is much better—and that matters more.

For the actual coding agent, I use a fork of Codex CLI with OpenAI’s gpt-5-codex-high model (here no open-source solution matches the frontier).

7. Layer 5: The iOS Shortcut Pipeline #

My iPhone’s Action Button runs a Shortcut that records audio and sends my voice to agent-cli on my NixOS machine for transcription and cleanup.

For the full recipe, see the iOS Shortcut Guide: agent-cli/iOS_Shortcut_Guide.md.

In short: I press the Action Button, the Shortcut records a snippet, sends it to agent-cli, and copies the cleaned text to my clipboard so I can paste it into Codex CLI.

The whole loop finishes fast enough that I can capture intent by voice and paste it into Codex CLI without typing.

8. Workflow in Practice #

Here’s the typical flow:

WireGuard auto-connects when I’m off home WiFi
Open Blink—my Mosh session is still alive from last time, Zellij panes intact
Dictate the change, paste into Codex CLI, review the agent’s response
Check diffs in an adjacent Zellij pane or different Blink tab with separate Mosh session
The agent commits and pushes to a branch, then opens a PR via gh for me to review and merge

On the phone, I aim for the smallest practical edits and initial implementations. Even on personal repos, I still open a PR—sometimes prompting alone gets me to a merge‑ready result. For open‑source with real users, I finish on the computer with a careful review and any final edits.

9. Conclusion and Further Reading #

This phone setup extends my Agentic Coding workflow to mobile. I connect to the same Zellij session on nixos, so when an idea hits, I pick up exactly where I left off—same environment, same context, no setup. Everything runs on my hardware (voice processing and automation) except the coding model itself. It’s the most effective mobile workflow I’ve had so far. It’s mostly open‑source not out of dogma, but because those tools are the best options for my needs. The one exception is the coding model: there’s no true open equivalent right now, and it makes no sense to buy 20× H100s just to self‑host a frontier model even if such a high quality model would be available open-source.

If you’re curious to dive deeper, here are a few related posts:

Agentic Coding
Dotfiles: A Practical, Cross-Platform Terminal Setup
Local AI Journey: RTX 3090 Edition
Terminal Ninja
Using LLMs Effectively (already outdated at this point!)

I’d love to hear what self-hosted tricks you’re using for mobile development—reach out if you adapt this stack or build something wild on top of it.

How a 3090 turned into two local‑first AI projects 🎮

Mon, 14 Jul 2025 00:00:00 +0000

Table of contents

1. Introduction: The Unplayed Games #

I’ve wanted to dabble with local AI for a long time, mainly because of the possibilities it unlocks. Local AI allows me to use all of my data—financial records, personal messages, location history, even transcribing everything I say—without the privacy concerns of sending it to a cloud provider. This level of deep, personal data analysis is only something I’d be comfortable with on my own hardware.

However, I couldn’t justify buying a powerful GPU just for an imagined AI project that I might never really start. Then it hit me: I could also use it for gaming (duh). Because I really enjoyed The Last of Us series, I thought, “Okay, why not play The Last of Us on PC?” This was the push I needed. So, a couple of weeks ago, I convinced myself to buy someone’s old gaming machine. After many deep research sessions with ChatGPT about the best bang-for-the-buck consumer-grade GPU, I landed on an NVIDIA 3090. I found an (old) beast of a machine, and my initial idea was set.

Click here to see the full system specs ($1350 for all this!)

GPU: ASUS ROG STRIX RTX 3090 OC (24GB)
CPU: AMD Ryzen 9 3900X (12-Core)
Motherboard: ASUS ROG Crosshair VIII Hero
Power Supply: Be Quiet! 1200W Dark Power Pro
RAM: 32GB G.Skill Trident Z Royal DDR4
Storage: 2TB Intel M.2 NVMe SSD
Case: Corsair 4000D Airflow
CPU Cooler: Thermalright Peerless Assassin 120

Fast forward four weeks. I haven’t played a single minute, but my machine is now indexing 200 GB of email archives and facing a serious performance bottleneck I’ll get to later. That “gaming” machine has been humming away, but not rendering virtual worlds. It’s been training, transcribing, and thinking.

I’m writing this post simply because I am so enthusiastic about local AI. I’ve found that local models are surprisingly powerful for specific use cases. Of course, they aren’t going to replace Gemini or Claude for coding, but for many other applications, they work incredibly well. Lately, I’ve been so obsessed with this that I’m sleeping significantly less and spending all my free hours working on it.

2. The Rocky Road to a Working Setup #

Before I could even get to the fun AI stuff, I had to wrestle with the machine itself. I started with Pop!_OS, which is supposed to be a great out of the box solution that “just works”. However, it’s been a long time since I used Linux on the desktop; my experience is almost exclusively with servers. I quickly installed some of the wrong NVIDIA drivers and ended up debugging stuff in Grub. I got frustrated with the different commands I had to run to set up everything and realized that reproducing this system was going to be a massive pain. Since I’ve grown to love Nix on my Mac with Nix-Darwin, I decided to switch the whole system to NixOS. This has been an amazing decision. Even though there are many pain points with Nix, I think it is well worth it. For those interested, you can see the complete setup, including all the local AI services, in my NixOS configuration on GitHub.

3. The First Victory: `agent-cli` 🐍 #

With a stable system, I quickly got started with a few scripts. These soon evolved into my first real project of this adventure: agent-cli.

agent-cli is a collection of local-first, AI-powered command-line agents that run entirely on your machine. The core philosophy is privacy; your data never leaves your computer. It’s designed for seamless integration with system-wide hotkeys. I’m now using it constantly.

Instead of typing, I now dictate almost everything. A quick hotkey, and agent-cli transcribe turns my speech into text. Another hotkey, and agent-cli autocorrect cleans it up using a local LLM running on Ollama. It has genuinely streamlined my workflow. For example, I can pipe the tools together, like correcting text from my clipboard and then having it spoken aloud: agent-cli autocorrect | agent-cli speak.

The toolkit includes:

autocorrect: Fixes grammar and spelling.
transcribe: Uses a local Whisper model for speech-to-text.
speak: Converts text to speech with a local TTS engine.
voice-edit: A voice-powered clipboard assistant.
assistant: A hands-free voice assistant using a wake word.
chat: A conversational AI with tool-calling capabilities.

It’s been a fantastic success in my personal workflow, and it’s all open-source for you to try. Contributions and ideas welcome!

Once I trusted my toolchain, the obvious next step was to feed all my data into it.

4. The Grand Ambition: `AIBrain` 🧠 #

My second package, AIBrain, is a much larger endeavor. The vision is to create a “life-OS”—a private, local AI that processes all my personal data (emails, calendar, files, photos, messages, browser history, location data) to generate summaries and answer questions, eventually allowing for queries like:

aibrain ask "When did I get into landscape photography?"
aibrain ask "How many countries did I visit between 2015-2025?"

This project led me to explore the landscape of agentic frameworks. I started with CrewAI, but I quickly found that despite the hype and funding, it had some painful problems (specifically this issue where things just fail without any helpful error messages). I decided to switch to LangGraph, which felt more robust. I also chose to not use PydanticAI (which I used in agent-cli) because its Ollama support is just a wrapper around the OpenAI-compatible API, which doesn’t expose all the useful options (I even opened a PR in Ollama to make its OpenAI compatible API more feature-complete, but it has not been merged yet).

The core infrastructure of AIBrain is ready. It can index emails, calendar events, and files in various formats (PDF, Excel, PowerPoint, images). However, I’ve hit a significant challenge.

5. Conclusion: The Journey Continues #

This past month has been a whirlwind. I set out to play games and ended up with two open-source AI packages and a much deeper appreciation for the power sitting in a consumer-grade GPU. It’s incredible what you can achieve with local hardware today.

agent-cli is a finished, polished tool that I use daily. AIBrain is a much bigger dream, and I’m still figuring out how to solve the performance puzzle to make it truly practical. Next up: smarter chunking with LangGraph, vector-DB swap-outs, and GPU scheduling tricks.

This experience has reinforced my passion for local-first software and open-source development. The future of personal, private AI is being built right now, not just in large corporate labs, but in the homes of hobbyists tinkering with their gaming rigs.

⭐ If this resonates, star the repos & drop issues/questions.

And who knows, maybe I’ll even find the time to play ‘The Last of Us’.

Nixos | Bas Nijholt

Friend-to-friend ZFS backups, version two

Why ZFS #

What we wanted #

ZFS already had most of it #

Rehearsing the permissions in a VM #

A gate that speaks syncoid #

Zones: a second wall in the kernel #

Who stops what #

What it cannot hide #

Setting it up #

Knowing when it stops working #

Restoring #

The first real push #

Tested like everything else #

AI scope creep or useful? #

References #

My declarative multi-machine homelab, explained

The big picture #

The machines, and NixOS #

What NixOS brings #

Deploying NixOS with comin #

What runs on it #

Containers and compose-farm #

What a container is #

Upstream’s Compose file, my folders #

Why I wrote compose-farm #

The front door: a reverse proxy #

What a reverse proxy brings #

Services announce themselves #

Services on other machines #

HTTPS and certificates #

What a certificate is #

Proving I own the domain without opening my network #

DNS: the phone book #

What DNS does #

Three answers for the same name #

Four ways in #

1. The home network #

2. WireGuard on the router #

3. Tailscale, via my own Headscale #

Why both? #

The Tailscale DNS problem #

4. The internet #

Keeping it safe: one allowlist #

DNS is not a lock #

Who is knocking #

Private by default #

Trusting my own network #

Sharing with friends and family #

Why not Cloudflare Tunnel or Tailscale’s own server? #

Declarative everything #

Why it matters #

Putting it together: adding a service #

Lessons learned #

References #

The last btrfs machine: migrating my PC to ZFS

Why touch a working machine #

The backup that wasn’t #

What the audit found #

The plan: one shared disko module #

Rehearsing in a VM, again #

The recovery kit #

Trusting the backup enough to wipe the disk #

The fifth question #

Restoring in stages #

Slimming down first #

The cutover #

References #

Migrating from TrueNAS to NixOS without losing data after the build-system rug pull

The itch before the last straw #

The last straw #

Why this was scarier than Proxmox #

Giving an agent the keys, carefully #

The NixOS scaffold #

The disko problem #

Rehearsing the dangerous part in a VM #

The parts that are not just storage #

Secrets stay out of the repo #

Where this leaves TrueNAS #

1. The Build Server (`nix-cache`) #

3. The Client (`nixcacheupdate`) #

6. Layer 4: `agent-cli` Server #

3. The First Victory: `agent-cli` 🐍 #

4. The Grand Ambition: `AIBrain` 🧠 #