Hardware | Bas Nijholt

Self-hosting AI does not save money, and I do it anyway

Fri, 02 Oct 2026 00:00:00 +0000

I have had this debate many times, so I am finally writing it down.

Whenever I say that self-hosting AI does not save money, people hear that I am against self-hosting. That is not the point at all. I am a massive fan of open-weight models. Qwen3.8 27B runs on two RTX 3090s at home (and 15 more models), and my phone dictation goes to Qwen3-ASR on the same machine. I think open-source AI is the best thing since sliced bread.

I don’t pretend it saves money. I do it for fun, for sovereignty, and for privacy, which I come back to at the end.

All benchmark numbers and prices below come from Artificial Analysis and OpenRouter as of September 30, 2026. I did the math for the hardware I own, and for the counterarguments I hear most. All assumptions are listed at the end.

Benchmarks are not everything, and a high score does not reliably predict how a model does on real work. They are still the best we have: all models are benchmaxed, tuned to do well on the popular benchmarks, so the scores at least work as a reference frame for comparing them with each other.

Update, October 2, 2026: after the discussion on r/LocalLLaMA, I added section 5 on Macs, DGX Sparks, and AMD AI Max machines.

Table of contents

1. The model I love, and the one that beats it on price #

My favorite local model right now is Qwen3.8 27B. It came out in August and it is Apache-2.0. People often say models like this run on a single gaming GPU, but that is only true after quantizing them. At full precision, Qwen3.8 27B needs about 54 GB of memory. Quantized to about 4 bits per weight, it fits on one 24 GB card, at some cost in quality. I still split it across both of my 3090s, because the second card leaves room for a larger context window.

I compare it with OpenAI’s GPT-6 Luna, the cheap tier released on September 22. Both models let you choose how long they think, but the settings do not mean the same thing for both. Luna’s token use grows more than 20 times from its lowest setting to its highest, while Qwen’s barely changes. Qwen on low already writes more tokens than Luna on xhigh.

Average output tokens per Intelligence Index task at each reasoning setting. Qwen3.8 27B has no high or max setting. Hover for scores.

So the name of a setting says little on its own. I compare both models at xhigh, the highest setting Qwen offers, and count the tokens each one actually uses. The extra tokens Qwen needs are part of what I am measuring. There, Qwen scores 33.7 on the Artificial Analysis Intelligence Index and Luna scores 34.6. For reference, Claude Opus 4.6, a frontier model from February, scores 31.9. That is an amazing feat in itself. When Opus 4.6 was the best model we had, I never thought that within the same year I could run essentially that level of intelligence in my own house.

Artificial Analysis also publishes how many tokens each model used to run the index, and what that cost. So the question is simple: what does it cost to run the whole benchmark once?¹

Cost of one run of the Artificial Analysis Intelligence Index. The two bars for hardware at home are electricity only: measured on my 3090s, estimated for the RTX PRO 6000.

Luna runs the whole benchmark for $67. Qwen at full precision, through the cheapest provider that does not keep your data (ZDR), costs $619. Two things cause that gap: providers charge almost four times as much per output token for Qwen, and Qwen generates almost three times as many tokens to get the same work done.

The second bar is my own machine: the electricity alone for running Qwen on my GPUs costs more than Luna’s entire bill.

The third bar is what it takes to match the API’s full precision at home: an RTX PRO 6000, a workstation card with 96 GB of memory, enough for the full model.² Its electricity alone costs twice Luna’s bill, and one run takes more than seven weeks. The card by itself sells for $10,000 to $20,000, so with a computer around it, you are paying for a pretty nice car. Buy a few of them to run agents in parallel, and you are building a small data center of your own.

The bars for hardware at home are electricity only, so they are the lowest these runs can cost: the cards come on top, and how much depends on how busy you keep them, which is what section 4 is about.

What I care about is the cheapest way to get answers of Qwen’s quality, from whichever model gives them.

2. “But my GPUs are already paid for” #

This is the argument I hear most, so assume the hardware is free and only the electricity counts.

I measured my own machine. It runs Qwen in vLLM, split across both cards, at about 107 tokens per second.³ The cards are capped at 270 W each,⁴ and the whole machine draws about 700 W. It then needs four weeks, running day and night, to get through the benchmark once. I also give my quantized copy the score Artificial Analysis measured through Alibaba’s API. I have not rerun the benchmark on it, and anything lost to quantization makes my machine look worse.

Electricity for one benchmark run of Qwen3.8 27B on my two 3090s, by electricity price. The dashed line is what Luna charges for the same work, with OpenAI’s hardware and margin included. Hover for values.

Above 14.5 cents per kWh, my electricity alone costs more than Luna’s API bill. Below it, I save a few dollars per run and wait four weeks instead of a few hours.

Those four weeks matter more than the few dollars. An API takes hundreds of requests at once, so even one benchmark run can finish in an hour or two. My setup works on one conversation at a time: when I sent it eight requests at once, seven of them waited in line.

Wall-clock time for one benchmark run. The API times use Artificial Analysis’s measured time per task for Luna; the parallel case assumes 100 requests at once. Log scale.

This is one more reason my coding agents run on APIs. When I run several agents in parallel, each of them should be as fast as if it were alone.

3. Why datacenters are so much better at this #

When my 3090s generate a token, they read all of the model’s weights from memory to produce one token for one conversation. Most of the chip’s compute sits idle while it waits for memory. A datacenter GPU reads the same weights once and produces a token for hundreds of conversations at the same time. This is called batching, and it is where most of the efficiency comes from.

Artificial Analysis measures this with AgentPerf, which replays real coding-agent sessions against datacenter hardware. It counts how many agents a system can serve while each one still gets at least 60 tokens per second.

Coding agents served per kW of accelerator power, each getting at least 60 tokens per second, from AgentPerf. The datacenter systems all run DeepSeek V4 Pro. The 3090 bar is my machine running Qwen3.8 27B, a much smaller model, with GPU power only, so the comparison favors my machine.

The rack of 36 GB300s serves 20 times more agents per kW than the 8-GPU H200 server. Part of that is newer chips, and part is software: NVIDIA tuned the B300 and GB300 setups itself, while Artificial Analysis configured the H200 one. The B300 and the GB300 share both the chip generation and the software. Most of the gap between those two bars comes from the rack itself: 36 GPUs on one fast interconnect keep more than a thousand agents in flight at once. My two 3090s serve one conversation at a time.

4. “But I have solar panels” #

Solar power is not free. With net metering, every kWh my GPUs burn is a kWh that does not get credited at the retail price. Without net metering, it is a kWh I do not sell, unless the surplus would go to waste anyway, which is the free-power case below. And the GPUs also run at night.

But fine, say the electricity is free and the only cost is the hardware. I was lucky and bought my two 3090s for about $750 each,⁵ and I write them off over three years. The GPUs only save money while they do work I would otherwise pay an API for, and while they sit idle they save nothing. So the question is how many hours a day they have to be busy before they pay for themselves.⁶

To make this fair to open models, imagine an open model that is exactly as efficient as Luna: the same quality from the same number of tokens, sold at Luna’s price.

How many hours a day my two $750 RTX 3090s have to be busy, every day for three years, before they cost less than a ZDR API. The host PC is not counted.

With free power, the GPUs pay for themselves if they are busy 4 hours a day, every day for three years. At 18 cents per kWh, they need 8.5. That uses what I paid for the cards; at today’s price of about $1,500 each, the 8.5 hours become 14.

And API prices keep falling. GPT-6 Luna costs 58% less per output token than GPT-5.6 Luna did in July. If the price halves again, no amount of use pays off at 18 cents per kWh. If it halves twice, the API costs less than my electricity alone.

5. “But my Mac, DGX Spark, or AI Max does save money” #

Several readers told me their machines do save them money: a MacBook Pro with an M5 Max, an NVIDIA DGX Spark, and a computer with AMD’s Ryzen AI Max+ 395. If you would have bought the machine anyway, that is true, for the reason in section 2: their electricity costs little compared with the API. Counting the hardware is a different story.

With 128 GB of memory, all three can hold a better model than my 3090s: Qwen3.8-Flash-Next, a sparse model with 180 billion parameters, of which only 6 billion work on each token. It scores 39.8, close to GPT-6 Luna on its highest setting (38.1), although at the Q2 a reader uses on their Mac it probably loses a few points.⁷ So how many years would each machine have to run nonstop before it pays for itself?

Years of nonstop generation before each machine pays for itself, compared with the most efficient API model at a similar score. Each machine runs the best model it can hold.

Even with free power, the 128 GB machines need six to ten years. The AMD machine draws only about 120 W, but it is slow too: one benchmark run of Qwen3.8 27B takes it four and a half months, and almost as much energy as my 3090s. The RTX PRO 6000 runs the same model about three times as fast but costs about twice as much, so it needs six to nine years. My 3090s pay off in under two years with free power, which is why the 3090 is still the value king, but at 18 cents per kWh they never do: the electricity for one run already costs more than Luna charges for it.

Ten years is a long time in AI. Better and more efficient open models will run on the same machines, but the frontier moves at the same pace, and API prices keep falling. Hardware gets more efficient too, in datacenters and at home, so a machine bought today also ends up competing with next year’s machines. These numbers only show how far apart the two sides start.

6. The bar keeps moving #

Section 4’s eight and a half hours a day would be easy for me: I run agents for longer than that, just not on Qwen. When Claude Opus 4.6 came out in February, I was perfectly happy with it. I thought it was all I would ever need, and I could not have imagined how much better models would get in half a year. Qwen3.8 27B now scores about the same as Opus 4.6, and I would no longer accept it for coding. Until GPT-6 Astra came out at the start of September, my go-to was GPT-5.6 Sol, which scores ten points higher than Qwen. I then used Astra until Claude Opus 5.5 came out less than three weeks later. Now I don’t even accept what was considered the best model a month ago.

Intelligence Index of the best model available on each date, of the models I used as my go-to, and of Qwen’s 27B models, which fit on a single 3090. Scores use the xhigh reasoning setting where a model has it.

The models that fit on my 3090s keep improving, but they stay about half a year behind the frontier, and my standard moves with the frontier.

7. A small company has it worse #

For me, this is a hobby, and a hobby is allowed to be inefficient. It gets worse once you try to use self-hosted models seriously, say for a team of ten developers.

If you stay fully self-hosted and want fast responses at peak, you have to buy hardware for the busiest hour of the year, not for the average one. The rest of the time, it sits idle.

An illustrative week of coding agents for a ten-person team. The hardware has to cover the busiest hour of the year, but a typical week uses only a fraction of it.

In this made-up but realistic week, the team uses 17% of what it paid for, so every token costs about six times more than it would at full load. You also need a spare GPU for when one dies, and someone who gets paged when it does. You could queue work or send the peaks to an API, but then you are paying for an API anyway.

An API provider has the opposite situation. It serves thousands of customers across every time zone, so its load curve is much flatter than yours. It fills the nights with discounted batch jobs (Luna’s batch tier is half price) and training runs. And more concurrent requests mean bigger batches, which is where the efficiency from section 3 comes from.

8. “They lose money on inference, so prices will go up” #

Another argument I hear is that API prices are subsidized by investors and will go up once those investors want their money back. I don’t think that holds for inference.

AgentPerf also reports how many tokens each GPU serves. On the GB300 rack, one GPU serving DeepSeek V4 Pro handles about 4 million output tokens and 530 million input tokens per hour. Most of that input is conversation history that coding agents send again with every step. Providers keep it in a cache and charge less than 1% of the normal input price for it.

At DeepSeek V4 Pro’s list prices, that GPU brings in at least $5.70 per hour, even if every input token is billed at the cache price. If 5% of the input misses the cache, it brings in about $17. Renting a B200, the closest GPU with a published rental price, costs $3.50 to $5.90 per hour from smaller cloud providers, and that price already includes their profit.

So the tokens pay for the hardware that serves them, even in the worst case. This leaves out costs like staff and training, so it does not tell you whether a lab makes money overall.

The money goes to training, research, free users, and flat-rate subscriptions for heavy users like me. When I wrote that I used $10,000 worth of API tokens for $200, that was list price, not what those tokens cost to serve.

And prices go down, not up. This is what the same level of intelligence has cost this year:

Output price per million tokens for models scoring between 30 and 35 on the Intelligence Index, by release date, as listed by Artificial Analysis. Log scale.

From Claude Opus 4.6 in February to GPT-6 Luna in September, the output price for this level of intelligence dropped by a factor of 50. Labs are in a race to the bottom on price.

9. “But they sell your data” #

Some people say APIs are cheap because the provider trains on your data or sells it. That is why I only used prices from providers that OpenRouter lists as zero data retention (ZDR).

Model	Provider	ZDR	Output ($ per million tokens)
GPT-6 Luna	OpenAI	no	0.50
GPT-6 Luna	Azure	yes	0.50
Qwen3.8 27B	AkashML (FP8)	yes	1.78
Qwen3.8 27B	DeepInfra (full precision)	yes	1.88
Qwen3.8 27B	Alibaba	no	2.55
Qwen3.8 27B	Cloudflare	no	3.20

Azure serves GPT-6 Luna with zero data retention at the same price as OpenAI. For Qwen3.8 27B, the cheapest endpoints are all ZDR, and the endpoints without ZDR cost more. The :free tier is where you pay with your data.

10. Where self-hosting does win: the same model #

If you compare running Qwen3.8 27B yourself with paying for Qwen3.8 27B through an API, self-hosting wins.

DeepInfra, the cheapest full-precision ZDR provider, charges $619 for one benchmark run, while my electricity costs $83. That compares the full model with my quantized copy, so the gap overstates my advantage by whatever quantization costs in quality. My machine pays for itself if it is busy 2.4 hours a day, or 1.5 hours with free power (the first group in the chart in section 4).

Providers charge a lot for a dense 27B model, because every token runs through all 27 billion parameters. Sparse open models, which use only a small part of their weights for each token, can be much cheaper to serve. If you specifically need Qwen and use it heavily, self-hosting it pays off. But you don’t need Qwen to get answers of Qwen’s quality. Luna gives you those for $67, with zero data retention.

11. So why do I do it? #

First, it is fun. I like knowing how the whole stack works, from the drivers in my NixOS configuration to how the layers are split between my two GPUs.

Second, nobody can take it away. Artificial Analysis already lists GPT-5.6 Luna as deprecated, less than three months after its release. My Qwen weights will still be on my disk in ten years, and no company can change their license or close their build system on me.

Third, some data should not leave my house. I would never send 200 GB of email, my messages, and my location history to an API, zero data retention or not. That is what my local AI projects are for.

Those are good reasons. Saving money is not one of them.

Appendix: all assumptions #

Show all assumptions, and which side each one favors

Assumptions that favor self-hosting

The host PC around the GPUs is not counted, and my 3090s count at the $750 each I paid, not the $1,500 they sell for today.
Quantized local copies get the score Artificial Analysis measured through an API, with no penalty for quantization.
The payback chart assumes each machine is busy 24 hours a day, with no idle time.
API prices stay where they are, although GPT-6 Luna’s output price is 58% lower than GPT-5.6 Luna’s, two and a half months earlier.
My time to set up and maintain the machine is not counted.
Every machine also gets a case with free electricity.

Assumptions that favor the API

The example electricity price is 18 cents per kWh, and some people pay less.
My machine serves one request at a time; batching several requests could raise its total throughput.
The hardware has no resale value at the end.
Waste heat that replaces heating in winter is not counted.
API outages, rate limits, and deprecated models are not counted.

Inputs

Benchmark: the Artificial Analysis Intelligence Index on September 30, 2026, at the xhigh reasoning setting, or the highest setting where noted. Its token counts per model are the unit of work.
API prices: the cheapest zero-data-retention endpoints on OpenRouter, with cached input billed at the cache price.
My machine: two RTX 3090s capped at 270 W, running Qwen3.8 27B in vLLM with AutoRound INT4 and DFlash2. Measured: 107 tokens per second, 1,600 tokens per second of prefill, and 540 W for both GPUs. Estimated: about 700 W for the whole machine under load and 150 W idle.
Break-even: the cards are written off over three years; busy hours save what the API would charge minus the electricity, and idle hours cost the idle power.
Other machines: prices, power, and speeds as in the footnotes. The M5 Max and RTX PRO 6000 numbers come from readers, and the DGX Spark and AMD Ryzen AI Max+ 395 speeds are estimated from Artificial Analysis measurements.
Datacenter: AgentPerf measures accelerator power only, and the B200 rental price stands in for the GB300, which has no published rental price.
Small company: the peak-load week is made up.

For each model, Artificial Analysis publishes how many input and output tokens the whole index took, and how many of the input tokens were read from a cache. I multiplied those by each provider’s prices, with cached input at the cache price. For Qwen that is 198 million output tokens and 820 million input tokens that miss the cache. For the bars at home, the same tokens divided by the machine’s speed give the run time, with the uncached input processed at the 1,600 tokens per second I measured on my 3090s and an estimated 2,000 on the RTX PRO 6000, and the run time times the machine’s power and the price per kWh gives the electricity. ↩︎
The RTX PRO 6000 has the same memory bandwidth as an RTX 5090, and at full precision every token reads three times as many bytes as in Q4_K_M. Scaling Artificial Analysis’s 5090 measurement gives about 48 tokens per second. I assume 600 W for the whole machine. ↩︎
I measured the model as I normally run it: vLLM with an AutoRound INT4 quant and DFlash2 speculative decoding, split across both cards. Single 1,500-token answers came out at 105 to 109 tokens per second, and reading a 22,000-token prompt ran at 1,600 tokens per second. Sending 2, 4, or 8 requests at once did not raise the total above about 110 tokens per second. The two GPUs drew 540 W together, each at its 270 W power limit; the 700 W adds my estimate for the rest of the machine, since I could not measure the CPU. The setup follows the recipes from club-3090, a community project that tunes LLM serving for RTX 3090s and publishes measured numbers, so I think it is about as fast as these cards get. Its benchmarks for this configuration at a similar power limit match mine: about 110 tokens per second for prose, and up to about 195 for code, where speculative decoding guesses more tokens right. Most of Qwen’s output on the benchmark is reasoning, which runs at the prose speed. ↩︎
The stock limit is 350 W. My cards sit in a normal desktop case with normal fans, and I worry that running both at full power for hours would shorten their life. The cap is a few lines in my NixOS configuration, and measurements from Puget Systems and r/LocalLLaMA put a 3090 at about 95% of its speed at 270 W. That is 77% of the power for 95% of the speed, so the cap also lowers my electricity cost per token. ↩︎
The 3090 is still the value king for VRAM per dollar, and $750 badly understates what mine are worth. That is what I paid more than a year ago; today a used 3090 sells for about $1,500. Even at that price it costs $62.50 per GB of VRAM, the same as an RTX 5090 at its $2,000 launch price, which is not what a 5090 sells for today. The right number for this calculation is what I could sell my cards for, and at $1,500 each the break-even points rise by about two thirds: at 18 cents per kWh, the same-model case moves from 2.4 to 4.0 hours per day, and the Luna-efficient case from 8.5 to 14.2 hours per day. ↩︎
While the cards are busy, they save what the API would have charged for the same work, minus the electricity at 700 W. While they are idle, the machine still draws about 150 W, of which the two GPUs with the model loaded take 85 W. The break-even point is the number of busy hours per day at which those savings cover the price of the cards, written off over three years, plus the idle power. ↩︎
The M5 Max numbers come from the reader: Qwen3.8-Flash-Next at Q2 (about 2.7 bits per weight), about 45 tokens per second, 1,250 tokens per second of prefill, and about 90 W, on a $7,000 laptop. The RTX PRO 6000 numbers come from another reader, u/TheAILegend, who measured Qwen3.8-Flash-Next at NVFP4 on theirs. For a single request, it stays at about 150 tokens per second from 4,000 to 200,000 tokens of context, with about 10,000 tokens per second of prefill and 266 W for the GPU plus about 160 W for the rest of the PC. With 32 short requests in parallel, the card reaches 590 tokens per second and pays off in about two years, which works for short batch jobs. At the long contexts of this benchmark, parallel requests are slower than one, so the chart uses the single-request speed. I use $15,000, the middle of the card’s price range. The DGX Spark and the AMD machine run Qwen3.8-Flash-Next at about 4.5 bits. For their speed, I took Artificial Analysis’s measurements of Qwen3.8 27B on each machine and doubled them, because the owner of the AMD machine finds Qwen3.8-Flash-Next about twice as fast as Qwen3.8 27B. That gives about 55 tokens per second on the DGX Spark ($6,950, about 150 W) and 47 on the AMD Ryzen AI Max+ 395 ($4,000, the price Artificial Analysis lists, about 120 W). All of them are compared with GPT-6 Luna on its highest setting, which runs the whole benchmark for $122. The 3090s run Qwen3.8 27B, as in the rest of this post, and are compared with Luna on xhigh. Artificial Analysis measured Qwen3.8 27B on the AMD machine at 23 tokens per second, so one benchmark run takes it 134 days and about 390 kWh, against about 460 kWh on my 3090s. Running Qwen3.8-Flash-Next on the 3090s with CPU expert offload, which club-3090 measured at 37 tokens per second, does not change their result: one run then takes 82 days, and at 18 cents per kWh its electricity costs more than Luna’s $122 unless the whole machine draws less than about 350 W. ↩︎

Switching from Ollama to llama-swap + llama.cpp on NixOS: the power user's choice 🦙

Sat, 29 Nov 2025 00:00:00 +0000

Table of contents

The draft I never published #

I started writing this blog post four months ago. Back then, I was frustrated because a specific model I wanted to try—gpt-oss-20b—was completely broken in Ollama. I went down the rabbit hole, set up llama-swap and llama.cpp, got the model working perfectly, wrote half this post… and then I stopped.

Why? Laziness. Once the novelty of that specific model wore off, I drifted back to Ollama. The real convenience of Ollama was never the ollama run command—I almost exclusively interact with my models via the API using my own tools like agent-cli. The convenience was ollama pull. Being able to grab a model with one command and have it “just work” was great.

But as a NixOS user, this imperative convenience is actually a flaw. I want my system to be declarative. I want my model configuration committed to git, not hiding in a local database. So I let convenience win for a while, but it always felt wrong.

But last month, that changed. I upgraded my hobbyist AI rig by adding a second RTX 3090. Suddenly, I had 48GB of VRAM and a whole new set of problems—and possibilities. I wanted to run massive models (like gpt-oss-120b or Qwen-3-VL-32B with a huge context window). I needed to balance layers between two GPUs. I needed to spill specific parts of the model into system RAM without crashing.

When you start pushing hardware to its limits, “magic” abstractions like Ollama stop being helpful and start getting in the way. I realized that llama.cpp isn’t just a backend; it is the place where development happens. If you want to use split-mode inference, granular CPU offloading, or the latest quantization tricks immediately, there is no alternative.

So, I dusted off this draft, updated my Nix configuration, and I’ve been running llama-swap full-time for a month. I’m finally ready to vouch for it.

The original trigger: broken models #

Four months ago, my frustration stemmed from compatibility. I was trying to run gpt-oss-20b, a model that required specific chat templates and GGUF handling that Ollama simply broke.

The maintainer of llama.cpp, Georgi Gerganov, explained it perfectly regarding the gpt-oss debacle: Ollama forked the ggml inference engine to rush out “day-1 support” for gpt-oss without coordinating with upstream. The result? Their implementation was not only incompatible with standard GGUF files but also significantly slower and unoptimized.

Benchmarks at the time showed Ollama had significant performance overhead—often 20-30% slower inference speeds compared to running llama.cpp directly.

This pattern means Ollama will always be behind llama.cpp. Meanwhile, I’ve had a pull request sitting unreviewed for over a month that fixes their broken OpenAI API. Compare that to llama.cpp: I submitted a PR there and it was merged in less than an hour. If you want the latest features now (and a responsive community), you go to the source.

The catalyst: dual GPUs and control #

With a single GPU, you usually either fit the model or you don’t. With two GPUs (and 64GB of system RAM), things get interesting. You enter the territory of heterogeneous inference.

Ollama tries to handle splitting automatically, but I found myself fighting it. I wanted to say: “Put 40 layers on GPU 0, 40 layers on GPU 1, and keep the KV cache here.” Or: “This model is slightly too big for VRAM; I want to offload exactly the last 10% to system RAM.”

With llama.cpp directly, this is just a command line flag. With llama-swap, I can define this behavior per-model in a config file:

"gpt-oss:120b":
 cmd: |
 ${pkgs.llama-cpp}/bin/llama-server
 -hf ggml-org/gpt-oss-120b-GGUF
 --port ${PORT}
 --ctx-size 65536
 --split-mode layer
 --tensor-split 3,1.3 # Tuned balance between my GPUs
 --n-cpu-moe 15 # Offload 15 experts to CPU RAM
 --threads 8
 --chat-template-kwargs '{"reasoning_effort": "high"}'

This level of granularity turned my rig from a “black box” into a tunable instrument. The --tensor-split and --n-cpu-moe flags allow me to run a 120B parameter model that technically shouldn’t fit, by carefully balancing the load across both GPUs and system RAM.

API compatibility and the “switch” #

One of the reasons I hesitated to switch was the fear of breaking my existing workflows. I use agent-cli and other custom Python scripts daily. But it turned out to be a non-issue. Both Ollama and llama-swap (wrapping llama-server) expose an OpenAI-compatible API.

Once I adopted the pattern of setting a custom OPENAI_BASE_URL in my projects, the backend became irrelevant. My tools don’t care if they are talking to api.openai.com, a local Ollama instance, or llama-swap. They just send JSON and get JSON back. This standardization meant I could swap the entire inference engine underneath my applications without changing a single line of their code.

My NixOS setup #

Since I manage my entire system with Nix, setting up llama-swap and pinning llama-cpp to the absolute bleeding edge was surprisingly elegant. I even wrote an auto-update script to ensure I’m always on the latest commit.

This is where my NixOS build cache shines. Unlike Ollama, which releases every few weeks, llama.cpp can have multiple releases a day. My cache server runs this update script automatically, builds the new binaries, and serves them to my workstation. I get the absolute latest performance improvements without ever compiling source code locally.

Compiler flags matter #

One critical lesson I learned: default builds matter. Initially, I was getting a measly 8 tokens/sec on gpt-oss:120b. It turned out the default llama.cpp package wasn’t enabling BLAS or native CPU optimizations, crippling the layers I offloaded to system RAM.

By overriding the package to enable blasSupport and passing -DGGML_NATIVE=ON (as seen below), performance jumped to 50 tokens/sec. This is another reason why I prefer managing this via Nix: I can enforce these compile-time flags declaratively.

Here’s my complete configuration (see package-overrides.nix and ai.nix):

# Override llama-cpp to latest version with CUDA support
llama-cpp =
 (pkgs.llama-cpp.override {
 cudaSupport = true;
 rocmSupport = false;
 metalSupport = false;
 # Enable BLAS for optimized CPU layer performance (OpenBLAS)
 blasSupport = true;
 }).overrideAttrs
 (oldAttrs: rec {
 version = "7205";
 src = pkgs.fetchFromGitHub {
 owner = "ggml-org";
 repo = "llama.cpp";
 tag = "b${version}";
 hash = "sha256-1CcYbc8RWAPVz8hoxKEmbAgQesC1oGFZ3fhfuU5vmOc=";
 leaveDotGit = true;
 postFetch = ''
 git -C "$out" rev-parse --short HEAD > $out/COMMIT
 find "$out" -name .git -print0 | xargs -0 rm -rf
 '';
 };
 # Enable native CPU optimizations (AVX, AVX2, etc.)
 cmakeFlags = (oldAttrs.cmakeFlags or []) ++ [
 "-DGGML_NATIVE=ON"
 ];
 # Disable Nix's march=native stripping
 preConfigure = ''
 export NIX_ENFORCE_NO_NATIVE=0
 ${oldAttrs.preConfigure or ""}
 '';
 });

# llama-swap from GitHub releases
llama-swap = pkgs.runCommand "llama-swap" { } ''
 mkdir -p $out/bin
 tar -xzf ${
 pkgs.fetchurl {
 url = "https://github.com/mostlygeek/llama-swap/releases/download/v175/llama-swap_175_linux_amd64.tar.gz";
 hash = "sha256-zeyVz0ldMxV4HKK+u5TtAozfRI6IJmeBo92IJTgkGrQ=";
 }
 } -C $out/bin
 chmod +x $out/bin/llama-swap
'';

# Configure llama-swap as a systemd service
systemd.services.llama-swap = {
 description = "llama-swap - OpenAI compatible proxy with automatic model swapping";
 after = [ "network.target" ];
 wantedBy = [ "multi-user.target" ];
 
 serviceConfig = {
 Type = "simple";
 User = "basnijholt";
 Group = "users";
 # Point to your declarative config file
 ExecStart = "${pkgs.llama-swap}/bin/llama-swap --config /etc/llama-swap/config.yaml --listen 0.0.0.0:9292 --watch-config";
 Restart = "always";
 RestartSec = 10;
 
 # Environment for CUDA support
 Environment = [
 "PATH=/run/current-system/sw/bin"
 "LD_LIBRARY_PATH=/run/opengl-driver/lib:/run/opengl-driver-32/lib"
 ];
 };
};

What I gained (and lost) #

Advantages of llama-swap + llama.cpp #

Heterogeneous compute: Perfectly balance loads between GPUs and CPU RAM.
Bleeding edge: I use features (like new quantization types) the day they land in llama.cpp.
Zero idle VRAM: Models unload completely when not in use.
Transparent operation: No hidden prompts or “helpful” formatting that breaks complex tasks.

The trade-off #

The main trade-off is “laziness.” You can’t just ollama pull new-model and have it appear. You have to find the GGUF on HuggingFace, decide on the quantization (Q4_K_M? Q6_K?), and add 5 lines of YAML to your config.

But honestly? That’s a feature, not a bug. It forces you to understand what you are running. It stops you from running a quantized model that is too stupid for your task just because it was the default. And for a NixOS user, putting that configuration into code instead of a hidden database is exactly how it should be.

Conclusion: maturing as an AI hobbyist #

My switch back to llama-swap wasn’t about fixing a bug; it was about outgrowing a tool. Ollama is the “starter bike” of local AI—fantastic for getting rolling. But when you add a second GPU, start caring about tokens per second, and want to maximize every gigabyte of your 48GB VRAM buffer, you need a manual transmission.

For anyone else building a hobbyist multi-GPU rig: stop fighting the abstraction. Go to the source. The control is worth it.

Check out my NixOS configuration for the complete setup!

Two years on the Glove80: a split ergonomic, programmable keyboard as a programmer ⌨️

Thu, 06 Nov 2025 00:00:00 +0000

Table of contents

Why I switched #

For roughly three years I used the Logitech Ergo K860. Despite liking it, I kept getting RSI (Repetitive Strain Injury) flare‑ups—pain and discomfort from repetitive typing motions. ~2 years ago I discovered that split ergonomic keyboards were a thing and went down the rabbit hole. I switched to the Glove80 in January 2024. Before I bought anything, I started watching videos from the channel If Coding Were Natural and from Ben Vallack. I read many reviews and first‑hand experiences. The more I watched and read, the more my interest in split ergonomic keyboards grew. Overwhelmingly, people seemed to praise the Glove80 for its comfort, so I decided to try it. I considered many alternatives like the Moonlander and Ergodox but chose the Glove80 for its lighter switches and curved key wells.

Before the Glove80, I had flirted with mechanical keyboards because the community is huge and very into them. Given the nerd that I am, I got nerd‑sniped by the mechanical‑keyboard scene. I assumed I would be into mechanical keyboards. In April 2019, I ordered a 60% board, the Anne Pro 2. After trying it, I returned it because it wasn’t an improvement for me and the reduced ergonomics weren’t worth it compared to the Logitech. The mechanical switches didn’t really excite me in practice as much as I had expected. What I actually needed was a split, columnar layout with thumb clusters and programmability. I didn’t know this yet.

Split ergonomic keyboards: What makes them different? #

Before diving into my experience, here’s what sets these keyboards apart from traditional ones.

Columnar layout #

Columns line up under each finger, so most movements are up/down instead of sideways. Compared to row‑staggered boards, this reduces lateral finger travel and awkward cross‑overs. Row‑staggered boards offset keys sideways, which encourages lateral travel rather than straight motions.

Vertical column‑stagger #

Each column is offset to match finger length, which reduces awkward reaches for ring and pinky fingers and keeps wrists more neutral. The middle finger column sits higher, the pinky column lower—matching the natural length differences between fingers.

Split halves #

Two halves let shoulders and wrists stay in a neutral, shoulder‑width posture. One‑piece boards tend to pull shoulders inward and bend wrists sideways (called ulnar deviation). With the split, I can position each half independently for my body.

Key wells #

Curved wells bring keycaps closer to your finger pads along natural arcs, so you extend and lift your fingers less. Flat boards make you lift and extend fingers more and move farther between rows. The Glove80’s wells are subtle but noticeable after hours of typing.

Thumb clusters #

Modifiers, Backspace, Enter, and other frequent keys can live under your strongest digits—the thumbs—instead of overloading pinkies. Big space bars under‑utilize thumbs and push more work to pinkies and corners. On the Glove80, my thumbs handle multiple keys that would normally require pinky reaches.

Tenting #

Tenting means raising one side of the keyboard to adjust the angle. I tent the halves to a mild angle so my forearms don’t have to twist as much (less pronation) and my wrists stay more neutral. Most standard keyboards cannot tent at all, which forces your palms completely flat and pulls your shoulders inward.

Switches #

I use Kailh Choc Red Pro switches on my Glove80. They are linear with a low actuation force, which feels light and comfortable for long typing. I don’t have much experience with other switches, so this is not a comparison—just what works for me.

Left half of my Glove80 showing the columnar columns, thumb cluster, and mild tenting.

Taken together, these differences mean less strain and more consistent typing for me than on row‑staggered boards.

The first month: Comfort vs. disruption #

After spending a month on the Glove80, I found it much more comfortable than expected. At the same time, it completely disrupted my typing strategy—what I had learned on a traditional layout didn’t transfer to the columnar layout of the Glove80.

What was frustrating #

In the first couple of days, my typing speed tanked and it was extremely frustrating. On the first day, I was at 22 wpm—slower than hunt-and-peck typing. I was constantly looking at the keyboard, trying to use the right fingers for the right keys. On a traditional keyboard I had my own strategy—basically three fingers per hand—and I was actually very fast with it. But that didn’t transfer at all to the columnar layout; using the “right” finger for each column felt natural once learned, but it wasn’t native to me at first. I did not want to give up and kept going despite the massive frustration. Maybe what kept me going was the sunk‑cost fallacy of $400 (which is mid-range for split ergonomic keyboards, they typically run 200-600 USD).

Practice tools that helped #

I started with Keybr to learn key‑by‑key. Once I finished that, I switched to using MonkeyType exclusively. After 1–2 months I had already reached ~85 wpm (words per minute—average typing speed is 40 wpm, programmers often type 60-80 wpm). I haven’t improved a whole lot since then, but I also haven’t done dedicated typing practice. While writing this post I tried MonkeyType again and I’m now ~90 wpm.

My Keybr progress during the first 4 weeks: starting at ~20 wpm, both speed and accuracy steadily improved over each lesson.

Beyond the tools, a few things helped with the transition: the Glove80’s columnar layout actually forces you to use the correct finger for each column—using the wrong finger feels completely unnatural and awkward, so I had to abandon my old three-finger strategy. I also had to accept the short-term speed dip while accuracy improved, and adding slight tenting helped keep my wrists more neutral.

Today, it feels completely natural for me to type on this keyboard.

Programmable features that make a difference #

These keyboards let you program behaviors that eliminate awkward reaches and repetitive strain. Here’s what I actually use:

Mod-tap: One key, two functions #

Keys do different things based on tap vs. hold. All my symbols work this way:

Tap 1 = “1”, hold for ~200ms = “!”
Tap , = “,”, hold = “<”
Tap . = “.”, hold = “>”

This keeps everything on the home row without reaching to a symbols row.

Sticky modifiers: Tap instead of hold #

All my modifiers are “sticky”—tap them once and they latch for the next key:

Tap Command + Shift, then tap T = reopens last tab
No more finger gymnastics holding three keys at once

With a single thumb movement on my left hand, I can reopen a tab quickly.

Layers: Multiple keyboards in one #

Multiple virtual keymaps on one keyboard. You can momentarily switch or toggle to a layer to get different keys under the same physical switches. Common patterns include a symbols layer, a navigation layer, and a numbers/function layer—all reachable from home row.

One of my layers: digits act as mod‑taps for symbols, and common modifiers live under the thumbs.

The “hyper” key shortcut #

If I hold the Home key, it acts as if I’m pressing Ctrl + Alt + Shift + Command simultaneously (the “hyper” key I mentioned earlier). Since few apps use that combo by default, I map it to custom commands:

Home + T → open Terminal
Home + B → open Browser
Home + C → open Calendar
Home + S → open Slack

This hyper key setup enabled my keyboard-first workflow across macOS (using Keyboard Maestro) and Linux (using Hyprland). The same muscle memory works across both systems, relieving me from tapping Command+Tab like a mad person.

More programmable tricks I use (details)

Double‑tap Shift → emulate mouse behavior on keyboard
- A quick double‑tap of Shift toggles my mouse behavior layer so I can move/scroll/click without leaving the home row.
Double‑tap Command → macOS‑style word navigation on Linux
- I use this to mirror macOS word‑jump behavior in Linux editors/terminals, so ⌘ + arrows behaves the same everywhere.
Combos for common keys
- E + D → Enter; R + F → Space. This lets me press Enter and Space with my left hand while my right hand is on the mouse, since those keys sit on the right side on many layouts. I also keep most app‑switch launchers on the left side so my left hand can trigger them while my right hand stays on the mouse.
Shift + Backspace → Delete
- Backspace morphs into Delete when Shift is held—handy on macOS/Linux.
Tap dance to momentary vs. toggle a layer
- Single tap for a momentary “Lower,” double tap to lock it on; keeps symbols/nav nearby without mode churn.
Mouse keys with speed profiles
- Separate slow/fast/warp cursor and scroll speeds for precision vs. movement.
Delete to start/end of line
- One chord deletes left to line start; another deletes right to line end—great for editors.
Tuning for fewer misfires
- Tap‑preferred hold‑taps and a ~200–220 ms tapping term match my timing.

None of this requires you to become a firmware expert on day one. Even a couple of well‑chosen mod‑taps, one sticky key, and a single “symbols + navigation” layer can remove a lot of awkward reaches.

What I’d tell my past self #

Expect an initial drop in speed; it’s normal.
Columnar layouts reward correct finger usage; looking down at first is fine.
Start simple: one or two mod‑taps, one sticky modifier, one extra layer.
Practice compounds. MonkeyType and Keybr were enough for me.
Don’t let sunk‑cost thinking scare you into quitting, but also don’t over‑engineer your layout on day one.
You won’t lose the ability to type on normal keyboards. When I type on my laptop’s built‑in keyboard, I just use my original three-finger strategy and have no issues. It’s like developing two different muscle memories based on the keyboard.
Gaming consideration: WASD sits off the natural columns on a columnar split. I briefly tried a shooter and it felt awkward—unless you’re willing to reprogram and relearn gaming muscle memory, FPS games don’t work well. I only played Factorio on it, which worked fine.

Where I am now #

As of November 2025, I’m ~90 wpm, and it feels completely natural to type on the Glove80. I reached 85 wpm within 1–2 months and have mostly stayed in that range since, with a small bump recently. The comfort gains are real, and the programmability (mod‑tap, sticky keys, and layers) turned out to be as exciting in practice as it sounded when I first discovered it. Was the $400 worth it? Absolutely! I probably spent many thousands of hours on it so far, so amortizing it over its lifetime, it’s well worth it for me. Plus it makes me feel like a leet hacker.

My current Glove80 layout: view it here.

What’s next? #

The Glove80 perfectly meets my needs—I have no practical reason to switch. But the nerd inside me is curious about other options in the split keyboard ecosystem. The Corne, Svalboard, and Cyboard Imprint all take quite different approaches and seem interesting to explore. Whether I’ll actually try them or stick with what works is an open question—sometimes the best keyboard is the one you’ve already mastered.

Community and support #

The MoErgo Glove80 community is extremely welcoming. Stephen, the designer, is very active and responsive—when I requested a small feature for the layout editor, it went live within a day.

How a 3090 turned into two local‑first AI projects 🎮

Mon, 14 Jul 2025 00:00:00 +0000

Table of contents

1. Introduction: The Unplayed Games #

I’ve wanted to dabble with local AI for a long time, mainly because of the possibilities it unlocks. Local AI allows me to use all of my data—financial records, personal messages, location history, even transcribing everything I say—without the privacy concerns of sending it to a cloud provider. This level of deep, personal data analysis is only something I’d be comfortable with on my own hardware.

However, I couldn’t justify buying a powerful GPU just for an imagined AI project that I might never really start. Then it hit me: I could also use it for gaming (duh). Because I really enjoyed The Last of Us series, I thought, “Okay, why not play The Last of Us on PC?” This was the push I needed. So, a couple of weeks ago, I convinced myself to buy someone’s old gaming machine. After many deep research sessions with ChatGPT about the best bang-for-the-buck consumer-grade GPU, I landed on an NVIDIA 3090. I found an (old) beast of a machine, and my initial idea was set.

Click here to see the full system specs ($1350 for all this!)

GPU: ASUS ROG STRIX RTX 3090 OC (24GB)
CPU: AMD Ryzen 9 3900X (12-Core)
Motherboard: ASUS ROG Crosshair VIII Hero
Power Supply: Be Quiet! 1200W Dark Power Pro
RAM: 32GB G.Skill Trident Z Royal DDR4
Storage: 2TB Intel M.2 NVMe SSD
Case: Corsair 4000D Airflow
CPU Cooler: Thermalright Peerless Assassin 120

Fast forward four weeks. I haven’t played a single minute, but my machine is now indexing 200 GB of email archives and facing a serious performance bottleneck I’ll get to later. That “gaming” machine has been humming away, but not rendering virtual worlds. It’s been training, transcribing, and thinking.

I’m writing this post simply because I am so enthusiastic about local AI. I’ve found that local models are surprisingly powerful for specific use cases. Of course, they aren’t going to replace Gemini or Claude for coding, but for many other applications, they work incredibly well. Lately, I’ve been so obsessed with this that I’m sleeping significantly less and spending all my free hours working on it.

2. The Rocky Road to a Working Setup #

Before I could even get to the fun AI stuff, I had to wrestle with the machine itself. I started with Pop!_OS, which is supposed to be a great out of the box solution that “just works”. However, it’s been a long time since I used Linux on the desktop; my experience is almost exclusively with servers. I quickly installed some of the wrong NVIDIA drivers and ended up debugging stuff in Grub. I got frustrated with the different commands I had to run to set up everything and realized that reproducing this system was going to be a massive pain. Since I’ve grown to love Nix on my Mac with Nix-Darwin, I decided to switch the whole system to NixOS. This has been an amazing decision. Even though there are many pain points with Nix, I think it is well worth it. For those interested, you can see the complete setup, including all the local AI services, in my NixOS configuration on GitHub.

3. The First Victory: `agent-cli` 🐍 #

With a stable system, I quickly got started with a few scripts. These soon evolved into my first real project of this adventure: agent-cli.

agent-cli is a collection of local-first, AI-powered command-line agents that run entirely on your machine. The core philosophy is privacy; your data never leaves your computer. It’s designed for seamless integration with system-wide hotkeys. I’m now using it constantly.

Instead of typing, I now dictate almost everything. A quick hotkey, and agent-cli transcribe turns my speech into text. Another hotkey, and agent-cli autocorrect cleans it up using a local LLM running on Ollama. It has genuinely streamlined my workflow. For example, I can pipe the tools together, like correcting text from my clipboard and then having it spoken aloud: agent-cli autocorrect | agent-cli speak.

The toolkit includes:

autocorrect: Fixes grammar and spelling.
transcribe: Uses a local Whisper model for speech-to-text.
speak: Converts text to speech with a local TTS engine.
voice-edit: A voice-powered clipboard assistant.
assistant: A hands-free voice assistant using a wake word.
chat: A conversational AI with tool-calling capabilities.

It’s been a fantastic success in my personal workflow, and it’s all open-source for you to try. Contributions and ideas welcome!

Once I trusted my toolchain, the obvious next step was to feed all my data into it.

4. The Grand Ambition: `AIBrain` 🧠 #

My second package, AIBrain, is a much larger endeavor. The vision is to create a “life-OS”—a private, local AI that processes all my personal data (emails, calendar, files, photos, messages, browser history, location data) to generate summaries and answer questions, eventually allowing for queries like:

aibrain ask "When did I get into landscape photography?"
aibrain ask "How many countries did I visit between 2015-2025?"

This project led me to explore the landscape of agentic frameworks. I started with CrewAI, but I quickly found that despite the hype and funding, it had some painful problems (specifically this issue where things just fail without any helpful error messages). I decided to switch to LangGraph, which felt more robust. I also chose to not use PydanticAI (which I used in agent-cli) because its Ollama support is just a wrapper around the OpenAI-compatible API, which doesn’t expose all the useful options (I even opened a PR in Ollama to make its OpenAI compatible API more feature-complete, but it has not been merged yet).

The core infrastructure of AIBrain is ready. It can index emails, calendar events, and files in various formats (PDF, Excel, PowerPoint, images). However, I’ve hit a significant challenge.

5. Conclusion: The Journey Continues #

This past month has been a whirlwind. I set out to play games and ended up with two open-source AI packages and a much deeper appreciation for the power sitting in a consumer-grade GPU. It’s incredible what you can achieve with local hardware today.

agent-cli is a finished, polished tool that I use daily. AIBrain is a much bigger dream, and I’m still figuring out how to solve the performance puzzle to make it truly practical. Next up: smarter chunking with LangGraph, vector-DB swap-outs, and GPU scheduling tricks.

This experience has reinforced my passion for local-first software and open-source development. The future of personal, private AI is being built right now, not just in large corporate labs, but in the homes of hobbyists tinkering with their gaming rigs.

⭐ If this resonates, star the repos & drop issues/questions.

And who knows, maybe I’ll even find the time to play ‘The Last of Us’.

My homelab: from a single Raspberry Pi to a Proxmox cluster and a dedicated NAS 🔧

Sat, 06 Jul 2024 00:00:00 +0000

In 2020, I began my homelab journey (even though I never heard of the term “homelab” at that point) with a simple yet powerful setup: Home Assistant running on Proxmox on a NUC8i3BEH. My Home Assistant instance, with over 100 automations and about 20 addons (docker containers), had outgrown the capabilities of my Raspberry Pi 4, leading me to upgrade to the NUC8i3BEH. You can check out my Home Assistant configuration.

This system was more than sufficient for running Home Assistant—arguably overkill. However, because I passionately develop addons and custom components, I needed the ability to quickly restart and reload everything while debugging. For a long time, I ran Proxmox, mainly because the Home Assistant community recommended it, but I only used it to host one extra Ubuntu VM and Home Assistant OS.

In 2022, I stumbled upon the Proxmox VE Helper-Scripts by tteck, which got me interested in the whole homelabber/self-hosting community. These scripts opened a new world of possibilities. With them, I expanded my Proxmox environment to run 25 different VMs and LXC containers, each performing a variety of tasks, from DNS servers and reverse proxies to local dashboards and backup software.

Initially, I used an external 2 TB drive for storage, but it quickly became insufficient as my data needs grew. After some research, I found the HP EliteDesk 800 G4 SFF, a renewed enterprise-grade desktop with slots for two 3.5-inch drives. At approximately $160 on Amazon, it offered excellent specs for a budget-friendly price. I decided to create a TrueNAS VM on this machine using Proxmox and passed through the two 16TB renewed enterprise-level HDDs (via the PCIe SATA controller), which I had purchased from serverpartdeals.com. I chose these drives based on their reliability as reported in the BackBlaze drive stats.

When I got the HP, I created a Proxmox cluster and was very excited about how easy it was to migrate containers and VMs from one machine to the other. However, I soon encountered a significant drawback: enterprise drives are notably loud. Since I had my Proxmox “cluster” (just the two machines) running behind the TV, the loudness was noticeable and disruptive. This noise was just the beginning of my troubles. The NFS server on the TrueNAS VM often crashed, causing the entire Proxmox system to lock up. Initially, I mounted the NFS drives directly on the Proxmox host and used bind-mounts to pass through the drives. Due to the locking behavior of NFS, any problem would cause the entire system to lock up. To prevent this, I started to use autofs to mount the drives in the LXC containers and VMs instead. This worked fine until Proxmox started its periodic backups. Any issue with the NFS during backup caused the whole process to hang, locking up the VMs and LXC containers. This could be partially avoided by using the “stop” backup mode, but it still resulted in almost daily debugging sessions where I had to restart Proxmox-related services.

After running all services as LXC containers, I realized that updating them and keeping up with all security updates was labor-intensive. So, I started to look into migrating everything to Docker Compose. This transition turned out great because I could manage my config files in a git repo and keep the relevant data separate, streamlining updates and maintenance.

To resolve these issues and because I was having a lot of fun, I decided to build a dedicated NAS running TrueNAS Scale, not that I really needed one, but mostly because I could. This new setup included high-performance components, ensuring both stability and quiet operation. I used “Manufacturer Recertified” Western Digital Ultrastar DC HC550 HDDs, known for their reliability and performance. I have 6 x 16TB = 96TB now with the option to add 2 more 3.5" drives, 2 nvme M.2 drives, and 1 SATA 2.5" SSD. Now, I run all services that previously relied on NFS locally on TrueNAS via a Debian systemd-nspawn container managed with a new script called Jailmaker, which leverages the systemd-nspawn program in TrueNAS Scale (not to be confused with FreeBSD jails). Inside this systemd-nspawn container, I run Debian with Docker. For managing Docker Compose files, I use Dockge, which provides a nice web UI for easy management.

Along the way, I got familiar with ZFS and grew to love it for its amazing snapshotting and data integrity features. ZFS has been a game-changer, providing robust data protection and ease of management. Initially, I used mirrored vdevs, but I recently switched to RAIDZ2 for the added storage capacity and the promise of OpenZFS 2.3 supporting expansion of RAIDZ vdevs. I also use ZFS for my Proxmox VMs and LXC containers, whose backups I store on the NAS via rsync in a cronjob to another jail running plain Debian (no more NFS on the Proxmox host for me…)

This journey from a simple NUC running Home Assistant to a Proxmox cluster and dedicated NAS has been very time-consuming but a lot of fun. The lessons learned and the improvements in stability and performance have made the effort worthwhile. If you’re considering building your own homelab, investing in reliable hardware and being prepared for a bit of trial and error can make all the difference.

Lessons learned #

Run your services in Docker containers, not LXC containers, for easier management and updates.
Do not have virtualized NAS (TrueNAS VM) on Proxmox, it’s a bad idea if you are also sharing the drives with Proxmox via NFS. There are many folks on the internet who run their NAS in a VM on Proxmox, seemingly without issues, but I was not one of them.
Do not encrypt the root ZFS dataset! Encrypt its children instead.
Use passphrase encryption instead of a generated key. It allows you to use the UI and rotating keys.
Do not remove the physical drive that is passed through in a Proxmox VM, because Proxmox will not boot properly afterwards.
Do not use random 100 character passwords for your Proxmox, you have only 60 seconds to enter it during boot.
When making Proxmox backups, send them to a different physical machine, not only the same machine that is being backed up.
When building a NAS or home server, consider noise levels, especially if the equipment will be in a living area. Enterprise-grade drives can be surprisingly loud.
When using Intel I226-V 2.5GbE controllers, be prepared for potential issues.
When planning your NAS storage size needs, remember to account for the extra space needed for redundancy (e.g., in RAIDZ2). Additionally, keep in mind that ZFS performs optimally when drives aren’t filled to capacity. It’s wise to plan for more storage than you think you’ll immediately need.
Buy as many used enterprise-grade hardware components as you can, they are often cheaper and more reliable than consumer-grade components.

Components #

For completeness, here are the components for each machine in my homelab:

NUC Components #

Name	Full Name	Server	Price	Quantity	Cost	Date
NUC8i3BEH	Intel NUC Kit NUC8i3BEH	NUC	278.3	1	278.3	2020-02-22
500GB M2 SSD	Samsung 970 EVO M.2 500GB	NUC	94.99	1	94.99	2020-02-21
32 GB RAM	TEAMGROUP 32GB DDR4 3200MHz	NUC	58.99	1	58.99	2024-05-01
4TB SSD	Crucial P3 Plus 4TB NVMe	NUC	226.99	1	226.99	2024-04-27

HP Components #

Name	Full Name	Server	Price	Quantity	Cost	Date
64 GB RAM	TEAMGROUP 32GB DDR4 3200MHz	HP	57.99	2	115.98	2024-05-02
HP EliteDesk 800 G4 SFF	HP EliteDesk 800 G4 SFF	HP	164	1	164	2024-04-27

Notes on the HP components:

Initially, I planned to use this machine as a NAS because it has two 3.5" drive bays, 2 M.2 slots, and the disc drive bay can be converted to a 2.5" drive bay. However, the 2 x 16TB drives I bought quickly filled up because I used a mirrored vdev, so I decided to build a dedicated NAS instead.

NAS Components #

Name	Full Name	Server	Price	Quantity	Cost	Date
16 TB HDD	Western Digital Ultrastar DC HC550 16TB	NAS	159.99	2	319.98	2024-05-01
16 TB HDD	Western Digital Ultrastar DC HC550 16TB	NAS	187.5	4	750	2024-06-21
Motherboard	Q670 8-bay NAS motherboard	NAS	203.21	1	203.21	2024-06-22
CPU i5 13500T	Intel Core i5-13500T	NAS	180.88	1	180.88	2024-06-22
PSU SFX	FSP Dagger Pro 650W	NAS	87.13	1	87.13	2024-06-21
Jonsbo N3 case	JONSBO N3 Mini-ITX NAS Case	NAS	187.51	1	187.51	2024-06-16
CPU cooler	Thermalright AXP90 X36 CPU cooler	NAS	21.83	1	21.83	2024-06-25
64 GB DDR5 RAM	CORSAIR VENGEANCE 64GB DDR5	NAS	162.13	1	162.13	2024-06-29

Notes on the NAS components:

The 13500T is a 35W TDP CPU I bought on eBay, which will help keep the system cool and quiet.
I bought the motherboard from AliExpress because it seems to be the only mini-ITX motherboard that supports 8 SATA drives, and also comes with 2x2.5GbE ports, 3xM.2 slots, and a full sized PCIe slot.
The motherboard uses Intel I225-V 2.5GbE controllers, which are notoriously bad, initially iperf3 maxed out at 100Mbps. The internet is full with people complaining about these controllers auto-negotiating to 100Mbps, however, for me it showed 1Gbps. After trying all suggestions I could find (there are few), as a last resort I connecting the NAS directly to my ASUS XT8 router instead of the NETGEAR GS305 switch, and now I got 1Gbps speeds. So I bought a new managed TL-SG108E switch, which now also gives me 1Gbps speeds. I’m not sure what the issue was, but I’m happy it’s resolved.

Hardware | Bas Nijholt

Self-hosting AI does not save money, and I do it anyway

1. The model I love, and the one that beats it on price #

2. “But my GPUs are already paid for” #

3. Why datacenters are so much better at this #

4. “But I have solar panels” #

5. “But my Mac, DGX Spark, or AI Max does save money” #

6. The bar keeps moving #

7. A small company has it worse #

8. “They lose money on inference, so prices will go up” #

9. “But they sell your data” #

10. Where self-hosting does win: the same model #

11. So why do I do it? #

Appendix: all assumptions #

Switching from Ollama to llama-swap + llama.cpp on NixOS: the power user's choice 🦙

The draft I never published #

The original trigger: broken models #

The catalyst: dual GPUs and control #

API compatibility and the “switch” #

My NixOS setup #

Compiler flags matter #

What I gained (and lost) #

Advantages of llama-swap + llama.cpp #

The trade-off #

Conclusion: maturing as an AI hobbyist #

Two years on the Glove80: a split ergonomic, programmable keyboard as a programmer ⌨️

Why I switched #

Split ergonomic keyboards: What makes them different? #

Columnar layout #

Vertical column‑stagger #

Split halves #

Key wells #

Thumb clusters #

Tenting #

Switches #

The first month: Comfort vs. disruption #

What was frustrating #

Practice tools that helped #

Programmable features that make a difference #

Mod-tap: One key, two functions #

Sticky modifiers: Tap instead of hold #

Layers: Multiple keyboards in one #

The “hyper” key shortcut #

What I’d tell my past self #

Where I am now #

What’s next? #

Community and support #

How a 3090 turned into two local‑first AI projects 🎮

1. Introduction: The Unplayed Games #

2. The Rocky Road to a Working Setup #

3. The First Victory: agent-cli 🐍 #

4. The Grand Ambition: AIBrain 🧠 #

5. Conclusion: The Journey Continues #

My homelab: from a single Raspberry Pi to a Proxmox cluster and a dedicated NAS 🔧

Lessons learned #

Components #

NUC Components #

HP Components #

NAS Components #

3. The First Victory: `agent-cli` 🐍 #

4. The Grand Ambition: `AIBrain` 🧠 #