Local-Ai | Bas Nijholt

Self-hosting AI does not save money, and I do it anyway

Fri, 02 Oct 2026 00:00:00 +0000

I have had this debate many times, so I am finally writing it down.

Whenever I say that self-hosting AI does not save money, people hear that I am against self-hosting. That is not the point at all. I am a massive fan of open-weight models. Qwen3.8 27B runs on two RTX 3090s at home (and 15 more models), and my phone dictation goes to Qwen3-ASR on the same machine. I think open-source AI is the best thing since sliced bread.

I don’t pretend it saves money. I do it for fun, for sovereignty, and for privacy, which I come back to at the end.

All benchmark numbers and prices below come from Artificial Analysis and OpenRouter as of September 30, 2026. I did the math for the hardware I own, and for the counterarguments I hear most. All assumptions are listed at the end.

Benchmarks are not everything, and a high score does not reliably predict how a model does on real work. They are still the best we have: all models are benchmaxed, tuned to do well on the popular benchmarks, so the scores at least work as a reference frame for comparing them with each other.

Update, October 2, 2026: after the discussion on r/LocalLLaMA, I added section 5 on Macs, DGX Sparks, and AMD AI Max machines.

Table of contents

1. The model I love, and the one that beats it on price #

My favorite local model right now is Qwen3.8 27B. It came out in August and it is Apache-2.0. People often say models like this run on a single gaming GPU, but that is only true after quantizing them. At full precision, Qwen3.8 27B needs about 54 GB of memory. Quantized to about 4 bits per weight, it fits on one 24 GB card, at some cost in quality. I still split it across both of my 3090s, because the second card leaves room for a larger context window.

I compare it with OpenAI’s GPT-6 Luna, the cheap tier released on September 22. Both models let you choose how long they think, but the settings do not mean the same thing for both. Luna’s token use grows more than 20 times from its lowest setting to its highest, while Qwen’s barely changes. Qwen on low already writes more tokens than Luna on xhigh.

Average output tokens per Intelligence Index task at each reasoning setting. Qwen3.8 27B has no high or max setting. Hover for scores.

So the name of a setting says little on its own. I compare both models at xhigh, the highest setting Qwen offers, and count the tokens each one actually uses. The extra tokens Qwen needs are part of what I am measuring. There, Qwen scores 33.7 on the Artificial Analysis Intelligence Index and Luna scores 34.6. For reference, Claude Opus 4.6, a frontier model from February, scores 31.9. That is an amazing feat in itself. When Opus 4.6 was the best model we had, I never thought that within the same year I could run essentially that level of intelligence in my own house.

Artificial Analysis also publishes how many tokens each model used to run the index, and what that cost. So the question is simple: what does it cost to run the whole benchmark once?¹

Cost of one run of the Artificial Analysis Intelligence Index. The two bars for hardware at home are electricity only: measured on my 3090s, estimated for the RTX PRO 6000.

Luna runs the whole benchmark for $67. Qwen at full precision, through the cheapest provider that does not keep your data (ZDR), costs $619. Two things cause that gap: providers charge almost four times as much per output token for Qwen, and Qwen generates almost three times as many tokens to get the same work done.

The second bar is my own machine: the electricity alone for running Qwen on my GPUs costs more than Luna’s entire bill.

The third bar is what it takes to match the API’s full precision at home: an RTX PRO 6000, a workstation card with 96 GB of memory, enough for the full model.² Its electricity alone costs twice Luna’s bill, and one run takes more than seven weeks. The card by itself sells for $10,000 to $20,000, so with a computer around it, you are paying for a pretty nice car. Buy a few of them to run agents in parallel, and you are building a small data center of your own.

The bars for hardware at home are electricity only, so they are the lowest these runs can cost: the cards come on top, and how much depends on how busy you keep them, which is what section 4 is about.

What I care about is the cheapest way to get answers of Qwen’s quality, from whichever model gives them.

2. “But my GPUs are already paid for” #

This is the argument I hear most, so assume the hardware is free and only the electricity counts.

I measured my own machine. It runs Qwen in vLLM, split across both cards, at about 107 tokens per second.³ The cards are capped at 270 W each,⁴ and the whole machine draws about 700 W. It then needs four weeks, running day and night, to get through the benchmark once. I also give my quantized copy the score Artificial Analysis measured through Alibaba’s API. I have not rerun the benchmark on it, and anything lost to quantization makes my machine look worse.

Electricity for one benchmark run of Qwen3.8 27B on my two 3090s, by electricity price. The dashed line is what Luna charges for the same work, with OpenAI’s hardware and margin included. Hover for values.

Above 14.5 cents per kWh, my electricity alone costs more than Luna’s API bill. Below it, I save a few dollars per run and wait four weeks instead of a few hours.

Those four weeks matter more than the few dollars. An API takes hundreds of requests at once, so even one benchmark run can finish in an hour or two. My setup works on one conversation at a time: when I sent it eight requests at once, seven of them waited in line.

Wall-clock time for one benchmark run. The API times use Artificial Analysis’s measured time per task for Luna; the parallel case assumes 100 requests at once. Log scale.

This is one more reason my coding agents run on APIs. When I run several agents in parallel, each of them should be as fast as if it were alone.

3. Why datacenters are so much better at this #

When my 3090s generate a token, they read all of the model’s weights from memory to produce one token for one conversation. Most of the chip’s compute sits idle while it waits for memory. A datacenter GPU reads the same weights once and produces a token for hundreds of conversations at the same time. This is called batching, and it is where most of the efficiency comes from.

Artificial Analysis measures this with AgentPerf, which replays real coding-agent sessions against datacenter hardware. It counts how many agents a system can serve while each one still gets at least 60 tokens per second.

Coding agents served per kW of accelerator power, each getting at least 60 tokens per second, from AgentPerf. The datacenter systems all run DeepSeek V4 Pro. The 3090 bar is my machine running Qwen3.8 27B, a much smaller model, with GPU power only, so the comparison favors my machine.

The rack of 36 GB300s serves 20 times more agents per kW than the 8-GPU H200 server. Part of that is newer chips, and part is software: NVIDIA tuned the B300 and GB300 setups itself, while Artificial Analysis configured the H200 one. The B300 and the GB300 share both the chip generation and the software. Most of the gap between those two bars comes from the rack itself: 36 GPUs on one fast interconnect keep more than a thousand agents in flight at once. My two 3090s serve one conversation at a time.

4. “But I have solar panels” #

Solar power is not free. With net metering, every kWh my GPUs burn is a kWh that does not get credited at the retail price. Without net metering, it is a kWh I do not sell, unless the surplus would go to waste anyway, which is the free-power case below. And the GPUs also run at night.

But fine, say the electricity is free and the only cost is the hardware. I was lucky and bought my two 3090s for about $750 each,⁵ and I write them off over three years. The GPUs only save money while they do work I would otherwise pay an API for, and while they sit idle they save nothing. So the question is how many hours a day they have to be busy before they pay for themselves.⁶

To make this fair to open models, imagine an open model that is exactly as efficient as Luna: the same quality from the same number of tokens, sold at Luna’s price.

How many hours a day my two $750 RTX 3090s have to be busy, every day for three years, before they cost less than a ZDR API. The host PC is not counted.

With free power, the GPUs pay for themselves if they are busy 4 hours a day, every day for three years. At 18 cents per kWh, they need 8.5. That uses what I paid for the cards; at today’s price of about $1,500 each, the 8.5 hours become 14.

And API prices keep falling. GPT-6 Luna costs 58% less per output token than GPT-5.6 Luna did in July. If the price halves again, no amount of use pays off at 18 cents per kWh. If it halves twice, the API costs less than my electricity alone.

5. “But my Mac, DGX Spark, or AI Max does save money” #

Several readers told me their machines do save them money: a MacBook Pro with an M5 Max, an NVIDIA DGX Spark, and a computer with AMD’s Ryzen AI Max+ 395. If you would have bought the machine anyway, that is true, for the reason in section 2: their electricity costs little compared with the API. Counting the hardware is a different story.

With 128 GB of memory, all three can hold a better model than my 3090s: Qwen3.8-Flash-Next, a sparse model with 180 billion parameters, of which only 6 billion work on each token. It scores 39.8, close to GPT-6 Luna on its highest setting (38.1), although at the Q2 a reader uses on their Mac it probably loses a few points.⁷ So how many years would each machine have to run nonstop before it pays for itself?

Years of nonstop generation before each machine pays for itself, compared with the most efficient API model at a similar score. Each machine runs the best model it can hold.

Even with free power, the 128 GB machines need six to ten years. The AMD machine draws only about 120 W, but it is slow too: one benchmark run of Qwen3.8 27B takes it four and a half months, and almost as much energy as my 3090s. The RTX PRO 6000 runs the same model about three times as fast but costs about twice as much, so it needs six to nine years. My 3090s pay off in under two years with free power, which is why the 3090 is still the value king, but at 18 cents per kWh they never do: the electricity for one run already costs more than Luna charges for it.

Ten years is a long time in AI. Better and more efficient open models will run on the same machines, but the frontier moves at the same pace, and API prices keep falling. Hardware gets more efficient too, in datacenters and at home, so a machine bought today also ends up competing with next year’s machines. These numbers only show how far apart the two sides start.

6. The bar keeps moving #

Section 4’s eight and a half hours a day would be easy for me: I run agents for longer than that, just not on Qwen. When Claude Opus 4.6 came out in February, I was perfectly happy with it. I thought it was all I would ever need, and I could not have imagined how much better models would get in half a year. Qwen3.8 27B now scores about the same as Opus 4.6, and I would no longer accept it for coding. Until GPT-6 Astra came out at the start of September, my go-to was GPT-5.6 Sol, which scores ten points higher than Qwen. I then used Astra until Claude Opus 5.5 came out less than three weeks later. Now I don’t even accept what was considered the best model a month ago.

Intelligence Index of the best model available on each date, of the models I used as my go-to, and of Qwen’s 27B models, which fit on a single 3090. Scores use the xhigh reasoning setting where a model has it.

The models that fit on my 3090s keep improving, but they stay about half a year behind the frontier, and my standard moves with the frontier.

7. A small company has it worse #

For me, this is a hobby, and a hobby is allowed to be inefficient. It gets worse once you try to use self-hosted models seriously, say for a team of ten developers.

If you stay fully self-hosted and want fast responses at peak, you have to buy hardware for the busiest hour of the year, not for the average one. The rest of the time, it sits idle.

An illustrative week of coding agents for a ten-person team. The hardware has to cover the busiest hour of the year, but a typical week uses only a fraction of it.

In this made-up but realistic week, the team uses 17% of what it paid for, so every token costs about six times more than it would at full load. You also need a spare GPU for when one dies, and someone who gets paged when it does. You could queue work or send the peaks to an API, but then you are paying for an API anyway.

An API provider has the opposite situation. It serves thousands of customers across every time zone, so its load curve is much flatter than yours. It fills the nights with discounted batch jobs (Luna’s batch tier is half price) and training runs. And more concurrent requests mean bigger batches, which is where the efficiency from section 3 comes from.

8. “They lose money on inference, so prices will go up” #

Another argument I hear is that API prices are subsidized by investors and will go up once those investors want their money back. I don’t think that holds for inference.

AgentPerf also reports how many tokens each GPU serves. On the GB300 rack, one GPU serving DeepSeek V4 Pro handles about 4 million output tokens and 530 million input tokens per hour. Most of that input is conversation history that coding agents send again with every step. Providers keep it in a cache and charge less than 1% of the normal input price for it.

At DeepSeek V4 Pro’s list prices, that GPU brings in at least $5.70 per hour, even if every input token is billed at the cache price. If 5% of the input misses the cache, it brings in about $17. Renting a B200, the closest GPU with a published rental price, costs $3.50 to $5.90 per hour from smaller cloud providers, and that price already includes their profit.

So the tokens pay for the hardware that serves them, even in the worst case. This leaves out costs like staff and training, so it does not tell you whether a lab makes money overall.

The money goes to training, research, free users, and flat-rate subscriptions for heavy users like me. When I wrote that I used $10,000 worth of API tokens for $200, that was list price, not what those tokens cost to serve.

And prices go down, not up. This is what the same level of intelligence has cost this year:

Output price per million tokens for models scoring between 30 and 35 on the Intelligence Index, by release date, as listed by Artificial Analysis. Log scale.

From Claude Opus 4.6 in February to GPT-6 Luna in September, the output price for this level of intelligence dropped by a factor of 50. Labs are in a race to the bottom on price.

9. “But they sell your data” #

Some people say APIs are cheap because the provider trains on your data or sells it. That is why I only used prices from providers that OpenRouter lists as zero data retention (ZDR).

Model	Provider	ZDR	Output ($ per million tokens)
GPT-6 Luna	OpenAI	no	0.50
GPT-6 Luna	Azure	yes	0.50
Qwen3.8 27B	AkashML (FP8)	yes	1.78
Qwen3.8 27B	DeepInfra (full precision)	yes	1.88
Qwen3.8 27B	Alibaba	no	2.55
Qwen3.8 27B	Cloudflare	no	3.20

Azure serves GPT-6 Luna with zero data retention at the same price as OpenAI. For Qwen3.8 27B, the cheapest endpoints are all ZDR, and the endpoints without ZDR cost more. The :free tier is where you pay with your data.

10. Where self-hosting does win: the same model #

If you compare running Qwen3.8 27B yourself with paying for Qwen3.8 27B through an API, self-hosting wins.

DeepInfra, the cheapest full-precision ZDR provider, charges $619 for one benchmark run, while my electricity costs $83. That compares the full model with my quantized copy, so the gap overstates my advantage by whatever quantization costs in quality. My machine pays for itself if it is busy 2.4 hours a day, or 1.5 hours with free power (the first group in the chart in section 4).

Providers charge a lot for a dense 27B model, because every token runs through all 27 billion parameters. Sparse open models, which use only a small part of their weights for each token, can be much cheaper to serve. If you specifically need Qwen and use it heavily, self-hosting it pays off. But you don’t need Qwen to get answers of Qwen’s quality. Luna gives you those for $67, with zero data retention.

11. So why do I do it? #

First, it is fun. I like knowing how the whole stack works, from the drivers in my NixOS configuration to how the layers are split between my two GPUs.

Second, nobody can take it away. Artificial Analysis already lists GPT-5.6 Luna as deprecated, less than three months after its release. My Qwen weights will still be on my disk in ten years, and no company can change their license or close their build system on me.

Third, some data should not leave my house. I would never send 200 GB of email, my messages, and my location history to an API, zero data retention or not. That is what my local AI projects are for.

Those are good reasons. Saving money is not one of them.

Appendix: all assumptions #

Show all assumptions, and which side each one favors

Assumptions that favor self-hosting

The host PC around the GPUs is not counted, and my 3090s count at the $750 each I paid, not the $1,500 they sell for today.
Quantized local copies get the score Artificial Analysis measured through an API, with no penalty for quantization.
The payback chart assumes each machine is busy 24 hours a day, with no idle time.
API prices stay where they are, although GPT-6 Luna’s output price is 58% lower than GPT-5.6 Luna’s, two and a half months earlier.
My time to set up and maintain the machine is not counted.
Every machine also gets a case with free electricity.

Assumptions that favor the API

The example electricity price is 18 cents per kWh, and some people pay less.
My machine serves one request at a time; batching several requests could raise its total throughput.
The hardware has no resale value at the end.
Waste heat that replaces heating in winter is not counted.
API outages, rate limits, and deprecated models are not counted.

Inputs

Benchmark: the Artificial Analysis Intelligence Index on September 30, 2026, at the xhigh reasoning setting, or the highest setting where noted. Its token counts per model are the unit of work.
API prices: the cheapest zero-data-retention endpoints on OpenRouter, with cached input billed at the cache price.
My machine: two RTX 3090s capped at 270 W, running Qwen3.8 27B in vLLM with AutoRound INT4 and DFlash2. Measured: 107 tokens per second, 1,600 tokens per second of prefill, and 540 W for both GPUs. Estimated: about 700 W for the whole machine under load and 150 W idle.
Break-even: the cards are written off over three years; busy hours save what the API would charge minus the electricity, and idle hours cost the idle power.
Other machines: prices, power, and speeds as in the footnotes. The M5 Max and RTX PRO 6000 numbers come from readers, and the DGX Spark and AMD Ryzen AI Max+ 395 speeds are estimated from Artificial Analysis measurements.
Datacenter: AgentPerf measures accelerator power only, and the B200 rental price stands in for the GB300, which has no published rental price.
Small company: the peak-load week is made up.

For each model, Artificial Analysis publishes how many input and output tokens the whole index took, and how many of the input tokens were read from a cache. I multiplied those by each provider’s prices, with cached input at the cache price. For Qwen that is 198 million output tokens and 820 million input tokens that miss the cache. For the bars at home, the same tokens divided by the machine’s speed give the run time, with the uncached input processed at the 1,600 tokens per second I measured on my 3090s and an estimated 2,000 on the RTX PRO 6000, and the run time times the machine’s power and the price per kWh gives the electricity. ↩︎
The RTX PRO 6000 has the same memory bandwidth as an RTX 5090, and at full precision every token reads three times as many bytes as in Q4_K_M. Scaling Artificial Analysis’s 5090 measurement gives about 48 tokens per second. I assume 600 W for the whole machine. ↩︎
I measured the model as I normally run it: vLLM with an AutoRound INT4 quant and DFlash2 speculative decoding, split across both cards. Single 1,500-token answers came out at 105 to 109 tokens per second, and reading a 22,000-token prompt ran at 1,600 tokens per second. Sending 2, 4, or 8 requests at once did not raise the total above about 110 tokens per second. The two GPUs drew 540 W together, each at its 270 W power limit; the 700 W adds my estimate for the rest of the machine, since I could not measure the CPU. The setup follows the recipes from club-3090, a community project that tunes LLM serving for RTX 3090s and publishes measured numbers, so I think it is about as fast as these cards get. Its benchmarks for this configuration at a similar power limit match mine: about 110 tokens per second for prose, and up to about 195 for code, where speculative decoding guesses more tokens right. Most of Qwen’s output on the benchmark is reasoning, which runs at the prose speed. ↩︎
The stock limit is 350 W. My cards sit in a normal desktop case with normal fans, and I worry that running both at full power for hours would shorten their life. The cap is a few lines in my NixOS configuration, and measurements from Puget Systems and r/LocalLLaMA put a 3090 at about 95% of its speed at 270 W. That is 77% of the power for 95% of the speed, so the cap also lowers my electricity cost per token. ↩︎
The 3090 is still the value king for VRAM per dollar, and $750 badly understates what mine are worth. That is what I paid more than a year ago; today a used 3090 sells for about $1,500. Even at that price it costs $62.50 per GB of VRAM, the same as an RTX 5090 at its $2,000 launch price, which is not what a 5090 sells for today. The right number for this calculation is what I could sell my cards for, and at $1,500 each the break-even points rise by about two thirds: at 18 cents per kWh, the same-model case moves from 2.4 to 4.0 hours per day, and the Luna-efficient case from 8.5 to 14.2 hours per day. ↩︎
While the cards are busy, they save what the API would have charged for the same work, minus the electricity at 700 W. While they are idle, the machine still draws about 150 W, of which the two GPUs with the model loaded take 85 W. The break-even point is the number of busy hours per day at which those savings cover the price of the cards, written off over three years, plus the idle power. ↩︎
The M5 Max numbers come from the reader: Qwen3.8-Flash-Next at Q2 (about 2.7 bits per weight), about 45 tokens per second, 1,250 tokens per second of prefill, and about 90 W, on a $7,000 laptop. The RTX PRO 6000 numbers come from another reader, u/TheAILegend, who measured Qwen3.8-Flash-Next at NVFP4 on theirs. For a single request, it stays at about 150 tokens per second from 4,000 to 200,000 tokens of context, with about 10,000 tokens per second of prefill and 266 W for the GPU plus about 160 W for the rest of the PC. With 32 short requests in parallel, the card reaches 590 tokens per second and pays off in about two years, which works for short batch jobs. At the long contexts of this benchmark, parallel requests are slower than one, so the chart uses the single-request speed. I use $15,000, the middle of the card’s price range. The DGX Spark and the AMD machine run Qwen3.8-Flash-Next at about 4.5 bits. For their speed, I took Artificial Analysis’s measurements of Qwen3.8 27B on each machine and doubled them, because the owner of the AMD machine finds Qwen3.8-Flash-Next about twice as fast as Qwen3.8 27B. That gives about 55 tokens per second on the DGX Spark ($6,950, about 150 W) and 47 on the AMD Ryzen AI Max+ 395 ($4,000, the price Artificial Analysis lists, about 120 W). All of them are compared with GPT-6 Luna on its highest setting, which runs the whole benchmark for $122. The 3090s run Qwen3.8 27B, as in the rest of this post, and are compared with Luna on xhigh. Artificial Analysis measured Qwen3.8 27B on the AMD machine at 23 tokens per second, so one benchmark run takes it 134 days and about 390 kWh, against about 460 kWh on my 3090s. Running Qwen3.8-Flash-Next on the 3090s with CPU expert offload, which club-3090 measured at 37 tokens per second, does not change their result: one run then takes 82 days, and at 18 cents per kWh its electricity costs more than Luna’s $122 unless the whole machine draws less than about 350 W. ↩︎

Frontier-level dictation on your iPhone keyboard, self-hosted

Tue, 22 Sep 2026 00:00:00 +0000

On a recent month-long trip visiting family in Europe, I did a lot of work from my phone, using my mobile coding workflow. It still boggles my mind that real, productive work from a phone is possible now. Most of that work is talking to coding agents, and the built-in iOS dictation is garbage for that. I had an iOS Shortcut that sent recordings to Agent CLI, my local AI toolbox, running at home instead. The problem was that the recording screen takes over the whole display, so I could not see the thing I was commenting on. The transcript then landed in my clipboard, and I had to paste it myself.

So I threw my principles overboard and installed Wispr Flow, a voice keyboard for iOS. I loved it instantly. iOS does not let keyboards use the microphone, so the keyboard’s button hands off to the Wispr Flow app, which keeps recording in the background. I was less happy about that, since it meant a closed-source app always had the microphone open and sent my voice to their servers. I was on the free plan, so I was probably the product. I also hit the free weekly word limit very quickly, and started looking for an alternative.

That is how I found Diction, which has since become my favorite app when I’m away from my laptop. It works the same way, background recording included. The difference is that it has a self-hosted mode with an open-source gateway, so my voice goes to a machine I own instead of their servers.

I run the gateway on my home machine in the U.S., reachable only over my private network. The gateway streams audio to Agent CLI, which runs Alibaba’s Qwen/Qwen3-ASR-1.7B-hf on an RTX 3090 behind an OpenAI-compatible transcription endpoint. There is no LLM cleanup step, so Diction’s Writing Style and Tones features do not work with this setup. I don’t miss them: Qwen’s raw transcription is already good enough for me.

1. Why Qwen? #

For the longest time I used Faster-Whisper with Whisper large-v3. Whisper is still the default choice for most people, but many newer models beat it. I tried some of them, including NVIDIA’s Parakeet. Parakeet is fast, but it cannot take custom instructions. Those instructions are how Diction’s My Words feature biases a transcription toward specialized names.

At the start of September, I looked at the Hugging Face Open ASR Leaderboard, which tests speech-to-text models on the same recordings and ranks them. The score is the word error rate (WER): the share of words a model gets wrong, counting words it swaps, drops, or makes up. Lower is better, and a WER of 4% means about one wrong word in every 25. The test recordings range from audiobooks and podcasts to meetings and earnings calls, so the average says more than any single clean benchmark.

These are the averages on the public English test sets as of September 19, 2026:

Model	WER	Can I run it at home?
Zoom Scribe v2 Pro	3.6%	No, paid API (#1 overall)
ElevenLabs Scribe v2	4.0%	No, paid API
Qwen3-ASR 1.7B	4.3%	Yes, Apache-2.0
AssemblyAI Universal-3.5 Pro	4.3%	No, paid API
NVIDIA Parakeet TDT 0.6B v3	4.9%	Yes
OpenAI Whisper large-v3	5.8%	Yes

I picked the best model on the board whose weights you can download: Qwen3-ASR 1.7B. It lands within a point of the best paid API, which is what I mean by frontier-level. Since I maintain Agent CLI, I told an agent to add a Qwen backend, and shortly afterward it had landed and was running on my server.

Agent CLI’s transcription server loads a model on the first request and unloads it after an idle timeout, so the model only takes VRAM while I use it. The same process serves an OpenAI-compatible API, which the Diction gateway talks to, and the Wyoming protocol that Home Assistant uses for local voice.

2. The recipe #

You need an NVIDIA GPU with the NVIDIA Container Toolkit, Docker Compose 2.30 or newer, and a private route from your iPhone to the server. I use Headscale, but Tailscale or plain WireGuard works just as well. Do not expose the gateway to the public internet.

Write the Compose file #

Save the following as compose.yaml in an empty directory. Agent CLI’s CI publishes a CUDA image with the Transformers backend and Qwen as the default model, so there is nothing to build. The file points Diction’s gateway at Agent CLI’s transcription port. WHISPER_TTL raises the idle timeout from the default five minutes to a day, so dictation rarely waits for a reload:

services:
 asr:
 image: ghcr.io/basnijholt/agent-cli-whisper:0.112.0-cuda-transformers
 environment:
 WHISPER_TTL: "86400"
 volumes:
 - model-cache:/home/whisper/.cache
 gpus: all
 restart: unless-stopped

 gateway:
 image: dictionlabs/gateway:v14.0
 depends_on:
 - asr
 environment:
 CUSTOM_BACKEND_URL: http://asr:10301
 CUSTOM_BACKEND_MODEL: Qwen/Qwen3-ASR-1.7B-hf
 CUSTOM_BACKEND_CANONICAL_ID: Qwen/Qwen3-ASR-1.7B-hf
 CUSTOM_BACKEND_NEEDS_WAV: "true"
 ports:
 - "8080:8080"
 restart: unless-stopped

volumes:
 model-cache:

Start it and connect the app #

Start both services:

docker compose up -d

The first transcription downloads Qwen into the model-cache volume, so that request can take a few minutes. docker compose logs -f shows the progress.

On the iPhone, go to Settings → General → Keyboard → Keyboards → Add New Keyboard → Diction. Tap Diction in the keyboard list, enable Allow Full Access, and grant microphone access when prompted. Then open Diction, go to Preferences → Mode → Self-Hosted, enter http://<private-server-ip>:8080, and tap Test connection.

On my machine, Qwen3-ASR 1.7B uses about 4.7 GiB of GPU memory and roughly 3 GiB of system memory while loaded.

3. The bar test #

A day later, I met a friend in a bar in the Netherlands and wanted to let him try it.

He was already on my tailnet for some of my other self-hosted services, so his phone only needed an ACL change to reach the gateway. I opened the terminal app on my phone, connected to an agent, and dictated that change. In less than a minute, Diction was connected and working on his phone. His speech crossed the Atlantic, was transcribed by Qwen on my home GPU, and appeared back in the app with barely any delay.

That moment sold the whole setup to me. I was in a bar without a laptop, and I used dictation to grant access to the dictation service.

SOTA RAG & Memory without the database: Files, Git, and simple folders

Wed, 10 Dec 2025 00:00:00 +0000

Table of contents

1. The amnesia problem #

If you’ve read my posts on agentic coding or my local AI journey, you know I am all-in on using AI to control my computer and write software. But one critical piece was missing: Context.

LLMs are powerful but stateless. Every time I start a new chat, the model forgets who I am, what I’m working on, and that I prefer functional Python code over object-oriented styles. Worse, it also forgets the thing I just told it five minutes ago in another window.

The standard solution to this is RAG (Retrieval-Augmented Generation) and Memory. But when I looked at the available tools, I hit a wall.

Most solutions fell into two buckets:

SaaS / Web UIs: Upload your files to a cloud service or use a specific web chat interface that locks you in.
Developer Libraries: “Just use LangChain/LlamaIndex!” they say. Great, now I have to write a Python script every time I want to ask a question about a PDF?

I didn’t want a library. I didn’t want a web UI. I wanted a folder. I wanted to drop a file into ~/my-docs, and have my AI instantly know about it. And I wanted my AI’s “memory” to be a file I can edit, not a hidden vector in a database I can’t inspect.

So I decided to build my own file-based solution.

2. Learning from failure: AI Journal #

This isn’t my first attempt at building a memory system.

A few months ago, I built AI Journal—an ambitious local-first journaling app where an AI would read your entries and build a “living self-model” of who you are. It had sophisticated features: a structured “persona core,” typed “claim atoms” with evidence links, four organizational layers (L1→L4), recency-aware scoring with decay functions, and Git-based time travel to “ask questions of your younger self.”

The architecture seemed solid. The problem was simpler: local models aren’t good enough yet.

I designed the whole thing to run entirely on Ollama with my dual RTX 3090s. But even with a 48GB of VRAM, the models couldn’t reliably extract structured claims, handle the multi-step reconciliation, or maintain coherence across long conversations. Whether the design was right, I still don’t know—but the execution definitely hit the wall of current local LLM capabilities.

The lesson: I tried to build the entire system at once without validating the core concepts and individual ideas first. This time, I took a different approach: tackle it problem by problem. Instead of designing a complete system upfront, I would validate each piece—chunking, retrieval, re-ranking, memory reconciliation—before combining them. The experience and UX would be my own, but the algorithms would be adapted from established projects.

3. How I studied SOTA: clone, read, re-implement #

My approach was more methodical this time. I cloned five major frameworks into the same workspace as agent-cli:

LlamaIndex: The OG RAG framework
LangChain: The kitchen-sink approach
Letta (formerly MemGPT): Stateful agents with long-term memory
Mem0: Focused specifically on memory extraction
PydanticAI: Clean, typed agent framework

I didn’t read through all this code myself—that would take months. Instead, I used AI coding agents to search the codebases and write summaries of the technical implementation details:

Gemini CLI (Gemini 3.0 Pro)
Claude Code (Opus 4.5)
Codex CLI (GPT-5.1-Codex-Max)

I fed those summaries into ChatGPT 5 Pro together with my high level plan of what I wanted to create, and it created an implementation plan. Then I took that plan back to the coding agents for implementation. I personally found Opus 4.5 the best coding model, but GPT‑5.1 Codex Max was the best at reviewing the code and actually identifying critical bugs.

For validation, I used clip-files to copy the agent-cli source code plus the architecture documents (like memory.md and rag.md) into ChatGPT 5 Pro to validate the approach.

The AI did the searching and implementing; I did the understanding and decision-making.

4. The dependency problem: why ONNX over PyTorch #

One thing became immediately clear when going through these frameworks: they’re massive.

Installing LlamaIndex with all its dependencies pulls in PyTorch. PyTorch alone is 2-3GB. Add transformers, sentence-transformers, and the kitchen sink, and you’re looking at an 8GB+ environment just to ask questions about some PDFs.

This felt excessive for a CLI tool.

One of my core design goals was minimal dependencies. I wanted agent-cli to install in seconds, not minutes. I wanted it to work on my Mac, my NixOS server, and my friend’s laptop without downloading half the internet.

The solution: ONNX Runtime.

ONNX Runtime is ~200MB and runs inference on basically anything—CPU, CUDA, even Apple Silicon. For embeddings, I use embeddinggemma running locally via llama.cpp. For re-ranking, I use a cross-encoder exported to ONNX (Xenova/ms-marco-MiniLM-L-6-v2).

The result: pip install agent-cli[rag] takes about 30 seconds and adds ~300MB. The key insight: embeddings and re-ranking are cheap operations. You don’t need a GPU for them—CPU is perfectly fine. PyTorch with CUDA support can easily reach 2-3GB+; ONNX Runtime is ~200MB and runs anywhere. Since I’m not training models, just running inference on small encoders, ONNX is the obvious choice.

5. RAG: just let me grep it (kind of) #

My requirement for RAG was simple: Files on disk are the source of truth.

I didn’t want to manage an ingestion pipeline or trigger APIs. I implemented a RAG proxy that uses OS-level file watchers (watchfiles). Here is the workflow:

I drop a PDF, Markdown, or Python file into ~/rag_docs.
The system detects the Create event.
It chunks the text and embeds it into a local ChromaDB instance.
It’s immediately searchable.

When moving or deleting files, the system updates the index accordingly.

The architecture: OpenAI compatibility #

The “secret sauce” isn’t the retrieval itself; it’s how I expose it. Instead of building a custom chat app, I built an OpenAI-compatible proxy.

flowchart LR Client["Client
(Cursor, Open WebUI)"] --> Retrieve subgraph Proxy ["RAG Proxy"] direction TB Retrieve --> Augment["Augment & Forward"] Chroma[("ChromaDB")] -.-> Retrieve Watcher["File Watcher"] --> Chroma end Augment --> Upstream["Upstream LLM"]

# Point any OpenAI-compatible tool at your local proxy
export OPENAI_BASE_URL=http://localhost:8000/v1

That’s it. Any tool that supports custom OpenAI endpoints—Cursor, Cline, Open WebUI, LibreChat—instantly gains the ability to “read” my document folder.

Two-stage retrieval: the SOTA approach #

I looked at LlamaIndex here. Standard “semantic search” (using just embeddings/bi-encoders) is often disappointing. It finds text that looks similar but isn’t actually relevant.

A common pattern in modern RAG systems is Two-Stage Retrieval.

Fast Stage: Retrieve 3x candidates using standard embeddings (text-embedding-3-small or local models like embeddinggemma).
Smart Stage: Use a Cross-Encoder to re-rank the results.

A cross-encoder looks at the query and the document together and outputs a relevance score. It’s computationally heavier, but since I’m running this locally on my RTX 3090 (or even CPU via ONNX), the 50ms latency penalty is worth it for results that are actually useful.

This isn’t a new idea, but implementing it cleanly with ONNX avoided the huge dependency tax.

6. Memory: files over databases #

This is where my approach diverges most from standard tools like Mem0. Most memory systems treat memories as database rows. If the AI remembers “User likes pizza,” that’s a vector in a DB. If it hallucinates and records “User hates pizza,” good luck fixing it. You need to write a script to query the DB and delete the row.

I prefer data I can inspect.

My memory system stores every fact as a Markdown file with YAML front matter.

---
id: "550e8400-e29b-41d4-a716-446655440000"
role: "memory"
created_at: "2025-12-15T10:00:00Z"
conversation_id: "default"
---
The user prefers using 'uv' for Python package management.

Git integration: versioning memory 🕰️ #

Because memories are just files, I added a feature that I haven’t seen anywhere else: Automatic Git Versioning.

Every time the AI learns a new fact, updates an old one, or deletes a contradiction, the system automatically commits the change to a local git repository.

This leads to incredible capabilities:

Inspectability: I can git log and see exactly when the AI learned something.
Correction: If the AI learns something wrong, I open the markdown file in VS Code, edit it, and save.
Reversion: Agent hallucinated wildly yesterday? git reset --hard HEAD~1.

The reconciliation loop #

The logic for storing memories isn’t just “append to file.” Digging into Mem0 (and Letta) showed me that memory systems need active reconciliation. When I say “Actually, I switched to nix instead of uv,” the system:

Retrieves existing memories about package managers.
Decides (using a lightweight LLM) whether to ADD, UPDATE, or DELETE.
Executes the file operation (moving the old fact to a deleted/ folder for audit trails).

This keeps the memory bank clean and contradictory-free, unlike simple vector stores that just accumulate conflicting junk over time.

The tiny detail that broke everything #

After implementing the memory system—mostly modeled after Mem0—I noticed it didn’t work as well. After a lot of debugging, I found a tiny difference: I was asking the model for structured data via a tool call, whereas Mem0 uses freeform JSON that the LLM returns directly.

Since I use PydanticAI, fixing this was a one-line change:

agent = Agent(
 model=model_cfg,
 system_prompt=UPDATE_MEMORY_PROMPT,
- output_type=list[MemoryDecision],
+ output_type=PromptedOutput(list[MemoryDecision]), # JSON mode instead of tool calls
 retries=3,
)

This is exactly why reading the source code matters—not just the algorithms, but the tiny details that make them work.

7. Why building this was necessary #

You might ask: “Bas, why write thousands of lines of code for this? Why not just use ChatGPT’s memory or install LangChain?”

Three reasons: Privacy, Integration, and Dependencies.

Privacy #

I want to feed my AI my financial records, my health data, and my private journals. I am not comfortable sending that to OpenAI’s long-term storage or some VC-backed startup’s vector cloud. With agent-cli, the vectors live in ~/.cache/agent-cli/chroma, the files live in ~/documents, and the only thing leaving my machine is the specific context needed for a specific query (if I use a cloud model) or nothing at all (if I use Ollama/llama.cpp).

Integration #

By building this as an API proxy, I solved the “fragmented tools” problem. Any tool that supports custom OpenAI-compatible endpoints works out of the box:

agent-cli chat in the terminal
Cursor or Cline for agentic coding
Open WebUI, LibreChat, or Lobe Chat in the browser

Because my system speaks “OpenAI,” all of these tools share the same memory and the same document index. If I tell the terminal agent “I’m working on Project X,” and then switch to Cursor, Cursor knows about Project X. I can try a new chat UI tomorrow without losing anything—just point it to localhost:8000.

Dependencies #

To me, it just doesn’t feel right that asking questions about a PDF requires downloading 8GB of PyTorch. By choosing ONNX Runtime over the PyTorch ecosystem, I kept the entire install lightweight and fast. This matters when you want to run the system on a cheap VPS or your friend’s laptop.

8. Try it yourself #

The entire system is open-source and available in agent-cli.

Try Memory #

Get an LLM that remembers you using Ollama.

# 1. Pull models and start proxy
ollama pull embeddinggemma:300m && ollama pull gpt-oss:20b
uvx -p 3.13 --from "agent-cli[memory]" agent-cli memory proxy \
 --memory-path ./my-memories \
 --openai-base-url http://localhost:11434/v1 \
 --embedding-model embeddinggemma:300m

# 2. Chat from the terminal (env vars point to proxy)
export OPENAI_BASE_URL=http://localhost:8100/v1 OPENAI_API_KEY=dummy

# Tell it something about yourself
uvx openai api chat.completions.create -m gpt-oss:20b -g user "My name is Alice and I love hiking"

# Later, ask if it remembers
uvx openai api chat.completions.create -m gpt-oss:20b -g user "What's my name?"

Try RAG #

Chat with your documents.

# 1. Start the RAG proxy (runs in foreground)
uvx -p 3.13 --from "agent-cli[rag]" agent-cli rag-proxy \
 --docs-folder ./my-docs \
 --openai-base-url http://localhost:11434/v1 \
 --embedding-model embeddinggemma:300m

# 2. Chat with your docs
OPENAI_BASE_URL=http://localhost:8000/v1 OPENAI_API_KEY=dummy \
 uvx openai api chat.completions.create \
 -m gpt-oss:20b \
 -g user "What does my documentation say about X?"

9. What I learned #

This journey reinforced a few beliefs:

Validate components first. AI Journal failed not because the ideas were wrong, but because I tried to build a complete system before validating individual components. This time I tackled problems one at a time—chunking, retrieval, re-ranking, memory reconciliation—and only combined them once each piece worked.
Study implementations, not just papers. Reading about two-stage retrieval is one thing. Reading how LlamaIndex actually implements it—the edge cases, the defaults, the performance tradeoffs—is far more valuable. Clone the repos. Read (or make AI read) the code.
Dependencies matter. A 30-second install versus a 10-minute download changes whether someone actually tries your tool. I am happy with the ONNX Runtime over PyTorch choice.
Files are the ultimate API. By stripping away the complex databases and proprietary interfaces, I ended up with a system that is robust, version-controllable, and surprisingly simple to reason about.
Use the best model for each job. I used 4 different AI models during development, each with different strengths. Gemini 3.0 Pro, Opus 4.5, GPT-5.1-Codex Max. I used all of them during the implementation phase, but each one has its own strengths. Fore example, Opus 4.5 was the best coding model, but GPT-5.1 Codex Max was the best at reviewing the code and actually identifying critical bugs (use the built-in /review feature).

Switching from Ollama to llama-swap + llama.cpp on NixOS: the power user's choice 🦙

Sat, 29 Nov 2025 00:00:00 +0000

Table of contents

The draft I never published #

I started writing this blog post four months ago. Back then, I was frustrated because a specific model I wanted to try—gpt-oss-20b—was completely broken in Ollama. I went down the rabbit hole, set up llama-swap and llama.cpp, got the model working perfectly, wrote half this post… and then I stopped.

Why? Laziness. Once the novelty of that specific model wore off, I drifted back to Ollama. The real convenience of Ollama was never the ollama run command—I almost exclusively interact with my models via the API using my own tools like agent-cli. The convenience was ollama pull. Being able to grab a model with one command and have it “just work” was great.

But as a NixOS user, this imperative convenience is actually a flaw. I want my system to be declarative. I want my model configuration committed to git, not hiding in a local database. So I let convenience win for a while, but it always felt wrong.

But last month, that changed. I upgraded my hobbyist AI rig by adding a second RTX 3090. Suddenly, I had 48GB of VRAM and a whole new set of problems—and possibilities. I wanted to run massive models (like gpt-oss-120b or Qwen-3-VL-32B with a huge context window). I needed to balance layers between two GPUs. I needed to spill specific parts of the model into system RAM without crashing.

When you start pushing hardware to its limits, “magic” abstractions like Ollama stop being helpful and start getting in the way. I realized that llama.cpp isn’t just a backend; it is the place where development happens. If you want to use split-mode inference, granular CPU offloading, or the latest quantization tricks immediately, there is no alternative.

So, I dusted off this draft, updated my Nix configuration, and I’ve been running llama-swap full-time for a month. I’m finally ready to vouch for it.

The original trigger: broken models #

Four months ago, my frustration stemmed from compatibility. I was trying to run gpt-oss-20b, a model that required specific chat templates and GGUF handling that Ollama simply broke.

The maintainer of llama.cpp, Georgi Gerganov, explained it perfectly regarding the gpt-oss debacle: Ollama forked the ggml inference engine to rush out “day-1 support” for gpt-oss without coordinating with upstream. The result? Their implementation was not only incompatible with standard GGUF files but also significantly slower and unoptimized.

Benchmarks at the time showed Ollama had significant performance overhead—often 20-30% slower inference speeds compared to running llama.cpp directly.

This pattern means Ollama will always be behind llama.cpp. Meanwhile, I’ve had a pull request sitting unreviewed for over a month that fixes their broken OpenAI API. Compare that to llama.cpp: I submitted a PR there and it was merged in less than an hour. If you want the latest features now (and a responsive community), you go to the source.

The catalyst: dual GPUs and control #

With a single GPU, you usually either fit the model or you don’t. With two GPUs (and 64GB of system RAM), things get interesting. You enter the territory of heterogeneous inference.

Ollama tries to handle splitting automatically, but I found myself fighting it. I wanted to say: “Put 40 layers on GPU 0, 40 layers on GPU 1, and keep the KV cache here.” Or: “This model is slightly too big for VRAM; I want to offload exactly the last 10% to system RAM.”

With llama.cpp directly, this is just a command line flag. With llama-swap, I can define this behavior per-model in a config file:

"gpt-oss:120b":
 cmd: |
 ${pkgs.llama-cpp}/bin/llama-server
 -hf ggml-org/gpt-oss-120b-GGUF
 --port ${PORT}
 --ctx-size 65536
 --split-mode layer
 --tensor-split 3,1.3 # Tuned balance between my GPUs
 --n-cpu-moe 15 # Offload 15 experts to CPU RAM
 --threads 8
 --chat-template-kwargs '{"reasoning_effort": "high"}'

This level of granularity turned my rig from a “black box” into a tunable instrument. The --tensor-split and --n-cpu-moe flags allow me to run a 120B parameter model that technically shouldn’t fit, by carefully balancing the load across both GPUs and system RAM.

API compatibility and the “switch” #

One of the reasons I hesitated to switch was the fear of breaking my existing workflows. I use agent-cli and other custom Python scripts daily. But it turned out to be a non-issue. Both Ollama and llama-swap (wrapping llama-server) expose an OpenAI-compatible API.

Once I adopted the pattern of setting a custom OPENAI_BASE_URL in my projects, the backend became irrelevant. My tools don’t care if they are talking to api.openai.com, a local Ollama instance, or llama-swap. They just send JSON and get JSON back. This standardization meant I could swap the entire inference engine underneath my applications without changing a single line of their code.

My NixOS setup #

Since I manage my entire system with Nix, setting up llama-swap and pinning llama-cpp to the absolute bleeding edge was surprisingly elegant. I even wrote an auto-update script to ensure I’m always on the latest commit.

This is where my NixOS build cache shines. Unlike Ollama, which releases every few weeks, llama.cpp can have multiple releases a day. My cache server runs this update script automatically, builds the new binaries, and serves them to my workstation. I get the absolute latest performance improvements without ever compiling source code locally.

Compiler flags matter #

One critical lesson I learned: default builds matter. Initially, I was getting a measly 8 tokens/sec on gpt-oss:120b. It turned out the default llama.cpp package wasn’t enabling BLAS or native CPU optimizations, crippling the layers I offloaded to system RAM.

By overriding the package to enable blasSupport and passing -DGGML_NATIVE=ON (as seen below), performance jumped to 50 tokens/sec. This is another reason why I prefer managing this via Nix: I can enforce these compile-time flags declaratively.

Here’s my complete configuration (see package-overrides.nix and ai.nix):

# Override llama-cpp to latest version with CUDA support
llama-cpp =
 (pkgs.llama-cpp.override {
 cudaSupport = true;
 rocmSupport = false;
 metalSupport = false;
 # Enable BLAS for optimized CPU layer performance (OpenBLAS)
 blasSupport = true;
 }).overrideAttrs
 (oldAttrs: rec {
 version = "7205";
 src = pkgs.fetchFromGitHub {
 owner = "ggml-org";
 repo = "llama.cpp";
 tag = "b${version}";
 hash = "sha256-1CcYbc8RWAPVz8hoxKEmbAgQesC1oGFZ3fhfuU5vmOc=";
 leaveDotGit = true;
 postFetch = ''
 git -C "$out" rev-parse --short HEAD > $out/COMMIT
 find "$out" -name .git -print0 | xargs -0 rm -rf
 '';
 };
 # Enable native CPU optimizations (AVX, AVX2, etc.)
 cmakeFlags = (oldAttrs.cmakeFlags or []) ++ [
 "-DGGML_NATIVE=ON"
 ];
 # Disable Nix's march=native stripping
 preConfigure = ''
 export NIX_ENFORCE_NO_NATIVE=0
 ${oldAttrs.preConfigure or ""}
 '';
 });

# llama-swap from GitHub releases
llama-swap = pkgs.runCommand "llama-swap" { } ''
 mkdir -p $out/bin
 tar -xzf ${
 pkgs.fetchurl {
 url = "https://github.com/mostlygeek/llama-swap/releases/download/v175/llama-swap_175_linux_amd64.tar.gz";
 hash = "sha256-zeyVz0ldMxV4HKK+u5TtAozfRI6IJmeBo92IJTgkGrQ=";
 }
 } -C $out/bin
 chmod +x $out/bin/llama-swap
'';

# Configure llama-swap as a systemd service
systemd.services.llama-swap = {
 description = "llama-swap - OpenAI compatible proxy with automatic model swapping";
 after = [ "network.target" ];
 wantedBy = [ "multi-user.target" ];
 
 serviceConfig = {
 Type = "simple";
 User = "basnijholt";
 Group = "users";
 # Point to your declarative config file
 ExecStart = "${pkgs.llama-swap}/bin/llama-swap --config /etc/llama-swap/config.yaml --listen 0.0.0.0:9292 --watch-config";
 Restart = "always";
 RestartSec = 10;
 
 # Environment for CUDA support
 Environment = [
 "PATH=/run/current-system/sw/bin"
 "LD_LIBRARY_PATH=/run/opengl-driver/lib:/run/opengl-driver-32/lib"
 ];
 };
};

What I gained (and lost) #

Advantages of llama-swap + llama.cpp #

Heterogeneous compute: Perfectly balance loads between GPUs and CPU RAM.
Bleeding edge: I use features (like new quantization types) the day they land in llama.cpp.
Zero idle VRAM: Models unload completely when not in use.
Transparent operation: No hidden prompts or “helpful” formatting that breaks complex tasks.

The trade-off #

The main trade-off is “laziness.” You can’t just ollama pull new-model and have it appear. You have to find the GGUF on HuggingFace, decide on the quantization (Q4_K_M? Q6_K?), and add 5 lines of YAML to your config.

But honestly? That’s a feature, not a bug. It forces you to understand what you are running. It stops you from running a quantized model that is too stupid for your task just because it was the default. And for a NixOS user, putting that configuration into code instead of a hidden database is exactly how it should be.

Conclusion: maturing as an AI hobbyist #

My switch back to llama-swap wasn’t about fixing a bug; it was about outgrowing a tool. Ollama is the “starter bike” of local AI—fantastic for getting rolling. But when you add a second GPU, start caring about tokens per second, and want to maximize every gigabyte of your 48GB VRAM buffer, you need a manual transmission.

For anyone else building a hobbyist multi-GPU rig: stop fighting the abstraction. Go to the source. The control is worth it.

Check out my NixOS configuration for the complete setup!

Using gpt-oss:20b in OpenCode with a larger context and high reasoning

Thu, 13 Nov 2025 00:00:00 +0000

I have hundreds of private journal entries stored locally, many with incorrect date formats. The correct date is in the filename, so I wanted to use an LLM to batch update them. However, I do not want to share my private journal with any third-party service. Instead, I used Ollama with gpt-oss:20b and OpenCode to do this all locally.

The naive solution of just using gpt-oss:20b with Ollama’s defaults doesn’t work (e.g., tool calls don’t work). The main issues is that Ollama sets the context window to only 4096 tokens and medium reasoning by default. It took me too long to figure out how to get gpt-oss:20b working well in OpenCode (mostly because the OpenAI compatible API of Ollama doesn’t allow setting the context window), so I’m writing the steps down (partly for my future self).

In the end I could simply run the following command to batch update my journal entries:

for file in ./*.md; do
 opencode -m ollama/gpt-oss-20b-high-32k run \
 "Use the date in the filename \"${file}\" and correct the 'date:' entry in the YAML front matter of this file"
done

This post describes how to:

Create a long-context variant of gpt-oss:20b in Ollama.
Configure OpenCode to use that model with high reasoning enabled by default.

1. Create a long-context model in Ollama #

Start an interactive session with gpt-oss:20b:

ollama run gpt-oss:20b

In the prompt:

>>> /set parameter num_ctx 32768
>>> /save gpt-oss-20b-32k
>>> /bye

This:

Sets the context window to 32k tokens (num_ctx 32768).
Saves a new model/tag named gpt-oss-20b-32k with that parameter stored.

Adjust num_ctx as needed for your GPU (e.g. 16384, 65536, etc.).

2. Configure OpenCode to use high reasoning #

OpenCode talks to Ollama via the OpenAI-compatible API. In:

~/.config/opencode/opencode.json

add a provider configuration similar to:

{
 "$schema": "https://opencode.ai/config.json",
 "provider": {
 "ollama": {
 "npm": "@ai-sdk/openai-compatible",
 "name": "Ollama (local)",
 "options": {
 "baseURL": "http://localhost:11434/v1",
 "apiKey": "ollama"
 },
 "models": {
 "gpt-oss-20b-high-32k": {
 "id": "gpt-oss-20b-32k",
 "options": {
 "extraBody": {
 "think": "high"
 }
 }
 }
 }
 }
 },

 "model": "ollama/gpt-oss-20b-high-32k"
}

Notes:

"id": "gpt-oss-20b-32k" refers to the custom Ollama model with the larger context window.
"think": "high" enables high reasoning for gpt-oss:20b on each request.

3. Select the model in OpenCode #

Run OpenCode:

opencode

Inside OpenCode:

> /models

Then select:

ollama/gpt-oss-20b-high-32k

From this point, OpenCode uses gpt-oss:20b with:

A larger context window (configured in Ollama), and
High reasoning enabled by default (via think: "high").

Coding from my phone 📱

Mon, 20 Oct 2025 00:00:00 +0000

I am dictating this post at ten kilometers altitude on a flight home from Mexico, Blink Shell open on my iPhone, while an agentic assistant on my NixOS machine writes an initial draft for this post. Admittedly, I’m a little addicted to agentic coding: when an idea pops up, I want to jump on it right away—even if that means doing it from a cramped seat in the sky. This is the workflow I’ve arrived at after trying many alternatives, it extends my agentic coding write-up and the self-hosted AI obsession in my local AI journey. It’s my backup plan for those moments when a computer simply isn’t nearby—the rest of the time I’m still at a keyboard like any other developer. This is in no way a replacement for a proper computer/laptop workflow but gets the job done when needed. It’s a personal, mostly open-source stack built around agent-cli—my local-first voice layer. Here’s how it works: I speak into my phone, the audio goes to FasterWhisper for transcription, Ollama cleans up the text, and it lands on my clipboard ready to paste. My iOS Shortcut handles this through a small HTTP interface that agent-cli exposes. The coding agent itself still uses the best proprietary model I can access.

Table of contents

1. Why Phone Coding Works Now #

For years I used iSH (a full Alpine Linux emulator) with SSH to hop into servers from my phone, but dropped connections were a constant frustration (Mosh fixes this, more on that later). Because coding on a phone keyboard is terrible, I kept it to tiny configuration tweaks or one-off fixes of a few characters. Agentic tools changed that: with a CLI coding agent, I don’t need to type the code—I describe the change, review the patch, and run it. That made meaningful work on the phone possible for the first time, for those moments when a computer isn’t around.

I tried VS Code in the browser, bounced between iSH and Terminus for SSH, and even lived inside a handful of in-browser terminal clients. I also spent time with mobile companions like Happy and Omnara, both designed to mirror Claude Code sessions on the phone, but they still felt like extra steps between me and my actual terminal. That friction pushed me toward a phone‑ready, self‑hosted workflow that still gives me raw SSH access to my own machine when the computer is out of reach.

This post is based on the way I develop software today. Your mileage may vary, but if you also care about privacy, open tooling, and reproducible environments, I think there are useful pieces here.

2. Constraints and Trade-offs #

This setup grew organically over time. What emerged are less strict requirements and more happy accidents that turned into core benefits:

Single trust boundary: Only the model provider (OpenAI) sees code context; audio and automation stay local, and I avoid any additional third‑party relays.
Resilient sessions: Connections should survive sleep and spotty networks.
Voice-friendly: Dictation should be accurate enough that I can trust it.
Reproducible config: The entire stack must live in my dotfiles and NixOS configuration.

To give you a sense of what I tried, here is the short comparison that convinced me to roll my own:

Approach	Pros	Why I moved on
VS Code in the browser	Familiar editor UI	Needs a steady connection and still lives outside my dotfiles comfort zone
iSH / Terminus SSH	Works without extra infrastructure	Laggy, no Mosh, and awkward keybindings
In-browser terminals	Instant access from anywhere	Poor copy/paste ergonomics and flaky mobile keyboards
Happy (Claude Code companion)	Push notifications, encrypted mobile UI for Claude Code	Requires wrapping every session with a separate CLI and still abstracts away my shell, and was quite buggy for me
Omnara (agent command center)	Centralizes Claude Code/Codex sessions with terminal replay	Proxies via their servers; I already trust OpenAI for the model and don’t want another third‑party handling my code

The stack below gives me the resilience of Mosh, the ergonomics of Zellij, and full control over the AI layer.

3. Layer 1: WireGuard From the Router #

WireGuard is a modern VPN that lets me securely access my home network from anywhere. I terminate WireGuard on my router so every device in the house (and on the road) can dial home with the same config. (If you want something easier to set up without opening ports, Tailscale is a more user-friendly option.)

Server: ASUS XT8 router with WireGuard enabled via the router UI.
Client: The WireGuard iOS app with On-Demand rules so the tunnel flips on whenever I am off trusted Wi-Fi.
DNS: All mobile sessions resolve through my home DNS, so git.nijho.lt and internal services resolve instantly.

This gives Blink a local-LAN address for my desktop nixos.

4. Layer 2: Blink Shell + Mosh for Durable Sessions #

What is Mosh? It’s like SSH, but it stays connected when the network changes or the phone sleeps, and it feels faster on bad connections.

Blink Shell is my daily driver on iOS because it pairs beautifully with Mosh and has great keyboard ergonomics.

I launch sessions using mosh bas@nixos -- zellij attach --create phone (Zellij is my terminal multiplexer—more on that in the next section).
Mosh smooths over spotty LTE and keeps my session alive when the phone sleeps.

Mosh keeps the session responsive across flaky networks and sleep.

5. Layer 3: Zellij Layouts #

I use Zellij as my terminal multiplexer for mobile work.

I stick to the defaults and use a couple of predefined layouts (e.g., a “phone” layout with editor + shell panes).

Ergonomics matter even more on a glass keyboard, so I lean on a few shell helpers:

zoxide means I can jump between repos with z foo instead of pecking long paths.
Single-character aliases like p for pytest and gs for git status live in configs/shell/10_aliases.sh, keeping command entry to a minimum.
Autocompletion plugins—zsh-autosuggestions and zsh-syntax-highlighting—are wired up in configs/shell/70_zsh_plugins.sh, so I get inline hints and colour cues while typing.

I break these tricks down in more depth in Terminal Ninja, explain how I sync them with my Dotfiles, and even package binaries like zoxide with Dotbins.

6. Layer 4: `agent-cli` Server #

agent-cli runs as a small server on my NixOS machine. In this workflow I only use transcribe from iOS Shortcuts: the phone records audio, the server transcribes and cleans it up, and I paste the text. (Agent-CLI has other features like autocorrect and chat, but I don’t use them for this workflow.) On this machine I run a long‑lived agent-cli user service (systemd) so Shortcuts can POST audio to it and get cleaned text back—ready to paste into Codex CLI. The service is defined in my dotfiles—see: configs/nixos/modules/user.nix (agent-cli service).

For more background on why I built it and how I use it in practice, see Local AI Journey (§3) and Agentic Coding.

The models and services run on the same box:

FasterWhisper via faster-whisper-server for high‑accuracy streaming transcription.
Ollama for on‑device rewrite/cleanup before sending prompts to the coding agent.

My iOS Shortcut dictation workflow is slower than Apple’s built-in dictation, but the accuracy is much better—and that matters more.

For the actual coding agent, I use a fork of Codex CLI with OpenAI’s gpt-5-codex-high model (here no open-source solution matches the frontier).

7. Layer 5: The iOS Shortcut Pipeline #

My iPhone’s Action Button runs a Shortcut that records audio and sends my voice to agent-cli on my NixOS machine for transcription and cleanup.

For the full recipe, see the iOS Shortcut Guide: agent-cli/iOS_Shortcut_Guide.md.

In short: I press the Action Button, the Shortcut records a snippet, sends it to agent-cli, and copies the cleaned text to my clipboard so I can paste it into Codex CLI.

The whole loop finishes fast enough that I can capture intent by voice and paste it into Codex CLI without typing.

8. Workflow in Practice #

Here’s the typical flow:

WireGuard auto-connects when I’m off home WiFi
Open Blink—my Mosh session is still alive from last time, Zellij panes intact
Dictate the change, paste into Codex CLI, review the agent’s response
Check diffs in an adjacent Zellij pane or different Blink tab with separate Mosh session
The agent commits and pushes to a branch, then opens a PR via gh for me to review and merge

On the phone, I aim for the smallest practical edits and initial implementations. Even on personal repos, I still open a PR—sometimes prompting alone gets me to a merge‑ready result. For open‑source with real users, I finish on the computer with a careful review and any final edits.

9. Conclusion and Further Reading #

This phone setup extends my Agentic Coding workflow to mobile. I connect to the same Zellij session on nixos, so when an idea hits, I pick up exactly where I left off—same environment, same context, no setup. Everything runs on my hardware (voice processing and automation) except the coding model itself. It’s the most effective mobile workflow I’ve had so far. It’s mostly open‑source not out of dogma, but because those tools are the best options for my needs. The one exception is the coding model: there’s no true open equivalent right now, and it makes no sense to buy 20× H100s just to self‑host a frontier model even if such a high quality model would be available open-source.

If you’re curious to dive deeper, here are a few related posts:

Agentic Coding
Dotfiles: A Practical, Cross-Platform Terminal Setup
Local AI Journey: RTX 3090 Edition
Terminal Ninja
Using LLMs Effectively (already outdated at this point!)

I’d love to hear what self-hosted tricks you’re using for mobile development—reach out if you adapt this stack or build something wild on top of it.

How a 3090 turned into two local‑first AI projects 🎮

Mon, 14 Jul 2025 00:00:00 +0000

Table of contents

1. Introduction: The Unplayed Games #

I’ve wanted to dabble with local AI for a long time, mainly because of the possibilities it unlocks. Local AI allows me to use all of my data—financial records, personal messages, location history, even transcribing everything I say—without the privacy concerns of sending it to a cloud provider. This level of deep, personal data analysis is only something I’d be comfortable with on my own hardware.

However, I couldn’t justify buying a powerful GPU just for an imagined AI project that I might never really start. Then it hit me: I could also use it for gaming (duh). Because I really enjoyed The Last of Us series, I thought, “Okay, why not play The Last of Us on PC?” This was the push I needed. So, a couple of weeks ago, I convinced myself to buy someone’s old gaming machine. After many deep research sessions with ChatGPT about the best bang-for-the-buck consumer-grade GPU, I landed on an NVIDIA 3090. I found an (old) beast of a machine, and my initial idea was set.

Click here to see the full system specs ($1350 for all this!)

GPU: ASUS ROG STRIX RTX 3090 OC (24GB)
CPU: AMD Ryzen 9 3900X (12-Core)
Motherboard: ASUS ROG Crosshair VIII Hero
Power Supply: Be Quiet! 1200W Dark Power Pro
RAM: 32GB G.Skill Trident Z Royal DDR4
Storage: 2TB Intel M.2 NVMe SSD
Case: Corsair 4000D Airflow
CPU Cooler: Thermalright Peerless Assassin 120

Fast forward four weeks. I haven’t played a single minute, but my machine is now indexing 200 GB of email archives and facing a serious performance bottleneck I’ll get to later. That “gaming” machine has been humming away, but not rendering virtual worlds. It’s been training, transcribing, and thinking.

I’m writing this post simply because I am so enthusiastic about local AI. I’ve found that local models are surprisingly powerful for specific use cases. Of course, they aren’t going to replace Gemini or Claude for coding, but for many other applications, they work incredibly well. Lately, I’ve been so obsessed with this that I’m sleeping significantly less and spending all my free hours working on it.

2. The Rocky Road to a Working Setup #

Before I could even get to the fun AI stuff, I had to wrestle with the machine itself. I started with Pop!_OS, which is supposed to be a great out of the box solution that “just works”. However, it’s been a long time since I used Linux on the desktop; my experience is almost exclusively with servers. I quickly installed some of the wrong NVIDIA drivers and ended up debugging stuff in Grub. I got frustrated with the different commands I had to run to set up everything and realized that reproducing this system was going to be a massive pain. Since I’ve grown to love Nix on my Mac with Nix-Darwin, I decided to switch the whole system to NixOS. This has been an amazing decision. Even though there are many pain points with Nix, I think it is well worth it. For those interested, you can see the complete setup, including all the local AI services, in my NixOS configuration on GitHub.

3. The First Victory: `agent-cli` 🐍 #

With a stable system, I quickly got started with a few scripts. These soon evolved into my first real project of this adventure: agent-cli.

agent-cli is a collection of local-first, AI-powered command-line agents that run entirely on your machine. The core philosophy is privacy; your data never leaves your computer. It’s designed for seamless integration with system-wide hotkeys. I’m now using it constantly.

Instead of typing, I now dictate almost everything. A quick hotkey, and agent-cli transcribe turns my speech into text. Another hotkey, and agent-cli autocorrect cleans it up using a local LLM running on Ollama. It has genuinely streamlined my workflow. For example, I can pipe the tools together, like correcting text from my clipboard and then having it spoken aloud: agent-cli autocorrect | agent-cli speak.

The toolkit includes:

autocorrect: Fixes grammar and spelling.
transcribe: Uses a local Whisper model for speech-to-text.
speak: Converts text to speech with a local TTS engine.
voice-edit: A voice-powered clipboard assistant.
assistant: A hands-free voice assistant using a wake word.
chat: A conversational AI with tool-calling capabilities.

It’s been a fantastic success in my personal workflow, and it’s all open-source for you to try. Contributions and ideas welcome!

Once I trusted my toolchain, the obvious next step was to feed all my data into it.

4. The Grand Ambition: `AIBrain` 🧠 #

My second package, AIBrain, is a much larger endeavor. The vision is to create a “life-OS”—a private, local AI that processes all my personal data (emails, calendar, files, photos, messages, browser history, location data) to generate summaries and answer questions, eventually allowing for queries like:

aibrain ask "When did I get into landscape photography?"
aibrain ask "How many countries did I visit between 2015-2025?"

This project led me to explore the landscape of agentic frameworks. I started with CrewAI, but I quickly found that despite the hype and funding, it had some painful problems (specifically this issue where things just fail without any helpful error messages). I decided to switch to LangGraph, which felt more robust. I also chose to not use PydanticAI (which I used in agent-cli) because its Ollama support is just a wrapper around the OpenAI-compatible API, which doesn’t expose all the useful options (I even opened a PR in Ollama to make its OpenAI compatible API more feature-complete, but it has not been merged yet).

The core infrastructure of AIBrain is ready. It can index emails, calendar events, and files in various formats (PDF, Excel, PowerPoint, images). However, I’ve hit a significant challenge.

5. Conclusion: The Journey Continues #

This past month has been a whirlwind. I set out to play games and ended up with two open-source AI packages and a much deeper appreciation for the power sitting in a consumer-grade GPU. It’s incredible what you can achieve with local hardware today.

agent-cli is a finished, polished tool that I use daily. AIBrain is a much bigger dream, and I’m still figuring out how to solve the performance puzzle to make it truly practical. Next up: smarter chunking with LangGraph, vector-DB swap-outs, and GPU scheduling tricks.

This experience has reinforced my passion for local-first software and open-source development. The future of personal, private AI is being built right now, not just in large corporate labs, but in the homes of hobbyists tinkering with their gaming rigs.

⭐ If this resonates, star the repos & drop issues/questions.

And who knows, maybe I’ll even find the time to play ‘The Last of Us’.

Local-Ai | Bas Nijholt

Self-hosting AI does not save money, and I do it anyway

1. The model I love, and the one that beats it on price #

2. “But my GPUs are already paid for” #

3. Why datacenters are so much better at this #

4. “But I have solar panels” #

5. “But my Mac, DGX Spark, or AI Max does save money” #

6. The bar keeps moving #

7. A small company has it worse #

8. “They lose money on inference, so prices will go up” #

9. “But they sell your data” #

10. Where self-hosting does win: the same model #

11. So why do I do it? #

Appendix: all assumptions #

Frontier-level dictation on your iPhone keyboard, self-hosted

1. Why Qwen? #

2. The recipe #

Write the Compose file #

Start it and connect the app #

3. The bar test #

SOTA RAG & Memory without the database: Files, Git, and simple folders

1. The amnesia problem #

2. Learning from failure: AI Journal #

3. How I studied SOTA: clone, read, re-implement #

4. The dependency problem: why ONNX over PyTorch #

5. RAG: just let me grep it (kind of) #

The architecture: OpenAI compatibility #

Two-stage retrieval: the SOTA approach #

6. Memory: files over databases #

Git integration: versioning memory 🕰️ #

The reconciliation loop #

The tiny detail that broke everything #

7. Why building this was necessary #

Privacy #

Integration #

Dependencies #

8. Try it yourself #

Try Memory #

Try RAG #

9. What I learned #

Switching from Ollama to llama-swap + llama.cpp on NixOS: the power user's choice 🦙

The draft I never published #

The original trigger: broken models #

The catalyst: dual GPUs and control #

API compatibility and the “switch” #

My NixOS setup #

Compiler flags matter #

What I gained (and lost) #

Advantages of llama-swap + llama.cpp #

The trade-off #

Conclusion: maturing as an AI hobbyist #

Using gpt-oss:20b in OpenCode with a larger context and high reasoning

1. Create a long-context model in Ollama #

2. Configure OpenCode to use high reasoning #

3. Select the model in OpenCode #

Coding from my phone 📱

1. Why Phone Coding Works Now #

2. Constraints and Trade-offs #

3. Layer 1: WireGuard From the Router #

4. Layer 2: Blink Shell + Mosh for Durable Sessions #

5. Layer 3: Zellij Layouts #

6. Layer 4: agent-cli Server #

7. Layer 5: The iOS Shortcut Pipeline #

8. Workflow in Practice #

9. Conclusion and Further Reading #

How a 3090 turned into two local‑first AI projects 🎮

1. Introduction: The Unplayed Games #

2. The Rocky Road to a Working Setup #

3. The First Victory: agent-cli 🐍 #

4. The Grand Ambition: AIBrain 🧠 #

5. Conclusion: The Journey Continues #

6. Layer 4: `agent-cli` Server #

3. The First Victory: `agent-cli` 🐍 #

4. The Grand Ambition: `AIBrain` 🧠 #