Speech-to-Text | Bas Nijholt

Frontier-level dictation on your iPhone keyboard, self-hosted

Tue, 22 Sep 2026 00:00:00 +0000

On a recent month-long trip visiting family in Europe, I did a lot of work from my phone, using my mobile coding workflow. It still boggles my mind that real, productive work from a phone is possible now. Most of that work is talking to coding agents, and the built-in iOS dictation is garbage for that. I had an iOS Shortcut that sent recordings to Agent CLI, my local AI toolbox, running at home instead. The problem was that the recording screen takes over the whole display, so I could not see the thing I was commenting on. The transcript then landed in my clipboard, and I had to paste it myself.

So I threw my principles overboard and installed Wispr Flow, a voice keyboard for iOS. I loved it instantly. iOS does not let keyboards use the microphone, so the keyboard’s button hands off to the Wispr Flow app, which keeps recording in the background. I was less happy about that, since it meant a closed-source app always had the microphone open and sent my voice to their servers. I was on the free plan, so I was probably the product. I also hit the free weekly word limit very quickly, and started looking for an alternative.

That is how I found Diction, which has since become my favorite app when I’m away from my laptop. It works the same way, background recording included. The difference is that it has a self-hosted mode with an open-source gateway, so my voice goes to a machine I own instead of their servers.

I run the gateway on my home machine in the U.S., reachable only over my private network. The gateway streams audio to Agent CLI, which runs Alibaba’s Qwen/Qwen3-ASR-1.7B-hf on an RTX 3090 behind an OpenAI-compatible transcription endpoint. There is no LLM cleanup step, so Diction’s Writing Style and Tones features do not work with this setup. I don’t miss them: Qwen’s raw transcription is already good enough for me.

1. Why Qwen? #

For the longest time I used Faster-Whisper with Whisper large-v3. Whisper is still the default choice for most people, but many newer models beat it. I tried some of them, including NVIDIA’s Parakeet. Parakeet is fast, but it cannot take custom instructions. Those instructions are how Diction’s My Words feature biases a transcription toward specialized names.

At the start of September, I looked at the Hugging Face Open ASR Leaderboard, which tests speech-to-text models on the same recordings and ranks them. The score is the word error rate (WER): the share of words a model gets wrong, counting words it swaps, drops, or makes up. Lower is better, and a WER of 4% means about one wrong word in every 25. The test recordings range from audiobooks and podcasts to meetings and earnings calls, so the average says more than any single clean benchmark.

These are the averages on the public English test sets as of September 19, 2026:

Model	WER	Can I run it at home?
Zoom Scribe v2 Pro	3.6%	No, paid API (#1 overall)
ElevenLabs Scribe v2	4.0%	No, paid API
Qwen3-ASR 1.7B	4.3%	Yes, Apache-2.0
AssemblyAI Universal-3.5 Pro	4.3%	No, paid API
NVIDIA Parakeet TDT 0.6B v3	4.9%	Yes
OpenAI Whisper large-v3	5.8%	Yes

I picked the best model on the board whose weights you can download: Qwen3-ASR 1.7B. It lands within a point of the best paid API, which is what I mean by frontier-level. Since I maintain Agent CLI, I told an agent to add a Qwen backend, and shortly afterward it had landed and was running on my server.

Agent CLI’s transcription server loads a model on the first request and unloads it after an idle timeout, so the model only takes VRAM while I use it. The same process serves an OpenAI-compatible API, which the Diction gateway talks to, and the Wyoming protocol that Home Assistant uses for local voice.

2. The recipe #

You need an NVIDIA GPU with the NVIDIA Container Toolkit, Docker Compose 2.30 or newer, and a private route from your iPhone to the server. I use Headscale, but Tailscale or plain WireGuard works just as well. Do not expose the gateway to the public internet.

Write the Compose file #

Save the following as compose.yaml in an empty directory. Agent CLI’s CI publishes a CUDA image with the Transformers backend and Qwen as the default model, so there is nothing to build. The file points Diction’s gateway at Agent CLI’s transcription port. WHISPER_TTL raises the idle timeout from the default five minutes to a day, so dictation rarely waits for a reload:

services:
 asr:
 image: ghcr.io/basnijholt/agent-cli-whisper:0.112.0-cuda-transformers
 environment:
 WHISPER_TTL: "86400"
 volumes:
 - model-cache:/home/whisper/.cache
 gpus: all
 restart: unless-stopped

 gateway:
 image: dictionlabs/gateway:v14.0
 depends_on:
 - asr
 environment:
 CUSTOM_BACKEND_URL: http://asr:10301
 CUSTOM_BACKEND_MODEL: Qwen/Qwen3-ASR-1.7B-hf
 CUSTOM_BACKEND_CANONICAL_ID: Qwen/Qwen3-ASR-1.7B-hf
 CUSTOM_BACKEND_NEEDS_WAV: "true"
 ports:
 - "8080:8080"
 restart: unless-stopped

volumes:
 model-cache:

Start it and connect the app #

Start both services:

docker compose up -d

The first transcription downloads Qwen into the model-cache volume, so that request can take a few minutes. docker compose logs -f shows the progress.

On the iPhone, go to Settings → General → Keyboard → Keyboards → Add New Keyboard → Diction. Tap Diction in the keyboard list, enable Allow Full Access, and grant microphone access when prompted. Then open Diction, go to Preferences → Mode → Self-Hosted, enter http://<private-server-ip>:8080, and tap Test connection.

On my machine, Qwen3-ASR 1.7B uses about 4.7 GiB of GPU memory and roughly 3 GiB of system memory while loaded.

3. The bar test #

A day later, I met a friend in a bar in the Netherlands and wanted to let him try it.

He was already on my tailnet for some of my other self-hosted services, so his phone only needed an ACL change to reach the gateway. I opened the terminal app on my phone, connected to an agent, and dictated that change. In less than a minute, Diction was connected and working on his phone. His speech crossed the Atlantic, was transcribed by Qwen on my home GPU, and appeared back in the app with barely any delay.

That moment sold the whole setup to me. I was in a bar without a laptop, and I used dictation to grant access to the dictation service.

Wyoming MLX Whisper

Fri, 05 Dec 2025 00:00:00 +0000

Coding from my phone 📱

Mon, 20 Oct 2025 00:00:00 +0000

I am dictating this post at ten kilometers altitude on a flight home from Mexico, Blink Shell open on my iPhone, while an agentic assistant on my NixOS machine writes an initial draft for this post. Admittedly, I’m a little addicted to agentic coding: when an idea pops up, I want to jump on it right away—even if that means doing it from a cramped seat in the sky. This is the workflow I’ve arrived at after trying many alternatives, it extends my agentic coding write-up and the self-hosted AI obsession in my local AI journey. It’s my backup plan for those moments when a computer simply isn’t nearby—the rest of the time I’m still at a keyboard like any other developer. This is in no way a replacement for a proper computer/laptop workflow but gets the job done when needed. It’s a personal, mostly open-source stack built around agent-cli—my local-first voice layer. Here’s how it works: I speak into my phone, the audio goes to FasterWhisper for transcription, Ollama cleans up the text, and it lands on my clipboard ready to paste. My iOS Shortcut handles this through a small HTTP interface that agent-cli exposes. The coding agent itself still uses the best proprietary model I can access.

Table of contents

1. Why Phone Coding Works Now #

For years I used iSH (a full Alpine Linux emulator) with SSH to hop into servers from my phone, but dropped connections were a constant frustration (Mosh fixes this, more on that later). Because coding on a phone keyboard is terrible, I kept it to tiny configuration tweaks or one-off fixes of a few characters. Agentic tools changed that: with a CLI coding agent, I don’t need to type the code—I describe the change, review the patch, and run it. That made meaningful work on the phone possible for the first time, for those moments when a computer isn’t around.

I tried VS Code in the browser, bounced between iSH and Terminus for SSH, and even lived inside a handful of in-browser terminal clients. I also spent time with mobile companions like Happy and Omnara, both designed to mirror Claude Code sessions on the phone, but they still felt like extra steps between me and my actual terminal. That friction pushed me toward a phone‑ready, self‑hosted workflow that still gives me raw SSH access to my own machine when the computer is out of reach.

This post is based on the way I develop software today. Your mileage may vary, but if you also care about privacy, open tooling, and reproducible environments, I think there are useful pieces here.

2. Constraints and Trade-offs #

This setup grew organically over time. What emerged are less strict requirements and more happy accidents that turned into core benefits:

Single trust boundary: Only the model provider (OpenAI) sees code context; audio and automation stay local, and I avoid any additional third‑party relays.
Resilient sessions: Connections should survive sleep and spotty networks.
Voice-friendly: Dictation should be accurate enough that I can trust it.
Reproducible config: The entire stack must live in my dotfiles and NixOS configuration.

To give you a sense of what I tried, here is the short comparison that convinced me to roll my own:

Approach	Pros	Why I moved on
VS Code in the browser	Familiar editor UI	Needs a steady connection and still lives outside my dotfiles comfort zone
iSH / Terminus SSH	Works without extra infrastructure	Laggy, no Mosh, and awkward keybindings
In-browser terminals	Instant access from anywhere	Poor copy/paste ergonomics and flaky mobile keyboards
Happy (Claude Code companion)	Push notifications, encrypted mobile UI for Claude Code	Requires wrapping every session with a separate CLI and still abstracts away my shell, and was quite buggy for me
Omnara (agent command center)	Centralizes Claude Code/Codex sessions with terminal replay	Proxies via their servers; I already trust OpenAI for the model and don’t want another third‑party handling my code

The stack below gives me the resilience of Mosh, the ergonomics of Zellij, and full control over the AI layer.

3. Layer 1: WireGuard From the Router #

WireGuard is a modern VPN that lets me securely access my home network from anywhere. I terminate WireGuard on my router so every device in the house (and on the road) can dial home with the same config. (If you want something easier to set up without opening ports, Tailscale is a more user-friendly option.)

Server: ASUS XT8 router with WireGuard enabled via the router UI.
Client: The WireGuard iOS app with On-Demand rules so the tunnel flips on whenever I am off trusted Wi-Fi.
DNS: All mobile sessions resolve through my home DNS, so git.nijho.lt and internal services resolve instantly.

This gives Blink a local-LAN address for my desktop nixos.

4. Layer 2: Blink Shell + Mosh for Durable Sessions #

What is Mosh? It’s like SSH, but it stays connected when the network changes or the phone sleeps, and it feels faster on bad connections.

Blink Shell is my daily driver on iOS because it pairs beautifully with Mosh and has great keyboard ergonomics.

I launch sessions using mosh bas@nixos -- zellij attach --create phone (Zellij is my terminal multiplexer—more on that in the next section).
Mosh smooths over spotty LTE and keeps my session alive when the phone sleeps.

Mosh keeps the session responsive across flaky networks and sleep.

5. Layer 3: Zellij Layouts #

I use Zellij as my terminal multiplexer for mobile work.

I stick to the defaults and use a couple of predefined layouts (e.g., a “phone” layout with editor + shell panes).

Ergonomics matter even more on a glass keyboard, so I lean on a few shell helpers:

zoxide means I can jump between repos with z foo instead of pecking long paths.
Single-character aliases like p for pytest and gs for git status live in configs/shell/10_aliases.sh, keeping command entry to a minimum.
Autocompletion plugins—zsh-autosuggestions and zsh-syntax-highlighting—are wired up in configs/shell/70_zsh_plugins.sh, so I get inline hints and colour cues while typing.

I break these tricks down in more depth in Terminal Ninja, explain how I sync them with my Dotfiles, and even package binaries like zoxide with Dotbins.

6. Layer 4: `agent-cli` Server #

agent-cli runs as a small server on my NixOS machine. In this workflow I only use transcribe from iOS Shortcuts: the phone records audio, the server transcribes and cleans it up, and I paste the text. (Agent-CLI has other features like autocorrect and chat, but I don’t use them for this workflow.) On this machine I run a long‑lived agent-cli user service (systemd) so Shortcuts can POST audio to it and get cleaned text back—ready to paste into Codex CLI. The service is defined in my dotfiles—see: configs/nixos/modules/user.nix (agent-cli service).

For more background on why I built it and how I use it in practice, see Local AI Journey (§3) and Agentic Coding.

The models and services run on the same box:

FasterWhisper via faster-whisper-server for high‑accuracy streaming transcription.
Ollama for on‑device rewrite/cleanup before sending prompts to the coding agent.

My iOS Shortcut dictation workflow is slower than Apple’s built-in dictation, but the accuracy is much better—and that matters more.

For the actual coding agent, I use a fork of Codex CLI with OpenAI’s gpt-5-codex-high model (here no open-source solution matches the frontier).

7. Layer 5: The iOS Shortcut Pipeline #

My iPhone’s Action Button runs a Shortcut that records audio and sends my voice to agent-cli on my NixOS machine for transcription and cleanup.

For the full recipe, see the iOS Shortcut Guide: agent-cli/iOS_Shortcut_Guide.md.

In short: I press the Action Button, the Shortcut records a snippet, sends it to agent-cli, and copies the cleaned text to my clipboard so I can paste it into Codex CLI.

The whole loop finishes fast enough that I can capture intent by voice and paste it into Codex CLI without typing.

8. Workflow in Practice #

Here’s the typical flow:

WireGuard auto-connects when I’m off home WiFi
Open Blink—my Mosh session is still alive from last time, Zellij panes intact
Dictate the change, paste into Codex CLI, review the agent’s response
Check diffs in an adjacent Zellij pane or different Blink tab with separate Mosh session
The agent commits and pushes to a branch, then opens a PR via gh for me to review and merge

On the phone, I aim for the smallest practical edits and initial implementations. Even on personal repos, I still open a PR—sometimes prompting alone gets me to a merge‑ready result. For open‑source with real users, I finish on the computer with a careful review and any final edits.

9. Conclusion and Further Reading #

This phone setup extends my Agentic Coding workflow to mobile. I connect to the same Zellij session on nixos, so when an idea hits, I pick up exactly where I left off—same environment, same context, no setup. Everything runs on my hardware (voice processing and automation) except the coding model itself. It’s the most effective mobile workflow I’ve had so far. It’s mostly open‑source not out of dogma, but because those tools are the best options for my needs. The one exception is the coding model: there’s no true open equivalent right now, and it makes no sense to buy 20× H100s just to self‑host a frontier model even if such a high quality model would be available open-source.

If you’re curious to dive deeper, here are a few related posts:

Agentic Coding
Dotfiles: A Practical, Cross-Platform Terminal Setup
Local AI Journey: RTX 3090 Edition
Terminal Ninja
Using LLMs Effectively (already outdated at this point!)

I’d love to hear what self-hosted tricks you’re using for mobile development—reach out if you adapt this stack or build something wild on top of it.

How a 3090 turned into two local‑first AI projects 🎮

Mon, 14 Jul 2025 00:00:00 +0000

Table of contents

1. Introduction: The Unplayed Games #

I’ve wanted to dabble with local AI for a long time, mainly because of the possibilities it unlocks. Local AI allows me to use all of my data—financial records, personal messages, location history, even transcribing everything I say—without the privacy concerns of sending it to a cloud provider. This level of deep, personal data analysis is only something I’d be comfortable with on my own hardware.

However, I couldn’t justify buying a powerful GPU just for an imagined AI project that I might never really start. Then it hit me: I could also use it for gaming (duh). Because I really enjoyed The Last of Us series, I thought, “Okay, why not play The Last of Us on PC?” This was the push I needed. So, a couple of weeks ago, I convinced myself to buy someone’s old gaming machine. After many deep research sessions with ChatGPT about the best bang-for-the-buck consumer-grade GPU, I landed on an NVIDIA 3090. I found an (old) beast of a machine, and my initial idea was set.

Click here to see the full system specs ($1350 for all this!)

GPU: ASUS ROG STRIX RTX 3090 OC (24GB)
CPU: AMD Ryzen 9 3900X (12-Core)
Motherboard: ASUS ROG Crosshair VIII Hero
Power Supply: Be Quiet! 1200W Dark Power Pro
RAM: 32GB G.Skill Trident Z Royal DDR4
Storage: 2TB Intel M.2 NVMe SSD
Case: Corsair 4000D Airflow
CPU Cooler: Thermalright Peerless Assassin 120

Fast forward four weeks. I haven’t played a single minute, but my machine is now indexing 200 GB of email archives and facing a serious performance bottleneck I’ll get to later. That “gaming” machine has been humming away, but not rendering virtual worlds. It’s been training, transcribing, and thinking.

I’m writing this post simply because I am so enthusiastic about local AI. I’ve found that local models are surprisingly powerful for specific use cases. Of course, they aren’t going to replace Gemini or Claude for coding, but for many other applications, they work incredibly well. Lately, I’ve been so obsessed with this that I’m sleeping significantly less and spending all my free hours working on it.

2. The Rocky Road to a Working Setup #

Before I could even get to the fun AI stuff, I had to wrestle with the machine itself. I started with Pop!_OS, which is supposed to be a great out of the box solution that “just works”. However, it’s been a long time since I used Linux on the desktop; my experience is almost exclusively with servers. I quickly installed some of the wrong NVIDIA drivers and ended up debugging stuff in Grub. I got frustrated with the different commands I had to run to set up everything and realized that reproducing this system was going to be a massive pain. Since I’ve grown to love Nix on my Mac with Nix-Darwin, I decided to switch the whole system to NixOS. This has been an amazing decision. Even though there are many pain points with Nix, I think it is well worth it. For those interested, you can see the complete setup, including all the local AI services, in my NixOS configuration on GitHub.

3. The First Victory: `agent-cli` 🐍 #

With a stable system, I quickly got started with a few scripts. These soon evolved into my first real project of this adventure: agent-cli.

agent-cli is a collection of local-first, AI-powered command-line agents that run entirely on your machine. The core philosophy is privacy; your data never leaves your computer. It’s designed for seamless integration with system-wide hotkeys. I’m now using it constantly.

Instead of typing, I now dictate almost everything. A quick hotkey, and agent-cli transcribe turns my speech into text. Another hotkey, and agent-cli autocorrect cleans it up using a local LLM running on Ollama. It has genuinely streamlined my workflow. For example, I can pipe the tools together, like correcting text from my clipboard and then having it spoken aloud: agent-cli autocorrect | agent-cli speak.

The toolkit includes:

autocorrect: Fixes grammar and spelling.
transcribe: Uses a local Whisper model for speech-to-text.
speak: Converts text to speech with a local TTS engine.
voice-edit: A voice-powered clipboard assistant.
assistant: A hands-free voice assistant using a wake word.
chat: A conversational AI with tool-calling capabilities.

It’s been a fantastic success in my personal workflow, and it’s all open-source for you to try. Contributions and ideas welcome!

Once I trusted my toolchain, the obvious next step was to feed all my data into it.

4. The Grand Ambition: `AIBrain` 🧠 #

My second package, AIBrain, is a much larger endeavor. The vision is to create a “life-OS”—a private, local AI that processes all my personal data (emails, calendar, files, photos, messages, browser history, location data) to generate summaries and answer questions, eventually allowing for queries like:

aibrain ask "When did I get into landscape photography?"
aibrain ask "How many countries did I visit between 2015-2025?"

This project led me to explore the landscape of agentic frameworks. I started with CrewAI, but I quickly found that despite the hype and funding, it had some painful problems (specifically this issue where things just fail without any helpful error messages). I decided to switch to LangGraph, which felt more robust. I also chose to not use PydanticAI (which I used in agent-cli) because its Ollama support is just a wrapper around the OpenAI-compatible API, which doesn’t expose all the useful options (I even opened a PR in Ollama to make its OpenAI compatible API more feature-complete, but it has not been merged yet).

The core infrastructure of AIBrain is ready. It can index emails, calendar events, and files in various formats (PDF, Excel, PowerPoint, images). However, I’ve hit a significant challenge.

5. Conclusion: The Journey Continues #

This past month has been a whirlwind. I set out to play games and ended up with two open-source AI packages and a much deeper appreciation for the power sitting in a consumer-grade GPU. It’s incredible what you can achieve with local hardware today.

agent-cli is a finished, polished tool that I use daily. AIBrain is a much bigger dream, and I’m still figuring out how to solve the performance puzzle to make it truly practical. Next up: smarter chunking with LangGraph, vector-DB swap-outs, and GPU scheduling tricks.

This experience has reinforced my passion for local-first software and open-source development. The future of personal, private AI is being built right now, not just in large corporate labs, but in the homes of hobbyists tinkering with their gaming rigs.

⭐ If this resonates, star the repos & drop issues/questions.

And who knows, maybe I’ll even find the time to play ‘The Last of Us’.

Agent CLI

Wed, 25 Jun 2025 00:00:00 +0000