Local-Ai

Whenever I say that self-hosting AI is not economical, people hear that I am against self-hosting. I am not: I run Qwen3.8 27B at home and think open-weight models are great. This post compares it with GPT-6 Luna on the same benchmark, and explains why batching makes datacenters win.

A small Docker recipe for running Diction’s streaming gateway against Qwen3-ASR through Agent CLI, plus a test from a bar in the Netherlands, an ocean away from the GPU.

I wanted my AI agents to remember me and read my documents, but existing solutions required complex APIs or opaque databases. After my ambitious AI Journal project hit the wall of local model limitations, I took a different approach: clone LlamaIndex, LangChain, Letta, Mem0, and PydanticAI, study how SOTA systems work, and re-implement the best parts with minimal dependencies and ONNX instead of PyTorch.

I started writing this post four months ago when a specific model broke in Ollama. I initially went back to Ollama out of laziness, but after upgrading to dual RTX 3090s, I realized that for serious multi-GPU inference and RAM offloading, you need the raw control of llama.cpp. Here is how I manage my new 48GB VRAM setup declaratively with NixOS.

I set up OpenAI’s gpt-oss:20b in Ollama with a larger context window and high reasoning in OpenCode to process my private journal entries locally.

My mostly self-hosted mobile development loop: WireGuard from an iPhone into my NixOS machine, persistent Blink/Mosh sessions, a systemd-managed agent-cli server, and an iOS Shortcut for FasterWhisper+Ollama dictation—paired with the best proprietary coding model available.

A personal journey of buying a gaming PC and accidentally falling down the rabbit hole of local, private AI. I share my experience building agent-cli and AIBrain, the tools I used, and the lessons I learned along the way.