Python | Bas Nijholt

zfs-tenant

Sat, 26 Sep 2026 00:00:00 +0000

pytest-shm

Fri, 25 Sep 2026 00:00:00 +0000

ZFS Unlock

Mon, 29 Jun 2026 00:00:00 +0000

MindRoom: AI agents that live in Matrix and work everywhere 🧠

Fri, 26 Jun 2026 00:00:00 +0000

Table of contents

1. The problem: your AI is trapped in apps #

Programming is my biggest passion in life. I’ve been actively involved in open source for over ten years now, maintaining 40+ installable packages across Python, JavaScript, and Rust. Some of these have over 100,000 users; others have exactly one—me. I mention this because it matters for what comes next: when AI coding tools arrived, I didn’t start from zero. As I wrote about in my agentic coding post, AI has completely transformed how I work—when I’m AFK, I regularly build full features or entire projects by dictating to my phone and routing it through my own transcription software.

Through all of this, I became deeply convinced that AI agents are the future. Not just a useful tool for programmers—a fundamental shift in how everyone will interact with computers.

But if that’s true, then the way agents work today is broken.

We’ve built artificial barriers between artificial intelligences. ChatGPT knows you in one tab, Claude knows you in another, your Slack bot knows you in a third. Your email AI can’t tell your calendar AI about that meeting invitation. Your coding assistant has no idea about the project specs sitting in your document AI. Switch platforms, and your AI starts from scratch.

Most agent frameworks require you to program them. That works for developers like me, but if agents are truly the future, they need to work for everyone. Non-programmers don’t want to write code—they just want a chat.

And then there’s privacy. I’m fine sharing my email with Gemini—Google already owns my Gmail. But do I want to send my financial data, my health records, or my personal notes to a cloud provider? Not really. I’d rather use a local model for sensitive tasks and pick the best, cheapest Chinese model for general-purpose deep research.

So I built MindRoom on a chat protocol—if people just want a chat, give them a chat. Agents live in the same rooms as you, collaborate in threads, and follow you across every platform. The first commit landed on July 29, 2025 (four months before OpenClaw’s first commit 😅), so by the time I’m publishing this MindRoom has been with me for nearly eleven months: first as an obsessive side project, then, since March, with much more sustained focus. At first I got so obsessed with it that I eventually burned out and had to step away completely.

Recently, seeing OpenClaw gain massive traction solving a similar problem reminded me that what I’d been building wasn’t just a niche obsession. So here I am, dusting off MindRoom and writing about what it is, how it works, and why I think the approach still matters.

Live chat.mindroom.chat Personal thread with the prompt: “Explain how MindRoom and what you can do.” Mind answers in the real Matrix room, using tool calls and the Codex-backed local MindRoom runtime.

2. What is MindRoom? #

MindRoom is an open-source system that creates AI agents living inside the Matrix protocol. If you’re not familiar with Matrix—it’s a federated, end-to-end encrypted communication standard. The same protocol used by the French government for 5.5 million civil servants, by German healthcare for 150K+ organizations, and by the Element app that millions of people use daily.

The key insight: Matrix has bridges to many major platforms—Slack, Telegram, Discord, WhatsApp, IRC, email, even SMS.

So if your AI agent lives in Matrix, it can reach you on any platform. One agent, every platform, continuous memory.

One practical caveat: Matrix bridges vary in maturity. Some, like the Telegram bridge, work very well, while others can be finicky. Your mileage may vary depending on which platforms you need.

Here’s what a typical setup looks like in config.yaml:

agents:
 code:
 display_name: CodeAgent
 role: Generate code, manage files, execute shell commands
 model: opus-4.8
 tools: [file, shell, github]
 instructions:
 - Always read files before modifying them.
 rooms: [lobby, dev]

 research:
 display_name: ResearchAgent
 role: Search the web, summarize papers, find information
 model: gpt-5.5
 tools: [tavily, arxiv, wikipedia]
 rooms: [lobby, research]

teams:
 super_team:
 display_name: Super Team
 agents: [code, research]
 mode: collaborate

Define your agents, give them tools and rooms, and they show up in Matrix as real users—with avatars, typing indicators, and online status.

MindRoom agents are real Matrix users with their own accounts, generated avatars, display names, room memberships, and online presence.

3. How it actually works #

At the core sits what I call the MultiAgentOrchestrator—a class in bot.py that boots every configured entity (router, agents, teams), provisions Matrix user accounts for each one via matrix-nio, and keeps sync loops alive. The agents themselves are powered by the Agno framework, which provides a unified interface across AI model providers.

When someone sends a message in a Matrix room:

Explicit mentions go directly to the named agent (@mindroom_code help me debug this)
Thread continuations keep the same agent responding (no weird context switches mid-conversation)
New conversations hit the router—an AI that analyzes the message and picks the best agent based on capabilities

All conversations happen in threads, which keeps rooms organized. Agents stream their responses in real-time, editing a single message as they think rather than spamming new ones. You see tool calls happening live:

🔧 Tool Call: search_web(query="matrix protocol bridges")
[waiting for result...]
✅ search_web result:
[results here]

The recording below starts in the main Personal room, sends the prompt Explain how MindRoom and what you can do., and then follows MindRoom as it opens the new thread and streams the reply to completion. It is sped up 2.5x to keep the flow skimmable.

4. Building on Matrix: the good and the tricky #

One of the best things about building on Matrix is what you get for free. End-to-end encryption, a deeply interoperable chat protocol with bridges to many platforms, a choice of clients, federation between organizations—all of that comes with the protocol. You don’t have to build any of it yourself.

But because Matrix has such a tight specification, it also brings challenges.

The protocol doesn’t support streaming. AI agents that think for 30 seconds before dumping a wall of text make for a terrible chat experience, so I hacked streaming in by rapidly editing the same message as new tokens arrive. An ⋯ marker shows while the agent is still thinking—a small touch, but it makes the experience feel responsive and alive.

There’s also a size limit on message content. That’s fine for human chat, but AI responses can get long—especially when tool calls and their results are included. I worked around this by using Matrix’s attachment feature: when a response exceeds the limit, the content continues in an attachment that gets updated as the message keeps streaming in. This required forking the Cinny chat client so that attachments display inline rather than as downloadable files, making the whole thing seamless.

5. Memory that follows you #

MindRoom currently supports two memory implementations.

The first is the more traditional semantic memory system built on Mem0’s AsyncMemory, with configurable embedding providers (OpenAI, Ollama, or HuggingFace) and vector storage via ChromaDB. This gives agents searchable memory for preferences, project context, decisions, and recurring facts.

The second is a file-based memory system inspired by the simplicity of OpenClaw-style agent files. Instead of burying identity in a database, durable context can live in plain Markdown files that travel with the agent. That portability matters: I can move the same agent identity from OpenClaw to Hermes to MindRoom without starting from scratch.

MindRoom also has a memory flush step that periodically extracts durable facts from conversations and writes them back into memory. So the chat transcript remains chat, while stable preferences, decisions, and identity details become reusable context for future turns.

A live Personal memory demo: MindRoom stores a harmless preference in one turn and retrieves it later from the same Matrix conversation.

6. 100+ built-in tool integrations #

Agents can use over 100 built-in integrations:

Category	Examples
Communication	Gmail, Slack, Telegram, Discord
Development	GitHub, Shell, Python, Docker
Search	Tavily, Wikipedia, Arxiv, DuckDuckGo
Productivity	Google Calendar, Jira, Linear, Todoist
Smart Home	Home Assistant
AI/ML	DALL-E, ElevenLabs, Replicate
Data	Pandas, SQL, DuckDB, Yahoo Finance
Web	Firecrawl, Crawl4ai, Browser automation

Tools are lazy-loaded and credential-managed, so an agent only loads what it needs.

7. Teams: agents that collaborate #

Single agents are useful, but sometimes you need a team. MindRoom supports two collaboration modes:

Coordinate mode: A lead agent orchestrates others. You ask a question, the lead delegates subtasks, collects results, and synthesizes a unified response.

Collaborate mode: All agents work on the same task in parallel, each providing their independent analysis. The system then merges their responses with a consensus summary.

In practice, you might have a research team where one agent searches academic papers, another checks industry news, and a third validates claims—all triggered by a single message. The live room view below shows a real team response in the same Matrix thread surface normal agents use: threads, routed agents, and room history are all first-class chat objects rather than a separate dashboard.

Live Lobby thread in chat.mindroom.chat showing a real Team Response from HomeAssistant and AgentBuilder. Team collaboration uses the same Matrix-native interface as regular agent chats, with routed agents and room membership visible beside the conversation.

8. Hot-reload: change config and plugins without downtime #

config.yaml is watched at runtime. When you edit it—add an agent, change a model, update instructions—MindRoom diffs the old and new config, gracefully restarts only the affected agents, and has them rejoin their rooms. No downtime, no full runtime restart required.

The same development loop applies to plugins. MindRoom has a rich Python hook system, and plugins can be live-developed while the system is running. Change a plugin file, save it, and MindRoom automatically reloads the plugin without restarting the Python runtime.

This sounds minor, but when you’re iterating on agent behavior or plugin hooks, being able to tweak code and config and see results in seconds significantly improves the development loop.

9. Voice, scheduling, and other features #

Some features I built purely because I wanted them for myself (a recurring theme in my projects, as anyone who’s read my local AI journey can attest). I even built Matty, a Matrix CLI client, from bed at midnight because I needed a way to interact with my agents from the terminal:

Voice messages: Audio messages in Matrix are auto-transcribed via Whisper and treated as regular text input. You can talk to your agents.
Scheduled tasks: Natural language scheduling (!schedule "check my email every morning at 9 AM") backed by cron jobs. Agents can run tasks in the background and escalate to you when needed.
DM support: Agents respond naturally in 1:1 conversations without needing mentions.
Cross-organization federation: Because Matrix is federated, two companies’ AI agents can collaborate in a shared room—something that’s hard to do on proprietary platforms.

10. The obsession, the SaaS dream, and the burnout #

I need to be honest about what actually happened with MindRoom, because “life got in the way” is a sanitized version of the story.

What really happened is that I got completely, utterly obsessed. Every single second I wasn’t working at my day job or sleeping, I was working on MindRoom. Many hundreds of hours went into it. I seriously considered quitting my job and starting an AI startup around it.

If you’ve read Armin Ronacher’s post Agent Psychosis: Are We Going Insane?, you’ll know exactly what I’m talking about. The dopamine hit from building with AI agents is incredibly real. As Armin writes: “You feel productive, you feel like everything is amazing, and if you hang out just with people that are into that stuff too, without any checks, you go deeper and deeper into the belief that this all makes perfect sense.”

That was me. I was building and building, shipping feature after feature, and it felt incredible. The codebase grew to over 1,000 commits. I built the core system, then a React dashboard, then I started on a full SaaS platform—Kubernetes deployments on Hetzner Cloud, a FastAPI backend, a Next.js 15 frontend, Stripe integration, Supabase auth, Helm charts for multi-tenant isolation.

And that’s where the enthusiasm started to erode. Not because the core idea was bad, but because the work shifted from building interesting things (agent orchestration, memory systems, routing intelligence) to grinding through SaaS boilerplate: GDPR compliance, payment processing, automated Kubernetes deployments, terms of service, cookie banners. The dopamine loop that kept me going at 2 AM broke when the work stopped being creative and started being compliance paperwork.

The hosted control plane lives at app.mindroom.chat. I am deliberately not turning this section into a dashboard tour; the interesting part, and the part I still care about most, is the Matrix-native runtime underneath it.

11. OpenClaw and why it matters #

OpenClaw takes a different approach to the same core problem. Where MindRoom builds on Matrix federation as the backbone, OpenClaw runs a local Gateway on your machine and connects to messaging platforms directly (WhatsApp via Baileys, Telegram via grammY, Discord via discord.js, and so on).

The similarities are striking:

Both give you a single AI assistant across all your messaging platforms
Both have persistent memory
Both support multiple tools and integrations
Both are open-source and privacy-focused
Both let you choose your own AI model

The differences are interesting too. OpenClaw is local-first and TypeScript-based with 380K+ stars and a massive community. MindRoom is Python-based, federation-first, and… well, let’s just say my GitHub star count is a few orders of magnitude lower.

But seeing OpenClaw validate the core idea—that people want AI assistants that aren’t trapped in apps—reminded me that what I’d been building matters. The federation angle is something OpenClaw doesn’t have: the ability for agents from different organizations to collaborate natively, with end-to-end encryption (Olm/Megolm), on a protocol that governments already trust and deploy at scale.

MindRoom uses Matrix as the federated backbone; OpenClaw takes a local-gateway, direct-connector approach.

And then there’s Moltbook—a social network built exclusively for AI agents that the internet was losing its mind over for a couple of days. I’m not personally convinced it’s the next big thing, but what caught my attention is that MindRoom could support something like it natively. Matrix is already federated—agents on different servers can already interact, join shared rooms, and collaborate across organizational boundaries. A bridge from a Reddit-like platform to Matrix and your agents could participate without any special integration.

Building agent-to-agent infrastructure from scratch also means building security from scratch—and that’s where new platforms tend to struggle. Matrix has spent years hardening its E2E encryption, and it’s already deployed by governments and healthcare organizations at scale. Building on that foundation means you inherit those security properties rather than hoping to get them right yourself.

12. What’s next #

I first drafted this post a couple of months ago, when MindRoom was still something I was trying to come back to. That part is already out of date. Since January I’ve been spending nearly every free hour on it again, and since March it has effectively been my full-time work. The codebase supports 8+ AI model providers—OpenAI, Anthropic, Ollama, Groq, Google, OpenRouter, DeepSeek, Cerebras—and the core architecture is solid. It also has real users now. My rough guess is that something like 30-100 people have tried MindRoom in one form or another, which is still early, but meaningfully different from “only I have touched this.” There’s already a Docker Compose file and a starter repository to get started, but I want to make it even simpler.

What I’m focusing on now:

Skills system: Building an ecosystem of reusable agent behaviors (already partially implemented with OpenClaw-compatible format)
Even easier onboarding: A docker compose up gets you running today, but I want a wizard that gets you from zero to working agents without touching a YAML file
Community: Turning the small group of early users into a real operator and contributor community
Scope discipline: Keeping the work sustainable now that MindRoom is effectively full-time, not slipping back into the obsessive 2 AM marathon from before

I’m deliberately leaving the SaaS ambitions aside for now. The core system is what matters, and that’s where the energy is going.

If you’re interested in AI agents that live in Matrix and work everywhere, check out MindRoom on GitHub.

Sometimes seeing someone else succeed with a similar idea is enough to remember why you started building in the first place. And sometimes the most important lesson from a project isn’t technical—it’s learning when to step away and when to come back.

Are you running AI agents on any messaging platforms? Have you tried Matrix for anything beyond regular chat? I’d love to hear about your setups!

Links and resources #

Matrix MCP

Tue, 05 May 2026 00:00:00 +0000

Privata

Tue, 05 May 2026 00:00:00 +0000

Auto-installing extras at runtime so users never have to think about dependencies

Wed, 11 Feb 2026 00:00:00 +0000

Table of contents

1. From one command to a dependency maze #

agent-cli started as a simple thing. Voice in, text out. It connected to a Wyoming Whisper server for speech-to-text and a Piper server for text-to-speech, basically a local, open-source Siri triggered from terminal hotkeys. I wrote about the early version in my local AI journey post.

And then, as tends to happen with side projects I use every day, I kept adding things.

First a RAG proxy with ChromaDB, so I could chat with my documents. Then a long-term memory system, so conversations could remember context across sessions. Server commands that run Kokoro TTS or Whisper locally, optimized for whatever hardware you have—Apple Silicon gets MLX, NVIDIA gets CTranslate2. Voice activity detection with Silero VAD. Audio speed adjustment. An agent orchestration system. systemd/launchd service management for running background daemons.

Each of these features pulled in wildly different dependency trees. torch and transformers for GPU-accelerated Whisper. sounddevice and numpy for microphone access. chromadb and onnxruntime for vector storage. kokoro with its entire spacy pipeline for neural TTS.

Today the project has 13 optional extras. The tool became much more useful, but dependency UX got progressively worse with every feature.

2. Why “just use extras” breaks down #

Python’s optional-dependencies in pyproject.toml is the standard mechanism for this, and it’s the right packaging primitive. You define groups, users install with pip install agent-cli[rag]. On paper, this is perfectly reasonable.

In practice, it falls apart when you have 13 of them:

[project.optional-dependencies]
audio = ["numpy", "sounddevice>=0.4.6", "agent-cli[wyoming]"]
llm = ["pydantic-ai-slim[openai,google,duckduckgo,vertexai]>=0.1.1"]
rag = ["pydantic-ai-slim[openai,google]>=0.1.1", "chromadb>=0.4.22", ...]
memory = ["pydantic-ai-slim[openai,google]>=0.1.1", "chromadb>=0.4.22", ...]
vad = ["onnxruntime>=1.16.0"]
faster-whisper = ["faster-whisper>=1.0.0"]
mlx-whisper = ["mlx-whisper>=0.4.0; sys_platform == 'darwin' and platform_machine == 'arm64'"]
kokoro = ["kokoro>=0.9.0", "soundfile>=0.12.0", ...]
piper = ["piper-tts>=1.2.0"]
# ... and more

The UX problem is obvious. If I want to transcribe voice to text, do I need audio? llm? vad? All three? What if I want the Whisper server to run locally—is that faster-whisper or mlx-whisper? Do I also need server for that?

Expecting users to study a dependency matrix before running a CLI command is a terrible experience. I wanted three things:

A lightweight base install that’s fast and doesn’t pull in heavy ML dependencies.
Commands that “just work” when first used.
A clear opt-out for people who don’t want automatic installs.

3. The decorator #

The core idea is simple. Each CLI command declares what it needs, and the decorator handles the rest:

@app.command("transcribe", rich_help_panel="Voice Commands")
@requires_extras("audio", "llm")
def transcribe(...):
 ...

That’s it from the command author’s perspective. One line, and you never think about dependency management in that command again.

Here are a few more examples from the codebase to show how this scales:

# Continuous transcription needs audio, voice activity detection, and an LLM
@requires_extras("audio", "vad", "llm")
def transcribe_live(...):
 ...

# TTS server needs the server framework + one of two TTS backends + Wyoming protocol
@requires_extras("server", "piper|kokoro", "wyoming")
def start_tts_server(...):
 ...

# Whisper server: any of three ASR backends will do
@requires_extras("server", "faster-whisper|mlx-whisper|whisper-transformers", "wyoming")
def start_whisper_server(...):
 ...

Notice the pipe syntax: "piper|kokoro" means “you need either Piper or Kokoro—any one of these satisfies the requirement.” On Apple Silicon, if you already have mlx-whisper installed, it won’t try to install faster-whisper instead.

The decorator itself is straightforward:

def requires_extras(*extras: str) -> Callable[[F], F]:
 def decorator(func: F) -> F:
 @functools.wraps(func)
 def wrapper(*args, **kwargs):
 if _check_and_install_extras(extras):
 raise typer.Exit(1)
 return func(*args, **kwargs)
 return wrapper
 return decorator

The real complexity lives in _check_and_install_extras, which handles several installation scenarios. But the interface for command authors is always a single line.

4. Checking without importing #

You don’t want to import torch just to check if it’s available—that takes seconds and loads hundreds of megabytes into memory. Instead, I use importlib.util.find_spec, which checks whether a module exists on the import path without executing any of its code. No heavy imports, no GPU initialization, no side effects. The check is essentially free.

5. The metadata bridge #

There’s a gap between what pyproject.toml knows and what the runtime check needs. pyproject.toml says the rag extra requires chromadb>=0.4.22—that’s the pip package name. But at runtime, I need to check Python import names. Sometimes these are the same (chromadb → import chromadb), but often they’re not: pydantic-ai-slim is imported as pydantic_ai, piper-tts as piper.

To bridge this, I generate an _extras.json file that maps each extra to its description and the Python import names needed for the runtime check:

{
 "audio": ["Audio recording/playback", ["sounddevice"]],
 "rag": ["RAG proxy (ChromaDB, embeddings)", ["chromadb", "pydantic_ai"]],
 "memory": ["Long-term memory proxy", ["chromadb", "yaml", "pydantic_ai"]],
 "vad": ["Voice Activity Detection (Silero VAD via ONNX)", ["onnxruntime"]],
 "faster-whisper": ["Whisper ASR via CTranslate2", ["faster_whisper"]]
}

This file is generated by a script that parses pyproject.toml and applies a mapping table for the package-name-to-import-name conversions. A pre-commit hook ensures it stays in sync—if you add an extra to pyproject.toml and forget to update the metadata, the commit fails.

I wish Python packaging had a standard way to discover a package’s top-level import name from its distribution name. It doesn’t, so the mapping table is the least-bad option I found.

6. Three environments, three strategies #

The hard part is that people run agent-cli in several different ways, and each needs a different installation strategy.

A) `uv tool install` — the persistent environment #

When installed via uv tool install agent-cli, the tool lives in a persistent environment under ~/.local/share/uv/tools/agent-cli/. This environment has a uv-receipt.toml file that tracks which extras are installed.

When auto-install triggers, the system reads the current extras from that receipt, merges in the new ones, and reinstalls:

def install_extras_impl(extras, *, quiet=False):
 if is_uv_tool_install():
 current_extras = _get_current_uv_tool_extras()
 new_extras = sorted(set(current_extras) | set(extras))
 return _install_via_uv_tool(new_extras, quiet=quiet)

So if you already had audio installed and now run a command that needs llm, it becomes uv tool install agent-cli[audio,llm] --force.

The key insight: uv tool upgrade reads uv-receipt.toml and preserves the extras list. Once auto-install adds an extra, it persists across future upgrades. You never have to install it again.

(Full implementation)

B) `uvx` — the ephemeral environment #

uvx is uv’s tool for running packages in temporary, disposable environments. Running uvx agent-cli transcribe creates a fresh environment, installs agent-cli, runs the command, and throws the environment away.

The problem: there’s nowhere to persist extras. No uv-receipt.toml to update, no virtualenv that survives the invocation.

The solution is one of my favorite tricks in this system: re-execute the entire command with the extras baked into the uvx invocation itself:

def _maybe_reexec_with_uvx(extras):
 if os.environ.get(_REEXEC_MARKER) or not _is_uvx_cache():
 return
 uvx_path = shutil.which("uvx")
 if not uvx_path:
 return
 extras_str = ",".join(extras)
 cmd = [uvx_path, "--python", "3.13",
 f"agent-cli[{extras_str}]", *sys.argv[1:]]
 _maybe_exec_with_marker(cmd, f"Re-running with extras: {extras_str}")

So uvx agent-cli transcribe transparently becomes uvx agent-cli[audio,llm] transcribe. The user sees a brief “Re-running with extras: audio, llm” message, and then the command works.

Environment detection is straightforward—uvx environments live under the uv cache directory:

def _is_uvx_cache():
 prefix_str = Path(sys.prefix).resolve().as_posix()
 return "/cache/uv/" in prefix_str or "/archive-v" in prefix_str

(Full implementation)

C) Regular virtualenv, `pipx`, or system install #

For plain pip install agent-cli, pipx, or virtualenv installs, the system falls back to direct installation. Each extra has a pre-generated requirements file with pinned versions (e.g., _requirements/audio.txt), and the install command adapts to whatever’s available:

def _install_cmd():
 in_venv = sys.prefix != sys.base_prefix
 if shutil.which("uv"):
 cmd = ["uv", "pip", "install", "--python", sys.executable]
 if not in_venv:
 cmd.append("--system")
 return cmd
 cmd = [sys.executable, "-m", "pip", "install"]
 if not in_venv:
 cmd.append("--user")
 return cmd

Prefers uv pip when available (fast), falls back to regular pip. Uses --system or --user as appropriate when not in a virtualenv. This is also the path that pipx installations take—pipx creates a regular virtualenv under the hood, so the auto-installer detects it as a venv and installs directly into it.

(Full implementation)

7. Preventing infinite re-execution loops #

There’s an obvious footgun here: what if the install succeeds but the package still isn’t importable for some reason? You’d get an infinite loop of install → re-exec → check → install → re-exec → …

The fix is a simple environment variable marker:

_REEXEC_MARKER = "_AGENT_CLI_REEXEC"

def _maybe_exec_with_marker(cmd, message):
 if os.environ.get(_REEXEC_MARKER):
 return # Already re-executed once, don't loop
 new_env = os.environ.copy()
 new_env[_REEXEC_MARKER] = "1"
 os.execvpe(cmd[0], cmd, new_env)

os.execvpe replaces the current process entirely—it never returns. The marker ensures we only do this once. If the second run still can’t find the packages, it falls back to a helpful error message instead of looping forever.

8. What users actually see #

With auto-install enabled (the default), a first run looks like this:

Auto-installing missing extras: audio, llm
Running: uv tool install agent-cli[audio,llm] --force --python 3.13 -q
Installation complete!
Re-running with installed extras...

And then the command runs normally. Next time, it runs instantly—no install needed.

With auto-install disabled, the user gets a clear message telling them exactly what to do:

This command requires the 'audio' extra (Audio recording/playback).

Install with:
 uv tool install -p 3.13 "agent-cli\[audio]"
 # or
 agent-cli install-extras audio

If you prefer explicit control, there’s also a manual install-extras command:

agent-cli install-extras rag memory # Install specific extras
agent-cli install-extras --all # Install everything
agent-cli install-extras --list # Show available extras with descriptions

You can opt out via environment variable or config file:

export AGENT_CLI_NO_AUTO_INSTALL=1

# ~/.config/agent-cli/config.toml
[settings]
auto_install_extras = false

9. Keeping everything in sync #

The runtime system depends on two generated artifacts:

_extras.json: maps extras to descriptions and import names (for the find_spec check)
_requirements/*.txt: pinned dependency files for each extra (for the pip/uv install)

_extras.json is generated by sync_extras.py (run manually or in CI). The requirements files are generated by sync_requirements.py, which runs as a pre-commit hook whenever pyproject.toml or uv.lock changes. A check script enforces that _extras.json stays consistent with pyproject.toml—if you add an extra but forget to update the metadata, the commit fails with a clear error.

This means the runtime system never drifts from the source of truth. pyproject.toml is the canonical definition, everything else is derived.

10. Why uv made this practical #

I want to be clear: this system would be much harder without uv.

uv tool install with persistent extras in uv-receipt.toml—extras survive upgrades automatically
uvx for ephemeral execution with extras specified inline
uv pip install that’s fast enough for runtime auto-install to feel acceptable
uv export for generating pinned requirements files from uv.lock

With traditional pip, auto-installing at runtime would take 30-60 seconds and feel terrible. With uv, it typically takes a second. That’s the difference between “annoying” and “barely noticeable.”

11. Tradeoffs #

This approach works well, but it’s still a tradeoff:

There’s metadata to maintain. The package-name-to-import-name mapping requires a lookup table with special cases like piper-tts → piper and pydantic-ai-slim → pydantic_ai. I wish pyproject.toml had a field for this.
The check only verifies importability, not version compatibility. find_spec tells you a module exists, not whether it’s the right version. In practice this hasn’t been a problem because uv’s pinned requirements handle versions, but it’s a theoretical gap.
Auto-install can fail in restricted environments. Corporate machines, read-only containers, air-gapped systems—the opt-out exists for a reason.
Re-exec is a CLI pattern, not a library pattern. os.execvpe replaces the process, which means any shell hooks or signal handlers set up before the decorator runs are lost. Fine for CLI tools, would be problematic for libraries.

12. Not dependency injection #

I originally called this “dependency injection” in my head, but that’s the wrong term entirely. Dependency injection is about providing object dependencies through constructors or function parameters. This is something different.

The best names I’ve come up with:

runtime optional dependency resolution — accurate but a mouthful
on-demand extras installation — emphasizes the user experience
lazy extras — short, probably what I’d use in conversation

If there’s a standard term for this pattern, I haven’t found it.

13. Conclusion #

The underlying principle is simple: the tool should know what it needs, and it should get it. Users shouldn’t have to study a matrix of 13 optional extras to figure out which pip command to run. They should just type the command they want, and it should work.

uv’s speed and tooling model made this practical. A decorator made it ergonomic for command authors. Pre-commit hooks made it maintainable. And three environment-detection strategies made it work regardless of how the user installed the tool.

If you’re building a Python CLI with optional features that keep growing, consider this pattern. Your users will thank you, mostly by never having to ask “which extras do I need?”

The code discussed here is part of agent-cli, an open-source suite of AI-powered command-line tools. The implementation lives primarily in agent_cli/core/deps.py.

trueloc

Thu, 15 Jan 2026 00:00:00 +0000

Compose Farm

Thu, 11 Dec 2025 00:00:00 +0000

SOTA RAG & Memory without the database: Files, Git, and simple folders

Wed, 10 Dec 2025 00:00:00 +0000

Table of contents

1. The amnesia problem #

If you’ve read my posts on agentic coding or my local AI journey, you know I am all-in on using AI to control my computer and write software. But one critical piece was missing: Context.

LLMs are powerful but stateless. Every time I start a new chat, the model forgets who I am, what I’m working on, and that I prefer functional Python code over object-oriented styles. Worse, it also forgets the thing I just told it five minutes ago in another window.

The standard solution to this is RAG (Retrieval-Augmented Generation) and Memory. But when I looked at the available tools, I hit a wall.

Most solutions fell into two buckets:

SaaS / Web UIs: Upload your files to a cloud service or use a specific web chat interface that locks you in.
Developer Libraries: “Just use LangChain/LlamaIndex!” they say. Great, now I have to write a Python script every time I want to ask a question about a PDF?

I didn’t want a library. I didn’t want a web UI. I wanted a folder. I wanted to drop a file into ~/my-docs, and have my AI instantly know about it. And I wanted my AI’s “memory” to be a file I can edit, not a hidden vector in a database I can’t inspect.

So I decided to build my own file-based solution.

2. Learning from failure: AI Journal #

This isn’t my first attempt at building a memory system.

A few months ago, I built AI Journal—an ambitious local-first journaling app where an AI would read your entries and build a “living self-model” of who you are. It had sophisticated features: a structured “persona core,” typed “claim atoms” with evidence links, four organizational layers (L1→L4), recency-aware scoring with decay functions, and Git-based time travel to “ask questions of your younger self.”

The architecture seemed solid. The problem was simpler: local models aren’t good enough yet.

I designed the whole thing to run entirely on Ollama with my dual RTX 3090s. But even with a 48GB of VRAM, the models couldn’t reliably extract structured claims, handle the multi-step reconciliation, or maintain coherence across long conversations. Whether the design was right, I still don’t know—but the execution definitely hit the wall of current local LLM capabilities.

The lesson: I tried to build the entire system at once without validating the core concepts and individual ideas first. This time, I took a different approach: tackle it problem by problem. Instead of designing a complete system upfront, I would validate each piece—chunking, retrieval, re-ranking, memory reconciliation—before combining them. The experience and UX would be my own, but the algorithms would be adapted from established projects.

3. How I studied SOTA: clone, read, re-implement #

My approach was more methodical this time. I cloned five major frameworks into the same workspace as agent-cli:

LlamaIndex: The OG RAG framework
LangChain: The kitchen-sink approach
Letta (formerly MemGPT): Stateful agents with long-term memory
Mem0: Focused specifically on memory extraction
PydanticAI: Clean, typed agent framework

I didn’t read through all this code myself—that would take months. Instead, I used AI coding agents to search the codebases and write summaries of the technical implementation details:

Gemini CLI (Gemini 3.0 Pro)
Claude Code (Opus 4.5)
Codex CLI (GPT-5.1-Codex-Max)

I fed those summaries into ChatGPT 5 Pro together with my high level plan of what I wanted to create, and it created an implementation plan. Then I took that plan back to the coding agents for implementation. I personally found Opus 4.5 the best coding model, but GPT‑5.1 Codex Max was the best at reviewing the code and actually identifying critical bugs.

For validation, I used clip-files to copy the agent-cli source code plus the architecture documents (like memory.md and rag.md) into ChatGPT 5 Pro to validate the approach.

The AI did the searching and implementing; I did the understanding and decision-making.

4. The dependency problem: why ONNX over PyTorch #

One thing became immediately clear when going through these frameworks: they’re massive.

Installing LlamaIndex with all its dependencies pulls in PyTorch. PyTorch alone is 2-3GB. Add transformers, sentence-transformers, and the kitchen sink, and you’re looking at an 8GB+ environment just to ask questions about some PDFs.

This felt excessive for a CLI tool.

One of my core design goals was minimal dependencies. I wanted agent-cli to install in seconds, not minutes. I wanted it to work on my Mac, my NixOS server, and my friend’s laptop without downloading half the internet.

The solution: ONNX Runtime.

ONNX Runtime is ~200MB and runs inference on basically anything—CPU, CUDA, even Apple Silicon. For embeddings, I use embeddinggemma running locally via llama.cpp. For re-ranking, I use a cross-encoder exported to ONNX (Xenova/ms-marco-MiniLM-L-6-v2).

The result: pip install agent-cli[rag] takes about 30 seconds and adds ~300MB. The key insight: embeddings and re-ranking are cheap operations. You don’t need a GPU for them—CPU is perfectly fine. PyTorch with CUDA support can easily reach 2-3GB+; ONNX Runtime is ~200MB and runs anywhere. Since I’m not training models, just running inference on small encoders, ONNX is the obvious choice.

5. RAG: just let me grep it (kind of) #

My requirement for RAG was simple: Files on disk are the source of truth.

I didn’t want to manage an ingestion pipeline or trigger APIs. I implemented a RAG proxy that uses OS-level file watchers (watchfiles). Here is the workflow:

I drop a PDF, Markdown, or Python file into ~/rag_docs.
The system detects the Create event.
It chunks the text and embeds it into a local ChromaDB instance.
It’s immediately searchable.

When moving or deleting files, the system updates the index accordingly.

The architecture: OpenAI compatibility #

The “secret sauce” isn’t the retrieval itself; it’s how I expose it. Instead of building a custom chat app, I built an OpenAI-compatible proxy.

flowchart LR Client["Client
(Cursor, Open WebUI)"] --> Retrieve subgraph Proxy ["RAG Proxy"] direction TB Retrieve --> Augment["Augment & Forward"] Chroma[("ChromaDB")] -.-> Retrieve Watcher["File Watcher"] --> Chroma end Augment --> Upstream["Upstream LLM"]

# Point any OpenAI-compatible tool at your local proxy
export OPENAI_BASE_URL=http://localhost:8000/v1

That’s it. Any tool that supports custom OpenAI endpoints—Cursor, Cline, Open WebUI, LibreChat—instantly gains the ability to “read” my document folder.

Two-stage retrieval: the SOTA approach #

I looked at LlamaIndex here. Standard “semantic search” (using just embeddings/bi-encoders) is often disappointing. It finds text that looks similar but isn’t actually relevant.

A common pattern in modern RAG systems is Two-Stage Retrieval.

Fast Stage: Retrieve 3x candidates using standard embeddings (text-embedding-3-small or local models like embeddinggemma).
Smart Stage: Use a Cross-Encoder to re-rank the results.

A cross-encoder looks at the query and the document together and outputs a relevance score. It’s computationally heavier, but since I’m running this locally on my RTX 3090 (or even CPU via ONNX), the 50ms latency penalty is worth it for results that are actually useful.

This isn’t a new idea, but implementing it cleanly with ONNX avoided the huge dependency tax.

6. Memory: files over databases #

This is where my approach diverges most from standard tools like Mem0. Most memory systems treat memories as database rows. If the AI remembers “User likes pizza,” that’s a vector in a DB. If it hallucinates and records “User hates pizza,” good luck fixing it. You need to write a script to query the DB and delete the row.

I prefer data I can inspect.

My memory system stores every fact as a Markdown file with YAML front matter.

---
id: "550e8400-e29b-41d4-a716-446655440000"
role: "memory"
created_at: "2025-12-15T10:00:00Z"
conversation_id: "default"
---
The user prefers using 'uv' for Python package management.

Git integration: versioning memory 🕰️ #

Because memories are just files, I added a feature that I haven’t seen anywhere else: Automatic Git Versioning.

Every time the AI learns a new fact, updates an old one, or deletes a contradiction, the system automatically commits the change to a local git repository.

This leads to incredible capabilities:

Inspectability: I can git log and see exactly when the AI learned something.
Correction: If the AI learns something wrong, I open the markdown file in VS Code, edit it, and save.
Reversion: Agent hallucinated wildly yesterday? git reset --hard HEAD~1.

The reconciliation loop #

The logic for storing memories isn’t just “append to file.” Digging into Mem0 (and Letta) showed me that memory systems need active reconciliation. When I say “Actually, I switched to nix instead of uv,” the system:

Retrieves existing memories about package managers.
Decides (using a lightweight LLM) whether to ADD, UPDATE, or DELETE.
Executes the file operation (moving the old fact to a deleted/ folder for audit trails).

This keeps the memory bank clean and contradictory-free, unlike simple vector stores that just accumulate conflicting junk over time.

The tiny detail that broke everything #

After implementing the memory system—mostly modeled after Mem0—I noticed it didn’t work as well. After a lot of debugging, I found a tiny difference: I was asking the model for structured data via a tool call, whereas Mem0 uses freeform JSON that the LLM returns directly.

Since I use PydanticAI, fixing this was a one-line change:

agent = Agent(
 model=model_cfg,
 system_prompt=UPDATE_MEMORY_PROMPT,
- output_type=list[MemoryDecision],
+ output_type=PromptedOutput(list[MemoryDecision]), # JSON mode instead of tool calls
 retries=3,
)

This is exactly why reading the source code matters—not just the algorithms, but the tiny details that make them work.

7. Why building this was necessary #

You might ask: “Bas, why write thousands of lines of code for this? Why not just use ChatGPT’s memory or install LangChain?”

Three reasons: Privacy, Integration, and Dependencies.

Privacy #

I want to feed my AI my financial records, my health data, and my private journals. I am not comfortable sending that to OpenAI’s long-term storage or some VC-backed startup’s vector cloud. With agent-cli, the vectors live in ~/.cache/agent-cli/chroma, the files live in ~/documents, and the only thing leaving my machine is the specific context needed for a specific query (if I use a cloud model) or nothing at all (if I use Ollama/llama.cpp).

Integration #

By building this as an API proxy, I solved the “fragmented tools” problem. Any tool that supports custom OpenAI-compatible endpoints works out of the box:

agent-cli chat in the terminal
Cursor or Cline for agentic coding
Open WebUI, LibreChat, or Lobe Chat in the browser

Because my system speaks “OpenAI,” all of these tools share the same memory and the same document index. If I tell the terminal agent “I’m working on Project X,” and then switch to Cursor, Cursor knows about Project X. I can try a new chat UI tomorrow without losing anything—just point it to localhost:8000.

Dependencies #

To me, it just doesn’t feel right that asking questions about a PDF requires downloading 8GB of PyTorch. By choosing ONNX Runtime over the PyTorch ecosystem, I kept the entire install lightweight and fast. This matters when you want to run the system on a cheap VPS or your friend’s laptop.

8. Try it yourself #

The entire system is open-source and available in agent-cli.

Try Memory #

Get an LLM that remembers you using Ollama.

# 1. Pull models and start proxy
ollama pull embeddinggemma:300m && ollama pull gpt-oss:20b
uvx -p 3.13 --from "agent-cli[memory]" agent-cli memory proxy \
 --memory-path ./my-memories \
 --openai-base-url http://localhost:11434/v1 \
 --embedding-model embeddinggemma:300m

# 2. Chat from the terminal (env vars point to proxy)
export OPENAI_BASE_URL=http://localhost:8100/v1 OPENAI_API_KEY=dummy

# Tell it something about yourself
uvx openai api chat.completions.create -m gpt-oss:20b -g user "My name is Alice and I love hiking"

# Later, ask if it remembers
uvx openai api chat.completions.create -m gpt-oss:20b -g user "What's my name?"

Try RAG #

Chat with your documents.

# 1. Start the RAG proxy (runs in foreground)
uvx -p 3.13 --from "agent-cli[rag]" agent-cli rag-proxy \
 --docs-folder ./my-docs \
 --openai-base-url http://localhost:11434/v1 \
 --embedding-model embeddinggemma:300m

# 2. Chat with your docs
OPENAI_BASE_URL=http://localhost:8000/v1 OPENAI_API_KEY=dummy \
 uvx openai api chat.completions.create \
 -m gpt-oss:20b \
 -g user "What does my documentation say about X?"

9. What I learned #

This journey reinforced a few beliefs:

Validate components first. AI Journal failed not because the ideas were wrong, but because I tried to build a complete system before validating individual components. This time I tackled problems one at a time—chunking, retrieval, re-ranking, memory reconciliation—and only combined them once each piece worked.
Study implementations, not just papers. Reading about two-stage retrieval is one thing. Reading how LlamaIndex actually implements it—the edge cases, the defaults, the performance tradeoffs—is far more valuable. Clone the repos. Read (or make AI read) the code.
Dependencies matter. A 30-second install versus a 10-minute download changes whether someone actually tries your tool. I am happy with the ONNX Runtime over PyTorch choice.
Files are the ultimate API. By stripping away the complex databases and proprietary interfaces, I ended up with a system that is robust, version-controllable, and surprisingly simple to reason about.
Use the best model for each job. I used 4 different AI models during development, each with different strengths. Gemini 3.0 Pro, Opus 4.5, GPT-5.1-Codex Max. I used all of them during the implementation phase, but each one has its own strengths. Fore example, Opus 4.5 was the best coding model, but GPT-5.1 Codex Max was the best at reviewing the code and actually identifying critical bugs (use the built-in /review feature).

Wyoming MLX Whisper

Fri, 05 Dec 2025 00:00:00 +0000

GitHub Backup Sync

Fri, 24 Oct 2025 00:00:00 +0000

Matty

Sat, 23 Aug 2025 00:00:00 +0000

MindRoom

Tue, 29 Jul 2025 00:00:00 +0000

How a 3090 turned into two local‑first AI projects 🎮

Mon, 14 Jul 2025 00:00:00 +0000

Table of contents

1. Introduction: The Unplayed Games #

I’ve wanted to dabble with local AI for a long time, mainly because of the possibilities it unlocks. Local AI allows me to use all of my data—financial records, personal messages, location history, even transcribing everything I say—without the privacy concerns of sending it to a cloud provider. This level of deep, personal data analysis is only something I’d be comfortable with on my own hardware.

However, I couldn’t justify buying a powerful GPU just for an imagined AI project that I might never really start. Then it hit me: I could also use it for gaming (duh). Because I really enjoyed The Last of Us series, I thought, “Okay, why not play The Last of Us on PC?” This was the push I needed. So, a couple of weeks ago, I convinced myself to buy someone’s old gaming machine. After many deep research sessions with ChatGPT about the best bang-for-the-buck consumer-grade GPU, I landed on an NVIDIA 3090. I found an (old) beast of a machine, and my initial idea was set.

Click here to see the full system specs ($1350 for all this!)

GPU: ASUS ROG STRIX RTX 3090 OC (24GB)
CPU: AMD Ryzen 9 3900X (12-Core)
Motherboard: ASUS ROG Crosshair VIII Hero
Power Supply: Be Quiet! 1200W Dark Power Pro
RAM: 32GB G.Skill Trident Z Royal DDR4
Storage: 2TB Intel M.2 NVMe SSD
Case: Corsair 4000D Airflow
CPU Cooler: Thermalright Peerless Assassin 120

Fast forward four weeks. I haven’t played a single minute, but my machine is now indexing 200 GB of email archives and facing a serious performance bottleneck I’ll get to later. That “gaming” machine has been humming away, but not rendering virtual worlds. It’s been training, transcribing, and thinking.

I’m writing this post simply because I am so enthusiastic about local AI. I’ve found that local models are surprisingly powerful for specific use cases. Of course, they aren’t going to replace Gemini or Claude for coding, but for many other applications, they work incredibly well. Lately, I’ve been so obsessed with this that I’m sleeping significantly less and spending all my free hours working on it.

2. The Rocky Road to a Working Setup #

Before I could even get to the fun AI stuff, I had to wrestle with the machine itself. I started with Pop!_OS, which is supposed to be a great out of the box solution that “just works”. However, it’s been a long time since I used Linux on the desktop; my experience is almost exclusively with servers. I quickly installed some of the wrong NVIDIA drivers and ended up debugging stuff in Grub. I got frustrated with the different commands I had to run to set up everything and realized that reproducing this system was going to be a massive pain. Since I’ve grown to love Nix on my Mac with Nix-Darwin, I decided to switch the whole system to NixOS. This has been an amazing decision. Even though there are many pain points with Nix, I think it is well worth it. For those interested, you can see the complete setup, including all the local AI services, in my NixOS configuration on GitHub.

3. The First Victory: `agent-cli` 🐍 #

With a stable system, I quickly got started with a few scripts. These soon evolved into my first real project of this adventure: agent-cli.

agent-cli is a collection of local-first, AI-powered command-line agents that run entirely on your machine. The core philosophy is privacy; your data never leaves your computer. It’s designed for seamless integration with system-wide hotkeys. I’m now using it constantly.

Instead of typing, I now dictate almost everything. A quick hotkey, and agent-cli transcribe turns my speech into text. Another hotkey, and agent-cli autocorrect cleans it up using a local LLM running on Ollama. It has genuinely streamlined my workflow. For example, I can pipe the tools together, like correcting text from my clipboard and then having it spoken aloud: agent-cli autocorrect | agent-cli speak.

The toolkit includes:

autocorrect: Fixes grammar and spelling.
transcribe: Uses a local Whisper model for speech-to-text.
speak: Converts text to speech with a local TTS engine.
voice-edit: A voice-powered clipboard assistant.
assistant: A hands-free voice assistant using a wake word.
chat: A conversational AI with tool-calling capabilities.

It’s been a fantastic success in my personal workflow, and it’s all open-source for you to try. Contributions and ideas welcome!

Once I trusted my toolchain, the obvious next step was to feed all my data into it.

4. The Grand Ambition: `AIBrain` 🧠 #

My second package, AIBrain, is a much larger endeavor. The vision is to create a “life-OS”—a private, local AI that processes all my personal data (emails, calendar, files, photos, messages, browser history, location data) to generate summaries and answer questions, eventually allowing for queries like:

aibrain ask "When did I get into landscape photography?"
aibrain ask "How many countries did I visit between 2015-2025?"

This project led me to explore the landscape of agentic frameworks. I started with CrewAI, but I quickly found that despite the hype and funding, it had some painful problems (specifically this issue where things just fail without any helpful error messages). I decided to switch to LangGraph, which felt more robust. I also chose to not use PydanticAI (which I used in agent-cli) because its Ollama support is just a wrapper around the OpenAI-compatible API, which doesn’t expose all the useful options (I even opened a PR in Ollama to make its OpenAI compatible API more feature-complete, but it has not been merged yet).

The core infrastructure of AIBrain is ready. It can index emails, calendar events, and files in various formats (PDF, Excel, PowerPoint, images). However, I’ve hit a significant challenge.

5. Conclusion: The Journey Continues #

This past month has been a whirlwind. I set out to play games and ended up with two open-source AI packages and a much deeper appreciation for the power sitting in a consumer-grade GPU. It’s incredible what you can achieve with local hardware today.

agent-cli is a finished, polished tool that I use daily. AIBrain is a much bigger dream, and I’m still figuring out how to solve the performance puzzle to make it truly practical. Next up: smarter chunking with LangGraph, vector-DB swap-outs, and GPU scheduling tricks.

This experience has reinforced my passion for local-first software and open-source development. The future of personal, private AI is being built right now, not just in large corporate labs, but in the homes of hobbyists tinkering with their gaming rigs.

⭐ If this resonates, star the repos & drop issues/questions.

And who knows, maybe I’ll even find the time to play ‘The Last of Us’.

Agent CLI

Wed, 25 Jun 2025 00:00:00 +0000

gh-download

Fri, 30 May 2025 00:00:00 +0000

Confessions of an asymmetric hypocrite: on Python dependencies

Sun, 27 Apr 2025 00:00:00 +0000

Introduction: The Dependency Dilemma 🤔 #

I just listened to a podcast “Build software that lasts!” with Bert Hubert (a fellow Dutch), who prefers to reduce the number of dependencies to an absolute minimum (also see his related blog posts [1] and [2]). While I agree one shouldn’t use is-even as a dependency (is-odd is a better choice 😏), I think there is a lot of value in reducing the number of dependencies but also in using the right tool for the job and not continuously re-inventing the wheel. The podcast got me thinking about my own stance on dependencies and how I actively avoid introducing dependencies into my libraries while expecting (hoping 🤞) others to depend on my work.

Like many software developers, I’ve spent countless hours wrestling with dependencies. Dependency hell is easy to fall into, especially in dynamic languages like Python. Those moments where installing or updating one package breaks another, or where a tiny library pulls in hundreds of transitive dependencies, are frustratingly common. This frustration partly motivated me to create unidep, a tool aimed at simplifying dependency management across different Python tools like pip and conda.

But working on unidep and my other projects forces me to think about my own relationship with dependencies. I’ve realized I operate with what feels like an “asymmetric hypocrisy.” My standards for the libraries I publish are extremely strict regarding which external dependencies I consume. Yet, as an author of over 30 Python packages, I inherently produce dependencies, hoping others will adopt my work. And then, when building applications or CLIs, my caution often disappears entirely.

This post dives into that asymmetry. It’s about the different hats I wear: the minimalist library author (as a dependency consumer), the hopeful library author (as a dependency producer), and the pragmatic application developer. In this post, I’ll explain why I take these different approaches and why it’s really about the situation, not a contradiction.

Table of contents

1. The Library Author Hat 🎩 (as Dependency Consumer): Maximum Caution #

When I build and publish an open-source Python library – like Adaptive, pipefunc, or unidep – my primary focus shifts to the users who will depend on my code. My choices directly impact their projects. This responsibility means I have to be very careful about adding external dependencies:

Minimize and Scrutinize: Fewer dependencies mean fewer potential conflicts for users. I aggressively minimize the core runtime dependency list.
Favor Battle-Tested Dependencies: I strongly prefer dependencies that are already widely adopted and trusted by the community. (I have been called that person that always wants the latest and greatest, the truth is that I am a sucker for shiny new things but typically only adopt them once they are already quite popular.) Think numpy or cloudpickle. These are less likely to cause unexpected issues or disappear overnight.
Low Transitive Dependency Count: A key criterion is how many other packages a potential dependency pulls in. Ideally, I choose dependencies with zero or very few further dependencies. For example, in pipefunc, dependencies like numpy, networkx, and cloudpickle are good examples – they are popular and don’t rely on many other things.
Optional Dependencies for Heavy Features: If a feature really needs a large or less common dependency, I try making it a optional extra (project.optional-dependencies). Users opt-in only if they need that specific capability.
Avoid Pinning (Mostly): As mentioned before, hard version pins (==1.2.3) in libraries are usually harmful. Minimum versions (>=1.2) or safe upper bounds (<2.0) are preferred.
License Awareness: The license of a dependency matters immensely, especially regarding compatibility and obligations (GPL considerations).

The guiding principle is: Do not cause pain for your users. A library should integrate smoothly into diverse environments.

2. The Library Author Hat 🎩 (as Dependency Producer): The Paradox #

Here’s where the “hypocrisy” might seem to surface. While I’m cautious about consuming dependencies for my libraries, I am also the author of over 30 packages. I invest time and effort into these tools, naturally hoping they prove useful and that others will adopt them – making my libraries dependencies in their projects.

Is it hypocritical to be wary of adding dependencies while simultaneously creating them for others? Perhaps slightly, but it’s also the nature of contributing to an ecosystem. I try to handle this by strictly following the rules from Section 1 for my own libraries. If I expect others to rely on my code, I feel responsible to make that dependency as low-risk and stable as possible. I want my libraries to be the kind of well-behaved, minimal-friction dependencies that I myself would be comfortable adding.

3. The Application Developer / Contributor Hat 🧢: Practicality is Key #

When the context shifts to building an application – like a command-line tool, a web service, or a personal project – my rules become much less strict. The key differences are isolation (keeping things separate) and how far the effects reach.

Isolation Reduces Risk: Applications often run in controlled, isolated environments. Docker, conda/micromamba (Python Environment Jungle post), or simple venvs contain the dependencies. A conflict within the application’s environment doesn’t break the user’s entire system or other unrelated projects.
CLIs as a Prime Example: Many tools I’ve built are CLIs (like tuitorial or helpers within my dotfiles). These might have tens of dependencies. Because they are installed into isolated environments (often via uv, pipx, or within a dedicated conda env), this complexity is okay. The risk is contained.
Developer Tools Welcome: Productivity boosters like ruff, mypy, etc., are fair game as development dependencies (My Python Project Toolkit). They don’t impact the end-user runtime (if packaged properly), so I am much more lenient with dev tools.
Focus on Functionality: The goal is often to deliver features or solve a problem efficiently. Leveraging existing libraries, even those with dependencies, is usually faster and more robust than reimplementing complex logic. It usually makes more sense to use existing libraries.

In this context, the focus changes from minimizing external impact (libraries) to maximizing internal capability and maintainability within a controlled scope (applications).

4. Conclusion: Not Hypocrisy, but Context-Driven Strategy 🙏 #

My approach to dependencies is “it depends on the situation”. I’m stricter about dependencies for my libraries than for my applications or personal tools. I am cautious about adopting dependencies while actively producing them for others.

But I don’t see this as hypocrisy. It’s a practical approach based on different situations and goals:

Libraries (as Consumer): Focus on keeping the software community stable and making things easy for users. Be conservative.
Libraries (as Producer): Create software responsibly, following the same high standards you expect from others.
Applications: Focus on adding features and developing quickly within separate environments. Be pragmatic.

Understanding the different risks and goals for each situation helps make good decisions about dependencies. It’s about choosing the right trade-offs for the specific situation, whether you’re building a foundational block for the community or a specific tool for a controlled environment.

How do you handle dependencies across different types of projects? Do you find yourself navigating similar questions? Share your thoughts!

Functions over classes? Why I prefer a simpler, functional style in Python 🐍

Mon, 14 Apr 2025 00:00:00 +0000

Over years of building software, from scientific simulation tools like Adaptive and pipefunc to package management packages like unidep, I’ve found myself preferring a certain coding style: using small, focused functions more than large classes with complex internal state and deep inheritance hierarchies. This isn’t about dogma, but about a pragmatic approach that, in my experience, leads to code that is easier to understand, test, and maintain.

This post outlines why I lean towards this functional-inspired style in my Python projects, while still recognizing where classes are the right tool.

Table of contents

1. Taming Complexity: Avoiding the Over-Engineering Trap 🧠 #

One of the biggest challenges in software development is managing complexity. As systems grow, keeping track of how different parts interact and understanding the internal state of objects can become a lot to keep track of mentally (high cognitive load).

I’ve often observed a pattern, especially when developers first start using object-oriented programming: the tendency to use classes everywhere. This can lead to over-engineering – introducing many layers of inheritance, abstract base classes with only a single implementation, and generally creating complexity when simplicity would suffice. Thinking you’re being clever by building complex class structures can sometimes just be a form of overgeneralization that makes the code harder, not easier, to follow.

Classes, particularly those with many methods modifying internal attributes (self.x, self.y, etc.), require you to constantly hold their state in your head. When you call a method object.do_something(), the outcome might depend not just on the arguments you pass, but also on the object’s history – what methods were called before, and in what order? This hidden state can make reasoning about the code’s behavior difficult and debugging tricky.

Functions, particularly when kept small and focused, tend to be more explicit. Ideally, a function takes inputs and produces outputs, with minimal side effects (or ideally, none - making them pure functions). Its behavior is primarily determined by its arguments. This makes the code flow easier to follow: data comes in, transformation happens, data goes out. There’s less hidden context to juggle mentally.

2. The Power of Small, Focused Functions 🎯 #

This leads directly to the preference for small functions, each doing one thing well (adhering to the Single Responsibility Principle). When functions are concise and have a clear purpose:

Readability Improves: It’s easier to understand what a small, well-named function does quickly.
Reusability Increases: Small, focused functions are more likely to be reusable in different parts of the codebase or even in other projects.
Refactoring is Simpler: Modifying or replacing a small unit of logic is less risky than altering a large, complex method within a class.

This philosophy is something I actively apply. Looking at the codebase for unidep, for example, you’ll find many utility functions in modules like _dependencies_parsing.py or _conda_env.py, each handling a specific, well-defined task.

3. Keeping Classes Simple & Using Them Where They Fit 🛠️ #

So, where do classes fit in this picture?

Firstly, as simple data structures – think NamedTuples or dataclasses (preferably with frozen=True). Their main role is to group related data attributes together, providing a clear structure for passing information around. They hold data, but don’t typically manage complex state transitions.

# Example of a simple data class from unidep
from dataclasses import dataclass
from pathlib import Path

@dataclass(frozen=True) # Use frozen=True for immutability
class Spec:
 name: str
 which: Literal["conda", "pip"]
 pin: str | None = None
 # ... other simple attributes ...

# Used to pass structured data, not to manage complex state
def process_spec(spec: Spec) -> Path:
 # ... function logic using spec data ...
 pass

Secondly, and importantly, classes are excellent for defining user-facing APIs. For the end-user of a library, having a single object with easily discoverable methods (object.<tab>) is often a great user experience.

A good example from my own work is the pipefunc package. It provides two main classes, Pipeline and PipeFunc, which encapsulate the core functionality. Users interact primarily with these objects and their methods.

# Simplified example from pipefunc's user perspective
from pipefunc import Pipeline

pipeline = Pipeline(...)
pipeline.add(some_function, ...)
results = pipeline.run("output_node")

However, inside the methods of these API classes (Pipeline.run, Pipeline.add, etc.), the actual logic is often delegated to smaller, internal functions. The class acts as a clean interface or facade, orchestrating calls to these more focused, functional units. This gives the user a nice API while keeping the internal implementation modular and testable.

I also actively avoid deep inheritance hierarchies. Composition (having objects contain other objects) often provides a more flexible and explicit way to combine functionality without the tight coupling and complexity inheritance can introduce.

4. The Testability Advantage ✅ #

One of the most significant practical benefits of prioritizing functions and simple classes is testability. Small, pure (or mostly pure) functions are incredibly easy to test:

Provide known inputs.
Assert expected outputs.
Minimal setup or mocking is required because there’s no complex internal state to manage.

Even when testing the methods of the API classes (like Pipeline in pipefunc), the fact that they often delegate to well-tested internal functions simplifies the process. You focus on testing the orchestration and interaction logic at the class level, knowing the underlying functional units are already verified.

Testing classes with intricate internal state inherently involves more setup (instantiating the object, getting it into the right state before calling the method under test) and potentially more complex assertions to verify state changes. Aiming for more functional units significantly simplifies the testing process.

5. It’s About Pragmatism, Not Dogma 🙏 #

This isn’t a wholesale rejection of object-oriented programming. It’s about choosing the right tool for the job and prioritizing clarity, testability, and maintainability. Classes are fantastic tools, especially for building clear APIs and structuring data. Deep inheritance and complex state management, however, should be used carefully, not just because the language allows it.

For much of the work I do – building libraries, tools, and performing data analysis or simulations – I’ve found that prioritizing functions, keeping API classes focused, minimizing internal state, and avoiding deep inheritance leads to code that I find easier to write, understand, debug, and maintain in the long run. The reduction in cognitive load and the ease of testing are strong points for how I work.

If you often find yourself struggling with complex state management or tangled inheritance trees, perhaps consider if a simpler, more functional internal structure, potentially fronted by a clean class-based API, might bring some clarity and simplicity to your projects too.

Vibe coding worked—until the 15‑hour cleanup

Fri, 21 Mar 2025 00:00:00 +0000

I tried “vibe coding” for 3 hours, my application worked, but then I spent 15 hours fixing it #

That’s not an exaggeration—it’s really what happened.

Recently, I tried “Vibe coding,” a term coined by Andrej Karpathy, a prominent AI researcher, co-founder of OpenAI, and former AI lead at Tesla. In his viral tweet from February 2025, Karpathy described vibe coding as a programming approach where you “fully give in to the vibes, embrace exponentials, and forget that the code even exists.” It’s essentially about letting AI handle the coding based on natural language prompts without closely reviewing the generated code.

Intrigued by the hype, I decided to experiment with vibe coding using Cursor on a new Python project I called dotbins (check out my detailed write-up here)." Dotbins manages CLI tool binaries within Git repositories, automatically downloading the correct binary from GitHub for the user’s platform and architecture, significantly simplifying my dotfiles and bootstrapping process. I am super excited about this project but it is not what this post is about.

Initially, it was fun to get something working so quickly. Within just three hours, I had a functioning prototype—without carefully examining the code. But I had unknowingly created a monster—a true spaghetti monster! 🍝

Once I started reviewing the AI-generated code, I realized it was totally incomprehensible. The AI had implemented the most complicated and roundabout ways to achieve simple tasks. It introduced redundant checks for impossible conditions, irrelevant error handling, and execution paths that would never trigger. The code worked superficially, but beneath the surface, it was chaotic and nearly impossible to understand or maintain.

This realization turned three hours of fun into fifteen hours of meticulous debugging, refactoring, and cleaning up the mess. Essentially, I had to rewrite the entire project.

My main takeaway: AI-assisted coding, or “vibe coding,” is incredibly powerful for quick prototyping but dangerous when used without oversight. According to this recent Y Combinator discussion, many startup founders are building products almost entirely with AI-generated code, “vibing” the code, meaning they let the AI write the code and don’t review it. For writing a couple of hundred lines of code it likely works pretty well, but for anything more complex, it’s a recipe for disaster. Personally, I would never invest in, build upon, or use such products—they’re essentially constructing technical prisons for themselves.

AI is fantastic for inspiration, quick prototyping, writing code, and refactoring—but letting it run wild without careful human review can quickly lead to severe technical debt and hidden bugs.

Thanks for reading!

dotbins: managing binary tools in your dotfiles 🧰

Tue, 11 Mar 2025 00:00:00 +0000

In my previous post about terminal productivity, I shared a setup combining tools like zsh, oh-my-zsh, zsh-z, and various plugins to create a highly efficient command-line environment. While that setup served me well for years, I’ve recently encountered a challenge: incorporating modern, compiled CLI tools across all my environments without administrative privileges.

Today, I’m excited to introduce dotbins - a solution I created to solve this exact problem.

The Modern CLI Tools Problem #

Over the past few years, we’ve seen an explosion of incredible, modern CLI tools written in languages like Rust and Go. Tools like:

zoxide (a smarter, faster alternative to zsh-z)
bat (a cat replacement with syntax highlighting)
eza (a modern ls alternative)
delta (a better diff viewer for Git)
fzf (a fuzzy finder for your terminal)
ripgrep (a faster, better grep)
atuin (a shell history with search and sync)

These tools significantly enhance terminal productivity, but they come with a catch: they’re compiled binaries, not simple scripts. While my previous setup worked well because all the tools were basic shell scripts I could include as submodules in my dotfiles repository, these modern alternatives require installation.

This became frustrating when working on:

Remote systems where I lacked admin/sudo permissions
Machines that I use only temporarily
Systems where I didn’t want to install package managers

I’d spend time carefully setting up my dotfiles with all my configurations, only to find that the actual tools they relied on weren’t available.

Enter `dotbins` 🧰 #

dotbins was born out of this frustration. It’s a utility that manages pre-compiled binaries for CLI tools across multiple platforms and architectures, right in your dotfiles repository.

The key insight: track pre-compiled binaries in a separate Git repository (using Git LFS for efficiency), include this repository as a submodule in your dotfiles, and ensure all your essential tools are immediately available on any system.

No package manager, no sudo, no problem.

How `dotbins` Works #

dotbins handles several critical tasks:

Downloads binaries directly from GitHub releases for your selected tools
Organizes them by platform (macOS, Linux) and architecture (amd64, arm64)
Tracks installed versions and update timestamps
Extracts binaries from archives in various formats
Integrates seamlessly with your shell via a simple PATH addition

The configuration is straightforward - a YAML file where you define which tools you want and how to get them:

tools:
 ripgrep:
 repo: BurntSushi/ripgrep
 extract_binary: true
 binary_name: rg
 binary_path: ripgrep-*/rg
 asset_patterns:
 linux: ripgrep-{version}-{arch}-unknown-linux-musl.tar.gz
 macos: ripgrep-{version}-{arch}-apple-darwin.tar.gz
 arch_map:
 amd64: x86_64
 arm64: aarch64

My Workflow: A Two-Repository Approach #

My approach uses two repositories:

Private dotfiles repository containing all my configurations
Public mydotbins repository containing all the binaries

I include mydotbins as a submodule in my dotfiles repository. This separation keeps large binary files out of my main dotfiles repository and allows me to share the binary collection publicly while keeping my personal configurations private.

When I clone my dotfiles on a new system, I get not just my configuration files, but also all the CLI tools I depend on for productivity.

Modern Alternatives to My Previous Setup #

With dotbins, I’ve been able to upgrade several tools from my previous setup:

Old Tool	Modern Alternative	Advantage
`zsh-z`	`zoxide`	Faster, smarter, works across shells
`cat`	`bat`	Syntax highlighting, Git integration
`ls`	`eza`	Better formatting, Git integration
`grep`	`ripgrep`	Much faster, respects `.gitignore`
Control + R	`atuin`	Shell history with search and sync
Default Git diff	`delta`	Syntax highlighting, better visual diffs
N/A	`fzf`	Fuzzy finding makes everything better
N/A	`atuin`	Shell history with search and sync

The best part? All of these are now automatically available on every machine where I clone my dotfiles, without any additional installation steps.

Getting Started with `dotbins` #

If you’d like to adopt this approach, here’s how to get started:

Install dotbins:

pip install dotbins
# Or better yet, use uv:
uvx dotbins

Create a configuration file at ~/.config/dotbins/config.yaml:

tools_dir: ~/.mydotbins/tools

platforms:
 linux:
 - amd64
 - arm64
 macos:
 - arm64

tools:
 # Add your favorite tools here
 fzf:
 repo: junegunn/fzf
 extract_binary: true
 binary_name: fzf
 binary_path: fzf
 asset_patterns: fzf-{version}-{platform}_{arch}.tar.gz
 platform_map:
 macos: darwin

Initialize and download your tools:
```
dotbins init
dotbins update
```

Add the shell integration to your .zshrc or .bashrc:

# dotbins - Add platform-specific binaries to PATH
_os=$(uname -s | tr '[:upper:]' '[:lower:]')
[[ "$_os" == "darwin" ]] && _os="macos"

_arch=$(uname -m)
[[ "$_arch" == "x86_64" ]] && _arch="amd64"
[[ "$_arch" == "aarch64" || "$_arch" == "arm64" ]] && _arch="arm64"

export PATH="$HOME/.mydotbins/tools/$_os/$_arch/bin:$PATH"

(Optional) Track your binaries in a separate Git repository with Git LFS for efficiency

Finding New Tools to Add #

Not sure which pattern to use for a new tool? The analyze command is here to help:

dotbins analyze sharkdp/bat

This will suggest a configuration for the tool based on its GitHub release assets.

Conclusion: The Evolution of My Terminal Setup #

With dotbins, my terminal productivity setup has evolved beyond what I described in my previous post. I can now seamlessly incorporate the best modern CLI tools without worrying about installation or administrative privileges.

The beauty of this approach is its portability—I get a consistent, powerful environment on any machine where I clone my dotfiles.

If you’re a terminal power user who works across multiple environments, give dotbins a try. It might just be the missing piece in your dotfiles setup!

dotbins

Sun, 09 Mar 2025 00:00:00 +0000

pixi-to-conda-lock

Wed, 26 Feb 2025 00:00:00 +0000

GPL for Python packaging ⚖️

Mon, 10 Feb 2025 00:00:00 +0000

Understanding GPL for Python Packaging #

Over the last 10 years–since I started working on and with open source projects–the GPL topic has frequently come up with friends and colleagues. I simply write this post as a reminder to myself and to have something concise and clear to link to later. Also, IANAL, so if you are relying on this post, please do your own research, I have included all the links I used to make this post.

The GNU General Public License (GPL) is widely used in the open-source world, but its implications can be unclear, especially regarding Python packaging. If you develop or distribute Python software, understanding GPL variants is crucial.

What is GPL? #

GPL is a strong copyleft license—meaning software using GPL-licensed code must also be released under the GPL. There are several key variants relevant to Python:

GPLv2 & GPLv3: These licenses require that software using GPL code (directly or indirectly) also be open-sourced under GPL upon distribution.(¹,²) If your Python application imports or depends on a GPL package, your code must adopt GPL licensing.
LGPL (Lesser GPL): LGPL is more permissive and is usually applied to libraries. It allows proprietary software to link or import the library without forcing the proprietary software itself to adopt LGPL.(³,⁴) You just need to ensure users can replace or modify the LGPL component.
AGPL (Affero GPL): AGPL is similar to GPLv3 but includes a critical extra clause: it applies even when users access the software remotely over a network (such as web services or APIs). If you use an AGPL-licensed Python library, your entire application must be open-sourced under AGPL, even if you’re only hosting it online.(⁵,⁶)

What Does “Linking” Mean in Python? #

GPL licenses often refer to “linking,” traditionally associated with compiled languages (e.g., C, C++). Python doesn’t link code in the traditional sense. However, it is generally agreed upon that importing a Python module counts as linking for licensing purposes.(¹,⁷) Therefore, directly importing a GPL-licensed Python package typically triggers the GPL’s copyleft requirements.

Why Does This Matter for Python Packaging? #

Python makes it easy to incorporate third-party libraries. But unintentionally using GPL-licensed libraries (especially GPL or AGPL) legally obligates you to open-source your proprietary code.

Direct Dependencies: If you directly import a GPL-licensed Python library, your software must be GPL-licensed upon distribution.(¹,²)
Indirect (Transitive) Dependencies: If any dependency in your Python project indirectly relies on GPL-licensed code, your entire software must also adopt GPL licensing upon distribution. Even a hidden, indirect GPL dependency triggers the GPL requirements.(⁸,⁹,¹⁰)

Recommendations #

Check Licenses Carefully: Always verify the licenses of both your direct and indirect dependencies to prevent unintended GPL licensing obligations.(¹¹,⁷)
Use Permissive or LGPL Licenses: When planning to keep your own software proprietary, prioritize libraries with permissive licenses (e.g., MIT, Apache, BSD) or LGPL-licensed libraries, as they allow for proprietary use without triggering GPL requirements.(³,⁴)
Separate GPL Code: If you must utilize GPL code, explicitly separate it from proprietary code. Using subprocess calls or isolated microservices is often recommended since it counts as “mere aggregation,” not linking, thus avoiding GPL obligations.(¹²,⁷)

By clearly understanding GPL licensing implications, you can confidently manage your Python project’s dependencies and safely distribute your software.

tuitorial: terminal-based interactive code tutorials 📚

Sat, 04 Jan 2025 00:00:00 +0000

While trying to demonstrate my pipefunc package (a tool for automating computational pipelines), I hit a common presentation problem: how do you effectively walk through code without endless repetition?

Traditional presentation tools like PowerPoint force you to copy-paste the same code snippets multiple times just to highlight different parts. Jupyter notebooks are great for self-paced learning but clunky for live presentations (too much text that the viewer will try to read). I needed something that would let me define code once and create multiple views highlighting different aspects.

So, during the holiday season, I did what any reasonable developer would do: procrastinated on my actual work and built a new tool instead!

What is tuitorial? #

tuitorial is a Python package that lets you create interactive, terminal-based code tutorials where you define the code ONCE, then create steps that highlight different parts based on rules (like regex patterns or line numbers). No more copy-paste presentations!

Key Features #

Write code once: Define your code once, then create multiple views highlighting different aspects
Rich highlighting options: From simple literal matches to complex regex patterns
Interactive navigation: Intuitive keyboard controls for stepping through tutorials
Beautiful terminal UI: Powered by Textual for a polished look
Flexible configuration: Choose between Python API or YAML format
Development mode: Live reloading for quick iterations

See It in Action #

Here’s a demo of tuitorial explaining pipefunc - a package that lets you create computational pipelines with automatic parallelization, caching, and dependency management:

Who Should Use This? #

While tuitorial isn’t for everyone, it’s perfect for:

Developers giving technical presentations
Teachers explaining code concepts
Teams creating interactive documentation
Anyone who loves terminal-based tools

Getting Started #

pip install tuitorial

Check out the documentation for examples and detailed usage instructions.

Final Thoughts #

Building tuitorial was a lot of fun! While it may not replace PowerPoint for your next business presentation, it offers a unique approach to code demonstrations that I hope others will find useful.

Try it out and let me know what you think!

Python environments that stuck: micromamba and uv 🐍

Fri, 03 Jan 2025 00:00:00 +0000

1. Introduction 🚀 #

Like many Python developers, I’ve spent considerable time trying different environment management tools to find what works best for me.

After exploring various options from virtualenv to Conda, I found that Conda-based solutions fit my needs well - mainly because they handle non-Python dependencies without much hassle.

While Conda itself can be slow when setting up environments, tools like Mamba and Pixi provide the same functionality with better performance.

In this post, I’ll share my experience with different Python environment tools and provide practical commands for setting up Python environments.

Table of contents

2. The Alternatives 🌟 #

Before we start look at this comparison tables for the most popular Python environment management tools.

I had to split the tables into two because of the number of columns.

Table 1: Python Environment Management Tools

Tool	Key Features	Native/3rd Party	Non-Python Deps
Pyenv	Python version management	3rd Party	❌
Pipenv	Package & env management	3rd Party	❌
Conda	Package & env management	3rd Party	✅
Mamba	Fast Conda replacement	3rd Party	✅
Micromamba	Lightweight Mamba	3rd Party	✅
Docker	Containerization	3rd Party	✅
Venv	Built-in env management	Native	❌
Virtualenv	Env management (pre-3.3)	3rd Party	❌
Homebrew	macOS/Linux package manager	3rd Party	✅
Pixi	Fast Conda alternative	3rd Party	✅
uv	Fast pip and venv replacement	3rd Party	❌
System Package Managers	OS-based package management	Native	✅

Table 2: Python Environment Management Tools

Tool	Ease of Use	Isolation	Cross-Platform Support	Community & Support	Performance
Pyenv	Medium	✅	Windows, macOS, Linux	Good	Good
Pipenv	Easy	✅	Windows, macOS, Linux	Good	Good
Conda	Easy	✅	Windows, macOS, Linux	Excellent	Average
Mamba	Easy	✅	Windows, macOS, Linux	Growing	Fast
Micromamba	Medium	✅	Windows, macOS, Linux	Growing	Fast
Docker	Hard	✅	Windows, macOS, Linux	Excellent	Good
Venv	Easy	✅	Windows, macOS, Linux	Excellent	Good
Virtualenv	Medium	✅	Windows, macOS, Linux	Good	Good
Homebrew	Easy	❌	macOS, Linux	Excellent	Good
Pixi	Easy	✅	Windows, macOS, Linux	Growing	Fast
uv	Easy	✅	Windows, macOS, Linux	Growing	Very Fast
System Package Managers	Easy	❌	Depends on OS	Depends on OS	Good

Now that we’ve seen the alternatives, let’s dive into setting up environments with the help of Conda and Mamba.

🐍 Pyenv #

A popular tool that enables you to easily manage multiple Python versions on a single system. It allows you to switch between different Python versions without interfering with system-level installations.

Setup Instructions

Install pyenv:
- macOS: brew install pyenv
- Ubuntu: curl https://pyenv.run | bash
Add the following lines to your shell configuration file (e.g., ~/.bashrc, ~/.zshrc):
```
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
```
Restart your shell or run source ~/.bashrc (or source ~/.zshrc)
Install a specific Python version: pyenv install 3.8.5
Set the Python version for your project: pyenv local 3.8.5

📦 Pipenv #

A package management and virtual environment management tool that combines the functionality of pip and virtualenv. It streamlines the process of installing and managing packages within isolated environments.

Setup Instructions

Install pipenv: pip install --user pipenv
Change to your project directory: cd my_project
Initialize a new virtual environment: pipenv --python 3.8
Install packages: pipenv install <package_name>
Activate the environment: pipenv shell

🌐 Conda #

A cross-platform package manager and environment management tool, often used in conjunction with Anaconda and Miniconda. Conda simplifies the installation and management of packages and dependencies, especially for data science and machine learning projects.

Setup Instructions

Download and install Miniconda: https://github.com/conda-forge/miniforge#miniforge3
Create a new environment: conda create --name my_env python=3.8
Activate the environment: conda activate my_env
Install packages: conda install <package_name>

🚀 Mamba #

Mamba is a fast, drop-in replacement for Conda, designed to resolve and install packages more quickly. It uses the same Conda repositories and environment files, making it easy to switch between the two.

Setup Instructions

Install Mambaforge with Mamba pre-installed:

Download and install Mambaforge with Mamba for your platform: https://github.com/conda-forge/miniforge#mambaforge

Alternatively, you can install Mamba within an existing Conda environment:
```
conda install -c conda-forge mamba
```
Create a new environment with Mamba:
```
mamba create --name my_env python=3.8
```
Activate the environment:
```
mamba activate my_env
```

Install packages using Mamba:

mamba install <package_name> -c conda-forge

By using Mamba instead of Conda, you can significantly speed up environment resolution and package installation while still benefiting from the Conda ecosystem.

🌌 Micromamba #

Micromamba is a lightweight and fast alternative to the Mamba package manager. It is a statically linked C++ executable that doesn’t require a base environment or come with a default Python version. It supports a subset of Mamba and Conda commands and is designed to resolve and install packages quickly. For those who need Conda features, you can install Conda using Micromamba with micromamba install conda.

Setup Instructions

Download the appropriate Micromamba binary for your platform from the official repository.
Make the binary executable and move it to a directory in your PATH:
```
chmod +x micromamba mv micromamba /usr/local/bin/
```

Create a new environment with Micromamba:

micromamba create -n my_env python -c conda-forge

Activate the environment:
```
micromamba activate my_env
```

Install packages using Micromamba:

micromamba install <package_name> -c conda-forge

If you need Conda’s features, you can install Conda within your Micromamba environment:
```
micromamba install conda
```

For example, if you need nb_conda_kernels to use your Micromamba environment in JupyterLab:

micromamba install conda nb_conda_kernels

And you are set up!

Micromamba is the perfect solution for those who want a fast and lightweight package manager while still being able to access Conda’s features when needed.

🐳 Docker #

A containerization platform that allows you to create lightweight, portable environments called containers. With Docker, you can package your Python application along with its dependencies, ensuring consistent execution across different systems.

Setup Instructions

Install Docker: https://docs.docker.com/get-docker/

Create a Dockerfile in your project directory with the following contents:

FROM python:3.11
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "your_script.py"]

Build the Docker image: docker build -t my_project .
Run the Docker container: docker run my_project

🏞️ Venv #

A built-in Python module available since Python 3.3, which allows you to create lightweight virtual environments. Venv simplifies the process of creating isolated environments for different projects, ensuring that package dependencies don’t conflict with each other.

Setup Instructions

Create a new virtual environment: python3 -m venv my_env
Activate the environment:
- macOS/Linux: source my_env/bin/activate
- Windows: my_envScriptsactivate
Install packages: pip install <package_name>

🌐 Virtualenv #

Virtualenv is a third-party Python environment management tool that predates the built-in venv module. It allows you to create isolated environments for different projects, ensuring that package dependencies don’t conflict with each other. Virtualenv is compatible with Python 2.7 and later versions, whereas venv is only available since Python 3.3.

Setup Instructions

Install virtualenv: pip install virtualenv
Create a new virtual environment: virtualenv my_env
Activate the environment:
- macOS/Linux: source my_env/bin/activate
- Windows: my_env\Scripts\activate
Install packages: pip install <package_name>

🍺 Homebrew #

Homebrew is a popular package manager for macOS and Linux that simplifies the installation and management of software. Though it is not a dedicated Python environment management tool, it does offer support for installing and managing multiple Python versions alongside other software.

Homebrew can be useful for installing and managing Python versions for system-level usage, but it’s not the best choice for managing isolated project-specific dependencies. For project-specific dependency management, it’s recommended to use other tools like venv, virtualenv, or Conda in conjunction with Homebrew-installed Python.

Setup Instructions

Install Homebrew (if not already installed):
- macOS: Follow the instructions at https://brew.sh
- Linux: Follow the instructions at https://docs.brew.sh/Homebrew-on-Linux
Install a specific Python version:
```
brew install python@3.11
```
Add the installed Python version to your PATH by adding the following line to your shell configuration file (e.g., ~/.bashrc, ~/.zshrc):
```
export PATH="/usr/local/opt/python@3.11/bin:$PATH"
```
Restart your shell or run source ~/.bashrc (or source ~/.zshrc) to apply the changes.
Verify the Python version: python3.11 --version

Keep in mind that Homebrew is mainly for installing and managing software at the system level. To manage project-specific dependencies and create isolated environments, combine Homebrew with other tools like venv or virtualenv for better dependency management.

💻 System package managers #

Some operating systems have their own package managers (e.g., apt for Ubuntu, pacman for Arch Linux, or Homebrew for macOS) that can be used to install and manage Python environments. However, using system package managers may lead to conflicts with other system packages and is generally not recommended for managing project-specific dependencies.

I will skip the setup instructions for this one, as it is just not recommended to do this!

🌟 Pixi #

Pixi is a new package manager built in Rust that uses Conda-forge packages but aims to be significantly faster and simpler than Conda. It’s developed by prefix.dev (the team behind Mamba) and focuses on providing a more streamlined experience. It is inspired by tools like Cargo, nix, and npm and aims to provide a modern package management experience for Python developers.

Setup Instructions

Install pixi following instructions at prefix.dev
Initialize a new project:
```
pixi init my-project
cd my-project
```
Add dependencies:
```
pixi add python numpy pandas
```
Run Python in the environment:
```
pixi run python
```

Create and run tasks (defined in pixi.toml):

# Add a task
pixi task add hello "python hello_world.py"

# Run the task
pixi run hello

Start an interactive shell in the environment:
```
pixi shell
```

Pixi can also be used as a global installation tool:

# Install tools globally
pixi global install starship
pixi global install fish

# Create a named environment with multiple packages
pixi global install --environment data-science-env \
 --expose python --expose jupyter \
 python jupyter numpy pandas

Key features:

Fast package resolution and installation
Uses Conda-forge packages
Built-in task runner
Global package installation support
GitHub Actions integration with automatic caching
Simple and intuitive CLI

⚡ uv #

uv is an extremely fast Python package installer and resolver written in Rust. It can be used as a drop-in replacement for pip and can generate virtual environments. While it is quite new it is already being used by many developers due to its speed and ease of use. Besides being fast, it is awesome because it is a single binary that can be used to also bootstrap Python itself.

Setup Instructions

Install uv:

curl -LsSf https://astral.sh/uv/install.sh | sh

Create a new virtual environment:
```
uv venv
```
Activate the environment:
- Unix/macOS: source .venv/bin/activate
- Windows: .venv\Scripts\activate
Install packages (much faster than pip):
```
uv pip install numpy pandas
```

It will automatically maintain a uv.lock file to ensure deterministic builds. Install this lock file with uv sync.

Key features of uv:

Up to 10x faster than pip
Built-in virtual environment support
Compatible with pip’s CLI interface
Global package caching
Deterministic builds

There is much more to uv, so check out the documentation for more details.

3. Conclusion 🏁 #

The choice you make for managing your Python environments ultimately depends on your use case. If you don’t have to frequently deal with non-Python dependencies, I would recommend using uv for its speed and simplicity. However, because I often work with projects that require non-Python dependencies, I’ve found that Micromamba is the perfect fit for me.

That being said, each tool has its own strengths and weaknesses, and it’s important to choose the one that works best for your specific needs. I hope this journey through Python environment management has been helpful and informative. Good luck in finding your perfect Python environment workflow! 🐍

4. Further Reading 📚 #

Here are some resources for further exploration:

Pyenv: Pyenv GitHub repository and documentation
Pipenv: Pipenv documentation
Conda: Conda documentation
Mamba: Mamba documentation
Micromamba: Micromamba documentation
venv: venv documentation
virtualenv: virtualenv documentation
Anaconda: Anaconda homepage
Miniconda: Miniconda documentation
Homebrew: Homebrew homepage
Pixi: Pixi documentation
uv: uv documentation

Home Assistant Ecosystem 🏠

Tue, 24 Dec 2024 00:00:00 +0000

(See my intro post)

There are not enough days in this advent calendar to highlight all my projects. So today is a collection of IoT/Home Assistant-related packages. Each has its own story.

📖 Origin Stories & Technical Details #

My Home Assistant obsession began as described in Day 10. Many projects emerged from this hobby:

aiokef: To control my KEF speakers, I reverse-engineered network traffic and decoded raw byte sequences, diving deep into proprietary protocols to “speak” the speaker’s language.
miflora: When @ChristianKuehnel couldn’t maintain this plant sensor library anymore, he transferred it to me. A perfect example of open source collaboration, it continues to help users monitor plant health and demonstrates how projects can thrive through community handovers.
addon-otmonitor: This HA add-on packages OpenTherm Monitor for the OpenTherm Gateway (OTGW). Using Docker to simplify deployment, it’s become a powerful open-source alternative to Nest or Ecobee, giving full control over heating systems.
home-assistant-macbook-touch-bar: Remember that useless touch bar on the old MacBook? I transformed it into a mini Home Assistant control panel for lights, temperature, and more - a classic case of hardware hacking giving new life to abandoned technology.
python-kasa: To modernize the popular pyHS100 library for TP-Link devices, I converted it to use asyncio. With @rytilahti, we created the python-kasa organization through community collaboration. While I haven’t contributed much since its inception, it’s now the standard for controlling Kasa devices.

📊 Impact & Lessons #

These projects serve thousands of users while teaching valuable lessons:

Open source empowers us: from KEF speaker integration to plant monitoring
Reverse engineering unlocks potential: turning proprietary protocols into open solutions
Community drives success: projects like miflora and python-kasa thrive through collaboration
Innovation through repurposing: giving new life to hardware like the MacBook Touch Bar
Modern tech matters: Docker simplifies deployment, async code boosts performance

Want to explore these projects? Check them out on GitHub:

This concludes our Advent of Open Source! I hope you’ve enjoyed this journey through my projects as much as I’ve enjoyed sharing them. Happy holidays and happy coding!

#OpenSource #HomeAssistant #IoT #Python #SmartHome

tuitorial

Tue, 24 Dec 2024 00:00:00 +0000

pfapack 🔢

Mon, 23 Dec 2024 00:00:00 +0000

(See my intro post)

Ever needed to calculate the square root of the determinant of a skew-symmetric matrix? Probably not, but if you’re in quantum physics, you might have! This is called the Pfaffian. Today’s project, pfapack, is a Python package that provides efficient routines for computing it. The underlying algorithms and original code implementations were developed by my PhD co-promotor Michael Wimmer. I created the pfapack package to make this powerful tool easily accessible.

📖 Origin Story #

Working in quantum physics, I frequently need to compute the Pfaffian. Michael had not only developed the algorithms but also wrote highly optimized C, FORTRAN, and Python implementations, distributing them as a tar file on his website. To make it easier for researchers working in Python, I packaged his existing Python code, added wrappers for his C and FORTRAN implementations, and set up automated builds for PyPI and conda-forge to make it easily installable.

🔧 Technical Highlights #

Python Wrapper for Efficient Code: User-friendly Python interface to Wimmer’s optimized C and FORTRAN.
Multiple Algorithms: Supports both the Parlett-Reid and Householder methods.
Optimized for Performance: Leverages low-level optimizations for speed.
Conda and Pip Installable: Easy installation via conda-forge and PyPI.
Automated Wheel Builds: Uses pypa/cibuildwheel for streamlined, automated builds and releases.
Cross-Platform Compatibility: Works on Linux, macOS, and Windows (with some caveats).

📊 Impact #

Makes specialized mathematical algorithms readily available to Python users.
Simplifies the use of highly optimized code.
15 GitHub stars.
Used in research projects that require efficient Pfaffian computations, particularly in quantum physics.

🎯 Challenges and Solutions #

Bridging Languages: Wrapping C and FORTRAN for Python integration.
Automated Builds: Setting up automated builds for multiple platforms was complex.
Windows Compatibility: As always Windows support is a PITA; it currently requires MSYS2 and MinGW64.

💡 Lessons Learned #

Making specialized tools accessible broadens their impact.
Wrapping low-level code in Python can improve usability.
Automated builds are essential for maintainability.
pypa/cibuildwheel is awesome for cross-platform builds.

🙏 Credits #

All the credits for the underlying algorithms and original code implementations go to Michael Wimmer. I simply made it easier to distribute and use in Python.

Want to compute Pfaffians efficiently in Python? Check out pfapack on GitHub or install it via pip install pfapack or conda install -c conda-forge pfapack!

#OpenSource #Python #ScientificComputing #Math #Pfaffian #FORTRAN #C

tiller-streamlit 💰

Sun, 22 Dec 2024 00:00:00 +0000

(See my intro post)

Tracking personal finances is crucial, especially when working towards financial independence (see my WenFIRE post on Day 4). Today’s project is tiller-streamlit, a Streamlit app that visualizes your financial data from Tiller. I host it on my local network under money.local using Traefik for easy access.

📖 Origin Story #

Used to use Intuit Mint, but when they shut down, I needed an alternative with full data control.
Tiller automatically populates your financial data into a Google Sheet that you own forever.
With 20+ accounts, manual updates are tedious – Tiller automates it.
Combined Tiller’s data with Streamlit’s ease of use, creating a personal finance dashboard.

🔧 Technical Highlights #

Powered by Streamlit: Turns Python into a web app with minimal code.
Tiller Integration: Connects to Tiller’s Google Sheets for automated data retrieval.
Interactive Visualizations: Uses Plotly and Altair for dynamic charts.
Customizable Analysis: Filter by categories, time periods, and more.
Net Worth Tracking: Calculates and visualizes net worth over time.
Dockerized Deployment: Easy deployment with Docker, and served locally with Traefik.

📊 Impact #

Primarily for personal use, it has significantly improved my understanding of my finances.
Great way to learn Streamlit and explore its capabilities.
Visualizing spending patterns helps identify areas to cut back and make informed financial decisions on the path to financial independence.

🎯 Challenges and Solutions #

Learning Streamlit: Excellent documentation and community support made it a smooth process.
Data Wrangling: Transforming Tiller’s raw data into a format suitable for visualization.
Deployment: Docker and Traefik simplified local deployment, making it accessible on my home network.

💡 Lessons Learned #

Data Ownership Matters: Owning your financial data gives you control and flexibility.
Visualization is Key: Seeing your finances visually makes a huge difference.
Streamlit is Powerful: Simplifies web app development.
Automation is Essential: Tiller’s automatic data population makes the whole process sustainable.
Docker and Traefik Simplify Local Deployment: Making it easy to access the app on my home network.

🔮 Future Plans #

Budgeting Features: Integrate budget tracking and forecasting.
Goal Setting: Set financial goals and track progress.
Investment Tracking: Incorporate investment data for a complete picture.

Want to visualize your own finances with Tiller and Streamlit? Check out tiller-streamlit on GitHub!

#OpenSource #Python #Streamlit #PersonalFinance #DataVisualization #Tiller #Traefik #Docker

Interactive Graph Visualization 🕸️

Sat, 21 Dec 2024 00:00:00 +0000

(See my intro post)

After creating pipefunc (Day 20), I wanted a way to visualize its (possibly large and complex) computational graphs interactively in Jupyter notebooks. What started as a simple visualization PR evolved into two standalone packages: graphviz-anywidget (Python) and graphvizsvg (JavaScript) - which unexpectedly introduced me to JavaScript development!

📖 Origin Story #

While working on pipefunc’s visualization features, I discovered that existing solutions didn’t quite meet my needs. However, I found several components that would solve my issue:

AnyWidget (interactive GraphViz in Jupyter)
@hpcc-js/wasm-graphviz (GraphViz in WASM)
d3-graphviz (interactive GraphViz in D3)
jquery.graphviz.svg (GraphViz SVG manipulation)

AnyWidget would work with ESM modules, however, I had no clue what that even was. The jquery.graphviz.svg package wasn’t one and hadn’t been maintained in 8 years.

With AI as my JavaScript mentor, I first rewrote jquery.graphviz.svg into a modern ESM module (graphvizsvg), adding comprehensive tests that uncovered bugs in the original. Then, using AnyWidget, I created graphviz-anywidget to connect everything together - combining the WASM-powered d3-graphviz with the interactive features from graphvizsvg into a seamless Python package.

To my surprise, I found JavaScript’s development tooling remarkably sophisticated - especially npm, which handles everything from linting to testing to bundling to publishing. Something we’re still missing in Python!

🔧 Technical Highlights #

graphviz-anywidget:

Interactive Graphviz visualization in Jupyter
WASM-powered graph rendering
Directional graph traversal
Node and edge highlighting
Works in JupyterLab, Notebook, and VS Code

graphvizsvg:

Modern ESM module for interactive SVG visualization
Automatic color transitions
Bi-directional graph traversal
Comprehensive test coverage
My first npm package!

📊 Impact #

Enhances pipefunc’s visualization capabilities
Modernizes legacy jQuery code for modern web development
Proves that diving into new technologies can yield useful tools

🎯 Challenges and Solutions #

Learning JavaScript/TypeScript from scratch
Making WASM work in Jupyter
Getting it to work in VS Code

💡 Lessons Learned #

Don’t let preconceptions limit your tech choices
Modern JavaScript development is actually enjoyable
AI can be an excellent programming mentor
Good visualization makes complex data accessible
Breaking features into separate packages increases reusability
Test coverage helps find bugs in even well-used code

Want to visualize graphs interactively? Check out graphviz-anywidget and graphvizsvg!

#OpenSource #JavaScript #Python #DataVisualization #WebDev

pipefunc 🕸️

Fri, 20 Dec 2024 00:00:00 +0000

(See my intro post)

Of all the projects I’m sharing this month, this one that started as a passion project excites me the most!

📖 Origin Story #

Sometimes the best projects are born from the most unexpected moments. More than 1½ year ago, while on parental leave with my twin boys, I found myself with small pockets of time during naps to work on a problem that had been bothering me for years: the tedious bookkeeping required in complex computational workflows. Every scientific computation project seemed to reinvent the same patterns - managing function dependencies, parameter sweeps, result caching, parallelization, and a lot of boilerplate to combine the resulting data. I wanted something that would let scientists focus on their science, not on pipeline management.

Example of a pipeline generated by PipeFunc

🔧 Technical Highlights #

Automatic DAG construction via very simple and lightweight syntax
N-dimensional parameter sweeps with automatic parallelization
Visual pipeline representation using NetworkX
Resource profiling (CPU, memory, time)
Type validation between pipeline stages
Ultra-fast: only 15 µs overhead per function
Flexible caching strategies (memory, disk, cloud)
Integration with scientific computing tools:
- NetworkX for graph algorithms
- NumPy for multi-dimensional arrays
- Xarray for labeled arrays
- Zarr for distributed storage
- Adaptive for parallel computing

📊 Impact #

230 GitHub stars
700+ tests with 100% coverage
Fully typed codebase
Comprehensive documentation
Tested on real workflow on SLURM cluster
Covered on Pycoder’s Weekly with >100k subscribers

🎯 Challenges and Solutions #

Balancing simplicity with power
Making complex workflows intuitive
Handling distributed computing edge cases
Ensuring type safety across the pipeline
Optimizing performance without sacrificing features

💡 Lessons Learned #

Sometimes the best time to code is during baby naps
Complex problems can have elegant solutions
Scientific computing needs better tooling
Good abstractions make hard things easy
Type hints and tests prevent headaches

🔮 Future Plans #

The journey is far from over. Plans include:

Enhanced cloud computing support
More interactive visualization options
Interactive pipeline debugging tools
Expanded parameter sweep capabilities

Want to simplify your computational workflows? Check out pipefunc on GitHub or read the documentation!

#OpenSource #Python #DataScience #ScientificComputing #Programming

fileup 📤

Thu, 19 Dec 2024 00:00:00 +0000

(See my intro post)

Sometimes the best tools are the ones you build for yourself and keep using for years. Today’s project is fileup, a tiny command-line utility that’s been making my life easier for 7 years - because sharing files shouldn’t require leaving the terminal!

📖 Origin Story #

Back in 2017, I found myself repeatedly going through the same tedious process: needing to share a file, opening a browser, navigating to a file sharing service, waiting for uploads, and copying URLs. As someone who lives in the terminal, this felt inefficient. I wanted something as simple as fu filename that would handle everything and put the URL in my clipboard. Seven years and countless file shares later, it’s still one of my most-used tools.

🔧 Technical Highlights #

Supports both FTP and SCP uploads
Smart URL generation:
- Jupyter notebooks → nbviewer links
- Images → markdown-formatted links
- Regular files → direct URLs
Automatic file expiration and cleanup
Zero dependencies - uses only Python standard library
Clipboard integration (macOS)
Simple INI configuration
Works as both a standalone script or installed package

📊 Impact #

This is one of those tools where GitHub stars (currently 0!) don’t tell the whole story:

Used almost daily for 7 years
Shared thousands of files
Saved countless minutes of browser interactions
Proves that utility trumps popularity in open source

💡 Lessons Learned #

Longevity comes from solving a real, persistent need
A tool doesn’t need to be revolutionary to be useful
Zero dependencies mean zero maintenance headaches
Sometimes the impact of a tool isn’t measured in stars
Simple tools that remove friction are worth maintaining

Want to share files from your terminal? Check out fileup on GitHub!

#OpenSource #Python #DevTools #CommandLine #Programming

Psychedelic Data Science 🍄

Wed, 18 Dec 2024 00:00:00 +0000

(See my intro post)

📝 Note: Running out of awesome projects and I want to save the best for last, so today just something fun.

While preparing for this advent calendar, I browsed through my 382 GitHub repositories and rediscovered this time capsule from 8 years ago. It explores the language of psychedelic experiences through data science - with an interesting hypothesis: do psychedelic experiences generate richer vocabulary compared to other substance reports?

📖 Origin Story #

Back in 2015, when I was still “young” (before back pain and two-day hangovers became a thing), I was learning data science and natural language processing. I had an amusing hypothesis: surely people describing their psychedelic experiences would use richer vocabulary than those writing about stimulants - I mean, who hasn’t pondered the linguistic complexity of “everything is connected” versus “I cleaned my entire apartment at 4 AM”? Erowid.org, with its thousands of detailed first-person narratives across different substances, provided the perfect dataset to test this theory.

🔧 Technical Highlights #

Natural language processing of experience reports
Vocabulary richness analysis across substance categories
TF-IDF vectorization for text analysis
Support Vector Machine classification of experience types
Web scraping with BeautifulSoup
K-means clustering for discovering common themes
Word cloud generation to visualize vocabulary differences

📊 Impact #

A learning project that helped understand:
- How different experiences shape language use
- Processing subjective experience narratives
- Document classification techniques
- The challenges of quantifying vocabulary richness
10 GitHub stars

🎯 Challenges and Solutions #

Implementing respectful web scraping
Controlling for report length and education level
Creating meaningful metrics for vocabulary richness
Visualizing linguistic patterns across categories

💡 Lessons Learned #

Putting things on GitHub means it’s less likely forgotten things get lost
Early data science projects often reveal our initial fascinations
Web scraping requires both technical skill and ethical consideration
Text analysis tools have evolved dramatically since 2015
Sometimes the most interesting projects are the ones you almost forgot about

Want to explore this intersection of psychedelics, language, and data science? Check out the project on GitHub!

#OpenSource #Python #DataScience #NLP #MachineLearning

Adaptive Scheduler 🚀

Tue, 17 Dec 2024 00:00:00 +0000

(See my intro post)

Ever tried to efficiently run 100,000 cores from a Jupyter notebook? No? Well, I have, and it led to some interesting discoveries about the limits of parallel computing tools.

📖 Origin Story #

During my 2018 internship at Microsoft Quantum, we faced a unique challenge: our adaptive sampling algorithms needed to run on massive computing clusters (50,000+ cores), but existing tools couldn’t handle it efficiently.

The problem was interesting: traditional parallel computing tools like Dask rely on a central scheduler process. Imagine 100,000 cores, each finishing a task every 10 seconds. That means every 10 microseconds, a core needs new work! With typical scheduler overhead being 1-50 milliseconds, workers would spend 99% of their time waiting rather than computing.

Usually, you’d solve this by batching work ahead of time. But with adaptive sampling (yesterday’s project), that’s impossible - you need the results of previous calculations to know what to compute next. It was like trying to conduct an orchestra where each musician needs personal instructions every few seconds!

🔧 Technical Highlights #

We created a solution that:

Acts as a meta-scheduler, launching multiple sub-schedulers (Dask, ipyparallel, mpi4py, etc.)
Minimizes communication between scheduling layers
Handles job management through a simple database
Supports automatic fault tolerance and data saving
Works with any cluster scheduler (SLURM, PBS)
Provides real-time progress tracking in Jupyter notebooks
Scales to 50,000+ cores without breaking a sweat

📊 Impact #

Used extensively for real quantum device simulations
26 GitHub stars (but don’t let that fool you - it’s highly specialized!)
Enables interactive supercomputing from a simple Jupyter notebook
Likely ran for many millions of core-hours across various projects
Actively maintained with 61 releases since 2018, proving its ongoing value

🎯 Challenges We Solved #

Scheduler overhead killing performance at scale
Data locality and fault tolerance
Real-time monitoring of massive parallel jobs
Making supercomputing accessible through notebooks
Automatic job recovery after crashes (e.g., when using spot VMs or Heisenbugs)

💡 Lessons Learned #

Sometimes you need to think outside the box
The best tools hide complexity behind simple interfaces
Real-time feedback is crucial, even for massive computations
When scaling up 1000x, you often need a completely different approach
Making things “just work” from a notebook is worth the effort

Want to run massive parallel computations efficiently? Check out Adaptive Scheduler on GitHub!

#OpenSource #Python #HPC #QuantumComputing #ParallelComputing

Adaptive 📈

Mon, 16 Dec 2024 00:00:00 +0000

(See my intro post)

During my PhD, I was running quantum transport simulations on our 800-core computational cluster. Traditional parameter sweeps using dense grids were hitting the cluster’s limits - wasting precious compute time on uninteresting regions while potentially missing important features. Most parameter spaces we explored were “boring” with occasional interesting regions that required higher sampling density.

📖 Origin Story #

Initially, in 2016, I had a collection of one-off adaptive sampling scripts. Together with Joseph Weston and the support of my PhD advisor Anton Akhmerov, we turned it into a full-fledged Python package “Adaptive”. What began as an internal tool to make our cluster usage more efficient slowly gained traction in the wider scientific computing community.

🔧 Technical Highlights #

Multiple adaptive sampling strategies (1D, 2D, ND, averaging stochastic functions)
Parallel execution support (local, MPI clusters, distributed systems)
Live plotting in Jupyter notebooks with real-time feedback
Customizable loss functions for different sampling priorities
Minimal dependencies

📊 Impact & Applications #

1164 GitHub stars and thousands of daily downloads
Cited in 20+ scientific publications
Dependency of many packages, most notably: Orange Quantum System’s Quantify software for controlling real quantum computers!

Used in cutting-edge research across fields:

Quantum Computing: Simulating and controlling superconducting qubits and trapped ion systems
Quantum Physics: Exploring topological quantum systems and novel materials
Materials Science: Studying graphene and other 2D materials
Engineering:
- Optimizing radar system performance
- Fluid dynamics simulations
Chemistry: Molecular modeling and vibrational spectroscopy
Most Unexpected Use: Found in a paper on predicting energy usage of wartime bungalows!

This range of applications shows how a tool built for quantum transport simulations can find surprising uses across disciplines, from fundamental physics to everyday buildings.

🎯 Challenges and Solutions #

Making parallel execution truly scalable while maintaining sampling intelligence
Creating intuitive visualization tools for complex parameter spaces
Handling stochastic functions and noisy data

💡 Lessons Learned #

Building a user base takes active effort - giving talks and engaging online made all the difference
Good documentation converts interested users into regular contributors
This was my first open source package and learned all about packaging, testing, and CI/CD

Want smarter parameter space exploration? Check out Adaptive on GitHub or try our interactive tutorial!

#OpenSource #Python #ScientificComputing #ParallelComputing #Programming

clip-files 📋

Sun, 15 Dec 2024 00:00:00 +0000

(See my intro post)

Sometimes the most useful tools are born from daily frustrations. When working with AI assistants like ChatGPT or Claude, I found myself constantly copying files to share context. After doing this manually dozens of times, I created a tool to automate it.

📖 Origin Story #

While contributing to open source, I frequently like to share entire (or large parts of) codebases with AI assistants for context or code review. Since the code is open source anyway, there’s no reason not to share the full context! But manually copying files, adding their paths for context, and ensuring I didn’t exceed token limits was tedious. Instead of continuing this manual process, I spent an evening creating clip-files - and have used it multiple times daily ever since.

In fact, I’m using it right now! The command (shown below) lets me share this entire advent calendar series with AI assistants for proofreading.

clip-files --initial-file ai.md --files introduction.md day*.md

🔧 Technical Highlights #

Uses tiktoken for accurate GPT token counting
Supports both directory traversal and specific file selection
Automatically generates a file index for reference
Allows custom initial instructions via file
Preserves file paths in output for context
Clean, typed, and documented codebase

📊 Impact #

While this tool has no GitHub stars (I haven’t promoted it), its real impact is personal - I use it multiple times daily for:

Sharing entire open source codebases with AI assistants
Getting comprehensive code reviews
Discussing bug fixes with full context
Documentation improvements
Open source contributions

The tool’s real value isn’t in its complexity (it’s just ~250 lines of code) but in how it removes friction from open source development. When working with open code, we might as well share all the context we can - and this tool makes that effortless.

🎯 Challenges and Solutions #

Maintaining consistent formatting across different input methods
Calculating accurate token counts for different components
Providing meaningful error messages for invalid inputs
Structuring output to be AI-assistant friendly

💡 Lessons Learned #

When working with open source code, sharing full context is valuable
Simple command-line tools can significantly improve daily workflows
Consistent formatting matters when working with AI assistants
Sometimes a 250-line tool can save hours of manual work

Want to make sharing code with AI assistants easier? Check out clip-files on GitHub!

#OpenSource #Python #DevTools #AI #Programming

PhD Thesis Cover Design 🎨

Sat, 14 Dec 2024 00:00:00 +0000

(See my intro post)

When most people think of open source, they think of software. But what about open-sourcing your PhD thesis and creating a generative art system for its cover?

📖 Origin Story #

With my PhD defense approaching and needing around 100 copies of my thesis, I faced a design challenge. As someone more comfortable with Python than Photoshop, I turned what started as a simple visualization idea into a full-blown generative art project with machine learning-based clustering.

Multiple printed copies of the thesis

🔧 Technical Highlights #

Generated 3,671 unique covers from quantum physics simulation data
Used VGG16 neural network to extract features from covers
Implemented K-means clustering to group similar designs
Created interactive IPywidgets for design selection
Built a Binder app for real-time color scheme adjustment
Added unique edition numbers (1/120, 2/120, etc.)
Made both thesis and cover design code fully open source

The project combines:

Adaptive sampling for physics simulations
Neural networks for image feature extraction
Interactive visualization tools
Parallel processing for batch generation
Version control for scientific writing

📊 Impact #

Thesis repository: 14 GitHub stars
Cover design repository: 36 GitHub stars
Real-world adoption: Dr. Tomas Pippia from TU Delft used the code to create his own thesis cover
~100 unique printed copies, each with its own generated cover and edition number

🎯 Challenges and Solutions #

Managing 3,671 unique PDF files and their conversions
Creating a clustering system for visually similar designs
Building interactive tools for non-technical collaborators
Ensuring consistent print quality across designs
Balancing artistic appeal with scientific relevance

💡 Lessons Learned #

Open source isn’t just about code - it’s about sharing knowledge
Python can be a powerful tool for creative projects
Machine learning can help solve unexpected problems
Good visualization requires both technical skill and artistic sense
Interactive tools make collaboration with non-programmers easier

📚 Fun Fact #

Due to COVID restrictions, I defended my thesis without any guests present. The silver lining? I still have about 100 beautifully unique copies at home, each with a different generated cover and edition number. If you’d like a physical copy of this piece of quantum physics meets generative art, just let me know!

Want to create your own thesis cover or see how I wrote mine? Check out the thesis cover project and thesis source!

#OpenSource #PhD #Python #MachineLearning #DataVisualization

UniDep 🧬

Fri, 13 Dec 2024 00:00:00 +0000

(See my intro post)

Ever tried to set up a Python project that needs both pip and conda packages? Then you know the pain of maintaining multiple requirement files. UniDep was born from years of frustration with this exact problem.

📖 Origin Story #

While working on large monorepos with hundreds of dependencies, I kept running into the same issue: maintaining separate files for conda (environment.yml), pip (requirements.txt), and Python packaging (pyproject.toml). Coming from languages like Rust, where package management is a joy, Python’s fragmented ecosystem can feel overwhelming, especially to newcomers (obligatory xkcd.com/1987).

What started as a simple tool to avoid duplicate dependency definitions turned into something much more powerful. In enterprise environments, it’s common to find monorepos with over 1000 dependencies spread across dozens of packages. Setting up these environments traditionally involves following lengthy setup guides, installing packages in the correct order, and hoping nothing breaks.

Then came the “aha” moment - what if we could install everything with a single command? After implementing UniDep, those complex setup guides became a one-liner: unidep install-all -e. It installs all dependencies and sets up every package in the monorepo in editable mode. What used to take hours now takes minutes.

The difficulty of managing pip and conda

🔧 Technical Highlights #

Single source of truth for all dependencies
Supports both requirements.yaml and pyproject.toml
Handles platform-specific dependencies
Generates conda-lock files both for entire monorepos and fully consistent lock files for its individual packages
Works with pip, conda, and mamba
Monorepo-friendly with local dependency support
Integrates with setuptools and hatchling
≥99% test coverage, fully-typed

📊 Impact #

221 GitHub stars
Featured on Python Bytes podcast
Saves countless hours of dependency management
Makes complex project setups accessible
Actively maintained with 70 releases

🎯 Challenges and Solutions #

Different package naming between pip and conda
Platform-specific dependencies
Cross-platform compatibility (Linux, Windows, MacOS)
Supporting pip-only workflows

💡 Lessons Learned #

Good developer experience is worth the effort
Sometimes you need to build a bridge between existing tools
The best tools make hard things easy
Dependency resolution is hard but can be simple for users

🔮 Future Plans #

Adding Pixi support - allowing seamless migration to Pixi without changing config files
Eventually, Pixi might make UniDep obsolete (and that’s okay!)
For now, UniDep bridges the gap between pip and conda, making life easier for developers

Want simpler dependency management? Check out UniDep on GitHub!

#OpenSource #Python #DevTools #Programming #PackageManagement #Conda

instacron 📸

Thu, 12 Dec 2024 00:00:00 +0000

(See my intro post)

Ever found yourself with beautiful photos but too lazy to post them? Instead of changing my habits, I did what any developer would do: spent way more time automating the process than it would have taken to post manually.

📖 Origin Story #

Back in 2017, I had hundreds of landscape photos from traveling through South America that I wanted to share on Instagram. The solution? Write a Python script that would randomly select photos and post them automatically, complete with philosophical quotes randomly selected from Bukowski, Einstein, or Dostoevsky because… it seemed to be the default of all “influencers” at the time. Nothing says “authentic travel experience” like an auto-generated Bukowski quote about drinking next to your serene mountain photo!

Instacron Example Post

🔧 Technical Highlights #

Randomly selects photos from a directory
Adds location data from EXIF metadata
Generates captions with random philosophical quotes
Adds two random emojis to match the “mood”
Automatically generates relevant hashtags
Tracks posting frequency
Uses cron for scheduled posting

📊 Impact #

21 GitHub stars
4143 Instagram followers
Successfully posted hundreds of photos
Generated some concerned DMs (more on that below!)

🎯 Challenges and Solutions #

Instagram’s ever-changing API
Handling rate limits and restrictions
Ensuring photos weren’t posted multiple times
Making captions interesting without human input
The delicate art of random emoji selection

😅 Funny Story #

The random quote and emoji selector once created an unexpectedly dramatic post. Picture this: a beautiful landscape photo paired with a deeply philosophical quote about solitude, and topped with a broken heart emoji (💔). Within hours, I started receiving concerned DMs from friends asking if everything was okay and whether my relationship had ended!

I had to explain that no, I wasn’t going through a breakup - my bot just had a flair for the dramatic! These occasional mismatches made the whole thing more entertaining.

💡 Lessons Learned #

Sometimes the best projects come from pure laziness
Automation doesn’t always save time, but it’s fun
Never underestimate the power of a misplaced emoji to trigger concerned messages
Some followers actually care about you (even if they’re worried about your bot-generated content)

While I haven’t used this project in 6 years (Instagram’s API changes eventually won), it remains a testament to the lengths we’ll go to avoid repetitive tasks. Check out instacron on GitHub!

#OpenSource #Python #Automation #Photography #Instagram

conda-forge Contributions 🐍

Wed, 11 Dec 2024 00:00:00 +0000

(See my intro post)

Unlike other projects in this advent calendar, I want to celebrate being part of something much bigger: the conda-forge community, where thousands of volunteers maintain the packages that power scientific Python.

📖 Origin Story #

I started with conda-forge in 2016 during my PhD work with Kwant, a quantum transport simulator. Installing it was a nightmare - it required system-level numerical libraries like MUMPS and Scotch which required a special config file to link. When I discovered conda-forge and its mission to make scientific software installation painless, I knew this was the solution!

The humble work of conda-forge maintainers

🔧 Technical Highlights #

Over the past 8 years, I’ve maintained over 40 build recipes, including:

Major scientific packages like VTK, MUMPS, and Scotch
Quantum physics tools like Qsim, Qcodes, and Kwant
Visualization libraries like HoloViews
Plus all the packages I’ve authored myself

The real magic of conda-forge is its infrastructure:

Automated builds across Linux, macOS, and Windows
Strict dependency version management
Comprehensive CI/CD pipelines
Community-driven quality control

📊 Impact #

What makes conda-forge truly special is how initial contributions blossom into collaborative efforts. I’ve experienced this firsthand: after creating the initial MUMPS feedstock, 25 other contributors joined in, each bringing their unique expertise.

A recent experience also illustrates this: I added CUDA support to both Google’s Qsim and IBM’s Cirq packages. Shortly after, Leo Fang, an engineer from NVIDIA who knows CUDA far better than I do, stepped up and made significant improvements. This is the beauty of open source - experts naturally gravitate to where they can make the biggest impact.

This collaborative approach solves a problem I’ve encountered at several companies: the tendency to maintain complicated build systems internally with custom hacks. Instead of each organization reinventing the wheel, conda-forge provides a shared platform where:

Maintenance burden is distributed
Expert knowledge flows freely
Build practices are standardized
Infrastructure is public and reusable

🎯 Challenges and Solutions #

Keeping up with upstream changes
Managing complex dependency trees
Cross-platform compatibility
Coordinating with upstream maintainers

💡 Lessons Learned #

Community effort beats solo work for infrastructure
Automation is crucial for reliability
Small contributions compound over time
Being part of something bigger is rewarding

Want to contribute to scientific Python? Check out conda-forge and the staged-recipes repository!

#OpenSource #Python #Scientific #Programming #Community

Home Assistant Config 🏠

Tue, 10 Dec 2024 00:00:00 +0000

(See my intro post)

Funny how things work out - initially, I didn’t care for home automation at all. But when in 2020 I discovered Home Assistant was written in Python, my favorite programming language, I suddenly found a new hobby. Over the years, I’ve built a comprehensive setup with over 137 automations (and counting!). These automations are triggered by everything from simple motion sensors to complex patterns of conditions and sensor states, creating a home that responds intelligently to our needs.

📖 Origin Story #

What started with basic light controls evolved into a complete home automation system. In the beginning, my wife was continuously annoyed by lights turning on at the wrong moment or not at all. However, as I debugged and refined the system, things started working exactly as expected, and she began to really appreciate it. When we moved from The Netherlands to the US in 2022 (and two more times since), we felt like we were back in the stone age having to manually control lights. The annoyance had shifted from buggy automations to not having Home Assistant fully installed yet!

Unfortunately, I’ve been less active lately with my Home Assistant setup because so many other projects draw my attention. One of these “projects” is my 1.5-year-old twin boys, who now have their own room with specialized automations, including 100% local cameras with ML pattern detection for monitoring without compromising privacy.

The evolution of home automation

🔧 Technical Highlights #

137 documented automations with auto-generated documentation
Presence detection in every room
Adaptive lighting throughout the house
Privacy-focused baby monitoring
Fully local processing
2,432 commits of continuous improvement

📊 Impact #

1,747 GitHub stars
185 forks
Used as reference by countless Home Assistant enthusiasts
Featured in Home Assistant community
Spawned several spin-off projects (like Adaptive Lighting from Day 2!)

💡 Lessons Learned #

Start small, expand gradually
Privacy and local processing matter
Automated documentation stays accurate
Home automation is never “finished”
Sometimes life (like twins!) brings new automation challenges

Want to explore my home automation setup? Check out the full config on GitHub!

#OpenSource #HomeAssistant #SmartHome #IoT #Python

yaml2bib 📚

Mon, 09 Dec 2024 00:00:00 +0000

(See my intro post)

Every academic writing in LaTeX knows the pain: managing BibTeX files with hundreds of citations, each potentially formatted slightly differently, with inconsistent journal abbreviations, and duplicated entries across chapters. While procrastinating on writing my PhD thesis, I became obsessed with solving this universal academic headache.

📖 Origin Story #

When writing a PhD thesis with multiple chapters that share citations, maintaining consistency is crucial but tedious. Different chapters might cite the same paper but with slightly different BibTeX entries - one using the full journal name, another using abbreviations, or worse, having slightly different titles or author lists. Rather than manually standardizing ~350 citations, I created yaml2bib: give it a list of DOIs in YAML format, and it generates a perfect BibTeX file with consistent journal abbreviations and formatting.

The pain of manual BibTeX management

🔧 Technical Highlights #

Fetches citation data from CrossRef using DOIs
Ensures consistent journal name abbreviations
Caches API responses to avoid repeated queries
Supports custom text replacements
Combines multiple YAML sources
Works as both CLI tool and Python library
Zero manual citation formatting needed

📊 Impact #

While this tool has only 11 GitHub stars, it saved me countless hours during thesis writing and helped ensure professional consistency in my citations. It’s particularly useful for:

Large academic documents with many citations
Multi-chapter works sharing references
Collaborative papers needing citation standardization
Anyone tired of manually formatting BibTeX entries

💡 Lessons Learned #

Procrastination can be productive if channeled right (story of my life…)
A single source of truth prevents inconsistencies
Good tools make you care about details you’d otherwise skip
Every LaTeX user has fought with BibTeX at some point

Want perfectly consistent citations? Check out yaml2bib on GitHub!

#OpenSource #Python #Academia #LaTeX #Research

Home Assistant Stream Deck YAML 🎮

Sun, 08 Dec 2024 00:00:00 +0000

(See my intro post)

What started as a personal hack to control my home (temperature, lights, TV, music) with a gifted Stream Deck evolved into something bigger. This project became my first real exploration of AI-assisted development, showing me how much faster I could work with AI as a tool. With 250+ stars and 9 contributors adding significant features, it’s grown beyond my initial vision.

📖 Origin Story #

While the Stream Deck’s official software is great for streamers, it doesn’t support Linux and its Home Assistant integration is not very programmable. After creating a simple solution using python-elgato-streamdeck, I realized others might be interested. Using AI as a development partner changed my workflow - it helped write tests, reviewed code, and generated use cases I hadn’t thought of. What really surprised me was asking AI to brainstorm example configurations - it generated over 30 creative use cases with complete code examples, which I then turned into unit tests. These tests now automatically generate the documentation using markdown-code-runner (from Day 5).

🔧 Technical Highlights #

YAML configuration with Jinja2 templating
Supports all Stream Deck models, including the Plus with dials
Cross-platform (Linux, macOS, Windows)
30+ example configurations (AI-generated, then human-refined)
Documentation auto-generated from code and unit tests
Available as a Home Assistant add-on
Community-contributed features like touchscreen support and dial controls

💡 Lessons Learned #

AI can significantly speed up development
Faster development enables broader project scope
Good examples make the best documentation
Tools building upon tools multiply productivity
Community contributions can take a project in exciting new directions

Want to control Home Assistant with a Stream Deck? Check out the project on GitHub!

#OpenSource #HomeAssistant #SmartHome #Python #AI

numthreads 🧵

Sat, 07 Dec 2024 00:00:00 +0000

(See my intro post)

Sometimes the smallest tools solve the most persistent problems. Today’s project is about taming automatic parallelization in scientific computing - a deceptively simple challenge that has cost countless CPU hours.

📖 Origin Story #

While working on high-performance computing clusters, I frequently encountered a counterintuitive problem: code running slower on multiple CPU cores than on a single one. Many scientific libraries (NumPy, SciPy) automatically parallelize operations, which sounds great but can actually harm performance when you’re already parallelizing at a higher level. After repeatedly explaining to colleagues why they needed to set various environment variables to disable this behavior, I created numthreads to solve it once and for all.

Overenthusiastic threading

🔧 Technical Highlights #

Controls thread count for major numerical libraries:
- OpenBLAS
- Intel’s Math Kernel Library (MKL)
- OpenMP
- NumExpr
- Accelerate
Zero dependencies
Tiny (≤7KB) yet solves a real problem
Both CLI and Python API

📊 Impact #

Over my scientific computing career, I’ve encountered this threading issue dozens of times. Since creating this package a year ago, I’ve helped at least 5 different researchers speed up their workflows, sometimes by orders of magnitude. The beauty lies in its simplicity - the fix is often as simple as:

eval $(numthreads 1)

💡 Lessons Learned #

The simplest solutions are often the best
More cores doesn’t always mean faster code
If you find yourself repeating the same instructions, write a tool
Even a 7KB package can save hours of computing time

Want better control over your numerical computations? Check out numthreads on GitHub!

#OpenSource #Python #HPC #ScientificComputing #Programming

calver-auto-release

Fri, 06 Dec 2024 00:00:00 +0000

Open Science Publications 🔬

Fri, 06 Dec 2024 00:00:00 +0000

(See my intro post)

Today’s post is about something I have strong opinions on: making scientific research reproducible. While many researchers talk about open science, actually making your work reproducible for anyone requires significant effort.

📖 Origin Story #

During my Ph.D. and subsequent research, I noticed a frustrating pattern in scientific publications: claims of “code available upon request” that often led nowhere, or code snippets that wouldn’t run without substantial modification. I made an effort that my publications would be different – each includes complete, runnable code that reproduces every figure and result.

🔧 Technical Highlights #

Multiple repositories covering quantum physics experiments and simulations:
- orbitalfield - “Orbital effect of magnetic field on the Majorana phase diagram” (94 citations)
- supercurrent-majorana-nanowire - “Supercurrent interference in few-mode nanowire Josephson junctions” (78 citations)
- zigzag-majoranas - “Enhanced proximity effect in zigzag-shaped Majorana Josephson junctions” (56 citations)
- azure-quantum-tgp - “Protocol to identify a topological superconducting phase in a three-terminal device” (52 citations)
- spin-orbit-nanowires - “Spin-orbit protection of induced superconductivity in Majorana nanowires” (81 citations)

🔄 Evolution of Best Practices #

What I consider best practices has evolved over time. Today, I would:

Use Pixi for universal lock files across operating systems and programming languages
Provide a minimal Docker container that just installs the pixi lock file
Create self-documenting Jupyter notebooks that reproduce every result
Include clear figure-to-code mapping for paper reproducibility

🎯 Challenges and Solutions #

Balancing code cleanliness with research deadlines
Managing large datasets efficiently
Ensuring long-term reproducibility
Making complex physics simulations accessible

💡 Lessons Learned #

Clean code takes time but saves more in the long run
Documentation is as crucial as the code itself
Lock files are essential for true reproducibility
Making code public improves its quality

Want to explore quantum physics simulations? Check out the repositories above, each linked to their corresponding papers with full reproduction instructions.

#OpenSource #OpenScience #Physics #QuantumComputing #Python

markdown-code-runner 📝

Thu, 05 Dec 2024 00:00:00 +0000

(See my intro post)

Documentation should be as close to the code as possible - that’s the principle of “information locality.” But how do you keep code examples and their outputs in sync? Enter markdown-code-runner, a tool that executes code blocks within Markdown files and updates their output in-place!

📖 Origin Story #

While maintaining various open source and work projects, I found myself constantly updating README or Wiki files with new command outputs, updated tables, and fresh examples. This manual process was error-prone and tedious. Instead of using external templating files or more advanced documentation generators like Sphinx and Quarto (which are often overkill for single-page projects), I wanted something simpler: code and its output living together in the same Markdown file, always in sync.

🔧 Technical Highlights #

Executes code blocks (Python and Bash) within Markdown files
Updates outputs in-place while preserving the original file structure
Supports hidden code blocks for generating content without showing the code
Works with any language by using file code blocks and bash execution
Zero external dependencies
GitHub Actions integration for automatic README updates
Preserves Markdown spec compatibility by hiding code in HTML comments ()

📊 Impact #

97 GitHub stars
Used in many of my other projects for auto-generating:
- Command-line help texts
- API documentation
- Example outputs
- Data tables
Helps maintain “information locality” - everything needed is in one file
Enables dynamic documentation that stays up-to-date automatically

🎯 Challenges and Solutions #

Parsing Markdown without breaking its structure
Supporting multiple programming languages
Making it work seamlessly with GitHub Actions
Handling complex outputs (tables, images, etc.)
Maintaining backward compatibility with existing Markdown files

💡 Lessons Learned #

Documentation is best when it’s close to what it documents
Automation should be invisible - the tool shouldn’t change how you write Markdown
Sometimes the simplest solution (using HTML comments) is the best
Good documentation tools should make maintaining docs easier, not harder
Auto-generated content should be clearly marked as such

Want to keep your documentation always in sync? Check out markdown-code-runner on GitHub!

#OpenSource #Python #Documentation #DevTools #Programming

WenFire: When Could I retire?! 🔥

Wed, 04 Dec 2024 00:00:00 +0000

(See my intro post)

While I absolutely love my job, I believe I’ll love it even more when my family’s livelihood doesn’t depend on it. That’s the dream of financial independence - doing what you love because you want to, not because you have to. This led me to create WenFire, a financial independence calculator that helps answer the eternal questions: “When can I retire?” and “How many days of extra work will that new iPhone or car cost me?”

📖 Origin Story #

There are many retirement calculators out there, but they all made assumptions I didn’t quite agree with (and I’m sure others won’t agree with my assumptions either!). I wanted something that matched my specific views on salary growth and investment returns. Plus, I had been itching to learn more about building web applications, so this project let me explore FastAPI, htmx, and Vega-Lite while solving a personal need.

🔧 Technical Highlights #

Built with FastAPI and Bootstrap
Interactive visualizations using Vega-Lite
Responsive and mobile-friendly design
Customizable assumptions for:
- Investment growth
- Inflation rates
- Salary increases
- Safe withdrawal rates
Allows parameter changes at certain dates
Calculate retirement impact of one-time expenses
All parameters are encoded in the URL for easy sharing

📊 Impact #

While this project only has 3 GitHub stars, its real value is personal - it helps me track my own journey to financial independence. It’s also been a great learning experience in web development, an area where I had little prior experience.

🎯 Challenges and Solutions #

Learning web development from scratch
Creating intuitive visualizations for financial data
Balancing simplicity with customization options
Making reasonable default assumptions
Ensuring mobile responsiveness

💡 Lessons Learned #

Financial modeling requires careful consideration of assumptions
Web development is a different beast from scientific computing
Sometimes the best way to learn new tech is to solve a personal problem
The best calculator is the one that matches your own assumptions

Want to calculate your path to financial independence? Try it at wenfire.nijho.lt or check out the source code!

#OpenSource #Python #FinancialIndependence #WebDev #FIRE

rsync-time-machine.py 🕒

Tue, 03 Dec 2024 00:00:00 +0000

(See my intro post)

Sometimes the best open source projects start with “I can do better than this.” That’s exactly what happened when I looked at laurent22/rsync-time-backup, a popular 600-line Bash script for Time Machine-style backups. While it worked, it had known bugs and was tricky to maintain due to Bash’s limitations. It also lacked tests, type hints (bash duh…), and proper error handling. So, I did what any reasonable developer would do: rewrote it in Python: rsync-time-machine.py! 🐍

📖 Origin Story #

I needed reliable backups that worked across platforms and filesystems, unlike Apple’s Time Machine. While many excellent backup tools exist, I was drawn to the simplicity of this approach: it just creates folders named YYYY-MM-DD-HH-MM-SS containing your files. No special tools needed for restoring or browsing - just your regular file explorer! I created a fully tested, typed, and documented Python port that maintains compatibility with the original while adding modern development practices.

🔧 Technical Highlights #

Creates incremental backups using hard links (saving space while keeping full snapshots)
Works on Linux, macOS, and Windows (via WSL or Cygwin)
Supports local and remote (SSH) backups
Smart backup expiration strategy
Built-in safety checks and locking mechanism
Pretty terminal output
Absolutely zero dependencies beyond Python itself - it just works!
Usable as a standalone .py script or pip installable package
100% test coverage

📊 Impact #

The project reached the front page of Y Combinator’s Hacker News, causing its popularity to explode:

381 GitHub stars
16 forks
6 contributors
Featured on Real Python podcast
Active discussion on Reddit’s r/commandline

🎯 Challenges and Solutions #

Maintaining compatibility with the original Bash script while improving the codebase
Making it work on Windows through WSL/Cygwin (surprisingly tricky!)
Handling various edge cases in filesystem operations
Implementing a robust backup expiration strategy
Adding comprehensive test coverage for file operations
Keeping the codebase dependency-free while maintaining functionality

💡 Lessons Learned #

Sometimes a rewrite is worth it, especially when adding modern development practices
Good documentation and test coverage are crucial for backup tools
The Python community appreciates well-structured ports of useful tools
Zero dependencies means zero headaches for users
Cross-platform compatibility (especially Windows) requires careful consideration
Simple, transparent solutions (like regular folders) can be better than complex ones

Want reliable, cross-platform Time Machine-style backups? Check out the project on GitHub!

#OpenSource #Python #Backups #DevTools #Programming

Adaptive Lighting 💡

Mon, 02 Dec 2024 00:00:00 +0000

(See my intro post)

Unlike yesterday’s Calendar of Life project, today we’re looking at one of my most popular contributions. It’s one of several Home Assistant and home automation related projects (Day 8, Day 10, Day 24) I’ll be sharing this month. While my coding journey started in scientific computing with physics simulations, discovering how to control my home environment through code sparked a new obsession. Home Assistant, currently the #1 open source project on GitHub, is the brain of my smart home and lets you control and automate everything from lights to thermostats, completely locally and privacy-focused.

Keep your lighting in sync with the sun, they said. It’ll be simple, they said…

📖 Origin Story #

Back in 2020, inspired by @claytonjn’s circadian_lighting Home Assistant component, I wanted to create a more comprehensive solution for automatically adjusting lights based on the time of day. What started as a contribution attempt to Home Assistant Core evolved into its own custom component when the PR wasn’t merged - handling all the edge cases made the code too complex for core. The standalone component gave us the freedom to implement features properly, becoming one of Home Assistant’s most installed custom integrations.

🔧 Technical Highlights #

While the concept is simple, making it work reliably is… interesting:

Handles unreliable Zigbee networks and flaky WiFi connections
Manages different light brands’ quirks (looking at you, IKEA bulbs 👀)
Provides smooth transitions between color temperatures
Detects manual overrides without disrupting user control
Supports sleep modes for bedtime routines
Configurable via YAML or the UI (only integration to support this!)

The integration has 38 parameters to fine-tune your lighting. Try our WebAssembly simulator to visualize your settings!

📊 Impact #

2000+ GitHub stars
103 contributors
512 issues
Tens of thousands of active users
Available in 20+ languages
One of the most installed custom integrations
Featured in several YouTube videos and podcasts
Dedicated parameter visualization web app

🎯 Challenges We Solved #

Lights randomly turning on (bulbs falsely reporting state)
Network congestion (solved with Zigbee grouping)
Different brands interpreting colors differently
Handling manual overrides gracefully

💡 Lessons Learned #

“It works on my setup” is just the beginning
Users surprise you (like using it for plants, or above the polar circle!)
Edge cases multiply quickly
Documentation is as important as code
Open source thrives on community effort

Want your lights to follow the sun? Check out the project on GitHub and try the simulator!

#OpenSource #HomeAssistant #SmartHome #IoT #Python

Calendar of Life 📅

Sun, 01 Dec 2024 00:00:00 +0000

(See my intro post.)

Visualizing the finite weeks of our lives, because apparently, I needed another reminder of my mortality!

📖 Origin Story #

You know how sometimes you’re just casually browsing the internet, not looking to have an existential crisis? Well, that’s exactly what happened when I stumbled upon Tim Urban’s Wait But Why post “Your Life in Weeks” and saw Kurzgesagt’s “Calendar of Your Life” poster. Instead of just feeling mildly anxious about the finite nature of existence like a normal person, I decided to cope by writing some Python code. Because that’s what we do, right?

🔧 Technical Highlights #

The project uses matplotlib to create a grid where:

Each row represents one year (up to 80 years, optimistically speaking!)
Each column represents one week (52 weeks per year)
Different colors show different life stages (early childhood, school, university, etc.)
The current week blinks in the animated version, just in case you needed a more urgent reminder of time passing
Supports both dark and light modes, because existential dread comes in all themes

One interesting technical detail is handling the fact that 52 weeks isn’t exactly one year. The code uses a small correction factor (days_per_week = 365.25 / 52) to ensure the visualization stays accurate over many years. Yes, I made sure your existential crisis would be mathematically precise.

📊 Impact #

With its whopping 5 GitHub stars, this project clearly hasn’t taken the world by storm. But hey, sometimes the best tools are the ones we make for ourselves, right?

🎯 Challenges and Solutions #

The main challenges were (this was actually a very simple one):

Accurately mapping life events to week numbers (turns out life doesn’t fit neatly into boxes)
Creating smooth animations for the current week indicator (because a blinking dot really drives home the whole “time is passing” thing)
Handling the 52 vs 52.18 weeks per year discrepancy (thanks, Earth’s orbit)
Making the visualization work well in both dark and light modes (for viewing your life calendar at any time of day)

💡 Lessons Learned #

This project taught me that sometimes the most impactful tools are the simplest ones. Sure, it’s just a bunch of dots on a grid, but each dot represents a week of life - past, present, or future. It’s a humbling reminder that while we can’t add more dots to our grid, we can try to make each one count.

Want to contemplate your own mortality in a more organized way? Check out the project on GitHub!

#OpenSource #Python #DataVisualization #Programming

opennb

Wed, 27 Nov 2024 00:00:00 +0000

graphviz-anywidget

Fri, 22 Nov 2024 00:00:00 +0000

clip-files

Thu, 13 Jun 2024 00:00:00 +0000

Tiller Streamlit

Thu, 08 Feb 2024 00:00:00 +0000

numthreads

Fri, 05 Jan 2024 00:00:00 +0000

UniDep

Sat, 25 Nov 2023 00:00:00 +0000

PipeFunc

Thu, 15 Jun 2023 00:00:00 +0000

rsync-time-machine.py

Thu, 11 May 2023 00:00:00 +0000

WenFire ⁉️🔥

Thu, 11 May 2023 00:00:00 +0000

Adaptive Lighting

Sat, 22 Apr 2023 00:00:00 +0000

CodeStructure

Sat, 22 Apr 2023 00:00:00 +0000

Home Assistant Streamdeck YAML

Sat, 22 Apr 2023 00:00:00 +0000

Markdown Code Runner

Sat, 22 Apr 2023 00:00:00 +0000

My favorite tools that keep my Python projects sane 🧰

Sat, 22 Apr 2023 00:00:00 +0000

🌟 1. Introduction #

As a passionate Python developer, I am always on the hunt for the best tools to streamline my development process and enhance the quality of my projects. Over time, I have discovered a set of tools that have become indispensable in my workflow, making my coding experience more efficient and enjoyable. Using the right developer tools for Python projects is crucial for maximizing productivity, ensuring code consistency, and maintaining high-quality code.

In this blog post, I will share these essential tools that I now use for every single new project. We will explore the benefits of each tool and discuss how they can help you supercharge your Python development. The tools covered in this post include pre-commit, Black, Ruff, Mypy, Versioningit, Pyproject.toml with PEP 621, Nox, pytest, GitHub Actions CI matrix, and Read the Docs, Sphinx, MyST-NB, and Jupytext notebooks for documentation. Let’s dive in and learn how to enhance your Python projects with these powerful tools!

Table of contents

🛠️ 2. Pre-commit #

One of the most common challenges developers face when working on a project, whether alone or in a team, is maintaining code quality and consistency. As projects grow, it becomes increasingly difficult to enforce coding standards and catch issues before they make their way into the codebase. This is where pre-commit comes in.

Pre-commit is a powerful tool that helps you manage and maintain code quality by automating checks before commits. It achieves this by setting up a series of hooks that run before each commit, ensuring that only code that meets your predefined standards is committed to the repository. By catching issues early, pre-commit helps you keep your code clean and consistent, making it easier to review and maintain.

One of the reasons pre-commit has gained widespread adoption is its flexibility and extensibility. You can choose from a vast range of built-in hooks, such as code formatters, linters, and security checkers, or even create your own custom hooks. This allows you to tailor pre-commit to the specific needs of your project, ensuring that it enforces the exact coding standards you require.

Using pre-commit is beneficial for both individual developers and teams. For solo developers, it helps maintain discipline and enforces coding standards, making it easier to onboard collaborators in the future. For teams, it streamlines the review process by catching issues before they are submitted for review, reducing the time spent on addressing code inconsistencies and style violations.

Let’s take a look at an example. Suppose your project follows the PEP 8 coding standard, and you want to ensure that all code committed to the repository adheres to these guidelines. You can configure pre-commit to run a PEP 8 linter, such as Flake8, before each commit. This way, any code that does not meet the PEP 8 standard will be flagged, and the commit will be blocked until the issues are resolved.

In summary, pre-commit is an invaluable tool for maintaining code quality and consistency in your Python projects. By automating checks before commits, it ensures that your codebase remains clean and adheres to your chosen standards, making your development process more efficient and enjoyable.

Adding pre-commit to your Python project is simple and straightforward. To get started, you’ll first need to install the pre-commit package using pip:

pip install pre-commit

Next, create a configuration file named .pre-commit-config.yaml in your project’s root directory. This file will define the hooks you want to run before each commit. For example, to use Black for code formatting and Flake8 for linting, your configuration file might look like this:

repos:
 - repo: https://github.com/psf/black
 rev: stable
 hooks:
 - id: black
 - repo: https://github.com/pycqa/flake8
 rev: master
 hooks:
 - id: flake8

Once you have set up the configuration file, run the following command to install the pre-commit hooks:

pre-commit install

Now, every time you attempt to make a commit, pre-commit will run the specified hooks. If any issues are found, the commit will be blocked, and you’ll be prompted to fix the problems before trying again. To run pre-commit manually on all files in your repository, use the following command:

pre-commit run --all-files

By incorporating pre-commit into your development process, you’ll be able to catch and fix issues before they make their way into your codebase, ensuring a consistent and high-quality Python project.

🎨 3. Black: auto-formatting for Python #

As a developer who has always been particular about coding style, I’ve had my fair share of discussions on the proper way to indent multiline statements and other formatting nuances. These conversations, although sometimes necessary, can take up valuable time that could be better spent on actual development. That’s where Black, the uncompromising code formatter, comes in and saves the day.

Adopting Black has made a significant impact on my development workflow. No more unnecessary discussions about formatting preferences; everyone on the team simply follows the rules set forth by Black. This consistency across the codebase not only makes the code more readable but also reduces the time spent on formatting debates.

One of the best aspects of Black is that you don’t need to worry about writing perfectly formatted code initially. You can write your code as quickly as you want, even if it’s poorly formatted, and then simply run Black to transform it into a beautifully formatted masterpiece. This can be a real time-saver, especially when working on large projects with many contributors.

To install Black, run the following command:

pip install black

You can then format your code by running Black in the terminal:

black your_file.py

Alternatively, you can use the VS Code command palette to run Black on the currently open file. However, the most efficient way to use Black is in combination with pre-commit, as mentioned in the previous section. This way, your code is automatically formatted every time you commit, ensuring a consistent codebase without any extra effort.

In summary, Black is an invaluable tool for enforcing consistent code formatting throughout your Python projects. By adopting it and integrating it with pre-commit, you can save time and focus on what truly matters: writing great code.

🚀 4. Ruff: the fastest Python linter and auto-formatter #

In the past, I’ve used several tools to maintain code quality, such as isort, pyupgrade, flake8, and pydocstyle. However, I’ve recently discovered a powerful new tool called Ruff), an extremely fast Python linter written in Rust. Ruff is not only capable of replacing all these tools but also executes much faster, which significantly improves the development workflow.

Ruff offers an impressive range of features and performance enhancements. Some of its standout features include:

10-100x faster than existing linters
Over 500 built-in rules
Autofix support for automatic error correction
Built-in caching to avoid re-analyzing unchanged files
First-party editor integrations for VS Code and more

Moreover, Ruff has gained widespread adoption in the Python community and is used in major open-source projects like Apache Airflow, FastAPI, Hugging Face, Pandas, and SciPy. The tool is backed by Astral, which provides active development and support.

One of the best aspects of Ruff is the native VS Code extension, which continuously scans your code and can fix problems as you write. This seamless integration makes it an indispensable part of any Python development workflow.

To install Ruff, simply run the following command:

pip install ruff

To use Ruff in your project, run the following command in the terminal:

ruff your_file.py

For the best experience, I highly recommend using Ruff in conjunction with pre-commit, as mentioned earlier. This ensures that your code is always linted and formatted before committing, maintaining a consistent and high-quality codebase.

In conclusion, Ruff is an incredibly fast and powerful Python linter that streamlines the development process by consolidating multiple tools into a single, efficient package. By adopting Ruff and integrating it with pre-commit, you can enjoy a more productive and enjoyable Python development experience.

🔍 5. Mypy: static type checking for Python (find bugs before they happen) #

As developers, we’re always looking for ways to improve the quality of our code and minimize the time spent on debugging. This is where Mypy comes in. Mypy is a powerful static type checker for Python, designed to catch potential runtime errors before they occur. By using Mypy, you can ensure that your code adheres to type hints and that it runs smoothly without unexpected issues.

Mypy offers several advantages to Python developers, including:

Early detection of potential type errors, preventing many common bugs
Improved code readability and self-documentation through type annotations
Easier refactoring and code maintenance, as type annotations provide clear contracts between functions and modules
Better collaboration and communication within teams, since type annotations make the code’s intent and expectations more explicit

Adding Mypy to an existing project may initially seem time-consuming and frustrating, as it might flag numerous issues that require attention. However, once you’ve addressed these concerns and properly set up Mypy, the benefits far outweigh the initial effort. In fact, the time you invest in fixing type-related issues will quickly be recovered as Mypy helps you prevent bugs and reduce debugging time.

For a long time, I too ignored Mypy and only typed my code without actually checking it with the tool. But after truly embracing Mypy and integrating it into my development process, I can confidently say that I will never go back. The value it brings to the table in terms of code quality, readability, and reliability is simply too great to ignore.

By proactively identifying and resolving type-related issues, Mypy not only saves you precious debugging time but also enhances the overall quality of your codebase. As you continue to rely on Mypy, you’ll find that it becomes an indispensable part of your Python development toolkit, ensuring that your projects remain robust and maintainable.

To start using Mypy in your project, first install it with the following command:

pip install mypy

Then, you can run Mypy on your Python files by executing the following command in your terminal:

mypy your_file.py

For even better integration, you can add Mypy as a pre-commit hook, ensuring that your code is always checked for type consistency before being committed. This further strengthens the code quality and helps prevent potential runtime errors.

In summary, incorporating Mypy into your Python development workflow adds an extra layer of safety and clarity, making it easier to write high-quality, reliable code. By combining Mypy with other tools like Ruff and pre-commit, you can create a robust development environment that promotes productivity and code quality.

🧪 6. Nox and pytest for testing: a powerful combination #

When developing software, it’s essential to ensure that your code is reliable and functions as expected. The key to achieving this goal is writing thorough tests, which not only help you identify issues early on but also prevent regressions in the future. While it’s often tempting to dive headfirst into coding, taking a step back and first writing tests for your solution can lead to more efficient and effective development. Even if this approach doesn’t come naturally, it’s crucial to write tests for every problem you encounter to keep your codebase stable and reliable.

In the realm of Python testing, Nox and pytest stand out as powerful tools that can greatly enhance your testing process. Nox is a versatile automation tool that simplifies running tests in multiple virtual environments, while pytest is a popular and feature-rich testing framework for Python. Together, these tools provide a formidable testing environment that simplifies test management and execution.

In the past, you may have used Tox, a tool that manages test environments with a text-based configuration file (e.g., in setup.cfg or pyproject.toml). While Tox has served developers well for many years, including myself, Nox offers a more flexible alternative. Instead of relying on a text-based configuration, Nox takes advantage of a Python configuration file, providing virtually limitless configurability. This added flexibility makes it easy to create complex test matrices with different versions and dependencies, and even accommodate special conditions using simple if statements. After happily using Tox for years, I discovered Nox and found that it elevated my testing workflow to a whole new level.

To get started with Nox and pytest, create a basic nox.py file and a sample test file for pytest. Here’s an example nox.py configuration:

import nox

@nox.session(python=["3.7", "3.8", "3.9"])
def tests(session):
 session.install("-r", "requirements.txt")
 session.install("-r", "test-requirements.txt")
 session.run("pytest")

And an example test file for pytest:

import pytest

def test_addition():
 assert 1 + 1 == 2

def test_subtraction():
 assert 3 - 1 == 2

Beyond basic testing functionality, pytest boasts powerful features such as fixtures, mocking and patching, and parameterization. Pytest fixtures enable you to create reusable resources for your tests, while mocking and patching allow you to replace parts of your system with fake objects, isolating your tests from external dependencies. Parameterization, implemented using pytest.mark.parametrize, enables running the same test with multiple sets of inputs and expected outputs.

In conclusion, leveraging Nox and pytest for testing offers numerous advantages for developers who strive to improve code reliability and automate their testing workflows. By harnessing Nox’s exceptional configurability and pytest’s extensive testing features, you’ll establish a robust testing environment that bolsters code quality and ensures optimal performance.

🏷️ 7. Versioningit: version management with Git tags #

(Optional, only related when making a Python package)

For developers seeking an automated and customizable package versioning solution, Versioningit offers a powerful yet flexible approach. It is a setuptools plugin that automatically determines your package’s version based on your version control repository’s tags. The real advantage of Versioningit lies in its ability to customize the version format and even override separate functions used for version extraction and calculation.

Versioningit makes your release workflow extremely simple, especially if you have set up an automated build and twine PyPI upload pipeline. To create a new release, you only need to tag a version, and then click on the “Release” button in GitHub.

One of the major advantages of Versioningit is its minimal boilerplate code compared to other tools like Versioneer. Additionally, the Python __version__ string works seamlessly with editable installs (pip install -e .), unlike setuptools_scm, which is a common issue faced by developers.

Over the years, I have explored many options for package versioning, including Versioneer, Miniver, setuptools_scm, bumpversion (or bump2version), and manually setting everything. Versioningit stands out due to its simplicity, customization options, and compatibility with various workflows.

Some key features of Versioningit include:

Support for Git, modern Git archives, and Mercurial
Customizable version formatting using template strings
Optional writing of the final version and other details to a file for loading at runtime
Custom setuptools commands for inserting the final version and other details into a source file at build time
Customizable functions for VCS querying, tag-to-version calculation, version bumping, version formatting, and writing the version to a file

To use Versioningit in your project, simply add it to your project’s pyproject.toml file in the build-system.requires key. Then, create a [tool.versioningit] table in your pyproject.toml file. You can get up and running with just the minimal configuration - an empty table.

Once you have a [tool.versioningit] table in your pyproject.toml and your repository has at least one tag, building your project with setuptools will automatically set your project’s version based on the latest tag in your Git repository. With Versioningit, you can customize the version format using placeholder strings, making it adaptable to your project’s specific needs.

In summary, Versioningit streamlines the package versioning process and simplifies the release workflow. By automating package versioning based on version control repository tags and providing extensive customization options, it helps make the management of package versions more efficient and less error-prone.

📝 8. Pyproject.toml and PEP 621: Configuration Consolidation #

The pyproject.toml file plays a crucial role in consolidating configurations and setup for various tools in a Python project, such as Black, Ruff, Mypy, Versioningit, Nox, and pytest. This adherence to PEP 621 standards streamlines the development process by providing a single source of truth for multiple tools, eliminating the need for additional configuration files like setup.cfg or setup.py.

By specifying packaging details directly in the pyproject.toml file, you can efficiently manage your project’s dependencies and metadata. For example, the Adaptive Scheduler package demonstrates this approach with a comprehensive pyproject.toml file containing all the necessary configurations.

(click to unfold):

[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools", "wheel", "versioningit"]

[project]
name = "adaptive_scheduler"
description = "Run many `adaptive.Learner`s on many cores (>10k) using `mpi4py.futures`, `ipyparallel`, `dask-mpi`, or `process-pool`."
requires-python = ">=3.8"
dynamic = ["version"]
maintainers = [{ name = "Bas Nijholt", email = "bas@nijho.lt" }]
license = { text = "BSD-3" }
classifiers = [
 "Development Status :: 2 - Pre-Alpha",
 "Intended Audience :: Developers",
 "Intended Audience :: Science/Research",
 "License :: OSI Approved :: BSD License",
 "Operating System :: OS Independent",
 "Programming Language :: Python",
 "Programming Language :: Python :: 3",
 "Programming Language :: Python :: 3.8",
 "Programming Language :: Python :: 3.9",
 "Programming Language :: Python :: 3.10",
 "Programming Language :: Python :: 3.11",
 "Topic :: Scientific/Engineering",
 "Topic :: System :: Distributed Computing",
]
dependencies = [
 "adaptive >= 0.14.1",
 "cloudpickle",
 "dill",
 "ipyparallel",
 "ipywidgets",
 "itables",
 "jinja2",
 "loky",
 "numpy",
 "pandas",
 "psutil",
 "pyarrow",
 "pyzmq",
 "rich",
 "structlog",
 "tinydb",
 "toolz",
 "tqdm",
 "versioningit",
]
[project.optional-dependencies]
all = ["dask-mpi", "mpi4py"]
test = ["pytest", "pytest-asyncio", "coverage", "pytest-cov"]
docs = [
 "myst-nb",
 "sphinx_fontawesome",
 "sphinx",
 "furo",
 "myst-parser",
 "emoji",
]
[project.urls]
homepage = "https://adaptive-scheduler.readthedocs.io/"
documentation = "https://adaptive-scheduler.readthedocs.io/"
repository = "https://github.com/basnijholt/adaptive-scheduler"

[project.readme]
content-type = "text/x-rst"
file = "README.rst"

[project.scripts]
adaptive-scheduler-launcher = "adaptive_scheduler._server_support.launcher:main"

[tool.versioningit]

[tool.versioningit.onbuild]
build-file = "adaptive_scheduler/_version.py"
source-file = "adaptive_scheduler/_version.py"

[tool.setuptools.packages.find]
include = ["adaptive_scheduler.*", "adaptive_scheduler"]

[tool.pytest.ini_options]
addopts = """
 -vvv
 --cov=adaptive_scheduler
 --cov-report term
 --cov-report html
 --cov-report xml
 --cov-fail-under=35
 --asyncio-mode=auto
"""

[tool.coverage.report]
exclude_lines = [
 "pragma: no cover",
 "raise NotImplementedError",
 "if TYPE_CHECKING:",
 "if __name__ == .__main__.:",
]

[tool.black]
line_length = 88

[tool.ruff]
line-length = 150
target-version = "py38"
select = ["ALL"]
ignore = [
 "T20", # flake8-print
 "ANN101", # Missing type annotation for {name} in method
 "S101", # Use of assert detected
 "PD901", # df is a bad variable name. Be kinder to your future self.
 "ANN401", # Dynamically typed expressions (typing.Any) are disallowed in {name}
 "D402", # First line should not be the function's signature
 "PLW0603", # Using the global statement to update `X` is discouraged
 "D401", # First line of docstring should be in imperative mood
 "SLF001", # Private member accessed
]

[tool.ruff.per-file-ignores]
"tests/*" = ["SLF001"]
"tests/test_examples.py" = ["E501"]
".github/*" = ["INP001"]

[tool.ruff.mccabe]
max-complexity = 18

[tool.mypy]
python_version = "3.8"

(from here)

As seen in the example, the pyproject.toml file includes details about the build system, project metadata, dependencies, optional dependencies, URLs, and configurations for various tools such as Versioningit, setuptools, pytest, coverage, Black, Ruff, and Mypy. By consolidating these configurations, you can maintain a more organized and maintainable project, making it easier to understand and update settings for different tools.

In summary, the pyproject.toml file and adherence to PEP 621 provide a unified approach to configuring and setting up your Python project, making it easier to manage dependencies, metadata, and tool-specific settings. This streamlined process ultimately leads to a more efficient and maintainable development workflow.

🌐 9. GitHub Actions CI Matrix: Testing on Multiple Platforms and Auto Publishing to PyPI #

GitHub Actions is an excellent tool for automating workflows and improving code quality through continuous integration (CI). By using a CI matrix, you can test your code on multiple platforms and Python versions, ensuring that your package remains compatible and stable across different environments.

To set up a .github/workflows/nox.yaml file for GitHub Actions, you can use the following simplified example:

name: nox

on:
 pull_request:
 push:
 branches: [main]

jobs:
 test:
 runs-on: ${{ matrix.platform }}
 strategy:
 fail-fast: false
 matrix:
 platform: [ubuntu-latest, macos-latest, windows-latest]
 python-version: ["3.8", "3.9", "3.10", "3.11"]

 steps:
 - uses: actions/checkout@v2
 - name: Set up Python ${{ matrix.python-version }}
 uses: actions/setup-python@v2
 with:
 python-version: ${{ matrix.python-version }}
 - name: Install dependencies
 run: pip install nox
 - name: Test with nox
 run: nox -e py${{ matrix.python-version }}

This configuration file specifies the platforms and Python versions for testing, as well as the steps to set up the environment, install dependencies, and run tests using nox. With this simple configuration, you set up tests on MacOS, Windows, Linux, and multiple Python versions, ensuring that your package remains compatible across different environments.

Another essential part of the CI/CD process is deploying your Python package. You can use the .github/workflows/publish-package.yaml file to automate the package deployment to PyPI whenever a new release is published:

name: Upload Python Package

on:
 release:
 types: [published]

jobs:
 deploy:

 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v2
 - name: Set up Python
 uses: actions/setup-python@v2
 with:
 python-version: '3.x'
 - name: Install dependencies
 run: |
 python -m pip install --upgrade pip
 pip install setuptools wheel twine build
 - name: Build and publish
 env:
 TWINE_USERNAME: __token__
 TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
 run: |
 python -m build
 twine upload dist/*

This workflow sets up Python, installs necessary dependencies, builds the package, and publishes it to PyPI using Twine. The process integrates seamlessly with versioningit, ensuring that the published package has the correct version number based on the git history.

In summary, leveraging GitHub Actions for continuous integration and automated workflows enhances code quality and reliability, while also simplifying the development and deployment processes. By testing across various platforms and Python versions, and automating package deployment, you can be confident that your package is well-maintained and accessible to users.

📚 10. Read the Docs, Sphinx, MyST-NB, and Jupytext notebooks for documentation #

Documentation is crucial for any software project, as it helps users understand how to use the package and provides guidance for contributors. Read the Docs, Sphinx, MyST-NB, and Jupytext notebooks are powerful tools that, when combined, facilitate the creation and maintenance of high-quality documentation.

tl;dr: check out the Adaptive documentation for an example of how to use these tools to create a comprehensive documentation website.

Read the Docs is a documentation hosting platform that automatically builds, version controls, and hosts your documentation for free. It integrates well with Sphinx and GitHub, allowing your documentation to be updated whenever you push changes to your repository.

Sphinx is a documentation generator that can transform reStructuredText files, MyST files, or Jupyter notebooks into various output formats, such as HTML, LaTeX, or PDF. It is highly extensible, supporting custom themes and plugins, and can automatically generate API documentation from your Python code.

MyST-NB is an extension to Sphinx that enables it to parse Jupyter notebooks and execute the code cells during the build process. It uses the MyST (short for Markedly Structured Text) markdown parser, which adds support for Sphinx roles and directives, making it a powerful alternative to reStructuredText.

Jupytext notebooks is a Jupyter extension that enables you to save Jupyter notebooks in Markdown format (or other text-based formats) instead of the default JSON format. By using Jupytext notebooks, you can store your code, explanations, and visualizations as plain text, which is easier to version control and can be seamlessly integrated with MyST-NB and Sphinx.

With MyST-NB and Jupytext, you can write your documentation using Jupyter notebooks in Markdown format. This approach combines the benefits of Jupyter notebooks, such as interactive code execution and rich output, with the simplicity of Markdown and the powerful features of Sphinx. The result is a highly readable, user-friendly, and maintainable documentation that caters to both users and developers.

To get started with MyST-NB, you’ll need to install it using pip:

pip install myst-nb

Next, configure your Sphinx project to use MyST-NB by adding the following lines to your conf.py file:

extensions = [
 "myst_nb",
]

Now, Sphinx will be able to parse and execute your Jupyter notebooks during the documentation build process, integrating them seamlessly with your existing documentation.

In summary, using Read the Docs, Sphinx, MyST-NB, and Jupytext notebooks for your documentation enables you to create high-quality, interactive, and easily maintainable documentation that benefits your project’s users and contributors. This combination of tools ensures that your documentation remains up-to-date, easily navigable, and engaging, ultimately enhancing the overall experience for your project’s audience.

For a full example, see how I setup Adaptive’s documentation using Sphinx, MyST-NB, and Jupytext notebooks. readthedocs.yaml, docs/source/conf.py, and here is an example notebook that is used in the documentation: docs/source/tutorial/tutorial.Learner1D.md

🎉 11. Conclusion #

In conclusion, incorporating these powerful tools into your Python development workflow can significantly enhance your productivity and code quality. By adopting pre-commit, Black, Ruff, Mypy, Versioningit, Pyproject.toml with PEP 621, Nox, pytest, GitHub Actions CI matrix, and Read the Docs, Sphinx, MyST-NB, and Jupytext notebooks for documentation, you create a streamlined, efficient, and reliable development environment.

These tools help maintain code consistency, catch issues early, automate package versioning, simplify testing, consolidate configurations, and create high-quality documentation. By integrating them into your projects, you can focus on writing great code while ensuring that your work remains robust, maintainable, and accessible to users and contributors alike.

Don’t hesitate to explore these tools and incorporate them into your Python development toolkit. They can transform your development experience, making it more enjoyable and efficient, ultimately leading to higher-quality projects and happier developers. Happy coding!

Calendar of Life

Thu, 03 Jun 2021 00:00:00 +0000

home-assistant-macbook-touch-bar

Tue, 09 Jun 2020 00:00:00 +0000

Thesis cover

Mon, 11 May 2020 15:00:00 +0000

Crafting unique Ph.D. thesis covers with Python 🎨

Sat, 04 Apr 2020 13:40:00 +0000

Originally posted on quantumtinkerer.tudelft.nl/blog/thesis-cover on 2020-04-04 13:40.

I will hand out around a hundred copies of my thesis to my defense committee, colleagues, and my family and friends. Let’s be honest, most people will probably not get further than attempting to read the summary and appreciating the cover. Four years of work has gone into generating the content of the thesis, so I figured, at least some thought has to go into the design of its cover. Unfortunately, I am by no means an expert on a graphical design or even competent enough to attempt to use any kind of graphic design software. But luckily for me, I do consider myself an expert in Python , and why not make the thesis design a fun process?

Initial Design: Adaptive Sampling and Data Visualization #

My vague plan was to somehow incorporate a plot resulting from a simulation run with Adaptive [source ], the software I wrote during my Ph.D. to adaptively sample functions in parallel and my proudest work. I even had a potential simulation result in mind that I would use for it, specifically, one that has been on the background of my good friend’s phone for some years and one that has earned me quite some internet points on Reddit. This plot (below) is one of the results of Chapter 4 of my thesis [source ] and this paper.

Your browser does not support the video tag.

I went back to the code that generates the plot and started hacking away in Matplotlib. Quickly, I realized that not only the plot above was going to look cool, but that many other data sets that I had also resulted in beautiful patterns. I hit the paradox of choice: I had a total of 3671 unique data sets (all for Majorana devices with either different parameter sweeps or different physical effects that were included in the model) and no idea how I was going to make a selection.

This is an early attempt (and rather ugly in hindsight)

I decided to postpone the choice of which data to use and just focus on how to plot the data first. I experimented with different colormaps, fading colormaps that gradually reveal the underlying triangulation better, and more. On a weekend night, after telling a friend I was with about my plans, we (as usually happens) started to program some of the ideas we came up with. We settled on a design where the triangulation would slowly fade from the top to bottom and where each triangle simultaneously transitioned from a solid color (the average of the values at its vertices) to densely interpolated values. See this Jupyter notebook which generated the final version of the cover. After we also chose a nice colormap (by simply creating the cover for each possible colormap,) the following was suggested: “You know what would be even cooler?! To make every cover unique.” Ten minutes later, I had started 3671 jobs on our computational cluster. When the files slowly started to appear on my computer (I have my local files synced with the cluster), we could not have been more excited. There were simply so many cool looking alternatives. Too many to manually select now, to be honest—a problem for later I thought.

Overcoming the Paradox of Choice: Clustering and Neural Networks #

Four months later, with the rest of my thesis completely done and about a day left on the deadline to submit my thesis, I found myself facing the selection problem I had successfully ignored for so long. There must be a smarter way to select the covers, I thought. Many of them look roughly similar (and some of them are even quite boring or not ecstatically pleasing), so it does not make sense to scroll through all of them. I could cluster them by similarity and then I would just have to choose the nicely looking clusters while being able to instantly discard many.

Finally, a good application to use neural networks!

I still had the 3671 covers in pdf format in some folder, so using the wonderful GNU Parallel, I converted and rasterized all of them to jpeg with

find . -iname \*pdf -print0 | parallel -0 sips -s format jpeg {} --out {.}.jpg

Then I used a pre-trained convolutional neural network model (VGG16) to extract features in a lower-dimensional representation. I fed the feature arrays to a K-Means clustering algorithm that sorted them in 80 clusters. To procrastinate a little more, I created an interactive widget using ipywidgets (see video below) to facilitate the selection process.

Your browser does not support the video tag.

These are the clusters that I selected in a grid

The selection process was done using this Jupyter notebook.

Finally, I copied the selected clusters into different folders and used the OS’s file explorer to manually select a few covers from each cluster.

Final Touches: Fine-Tuning Colors and Adding Edition Numbers #

Then I printed an initial version of my thesis, however, I wasn’t really happy with the colors.

To choose the right color scheme I got help from a non-tech savvy friend who went to art school. Due to the COVID-19 crisis, we couldn’t sit side by side, so I created this Binder “app” with sliders.

Your browser does not support the video tag.

Additionally, since all of them were different anyway, I gave them an identifier and added

edition n of 120

to the back.

These are a few examples of the final result (click on one of the images to open gallery view)

A few days later, this was delivered to my doorstep

And finally I mailed them to my Ph.D. committee

Now it’s time to start preparing for my defense at May 11, 2020 at 15:00!

Notes #

The data and code to generate all printed covers is available at basnijholt/thesis-cover
The simulation code that generates the data is available at basnijholt/spin-orbit-nanowires
The video of the Reddit post is generated using this code

pfapack

Mon, 02 Mar 2020 00:00:00 +0000

slurm-usage

Tue, 14 Jan 2020 00:00:00 +0000

Adaptive-scheduler