Open Source | Bas Nijholt

MindRoom: AI agents that live in Matrix and work everywhere 🧠

Fri, 26 Jun 2026 00:00:00 +0000

Table of contents

1. The problem: your AI is trapped in apps #

Programming is my biggest passion in life. I’ve been actively involved in open source for over ten years now, maintaining 40+ installable packages across Python, JavaScript, and Rust. Some of these have over 100,000 users; others have exactly one—me. I mention this because it matters for what comes next: when AI coding tools arrived, I didn’t start from zero. As I wrote about in my agentic coding post, AI has completely transformed how I work—when I’m AFK, I regularly build full features or entire projects by dictating to my phone and routing it through my own transcription software.

Through all of this, I became deeply convinced that AI agents are the future. Not just a useful tool for programmers—a fundamental shift in how everyone will interact with computers.

But if that’s true, then the way agents work today is broken.

We’ve built artificial barriers between artificial intelligences. ChatGPT knows you in one tab, Claude knows you in another, your Slack bot knows you in a third. Your email AI can’t tell your calendar AI about that meeting invitation. Your coding assistant has no idea about the project specs sitting in your document AI. Switch platforms, and your AI starts from scratch.

Most agent frameworks require you to program them. That works for developers like me, but if agents are truly the future, they need to work for everyone. Non-programmers don’t want to write code—they just want a chat.

And then there’s privacy. I’m fine sharing my email with Gemini—Google already owns my Gmail. But do I want to send my financial data, my health records, or my personal notes to a cloud provider? Not really. I’d rather use a local model for sensitive tasks and pick the best, cheapest Chinese model for general-purpose deep research.

So I built MindRoom on a chat protocol—if people just want a chat, give them a chat. Agents live in the same rooms as you, collaborate in threads, and follow you across every platform. The first commit landed on July 29, 2025 (four months before OpenClaw’s first commit 😅), so by the time I’m publishing this MindRoom has been with me for nearly eleven months: first as an obsessive side project, then, since March, with much more sustained focus. At first I got so obsessed with it that I eventually burned out and had to step away completely.

Recently, seeing OpenClaw gain massive traction solving a similar problem reminded me that what I’d been building wasn’t just a niche obsession. So here I am, dusting off MindRoom and writing about what it is, how it works, and why I think the approach still matters.

Live chat.mindroom.chat Personal thread with the prompt: “Explain how MindRoom and what you can do.” Mind answers in the real Matrix room, using tool calls and the Codex-backed local MindRoom runtime.

2. What is MindRoom? #

MindRoom is an open-source system that creates AI agents living inside the Matrix protocol. If you’re not familiar with Matrix—it’s a federated, end-to-end encrypted communication standard. The same protocol used by the French government for 5.5 million civil servants, by German healthcare for 150K+ organizations, and by the Element app that millions of people use daily.

The key insight: Matrix has bridges to many major platforms—Slack, Telegram, Discord, WhatsApp, IRC, email, even SMS.

So if your AI agent lives in Matrix, it can reach you on any platform. One agent, every platform, continuous memory.

One practical caveat: Matrix bridges vary in maturity. Some, like the Telegram bridge, work very well, while others can be finicky. Your mileage may vary depending on which platforms you need.

Here’s what a typical setup looks like in config.yaml:

agents:
 code:
 display_name: CodeAgent
 role: Generate code, manage files, execute shell commands
 model: opus-4.8
 tools: [file, shell, github]
 instructions:
 - Always read files before modifying them.
 rooms: [lobby, dev]

 research:
 display_name: ResearchAgent
 role: Search the web, summarize papers, find information
 model: gpt-5.5
 tools: [tavily, arxiv, wikipedia]
 rooms: [lobby, research]

teams:
 super_team:
 display_name: Super Team
 agents: [code, research]
 mode: collaborate

Define your agents, give them tools and rooms, and they show up in Matrix as real users—with avatars, typing indicators, and online status.

MindRoom agents are real Matrix users with their own accounts, generated avatars, display names, room memberships, and online presence.

3. How it actually works #

At the core sits what I call the MultiAgentOrchestrator—a class in bot.py that boots every configured entity (router, agents, teams), provisions Matrix user accounts for each one via matrix-nio, and keeps sync loops alive. The agents themselves are powered by the Agno framework, which provides a unified interface across AI model providers.

When someone sends a message in a Matrix room:

Explicit mentions go directly to the named agent (@mindroom_code help me debug this)
Thread continuations keep the same agent responding (no weird context switches mid-conversation)
New conversations hit the router—an AI that analyzes the message and picks the best agent based on capabilities

All conversations happen in threads, which keeps rooms organized. Agents stream their responses in real-time, editing a single message as they think rather than spamming new ones. You see tool calls happening live:

🔧 Tool Call: search_web(query="matrix protocol bridges")
[waiting for result...]
✅ search_web result:
[results here]

The recording below starts in the main Personal room, sends the prompt Explain how MindRoom and what you can do., and then follows MindRoom as it opens the new thread and streams the reply to completion. It is sped up 2.5x to keep the flow skimmable.

4. Building on Matrix: the good and the tricky #

One of the best things about building on Matrix is what you get for free. End-to-end encryption, a deeply interoperable chat protocol with bridges to many platforms, a choice of clients, federation between organizations—all of that comes with the protocol. You don’t have to build any of it yourself.

But because Matrix has such a tight specification, it also brings challenges.

The protocol doesn’t support streaming. AI agents that think for 30 seconds before dumping a wall of text make for a terrible chat experience, so I hacked streaming in by rapidly editing the same message as new tokens arrive. An ⋯ marker shows while the agent is still thinking—a small touch, but it makes the experience feel responsive and alive.

There’s also a size limit on message content. That’s fine for human chat, but AI responses can get long—especially when tool calls and their results are included. I worked around this by using Matrix’s attachment feature: when a response exceeds the limit, the content continues in an attachment that gets updated as the message keeps streaming in. This required forking the Cinny chat client so that attachments display inline rather than as downloadable files, making the whole thing seamless.

5. Memory that follows you #

MindRoom currently supports two memory implementations.

The first is the more traditional semantic memory system built on Mem0’s AsyncMemory, with configurable embedding providers (OpenAI, Ollama, or HuggingFace) and vector storage via ChromaDB. This gives agents searchable memory for preferences, project context, decisions, and recurring facts.

The second is a file-based memory system inspired by the simplicity of OpenClaw-style agent files. Instead of burying identity in a database, durable context can live in plain Markdown files that travel with the agent. That portability matters: I can move the same agent identity from OpenClaw to Hermes to MindRoom without starting from scratch.

MindRoom also has a memory flush step that periodically extracts durable facts from conversations and writes them back into memory. So the chat transcript remains chat, while stable preferences, decisions, and identity details become reusable context for future turns.

A live Personal memory demo: MindRoom stores a harmless preference in one turn and retrieves it later from the same Matrix conversation.

6. 100+ built-in tool integrations #

Agents can use over 100 built-in integrations:

Category	Examples
Communication	Gmail, Slack, Telegram, Discord
Development	GitHub, Shell, Python, Docker
Search	Tavily, Wikipedia, Arxiv, DuckDuckGo
Productivity	Google Calendar, Jira, Linear, Todoist
Smart Home	Home Assistant
AI/ML	DALL-E, ElevenLabs, Replicate
Data	Pandas, SQL, DuckDB, Yahoo Finance
Web	Firecrawl, Crawl4ai, Browser automation

Tools are lazy-loaded and credential-managed, so an agent only loads what it needs.

7. Teams: agents that collaborate #

Single agents are useful, but sometimes you need a team. MindRoom supports two collaboration modes:

Coordinate mode: A lead agent orchestrates others. You ask a question, the lead delegates subtasks, collects results, and synthesizes a unified response.

Collaborate mode: All agents work on the same task in parallel, each providing their independent analysis. The system then merges their responses with a consensus summary.

In practice, you might have a research team where one agent searches academic papers, another checks industry news, and a third validates claims—all triggered by a single message. The live room view below shows a real team response in the same Matrix thread surface normal agents use: threads, routed agents, and room history are all first-class chat objects rather than a separate dashboard.

Live Lobby thread in chat.mindroom.chat showing a real Team Response from HomeAssistant and AgentBuilder. Team collaboration uses the same Matrix-native interface as regular agent chats, with routed agents and room membership visible beside the conversation.

8. Hot-reload: change config and plugins without downtime #

config.yaml is watched at runtime. When you edit it—add an agent, change a model, update instructions—MindRoom diffs the old and new config, gracefully restarts only the affected agents, and has them rejoin their rooms. No downtime, no full runtime restart required.

The same development loop applies to plugins. MindRoom has a rich Python hook system, and plugins can be live-developed while the system is running. Change a plugin file, save it, and MindRoom automatically reloads the plugin without restarting the Python runtime.

This sounds minor, but when you’re iterating on agent behavior or plugin hooks, being able to tweak code and config and see results in seconds significantly improves the development loop.

9. Voice, scheduling, and other features #

Some features I built purely because I wanted them for myself (a recurring theme in my projects, as anyone who’s read my local AI journey can attest). I even built Matty, a Matrix CLI client, from bed at midnight because I needed a way to interact with my agents from the terminal:

Voice messages: Audio messages in Matrix are auto-transcribed via Whisper and treated as regular text input. You can talk to your agents.
Scheduled tasks: Natural language scheduling (!schedule "check my email every morning at 9 AM") backed by cron jobs. Agents can run tasks in the background and escalate to you when needed.
DM support: Agents respond naturally in 1:1 conversations without needing mentions.
Cross-organization federation: Because Matrix is federated, two companies’ AI agents can collaborate in a shared room—something that’s hard to do on proprietary platforms.

10. The obsession, the SaaS dream, and the burnout #

I need to be honest about what actually happened with MindRoom, because “life got in the way” is a sanitized version of the story.

What really happened is that I got completely, utterly obsessed. Every single second I wasn’t working at my day job or sleeping, I was working on MindRoom. Many hundreds of hours went into it. I seriously considered quitting my job and starting an AI startup around it.

If you’ve read Armin Ronacher’s post Agent Psychosis: Are We Going Insane?, you’ll know exactly what I’m talking about. The dopamine hit from building with AI agents is incredibly real. As Armin writes: “You feel productive, you feel like everything is amazing, and if you hang out just with people that are into that stuff too, without any checks, you go deeper and deeper into the belief that this all makes perfect sense.”

That was me. I was building and building, shipping feature after feature, and it felt incredible. The codebase grew to over 1,000 commits. I built the core system, then a React dashboard, then I started on a full SaaS platform—Kubernetes deployments on Hetzner Cloud, a FastAPI backend, a Next.js 15 frontend, Stripe integration, Supabase auth, Helm charts for multi-tenant isolation.

And that’s where the enthusiasm started to erode. Not because the core idea was bad, but because the work shifted from building interesting things (agent orchestration, memory systems, routing intelligence) to grinding through SaaS boilerplate: GDPR compliance, payment processing, automated Kubernetes deployments, terms of service, cookie banners. The dopamine loop that kept me going at 2 AM broke when the work stopped being creative and started being compliance paperwork.

The hosted control plane lives at app.mindroom.chat. I am deliberately not turning this section into a dashboard tour; the interesting part, and the part I still care about most, is the Matrix-native runtime underneath it.

11. OpenClaw and why it matters #

OpenClaw takes a different approach to the same core problem. Where MindRoom builds on Matrix federation as the backbone, OpenClaw runs a local Gateway on your machine and connects to messaging platforms directly (WhatsApp via Baileys, Telegram via grammY, Discord via discord.js, and so on).

The similarities are striking:

Both give you a single AI assistant across all your messaging platforms
Both have persistent memory
Both support multiple tools and integrations
Both are open-source and privacy-focused
Both let you choose your own AI model

The differences are interesting too. OpenClaw is local-first and TypeScript-based with 380K+ stars and a massive community. MindRoom is Python-based, federation-first, and… well, let’s just say my GitHub star count is a few orders of magnitude lower.

But seeing OpenClaw validate the core idea—that people want AI assistants that aren’t trapped in apps—reminded me that what I’d been building matters. The federation angle is something OpenClaw doesn’t have: the ability for agents from different organizations to collaborate natively, with end-to-end encryption (Olm/Megolm), on a protocol that governments already trust and deploy at scale.

MindRoom uses Matrix as the federated backbone; OpenClaw takes a local-gateway, direct-connector approach.

And then there’s Moltbook—a social network built exclusively for AI agents that the internet was losing its mind over for a couple of days. I’m not personally convinced it’s the next big thing, but what caught my attention is that MindRoom could support something like it natively. Matrix is already federated—agents on different servers can already interact, join shared rooms, and collaborate across organizational boundaries. A bridge from a Reddit-like platform to Matrix and your agents could participate without any special integration.

Building agent-to-agent infrastructure from scratch also means building security from scratch—and that’s where new platforms tend to struggle. Matrix has spent years hardening its E2E encryption, and it’s already deployed by governments and healthcare organizations at scale. Building on that foundation means you inherit those security properties rather than hoping to get them right yourself.

12. What’s next #

I first drafted this post a couple of months ago, when MindRoom was still something I was trying to come back to. That part is already out of date. Since January I’ve been spending nearly every free hour on it again, and since March it has effectively been my full-time work. The codebase supports 8+ AI model providers—OpenAI, Anthropic, Ollama, Groq, Google, OpenRouter, DeepSeek, Cerebras—and the core architecture is solid. It also has real users now. My rough guess is that something like 30-100 people have tried MindRoom in one form or another, which is still early, but meaningfully different from “only I have touched this.” There’s already a Docker Compose file and a starter repository to get started, but I want to make it even simpler.

What I’m focusing on now:

Skills system: Building an ecosystem of reusable agent behaviors (already partially implemented with OpenClaw-compatible format)
Even easier onboarding: A docker compose up gets you running today, but I want a wizard that gets you from zero to working agents without touching a YAML file
Community: Turning the small group of early users into a real operator and contributor community
Scope discipline: Keeping the work sustainable now that MindRoom is effectively full-time, not slipping back into the obsessive 2 AM marathon from before

I’m deliberately leaving the SaaS ambitions aside for now. The core system is what matters, and that’s where the energy is going.

If you’re interested in AI agents that live in Matrix and work everywhere, check out MindRoom on GitHub.

Sometimes seeing someone else succeed with a similar idea is enough to remember why you started building in the first place. And sometimes the most important lesson from a project isn’t technical—it’s learning when to step away and when to come back.

Are you running AI agents on any messaging platforms? Have you tried Matrix for anything beyond regular chat? I’d love to hear about your setups!

Links and resources #

tuitorial: terminal-based interactive code tutorials 📚

Sat, 04 Jan 2025 00:00:00 +0000

While trying to demonstrate my pipefunc package (a tool for automating computational pipelines), I hit a common presentation problem: how do you effectively walk through code without endless repetition?

Traditional presentation tools like PowerPoint force you to copy-paste the same code snippets multiple times just to highlight different parts. Jupyter notebooks are great for self-paced learning but clunky for live presentations (too much text that the viewer will try to read). I needed something that would let me define code once and create multiple views highlighting different aspects.

So, during the holiday season, I did what any reasonable developer would do: procrastinated on my actual work and built a new tool instead!

What is tuitorial? #

tuitorial is a Python package that lets you create interactive, terminal-based code tutorials where you define the code ONCE, then create steps that highlight different parts based on rules (like regex patterns or line numbers). No more copy-paste presentations!

Key Features #

Write code once: Define your code once, then create multiple views highlighting different aspects
Rich highlighting options: From simple literal matches to complex regex patterns
Interactive navigation: Intuitive keyboard controls for stepping through tutorials
Beautiful terminal UI: Powered by Textual for a polished look
Flexible configuration: Choose between Python API or YAML format
Development mode: Live reloading for quick iterations

See It in Action #

Here’s a demo of tuitorial explaining pipefunc - a package that lets you create computational pipelines with automatic parallelization, caching, and dependency management:

Who Should Use This? #

While tuitorial isn’t for everyone, it’s perfect for:

Developers giving technical presentations
Teachers explaining code concepts
Teams creating interactive documentation
Anyone who loves terminal-based tools

Getting Started #

pip install tuitorial

Check out the documentation for examples and detailed usage instructions.

Final Thoughts #

Building tuitorial was a lot of fun! While it may not replace PowerPoint for your next business presentation, it offers a unique approach to code demonstrations that I hope others will find useful.

Try it out and let me know what you think!

Home Assistant Ecosystem 🏠

Tue, 24 Dec 2024 00:00:00 +0000

(See my intro post)

There are not enough days in this advent calendar to highlight all my projects. So today is a collection of IoT/Home Assistant-related packages. Each has its own story.

📖 Origin Stories & Technical Details #

My Home Assistant obsession began as described in Day 10. Many projects emerged from this hobby:

aiokef: To control my KEF speakers, I reverse-engineered network traffic and decoded raw byte sequences, diving deep into proprietary protocols to “speak” the speaker’s language.
miflora: When @ChristianKuehnel couldn’t maintain this plant sensor library anymore, he transferred it to me. A perfect example of open source collaboration, it continues to help users monitor plant health and demonstrates how projects can thrive through community handovers.
addon-otmonitor: This HA add-on packages OpenTherm Monitor for the OpenTherm Gateway (OTGW). Using Docker to simplify deployment, it’s become a powerful open-source alternative to Nest or Ecobee, giving full control over heating systems.
home-assistant-macbook-touch-bar: Remember that useless touch bar on the old MacBook? I transformed it into a mini Home Assistant control panel for lights, temperature, and more - a classic case of hardware hacking giving new life to abandoned technology.
python-kasa: To modernize the popular pyHS100 library for TP-Link devices, I converted it to use asyncio. With @rytilahti, we created the python-kasa organization through community collaboration. While I haven’t contributed much since its inception, it’s now the standard for controlling Kasa devices.

📊 Impact & Lessons #

These projects serve thousands of users while teaching valuable lessons:

Open source empowers us: from KEF speaker integration to plant monitoring
Reverse engineering unlocks potential: turning proprietary protocols into open solutions
Community drives success: projects like miflora and python-kasa thrive through collaboration
Innovation through repurposing: giving new life to hardware like the MacBook Touch Bar
Modern tech matters: Docker simplifies deployment, async code boosts performance

Want to explore these projects? Check them out on GitHub:

This concludes our Advent of Open Source! I hope you’ve enjoyed this journey through my projects as much as I’ve enjoyed sharing them. Happy holidays and happy coding!

#OpenSource #HomeAssistant #IoT #Python #SmartHome

pfapack 🔢

Mon, 23 Dec 2024 00:00:00 +0000

(See my intro post)

Ever needed to calculate the square root of the determinant of a skew-symmetric matrix? Probably not, but if you’re in quantum physics, you might have! This is called the Pfaffian. Today’s project, pfapack, is a Python package that provides efficient routines for computing it. The underlying algorithms and original code implementations were developed by my PhD co-promotor Michael Wimmer. I created the pfapack package to make this powerful tool easily accessible.

📖 Origin Story #

Working in quantum physics, I frequently need to compute the Pfaffian. Michael had not only developed the algorithms but also wrote highly optimized C, FORTRAN, and Python implementations, distributing them as a tar file on his website. To make it easier for researchers working in Python, I packaged his existing Python code, added wrappers for his C and FORTRAN implementations, and set up automated builds for PyPI and conda-forge to make it easily installable.

🔧 Technical Highlights #

Python Wrapper for Efficient Code: User-friendly Python interface to Wimmer’s optimized C and FORTRAN.
Multiple Algorithms: Supports both the Parlett-Reid and Householder methods.
Optimized for Performance: Leverages low-level optimizations for speed.
Conda and Pip Installable: Easy installation via conda-forge and PyPI.
Automated Wheel Builds: Uses pypa/cibuildwheel for streamlined, automated builds and releases.
Cross-Platform Compatibility: Works on Linux, macOS, and Windows (with some caveats).

📊 Impact #

Makes specialized mathematical algorithms readily available to Python users.
Simplifies the use of highly optimized code.
15 GitHub stars.
Used in research projects that require efficient Pfaffian computations, particularly in quantum physics.

🎯 Challenges and Solutions #

Bridging Languages: Wrapping C and FORTRAN for Python integration.
Automated Builds: Setting up automated builds for multiple platforms was complex.
Windows Compatibility: As always Windows support is a PITA; it currently requires MSYS2 and MinGW64.

💡 Lessons Learned #

Making specialized tools accessible broadens their impact.
Wrapping low-level code in Python can improve usability.
Automated builds are essential for maintainability.
pypa/cibuildwheel is awesome for cross-platform builds.

🙏 Credits #

All the credits for the underlying algorithms and original code implementations go to Michael Wimmer. I simply made it easier to distribute and use in Python.

Want to compute Pfaffians efficiently in Python? Check out pfapack on GitHub or install it via pip install pfapack or conda install -c conda-forge pfapack!

#OpenSource #Python #ScientificComputing #Math #Pfaffian #FORTRAN #C

tiller-streamlit 💰

Sun, 22 Dec 2024 00:00:00 +0000

(See my intro post)

Tracking personal finances is crucial, especially when working towards financial independence (see my WenFIRE post on Day 4). Today’s project is tiller-streamlit, a Streamlit app that visualizes your financial data from Tiller. I host it on my local network under money.local using Traefik for easy access.

📖 Origin Story #

Used to use Intuit Mint, but when they shut down, I needed an alternative with full data control.
Tiller automatically populates your financial data into a Google Sheet that you own forever.
With 20+ accounts, manual updates are tedious – Tiller automates it.
Combined Tiller’s data with Streamlit’s ease of use, creating a personal finance dashboard.

🔧 Technical Highlights #

Powered by Streamlit: Turns Python into a web app with minimal code.
Tiller Integration: Connects to Tiller’s Google Sheets for automated data retrieval.
Interactive Visualizations: Uses Plotly and Altair for dynamic charts.
Customizable Analysis: Filter by categories, time periods, and more.
Net Worth Tracking: Calculates and visualizes net worth over time.
Dockerized Deployment: Easy deployment with Docker, and served locally with Traefik.

📊 Impact #

Primarily for personal use, it has significantly improved my understanding of my finances.
Great way to learn Streamlit and explore its capabilities.
Visualizing spending patterns helps identify areas to cut back and make informed financial decisions on the path to financial independence.

🎯 Challenges and Solutions #

Learning Streamlit: Excellent documentation and community support made it a smooth process.
Data Wrangling: Transforming Tiller’s raw data into a format suitable for visualization.
Deployment: Docker and Traefik simplified local deployment, making it accessible on my home network.

💡 Lessons Learned #

Data Ownership Matters: Owning your financial data gives you control and flexibility.
Visualization is Key: Seeing your finances visually makes a huge difference.
Streamlit is Powerful: Simplifies web app development.
Automation is Essential: Tiller’s automatic data population makes the whole process sustainable.
Docker and Traefik Simplify Local Deployment: Making it easy to access the app on my home network.

🔮 Future Plans #

Budgeting Features: Integrate budget tracking and forecasting.
Goal Setting: Set financial goals and track progress.
Investment Tracking: Incorporate investment data for a complete picture.

Want to visualize your own finances with Tiller and Streamlit? Check out tiller-streamlit on GitHub!

#OpenSource #Python #Streamlit #PersonalFinance #DataVisualization #Tiller #Traefik #Docker

Interactive Graph Visualization 🕸️

Sat, 21 Dec 2024 00:00:00 +0000

(See my intro post)

After creating pipefunc (Day 20), I wanted a way to visualize its (possibly large and complex) computational graphs interactively in Jupyter notebooks. What started as a simple visualization PR evolved into two standalone packages: graphviz-anywidget (Python) and graphvizsvg (JavaScript) - which unexpectedly introduced me to JavaScript development!

📖 Origin Story #

While working on pipefunc’s visualization features, I discovered that existing solutions didn’t quite meet my needs. However, I found several components that would solve my issue:

AnyWidget (interactive GraphViz in Jupyter)
@hpcc-js/wasm-graphviz (GraphViz in WASM)
d3-graphviz (interactive GraphViz in D3)
jquery.graphviz.svg (GraphViz SVG manipulation)

AnyWidget would work with ESM modules, however, I had no clue what that even was. The jquery.graphviz.svg package wasn’t one and hadn’t been maintained in 8 years.

With AI as my JavaScript mentor, I first rewrote jquery.graphviz.svg into a modern ESM module (graphvizsvg), adding comprehensive tests that uncovered bugs in the original. Then, using AnyWidget, I created graphviz-anywidget to connect everything together - combining the WASM-powered d3-graphviz with the interactive features from graphvizsvg into a seamless Python package.

To my surprise, I found JavaScript’s development tooling remarkably sophisticated - especially npm, which handles everything from linting to testing to bundling to publishing. Something we’re still missing in Python!

🔧 Technical Highlights #

graphviz-anywidget:

Interactive Graphviz visualization in Jupyter
WASM-powered graph rendering
Directional graph traversal
Node and edge highlighting
Works in JupyterLab, Notebook, and VS Code

graphvizsvg:

Modern ESM module for interactive SVG visualization
Automatic color transitions
Bi-directional graph traversal
Comprehensive test coverage
My first npm package!

📊 Impact #

Enhances pipefunc’s visualization capabilities
Modernizes legacy jQuery code for modern web development
Proves that diving into new technologies can yield useful tools

🎯 Challenges and Solutions #

Learning JavaScript/TypeScript from scratch
Making WASM work in Jupyter
Getting it to work in VS Code

💡 Lessons Learned #

Don’t let preconceptions limit your tech choices
Modern JavaScript development is actually enjoyable
AI can be an excellent programming mentor
Good visualization makes complex data accessible
Breaking features into separate packages increases reusability
Test coverage helps find bugs in even well-used code

Want to visualize graphs interactively? Check out graphviz-anywidget and graphvizsvg!

#OpenSource #JavaScript #Python #DataVisualization #WebDev

pipefunc 🕸️

Fri, 20 Dec 2024 00:00:00 +0000

(See my intro post)

Of all the projects I’m sharing this month, this one that started as a passion project excites me the most!

📖 Origin Story #

Sometimes the best projects are born from the most unexpected moments. More than 1½ year ago, while on parental leave with my twin boys, I found myself with small pockets of time during naps to work on a problem that had been bothering me for years: the tedious bookkeeping required in complex computational workflows. Every scientific computation project seemed to reinvent the same patterns - managing function dependencies, parameter sweeps, result caching, parallelization, and a lot of boilerplate to combine the resulting data. I wanted something that would let scientists focus on their science, not on pipeline management.

Example of a pipeline generated by PipeFunc

🔧 Technical Highlights #

Automatic DAG construction via very simple and lightweight syntax
N-dimensional parameter sweeps with automatic parallelization
Visual pipeline representation using NetworkX
Resource profiling (CPU, memory, time)
Type validation between pipeline stages
Ultra-fast: only 15 µs overhead per function
Flexible caching strategies (memory, disk, cloud)
Integration with scientific computing tools:
- NetworkX for graph algorithms
- NumPy for multi-dimensional arrays
- Xarray for labeled arrays
- Zarr for distributed storage
- Adaptive for parallel computing

📊 Impact #

230 GitHub stars
700+ tests with 100% coverage
Fully typed codebase
Comprehensive documentation
Tested on real workflow on SLURM cluster
Covered on Pycoder’s Weekly with >100k subscribers

🎯 Challenges and Solutions #

Balancing simplicity with power
Making complex workflows intuitive
Handling distributed computing edge cases
Ensuring type safety across the pipeline
Optimizing performance without sacrificing features

💡 Lessons Learned #

Sometimes the best time to code is during baby naps
Complex problems can have elegant solutions
Scientific computing needs better tooling
Good abstractions make hard things easy
Type hints and tests prevent headaches

🔮 Future Plans #

The journey is far from over. Plans include:

Enhanced cloud computing support
More interactive visualization options
Interactive pipeline debugging tools
Expanded parameter sweep capabilities

Want to simplify your computational workflows? Check out pipefunc on GitHub or read the documentation!

#OpenSource #Python #DataScience #ScientificComputing #Programming

fileup 📤

Thu, 19 Dec 2024 00:00:00 +0000

(See my intro post)

Sometimes the best tools are the ones you build for yourself and keep using for years. Today’s project is fileup, a tiny command-line utility that’s been making my life easier for 7 years - because sharing files shouldn’t require leaving the terminal!

📖 Origin Story #

Back in 2017, I found myself repeatedly going through the same tedious process: needing to share a file, opening a browser, navigating to a file sharing service, waiting for uploads, and copying URLs. As someone who lives in the terminal, this felt inefficient. I wanted something as simple as fu filename that would handle everything and put the URL in my clipboard. Seven years and countless file shares later, it’s still one of my most-used tools.

🔧 Technical Highlights #

Supports both FTP and SCP uploads
Smart URL generation:
- Jupyter notebooks → nbviewer links
- Images → markdown-formatted links
- Regular files → direct URLs
Automatic file expiration and cleanup
Zero dependencies - uses only Python standard library
Clipboard integration (macOS)
Simple INI configuration
Works as both a standalone script or installed package

📊 Impact #

This is one of those tools where GitHub stars (currently 0!) don’t tell the whole story:

Used almost daily for 7 years
Shared thousands of files
Saved countless minutes of browser interactions
Proves that utility trumps popularity in open source

💡 Lessons Learned #

Longevity comes from solving a real, persistent need
A tool doesn’t need to be revolutionary to be useful
Zero dependencies mean zero maintenance headaches
Sometimes the impact of a tool isn’t measured in stars
Simple tools that remove friction are worth maintaining

Want to share files from your terminal? Check out fileup on GitHub!

#OpenSource #Python #DevTools #CommandLine #Programming

Psychedelic Data Science 🍄

Wed, 18 Dec 2024 00:00:00 +0000

(See my intro post)

📝 Note: Running out of awesome projects and I want to save the best for last, so today just something fun.

While preparing for this advent calendar, I browsed through my 382 GitHub repositories and rediscovered this time capsule from 8 years ago. It explores the language of psychedelic experiences through data science - with an interesting hypothesis: do psychedelic experiences generate richer vocabulary compared to other substance reports?

📖 Origin Story #

Back in 2015, when I was still “young” (before back pain and two-day hangovers became a thing), I was learning data science and natural language processing. I had an amusing hypothesis: surely people describing their psychedelic experiences would use richer vocabulary than those writing about stimulants - I mean, who hasn’t pondered the linguistic complexity of “everything is connected” versus “I cleaned my entire apartment at 4 AM”? Erowid.org, with its thousands of detailed first-person narratives across different substances, provided the perfect dataset to test this theory.

🔧 Technical Highlights #

Natural language processing of experience reports
Vocabulary richness analysis across substance categories
TF-IDF vectorization for text analysis
Support Vector Machine classification of experience types
Web scraping with BeautifulSoup
K-means clustering for discovering common themes
Word cloud generation to visualize vocabulary differences

📊 Impact #

A learning project that helped understand:
- How different experiences shape language use
- Processing subjective experience narratives
- Document classification techniques
- The challenges of quantifying vocabulary richness
10 GitHub stars

🎯 Challenges and Solutions #

Implementing respectful web scraping
Controlling for report length and education level
Creating meaningful metrics for vocabulary richness
Visualizing linguistic patterns across categories

💡 Lessons Learned #

Putting things on GitHub means it’s less likely forgotten things get lost
Early data science projects often reveal our initial fascinations
Web scraping requires both technical skill and ethical consideration
Text analysis tools have evolved dramatically since 2015
Sometimes the most interesting projects are the ones you almost forgot about

Want to explore this intersection of psychedelics, language, and data science? Check out the project on GitHub!

#OpenSource #Python #DataScience #NLP #MachineLearning

Adaptive Scheduler 🚀

Tue, 17 Dec 2024 00:00:00 +0000

(See my intro post)

Ever tried to efficiently run 100,000 cores from a Jupyter notebook? No? Well, I have, and it led to some interesting discoveries about the limits of parallel computing tools.

📖 Origin Story #

During my 2018 internship at Microsoft Quantum, we faced a unique challenge: our adaptive sampling algorithms needed to run on massive computing clusters (50,000+ cores), but existing tools couldn’t handle it efficiently.

The problem was interesting: traditional parallel computing tools like Dask rely on a central scheduler process. Imagine 100,000 cores, each finishing a task every 10 seconds. That means every 10 microseconds, a core needs new work! With typical scheduler overhead being 1-50 milliseconds, workers would spend 99% of their time waiting rather than computing.

Usually, you’d solve this by batching work ahead of time. But with adaptive sampling (yesterday’s project), that’s impossible - you need the results of previous calculations to know what to compute next. It was like trying to conduct an orchestra where each musician needs personal instructions every few seconds!

🔧 Technical Highlights #

We created a solution that:

Acts as a meta-scheduler, launching multiple sub-schedulers (Dask, ipyparallel, mpi4py, etc.)
Minimizes communication between scheduling layers
Handles job management through a simple database
Supports automatic fault tolerance and data saving
Works with any cluster scheduler (SLURM, PBS)
Provides real-time progress tracking in Jupyter notebooks
Scales to 50,000+ cores without breaking a sweat

📊 Impact #

Used extensively for real quantum device simulations
26 GitHub stars (but don’t let that fool you - it’s highly specialized!)
Enables interactive supercomputing from a simple Jupyter notebook
Likely ran for many millions of core-hours across various projects
Actively maintained with 61 releases since 2018, proving its ongoing value

🎯 Challenges We Solved #

Scheduler overhead killing performance at scale
Data locality and fault tolerance
Real-time monitoring of massive parallel jobs
Making supercomputing accessible through notebooks
Automatic job recovery after crashes (e.g., when using spot VMs or Heisenbugs)

💡 Lessons Learned #

Sometimes you need to think outside the box
The best tools hide complexity behind simple interfaces
Real-time feedback is crucial, even for massive computations
When scaling up 1000x, you often need a completely different approach
Making things “just work” from a notebook is worth the effort

Want to run massive parallel computations efficiently? Check out Adaptive Scheduler on GitHub!

#OpenSource #Python #HPC #QuantumComputing #ParallelComputing

Adaptive 📈

Mon, 16 Dec 2024 00:00:00 +0000

(See my intro post)

During my PhD, I was running quantum transport simulations on our 800-core computational cluster. Traditional parameter sweeps using dense grids were hitting the cluster’s limits - wasting precious compute time on uninteresting regions while potentially missing important features. Most parameter spaces we explored were “boring” with occasional interesting regions that required higher sampling density.

📖 Origin Story #

Initially, in 2016, I had a collection of one-off adaptive sampling scripts. Together with Joseph Weston and the support of my PhD advisor Anton Akhmerov, we turned it into a full-fledged Python package “Adaptive”. What began as an internal tool to make our cluster usage more efficient slowly gained traction in the wider scientific computing community.

🔧 Technical Highlights #

Multiple adaptive sampling strategies (1D, 2D, ND, averaging stochastic functions)
Parallel execution support (local, MPI clusters, distributed systems)
Live plotting in Jupyter notebooks with real-time feedback
Customizable loss functions for different sampling priorities
Minimal dependencies

📊 Impact & Applications #

1164 GitHub stars and thousands of daily downloads
Cited in 20+ scientific publications
Dependency of many packages, most notably: Orange Quantum System’s Quantify software for controlling real quantum computers!

Used in cutting-edge research across fields:

Quantum Computing: Simulating and controlling superconducting qubits and trapped ion systems
Quantum Physics: Exploring topological quantum systems and novel materials
Materials Science: Studying graphene and other 2D materials
Engineering:
- Optimizing radar system performance
- Fluid dynamics simulations
Chemistry: Molecular modeling and vibrational spectroscopy
Most Unexpected Use: Found in a paper on predicting energy usage of wartime bungalows!

This range of applications shows how a tool built for quantum transport simulations can find surprising uses across disciplines, from fundamental physics to everyday buildings.

🎯 Challenges and Solutions #

Making parallel execution truly scalable while maintaining sampling intelligence
Creating intuitive visualization tools for complex parameter spaces
Handling stochastic functions and noisy data

💡 Lessons Learned #

Building a user base takes active effort - giving talks and engaging online made all the difference
Good documentation converts interested users into regular contributors
This was my first open source package and learned all about packaging, testing, and CI/CD

Want smarter parameter space exploration? Check out Adaptive on GitHub or try our interactive tutorial!

#OpenSource #Python #ScientificComputing #ParallelComputing #Programming

clip-files 📋

Sun, 15 Dec 2024 00:00:00 +0000

(See my intro post)

Sometimes the most useful tools are born from daily frustrations. When working with AI assistants like ChatGPT or Claude, I found myself constantly copying files to share context. After doing this manually dozens of times, I created a tool to automate it.

📖 Origin Story #

While contributing to open source, I frequently like to share entire (or large parts of) codebases with AI assistants for context or code review. Since the code is open source anyway, there’s no reason not to share the full context! But manually copying files, adding their paths for context, and ensuring I didn’t exceed token limits was tedious. Instead of continuing this manual process, I spent an evening creating clip-files - and have used it multiple times daily ever since.

In fact, I’m using it right now! The command (shown below) lets me share this entire advent calendar series with AI assistants for proofreading.

clip-files --initial-file ai.md --files introduction.md day*.md

🔧 Technical Highlights #

Uses tiktoken for accurate GPT token counting
Supports both directory traversal and specific file selection
Automatically generates a file index for reference
Allows custom initial instructions via file
Preserves file paths in output for context
Clean, typed, and documented codebase

📊 Impact #

While this tool has no GitHub stars (I haven’t promoted it), its real impact is personal - I use it multiple times daily for:

Sharing entire open source codebases with AI assistants
Getting comprehensive code reviews
Discussing bug fixes with full context
Documentation improvements
Open source contributions

The tool’s real value isn’t in its complexity (it’s just ~250 lines of code) but in how it removes friction from open source development. When working with open code, we might as well share all the context we can - and this tool makes that effortless.

🎯 Challenges and Solutions #

Maintaining consistent formatting across different input methods
Calculating accurate token counts for different components
Providing meaningful error messages for invalid inputs
Structuring output to be AI-assistant friendly

💡 Lessons Learned #

When working with open source code, sharing full context is valuable
Simple command-line tools can significantly improve daily workflows
Consistent formatting matters when working with AI assistants
Sometimes a 250-line tool can save hours of manual work

Want to make sharing code with AI assistants easier? Check out clip-files on GitHub!

#OpenSource #Python #DevTools #AI #Programming

PhD Thesis Cover Design 🎨

Sat, 14 Dec 2024 00:00:00 +0000

(See my intro post)

When most people think of open source, they think of software. But what about open-sourcing your PhD thesis and creating a generative art system for its cover?

📖 Origin Story #

With my PhD defense approaching and needing around 100 copies of my thesis, I faced a design challenge. As someone more comfortable with Python than Photoshop, I turned what started as a simple visualization idea into a full-blown generative art project with machine learning-based clustering.

Multiple printed copies of the thesis

🔧 Technical Highlights #

Generated 3,671 unique covers from quantum physics simulation data
Used VGG16 neural network to extract features from covers
Implemented K-means clustering to group similar designs
Created interactive IPywidgets for design selection
Built a Binder app for real-time color scheme adjustment
Added unique edition numbers (1/120, 2/120, etc.)
Made both thesis and cover design code fully open source

The project combines:

Adaptive sampling for physics simulations
Neural networks for image feature extraction
Interactive visualization tools
Parallel processing for batch generation
Version control for scientific writing

📊 Impact #

Thesis repository: 14 GitHub stars
Cover design repository: 36 GitHub stars
Real-world adoption: Dr. Tomas Pippia from TU Delft used the code to create his own thesis cover
~100 unique printed copies, each with its own generated cover and edition number

🎯 Challenges and Solutions #

Managing 3,671 unique PDF files and their conversions
Creating a clustering system for visually similar designs
Building interactive tools for non-technical collaborators
Ensuring consistent print quality across designs
Balancing artistic appeal with scientific relevance

💡 Lessons Learned #

Open source isn’t just about code - it’s about sharing knowledge
Python can be a powerful tool for creative projects
Machine learning can help solve unexpected problems
Good visualization requires both technical skill and artistic sense
Interactive tools make collaboration with non-programmers easier

📚 Fun Fact #

Due to COVID restrictions, I defended my thesis without any guests present. The silver lining? I still have about 100 beautifully unique copies at home, each with a different generated cover and edition number. If you’d like a physical copy of this piece of quantum physics meets generative art, just let me know!

Want to create your own thesis cover or see how I wrote mine? Check out the thesis cover project and thesis source!

#OpenSource #PhD #Python #MachineLearning #DataVisualization

UniDep 🧬

Fri, 13 Dec 2024 00:00:00 +0000

(See my intro post)

Ever tried to set up a Python project that needs both pip and conda packages? Then you know the pain of maintaining multiple requirement files. UniDep was born from years of frustration with this exact problem.

📖 Origin Story #

While working on large monorepos with hundreds of dependencies, I kept running into the same issue: maintaining separate files for conda (environment.yml), pip (requirements.txt), and Python packaging (pyproject.toml). Coming from languages like Rust, where package management is a joy, Python’s fragmented ecosystem can feel overwhelming, especially to newcomers (obligatory xkcd.com/1987).

What started as a simple tool to avoid duplicate dependency definitions turned into something much more powerful. In enterprise environments, it’s common to find monorepos with over 1000 dependencies spread across dozens of packages. Setting up these environments traditionally involves following lengthy setup guides, installing packages in the correct order, and hoping nothing breaks.

Then came the “aha” moment - what if we could install everything with a single command? After implementing UniDep, those complex setup guides became a one-liner: unidep install-all -e. It installs all dependencies and sets up every package in the monorepo in editable mode. What used to take hours now takes minutes.

The difficulty of managing pip and conda

🔧 Technical Highlights #

Single source of truth for all dependencies
Supports both requirements.yaml and pyproject.toml
Handles platform-specific dependencies
Generates conda-lock files both for entire monorepos and fully consistent lock files for its individual packages
Works with pip, conda, and mamba
Monorepo-friendly with local dependency support
Integrates with setuptools and hatchling
≥99% test coverage, fully-typed

📊 Impact #

221 GitHub stars
Featured on Python Bytes podcast
Saves countless hours of dependency management
Makes complex project setups accessible
Actively maintained with 70 releases

🎯 Challenges and Solutions #

Different package naming between pip and conda
Platform-specific dependencies
Cross-platform compatibility (Linux, Windows, MacOS)
Supporting pip-only workflows

💡 Lessons Learned #

Good developer experience is worth the effort
Sometimes you need to build a bridge between existing tools
The best tools make hard things easy
Dependency resolution is hard but can be simple for users

🔮 Future Plans #

Adding Pixi support - allowing seamless migration to Pixi without changing config files
Eventually, Pixi might make UniDep obsolete (and that’s okay!)
For now, UniDep bridges the gap between pip and conda, making life easier for developers

Want simpler dependency management? Check out UniDep on GitHub!

#OpenSource #Python #DevTools #Programming #PackageManagement #Conda

instacron 📸

Thu, 12 Dec 2024 00:00:00 +0000

(See my intro post)

Ever found yourself with beautiful photos but too lazy to post them? Instead of changing my habits, I did what any developer would do: spent way more time automating the process than it would have taken to post manually.

📖 Origin Story #

Back in 2017, I had hundreds of landscape photos from traveling through South America that I wanted to share on Instagram. The solution? Write a Python script that would randomly select photos and post them automatically, complete with philosophical quotes randomly selected from Bukowski, Einstein, or Dostoevsky because… it seemed to be the default of all “influencers” at the time. Nothing says “authentic travel experience” like an auto-generated Bukowski quote about drinking next to your serene mountain photo!

Instacron Example Post

🔧 Technical Highlights #

Randomly selects photos from a directory
Adds location data from EXIF metadata
Generates captions with random philosophical quotes
Adds two random emojis to match the “mood”
Automatically generates relevant hashtags
Tracks posting frequency
Uses cron for scheduled posting

📊 Impact #

21 GitHub stars
4143 Instagram followers
Successfully posted hundreds of photos
Generated some concerned DMs (more on that below!)

🎯 Challenges and Solutions #

Instagram’s ever-changing API
Handling rate limits and restrictions
Ensuring photos weren’t posted multiple times
Making captions interesting without human input
The delicate art of random emoji selection

😅 Funny Story #

The random quote and emoji selector once created an unexpectedly dramatic post. Picture this: a beautiful landscape photo paired with a deeply philosophical quote about solitude, and topped with a broken heart emoji (💔). Within hours, I started receiving concerned DMs from friends asking if everything was okay and whether my relationship had ended!

I had to explain that no, I wasn’t going through a breakup - my bot just had a flair for the dramatic! These occasional mismatches made the whole thing more entertaining.

💡 Lessons Learned #

Sometimes the best projects come from pure laziness
Automation doesn’t always save time, but it’s fun
Never underestimate the power of a misplaced emoji to trigger concerned messages
Some followers actually care about you (even if they’re worried about your bot-generated content)

While I haven’t used this project in 6 years (Instagram’s API changes eventually won), it remains a testament to the lengths we’ll go to avoid repetitive tasks. Check out instacron on GitHub!

#OpenSource #Python #Automation #Photography #Instagram

conda-forge Contributions 🐍

Wed, 11 Dec 2024 00:00:00 +0000

(See my intro post)

Unlike other projects in this advent calendar, I want to celebrate being part of something much bigger: the conda-forge community, where thousands of volunteers maintain the packages that power scientific Python.

📖 Origin Story #

I started with conda-forge in 2016 during my PhD work with Kwant, a quantum transport simulator. Installing it was a nightmare - it required system-level numerical libraries like MUMPS and Scotch which required a special config file to link. When I discovered conda-forge and its mission to make scientific software installation painless, I knew this was the solution!

The humble work of conda-forge maintainers

🔧 Technical Highlights #

Over the past 8 years, I’ve maintained over 40 build recipes, including:

Major scientific packages like VTK, MUMPS, and Scotch
Quantum physics tools like Qsim, Qcodes, and Kwant
Visualization libraries like HoloViews
Plus all the packages I’ve authored myself

The real magic of conda-forge is its infrastructure:

Automated builds across Linux, macOS, and Windows
Strict dependency version management
Comprehensive CI/CD pipelines
Community-driven quality control

📊 Impact #

What makes conda-forge truly special is how initial contributions blossom into collaborative efforts. I’ve experienced this firsthand: after creating the initial MUMPS feedstock, 25 other contributors joined in, each bringing their unique expertise.

A recent experience also illustrates this: I added CUDA support to both Google’s Qsim and IBM’s Cirq packages. Shortly after, Leo Fang, an engineer from NVIDIA who knows CUDA far better than I do, stepped up and made significant improvements. This is the beauty of open source - experts naturally gravitate to where they can make the biggest impact.

This collaborative approach solves a problem I’ve encountered at several companies: the tendency to maintain complicated build systems internally with custom hacks. Instead of each organization reinventing the wheel, conda-forge provides a shared platform where:

Maintenance burden is distributed
Expert knowledge flows freely
Build practices are standardized
Infrastructure is public and reusable

🎯 Challenges and Solutions #

Keeping up with upstream changes
Managing complex dependency trees
Cross-platform compatibility
Coordinating with upstream maintainers

💡 Lessons Learned #

Community effort beats solo work for infrastructure
Automation is crucial for reliability
Small contributions compound over time
Being part of something bigger is rewarding

Want to contribute to scientific Python? Check out conda-forge and the staged-recipes repository!

#OpenSource #Python #Scientific #Programming #Community

Home Assistant Config 🏠

Tue, 10 Dec 2024 00:00:00 +0000

(See my intro post)

Funny how things work out - initially, I didn’t care for home automation at all. But when in 2020 I discovered Home Assistant was written in Python, my favorite programming language, I suddenly found a new hobby. Over the years, I’ve built a comprehensive setup with over 137 automations (and counting!). These automations are triggered by everything from simple motion sensors to complex patterns of conditions and sensor states, creating a home that responds intelligently to our needs.

📖 Origin Story #

What started with basic light controls evolved into a complete home automation system. In the beginning, my wife was continuously annoyed by lights turning on at the wrong moment or not at all. However, as I debugged and refined the system, things started working exactly as expected, and she began to really appreciate it. When we moved from The Netherlands to the US in 2022 (and two more times since), we felt like we were back in the stone age having to manually control lights. The annoyance had shifted from buggy automations to not having Home Assistant fully installed yet!

Unfortunately, I’ve been less active lately with my Home Assistant setup because so many other projects draw my attention. One of these “projects” is my 1.5-year-old twin boys, who now have their own room with specialized automations, including 100% local cameras with ML pattern detection for monitoring without compromising privacy.

The evolution of home automation

🔧 Technical Highlights #

137 documented automations with auto-generated documentation
Presence detection in every room
Adaptive lighting throughout the house
Privacy-focused baby monitoring
Fully local processing
2,432 commits of continuous improvement

📊 Impact #

1,747 GitHub stars
185 forks
Used as reference by countless Home Assistant enthusiasts
Featured in Home Assistant community
Spawned several spin-off projects (like Adaptive Lighting from Day 2!)

💡 Lessons Learned #

Start small, expand gradually
Privacy and local processing matter
Automated documentation stays accurate
Home automation is never “finished”
Sometimes life (like twins!) brings new automation challenges

Want to explore my home automation setup? Check out the full config on GitHub!

#OpenSource #HomeAssistant #SmartHome #IoT #Python

yaml2bib 📚

Mon, 09 Dec 2024 00:00:00 +0000

(See my intro post)

Every academic writing in LaTeX knows the pain: managing BibTeX files with hundreds of citations, each potentially formatted slightly differently, with inconsistent journal abbreviations, and duplicated entries across chapters. While procrastinating on writing my PhD thesis, I became obsessed with solving this universal academic headache.

📖 Origin Story #

When writing a PhD thesis with multiple chapters that share citations, maintaining consistency is crucial but tedious. Different chapters might cite the same paper but with slightly different BibTeX entries - one using the full journal name, another using abbreviations, or worse, having slightly different titles or author lists. Rather than manually standardizing ~350 citations, I created yaml2bib: give it a list of DOIs in YAML format, and it generates a perfect BibTeX file with consistent journal abbreviations and formatting.

The pain of manual BibTeX management

🔧 Technical Highlights #

Fetches citation data from CrossRef using DOIs
Ensures consistent journal name abbreviations
Caches API responses to avoid repeated queries
Supports custom text replacements
Combines multiple YAML sources
Works as both CLI tool and Python library
Zero manual citation formatting needed

📊 Impact #

While this tool has only 11 GitHub stars, it saved me countless hours during thesis writing and helped ensure professional consistency in my citations. It’s particularly useful for:

Large academic documents with many citations
Multi-chapter works sharing references
Collaborative papers needing citation standardization
Anyone tired of manually formatting BibTeX entries

💡 Lessons Learned #

Procrastination can be productive if channeled right (story of my life…)
A single source of truth prevents inconsistencies
Good tools make you care about details you’d otherwise skip
Every LaTeX user has fought with BibTeX at some point

Want perfectly consistent citations? Check out yaml2bib on GitHub!

#OpenSource #Python #Academia #LaTeX #Research

Home Assistant Stream Deck YAML 🎮

Sun, 08 Dec 2024 00:00:00 +0000

(See my intro post)

What started as a personal hack to control my home (temperature, lights, TV, music) with a gifted Stream Deck evolved into something bigger. This project became my first real exploration of AI-assisted development, showing me how much faster I could work with AI as a tool. With 250+ stars and 9 contributors adding significant features, it’s grown beyond my initial vision.

📖 Origin Story #

While the Stream Deck’s official software is great for streamers, it doesn’t support Linux and its Home Assistant integration is not very programmable. After creating a simple solution using python-elgato-streamdeck, I realized others might be interested. Using AI as a development partner changed my workflow - it helped write tests, reviewed code, and generated use cases I hadn’t thought of. What really surprised me was asking AI to brainstorm example configurations - it generated over 30 creative use cases with complete code examples, which I then turned into unit tests. These tests now automatically generate the documentation using markdown-code-runner (from Day 5).

🔧 Technical Highlights #

YAML configuration with Jinja2 templating
Supports all Stream Deck models, including the Plus with dials
Cross-platform (Linux, macOS, Windows)
30+ example configurations (AI-generated, then human-refined)
Documentation auto-generated from code and unit tests
Available as a Home Assistant add-on
Community-contributed features like touchscreen support and dial controls

💡 Lessons Learned #

AI can significantly speed up development
Faster development enables broader project scope
Good examples make the best documentation
Tools building upon tools multiply productivity
Community contributions can take a project in exciting new directions

Want to control Home Assistant with a Stream Deck? Check out the project on GitHub!

#OpenSource #HomeAssistant #SmartHome #Python #AI

numthreads 🧵

Sat, 07 Dec 2024 00:00:00 +0000

(See my intro post)

Sometimes the smallest tools solve the most persistent problems. Today’s project is about taming automatic parallelization in scientific computing - a deceptively simple challenge that has cost countless CPU hours.

📖 Origin Story #

While working on high-performance computing clusters, I frequently encountered a counterintuitive problem: code running slower on multiple CPU cores than on a single one. Many scientific libraries (NumPy, SciPy) automatically parallelize operations, which sounds great but can actually harm performance when you’re already parallelizing at a higher level. After repeatedly explaining to colleagues why they needed to set various environment variables to disable this behavior, I created numthreads to solve it once and for all.

Overenthusiastic threading

🔧 Technical Highlights #

Controls thread count for major numerical libraries:
- OpenBLAS
- Intel’s Math Kernel Library (MKL)
- OpenMP
- NumExpr
- Accelerate
Zero dependencies
Tiny (≤7KB) yet solves a real problem
Both CLI and Python API

📊 Impact #

Over my scientific computing career, I’ve encountered this threading issue dozens of times. Since creating this package a year ago, I’ve helped at least 5 different researchers speed up their workflows, sometimes by orders of magnitude. The beauty lies in its simplicity - the fix is often as simple as:

eval $(numthreads 1)

💡 Lessons Learned #

The simplest solutions are often the best
More cores doesn’t always mean faster code
If you find yourself repeating the same instructions, write a tool
Even a 7KB package can save hours of computing time

Want better control over your numerical computations? Check out numthreads on GitHub!

#OpenSource #Python #HPC #ScientificComputing #Programming

Open Science Publications 🔬

Fri, 06 Dec 2024 00:00:00 +0000

(See my intro post)

Today’s post is about something I have strong opinions on: making scientific research reproducible. While many researchers talk about open science, actually making your work reproducible for anyone requires significant effort.

📖 Origin Story #

During my Ph.D. and subsequent research, I noticed a frustrating pattern in scientific publications: claims of “code available upon request” that often led nowhere, or code snippets that wouldn’t run without substantial modification. I made an effort that my publications would be different – each includes complete, runnable code that reproduces every figure and result.

🔧 Technical Highlights #

Multiple repositories covering quantum physics experiments and simulations:
- orbitalfield - “Orbital effect of magnetic field on the Majorana phase diagram” (94 citations)
- supercurrent-majorana-nanowire - “Supercurrent interference in few-mode nanowire Josephson junctions” (78 citations)
- zigzag-majoranas - “Enhanced proximity effect in zigzag-shaped Majorana Josephson junctions” (56 citations)
- azure-quantum-tgp - “Protocol to identify a topological superconducting phase in a three-terminal device” (52 citations)
- spin-orbit-nanowires - “Spin-orbit protection of induced superconductivity in Majorana nanowires” (81 citations)

🔄 Evolution of Best Practices #

What I consider best practices has evolved over time. Today, I would:

Use Pixi for universal lock files across operating systems and programming languages
Provide a minimal Docker container that just installs the pixi lock file
Create self-documenting Jupyter notebooks that reproduce every result
Include clear figure-to-code mapping for paper reproducibility

🎯 Challenges and Solutions #

Balancing code cleanliness with research deadlines
Managing large datasets efficiently
Ensuring long-term reproducibility
Making complex physics simulations accessible

💡 Lessons Learned #

Clean code takes time but saves more in the long run
Documentation is as crucial as the code itself
Lock files are essential for true reproducibility
Making code public improves its quality

Want to explore quantum physics simulations? Check out the repositories above, each linked to their corresponding papers with full reproduction instructions.

#OpenSource #OpenScience #Physics #QuantumComputing #Python

markdown-code-runner 📝

Thu, 05 Dec 2024 00:00:00 +0000

(See my intro post)

Documentation should be as close to the code as possible - that’s the principle of “information locality.” But how do you keep code examples and their outputs in sync? Enter markdown-code-runner, a tool that executes code blocks within Markdown files and updates their output in-place!

📖 Origin Story #

While maintaining various open source and work projects, I found myself constantly updating README or Wiki files with new command outputs, updated tables, and fresh examples. This manual process was error-prone and tedious. Instead of using external templating files or more advanced documentation generators like Sphinx and Quarto (which are often overkill for single-page projects), I wanted something simpler: code and its output living together in the same Markdown file, always in sync.

🔧 Technical Highlights #

Executes code blocks (Python and Bash) within Markdown files
Updates outputs in-place while preserving the original file structure
Supports hidden code blocks for generating content without showing the code
Works with any language by using file code blocks and bash execution
Zero external dependencies
GitHub Actions integration for automatic README updates
Preserves Markdown spec compatibility by hiding code in HTML comments ()

📊 Impact #

97 GitHub stars
Used in many of my other projects for auto-generating:
- Command-line help texts
- API documentation
- Example outputs
- Data tables
Helps maintain “information locality” - everything needed is in one file
Enables dynamic documentation that stays up-to-date automatically

🎯 Challenges and Solutions #

Parsing Markdown without breaking its structure
Supporting multiple programming languages
Making it work seamlessly with GitHub Actions
Handling complex outputs (tables, images, etc.)
Maintaining backward compatibility with existing Markdown files

💡 Lessons Learned #

Documentation is best when it’s close to what it documents
Automation should be invisible - the tool shouldn’t change how you write Markdown
Sometimes the simplest solution (using HTML comments) is the best
Good documentation tools should make maintaining docs easier, not harder
Auto-generated content should be clearly marked as such

Want to keep your documentation always in sync? Check out markdown-code-runner on GitHub!

#OpenSource #Python #Documentation #DevTools #Programming

WenFire: When Could I retire?! 🔥

Wed, 04 Dec 2024 00:00:00 +0000

(See my intro post)

While I absolutely love my job, I believe I’ll love it even more when my family’s livelihood doesn’t depend on it. That’s the dream of financial independence - doing what you love because you want to, not because you have to. This led me to create WenFire, a financial independence calculator that helps answer the eternal questions: “When can I retire?” and “How many days of extra work will that new iPhone or car cost me?”

📖 Origin Story #

There are many retirement calculators out there, but they all made assumptions I didn’t quite agree with (and I’m sure others won’t agree with my assumptions either!). I wanted something that matched my specific views on salary growth and investment returns. Plus, I had been itching to learn more about building web applications, so this project let me explore FastAPI, htmx, and Vega-Lite while solving a personal need.

🔧 Technical Highlights #

Built with FastAPI and Bootstrap
Interactive visualizations using Vega-Lite
Responsive and mobile-friendly design
Customizable assumptions for:
- Investment growth
- Inflation rates
- Salary increases
- Safe withdrawal rates
Allows parameter changes at certain dates
Calculate retirement impact of one-time expenses
All parameters are encoded in the URL for easy sharing

📊 Impact #

While this project only has 3 GitHub stars, its real value is personal - it helps me track my own journey to financial independence. It’s also been a great learning experience in web development, an area where I had little prior experience.

🎯 Challenges and Solutions #

Learning web development from scratch
Creating intuitive visualizations for financial data
Balancing simplicity with customization options
Making reasonable default assumptions
Ensuring mobile responsiveness

💡 Lessons Learned #

Financial modeling requires careful consideration of assumptions
Web development is a different beast from scientific computing
Sometimes the best way to learn new tech is to solve a personal problem
The best calculator is the one that matches your own assumptions

Want to calculate your path to financial independence? Try it at wenfire.nijho.lt or check out the source code!

#OpenSource #Python #FinancialIndependence #WebDev #FIRE

rsync-time-machine.py 🕒

Tue, 03 Dec 2024 00:00:00 +0000

(See my intro post)

Sometimes the best open source projects start with “I can do better than this.” That’s exactly what happened when I looked at laurent22/rsync-time-backup, a popular 600-line Bash script for Time Machine-style backups. While it worked, it had known bugs and was tricky to maintain due to Bash’s limitations. It also lacked tests, type hints (bash duh…), and proper error handling. So, I did what any reasonable developer would do: rewrote it in Python: rsync-time-machine.py! 🐍

📖 Origin Story #

I needed reliable backups that worked across platforms and filesystems, unlike Apple’s Time Machine. While many excellent backup tools exist, I was drawn to the simplicity of this approach: it just creates folders named YYYY-MM-DD-HH-MM-SS containing your files. No special tools needed for restoring or browsing - just your regular file explorer! I created a fully tested, typed, and documented Python port that maintains compatibility with the original while adding modern development practices.

🔧 Technical Highlights #

Creates incremental backups using hard links (saving space while keeping full snapshots)
Works on Linux, macOS, and Windows (via WSL or Cygwin)
Supports local and remote (SSH) backups
Smart backup expiration strategy
Built-in safety checks and locking mechanism
Pretty terminal output
Absolutely zero dependencies beyond Python itself - it just works!
Usable as a standalone .py script or pip installable package
100% test coverage

📊 Impact #

The project reached the front page of Y Combinator’s Hacker News, causing its popularity to explode:

381 GitHub stars
16 forks
6 contributors
Featured on Real Python podcast
Active discussion on Reddit’s r/commandline

🎯 Challenges and Solutions #

Maintaining compatibility with the original Bash script while improving the codebase
Making it work on Windows through WSL/Cygwin (surprisingly tricky!)
Handling various edge cases in filesystem operations
Implementing a robust backup expiration strategy
Adding comprehensive test coverage for file operations
Keeping the codebase dependency-free while maintaining functionality

💡 Lessons Learned #

Sometimes a rewrite is worth it, especially when adding modern development practices
Good documentation and test coverage are crucial for backup tools
The Python community appreciates well-structured ports of useful tools
Zero dependencies means zero headaches for users
Cross-platform compatibility (especially Windows) requires careful consideration
Simple, transparent solutions (like regular folders) can be better than complex ones

Want reliable, cross-platform Time Machine-style backups? Check out the project on GitHub!

#OpenSource #Python #Backups #DevTools #Programming

Adaptive Lighting 💡

Mon, 02 Dec 2024 00:00:00 +0000

(See my intro post)

Unlike yesterday’s Calendar of Life project, today we’re looking at one of my most popular contributions. It’s one of several Home Assistant and home automation related projects (Day 8, Day 10, Day 24) I’ll be sharing this month. While my coding journey started in scientific computing with physics simulations, discovering how to control my home environment through code sparked a new obsession. Home Assistant, currently the #1 open source project on GitHub, is the brain of my smart home and lets you control and automate everything from lights to thermostats, completely locally and privacy-focused.

Keep your lighting in sync with the sun, they said. It’ll be simple, they said…

📖 Origin Story #

Back in 2020, inspired by @claytonjn’s circadian_lighting Home Assistant component, I wanted to create a more comprehensive solution for automatically adjusting lights based on the time of day. What started as a contribution attempt to Home Assistant Core evolved into its own custom component when the PR wasn’t merged - handling all the edge cases made the code too complex for core. The standalone component gave us the freedom to implement features properly, becoming one of Home Assistant’s most installed custom integrations.

🔧 Technical Highlights #

While the concept is simple, making it work reliably is… interesting:

Handles unreliable Zigbee networks and flaky WiFi connections
Manages different light brands’ quirks (looking at you, IKEA bulbs 👀)
Provides smooth transitions between color temperatures
Detects manual overrides without disrupting user control
Supports sleep modes for bedtime routines
Configurable via YAML or the UI (only integration to support this!)

The integration has 38 parameters to fine-tune your lighting. Try our WebAssembly simulator to visualize your settings!

📊 Impact #

2000+ GitHub stars
103 contributors
512 issues
Tens of thousands of active users
Available in 20+ languages
One of the most installed custom integrations
Featured in several YouTube videos and podcasts
Dedicated parameter visualization web app

🎯 Challenges We Solved #

Lights randomly turning on (bulbs falsely reporting state)
Network congestion (solved with Zigbee grouping)
Different brands interpreting colors differently
Handling manual overrides gracefully

💡 Lessons Learned #

“It works on my setup” is just the beginning
Users surprise you (like using it for plants, or above the polar circle!)
Edge cases multiply quickly
Documentation is as important as code
Open source thrives on community effort

Want your lights to follow the sun? Check out the project on GitHub and try the simulator!

#OpenSource #HomeAssistant #SmartHome #IoT #Python

Calendar of Life 📅

Sun, 01 Dec 2024 00:00:00 +0000

(See my intro post.)

Visualizing the finite weeks of our lives, because apparently, I needed another reminder of my mortality!

📖 Origin Story #

You know how sometimes you’re just casually browsing the internet, not looking to have an existential crisis? Well, that’s exactly what happened when I stumbled upon Tim Urban’s Wait But Why post “Your Life in Weeks” and saw Kurzgesagt’s “Calendar of Your Life” poster. Instead of just feeling mildly anxious about the finite nature of existence like a normal person, I decided to cope by writing some Python code. Because that’s what we do, right?

🔧 Technical Highlights #

The project uses matplotlib to create a grid where:

Each row represents one year (up to 80 years, optimistically speaking!)
Each column represents one week (52 weeks per year)
Different colors show different life stages (early childhood, school, university, etc.)
The current week blinks in the animated version, just in case you needed a more urgent reminder of time passing
Supports both dark and light modes, because existential dread comes in all themes

One interesting technical detail is handling the fact that 52 weeks isn’t exactly one year. The code uses a small correction factor (days_per_week = 365.25 / 52) to ensure the visualization stays accurate over many years. Yes, I made sure your existential crisis would be mathematically precise.

📊 Impact #

With its whopping 5 GitHub stars, this project clearly hasn’t taken the world by storm. But hey, sometimes the best tools are the ones we make for ourselves, right?

🎯 Challenges and Solutions #

The main challenges were (this was actually a very simple one):

Accurately mapping life events to week numbers (turns out life doesn’t fit neatly into boxes)
Creating smooth animations for the current week indicator (because a blinking dot really drives home the whole “time is passing” thing)
Handling the 52 vs 52.18 weeks per year discrepancy (thanks, Earth’s orbit)
Making the visualization work well in both dark and light modes (for viewing your life calendar at any time of day)

💡 Lessons Learned #

This project taught me that sometimes the most impactful tools are the simplest ones. Sure, it’s just a bunch of dots on a grid, but each dot represents a week of life - past, present, or future. It’s a humbling reminder that while we can’t add more dots to our grid, we can try to make each one count.

Want to contemplate your own mortality in a more organized way? Check out the project on GitHub!

#OpenSource #Python #DataVisualization #Programming