Packaging | Bas Nijholt

Auto-installing extras at runtime so users never have to think about dependencies

Wed, 11 Feb 2026 00:00:00 +0000

Table of contents

1. From one command to a dependency maze #

agent-cli started as a simple thing. Voice in, text out. It connected to a Wyoming Whisper server for speech-to-text and a Piper server for text-to-speech, basically a local, open-source Siri triggered from terminal hotkeys. I wrote about the early version in my local AI journey post.

And then, as tends to happen with side projects I use every day, I kept adding things.

First a RAG proxy with ChromaDB, so I could chat with my documents. Then a long-term memory system, so conversations could remember context across sessions. Server commands that run Kokoro TTS or Whisper locally, optimized for whatever hardware you have—Apple Silicon gets MLX, NVIDIA gets CTranslate2. Voice activity detection with Silero VAD. Audio speed adjustment. An agent orchestration system. systemd/launchd service management for running background daemons.

Each of these features pulled in wildly different dependency trees. torch and transformers for GPU-accelerated Whisper. sounddevice and numpy for microphone access. chromadb and onnxruntime for vector storage. kokoro with its entire spacy pipeline for neural TTS.

Today the project has 13 optional extras. The tool became much more useful, but dependency UX got progressively worse with every feature.

2. Why “just use extras” breaks down #

Python’s optional-dependencies in pyproject.toml is the standard mechanism for this, and it’s the right packaging primitive. You define groups, users install with pip install agent-cli[rag]. On paper, this is perfectly reasonable.

In practice, it falls apart when you have 13 of them:

[project.optional-dependencies]
audio = ["numpy", "sounddevice>=0.4.6", "agent-cli[wyoming]"]
llm = ["pydantic-ai-slim[openai,google,duckduckgo,vertexai]>=0.1.1"]
rag = ["pydantic-ai-slim[openai,google]>=0.1.1", "chromadb>=0.4.22", ...]
memory = ["pydantic-ai-slim[openai,google]>=0.1.1", "chromadb>=0.4.22", ...]
vad = ["onnxruntime>=1.16.0"]
faster-whisper = ["faster-whisper>=1.0.0"]
mlx-whisper = ["mlx-whisper>=0.4.0; sys_platform == 'darwin' and platform_machine == 'arm64'"]
kokoro = ["kokoro>=0.9.0", "soundfile>=0.12.0", ...]
piper = ["piper-tts>=1.2.0"]
# ... and more

The UX problem is obvious. If I want to transcribe voice to text, do I need audio? llm? vad? All three? What if I want the Whisper server to run locally—is that faster-whisper or mlx-whisper? Do I also need server for that?

Expecting users to study a dependency matrix before running a CLI command is a terrible experience. I wanted three things:

A lightweight base install that’s fast and doesn’t pull in heavy ML dependencies.
Commands that “just work” when first used.
A clear opt-out for people who don’t want automatic installs.

3. The decorator #

The core idea is simple. Each CLI command declares what it needs, and the decorator handles the rest:

@app.command("transcribe", rich_help_panel="Voice Commands")
@requires_extras("audio", "llm")
def transcribe(...):
 ...

That’s it from the command author’s perspective. One line, and you never think about dependency management in that command again.

Here are a few more examples from the codebase to show how this scales:

# Continuous transcription needs audio, voice activity detection, and an LLM
@requires_extras("audio", "vad", "llm")
def transcribe_live(...):
 ...

# TTS server needs the server framework + one of two TTS backends + Wyoming protocol
@requires_extras("server", "piper|kokoro", "wyoming")
def start_tts_server(...):
 ...

# Whisper server: any of three ASR backends will do
@requires_extras("server", "faster-whisper|mlx-whisper|whisper-transformers", "wyoming")
def start_whisper_server(...):
 ...

Notice the pipe syntax: "piper|kokoro" means “you need either Piper or Kokoro—any one of these satisfies the requirement.” On Apple Silicon, if you already have mlx-whisper installed, it won’t try to install faster-whisper instead.

The decorator itself is straightforward:

def requires_extras(*extras: str) -> Callable[[F], F]:
 def decorator(func: F) -> F:
 @functools.wraps(func)
 def wrapper(*args, **kwargs):
 if _check_and_install_extras(extras):
 raise typer.Exit(1)
 return func(*args, **kwargs)
 return wrapper
 return decorator

The real complexity lives in _check_and_install_extras, which handles several installation scenarios. But the interface for command authors is always a single line.

4. Checking without importing #

You don’t want to import torch just to check if it’s available—that takes seconds and loads hundreds of megabytes into memory. Instead, I use importlib.util.find_spec, which checks whether a module exists on the import path without executing any of its code. No heavy imports, no GPU initialization, no side effects. The check is essentially free.

5. The metadata bridge #

There’s a gap between what pyproject.toml knows and what the runtime check needs. pyproject.toml says the rag extra requires chromadb>=0.4.22—that’s the pip package name. But at runtime, I need to check Python import names. Sometimes these are the same (chromadb → import chromadb), but often they’re not: pydantic-ai-slim is imported as pydantic_ai, piper-tts as piper.

To bridge this, I generate an _extras.json file that maps each extra to its description and the Python import names needed for the runtime check:

{
 "audio": ["Audio recording/playback", ["sounddevice"]],
 "rag": ["RAG proxy (ChromaDB, embeddings)", ["chromadb", "pydantic_ai"]],
 "memory": ["Long-term memory proxy", ["chromadb", "yaml", "pydantic_ai"]],
 "vad": ["Voice Activity Detection (Silero VAD via ONNX)", ["onnxruntime"]],
 "faster-whisper": ["Whisper ASR via CTranslate2", ["faster_whisper"]]
}

This file is generated by a script that parses pyproject.toml and applies a mapping table for the package-name-to-import-name conversions. A pre-commit hook ensures it stays in sync—if you add an extra to pyproject.toml and forget to update the metadata, the commit fails.

I wish Python packaging had a standard way to discover a package’s top-level import name from its distribution name. It doesn’t, so the mapping table is the least-bad option I found.

6. Three environments, three strategies #

The hard part is that people run agent-cli in several different ways, and each needs a different installation strategy.

A) `uv tool install` — the persistent environment #

When installed via uv tool install agent-cli, the tool lives in a persistent environment under ~/.local/share/uv/tools/agent-cli/. This environment has a uv-receipt.toml file that tracks which extras are installed.

When auto-install triggers, the system reads the current extras from that receipt, merges in the new ones, and reinstalls:

def install_extras_impl(extras, *, quiet=False):
 if is_uv_tool_install():
 current_extras = _get_current_uv_tool_extras()
 new_extras = sorted(set(current_extras) | set(extras))
 return _install_via_uv_tool(new_extras, quiet=quiet)

So if you already had audio installed and now run a command that needs llm, it becomes uv tool install agent-cli[audio,llm] --force.

The key insight: uv tool upgrade reads uv-receipt.toml and preserves the extras list. Once auto-install adds an extra, it persists across future upgrades. You never have to install it again.

(Full implementation)

B) `uvx` — the ephemeral environment #

uvx is uv’s tool for running packages in temporary, disposable environments. Running uvx agent-cli transcribe creates a fresh environment, installs agent-cli, runs the command, and throws the environment away.

The problem: there’s nowhere to persist extras. No uv-receipt.toml to update, no virtualenv that survives the invocation.

The solution is one of my favorite tricks in this system: re-execute the entire command with the extras baked into the uvx invocation itself:

def _maybe_reexec_with_uvx(extras):
 if os.environ.get(_REEXEC_MARKER) or not _is_uvx_cache():
 return
 uvx_path = shutil.which("uvx")
 if not uvx_path:
 return
 extras_str = ",".join(extras)
 cmd = [uvx_path, "--python", "3.13",
 f"agent-cli[{extras_str}]", *sys.argv[1:]]
 _maybe_exec_with_marker(cmd, f"Re-running with extras: {extras_str}")

So uvx agent-cli transcribe transparently becomes uvx agent-cli[audio,llm] transcribe. The user sees a brief “Re-running with extras: audio, llm” message, and then the command works.

Environment detection is straightforward—uvx environments live under the uv cache directory:

def _is_uvx_cache():
 prefix_str = Path(sys.prefix).resolve().as_posix()
 return "/cache/uv/" in prefix_str or "/archive-v" in prefix_str

(Full implementation)

C) Regular virtualenv, `pipx`, or system install #

For plain pip install agent-cli, pipx, or virtualenv installs, the system falls back to direct installation. Each extra has a pre-generated requirements file with pinned versions (e.g., _requirements/audio.txt), and the install command adapts to whatever’s available:

def _install_cmd():
 in_venv = sys.prefix != sys.base_prefix
 if shutil.which("uv"):
 cmd = ["uv", "pip", "install", "--python", sys.executable]
 if not in_venv:
 cmd.append("--system")
 return cmd
 cmd = [sys.executable, "-m", "pip", "install"]
 if not in_venv:
 cmd.append("--user")
 return cmd

Prefers uv pip when available (fast), falls back to regular pip. Uses --system or --user as appropriate when not in a virtualenv. This is also the path that pipx installations take—pipx creates a regular virtualenv under the hood, so the auto-installer detects it as a venv and installs directly into it.

(Full implementation)

7. Preventing infinite re-execution loops #

There’s an obvious footgun here: what if the install succeeds but the package still isn’t importable for some reason? You’d get an infinite loop of install → re-exec → check → install → re-exec → …

The fix is a simple environment variable marker:

_REEXEC_MARKER = "_AGENT_CLI_REEXEC"

def _maybe_exec_with_marker(cmd, message):
 if os.environ.get(_REEXEC_MARKER):
 return # Already re-executed once, don't loop
 new_env = os.environ.copy()
 new_env[_REEXEC_MARKER] = "1"
 os.execvpe(cmd[0], cmd, new_env)

os.execvpe replaces the current process entirely—it never returns. The marker ensures we only do this once. If the second run still can’t find the packages, it falls back to a helpful error message instead of looping forever.

8. What users actually see #

With auto-install enabled (the default), a first run looks like this:

Auto-installing missing extras: audio, llm
Running: uv tool install agent-cli[audio,llm] --force --python 3.13 -q
Installation complete!
Re-running with installed extras...

And then the command runs normally. Next time, it runs instantly—no install needed.

With auto-install disabled, the user gets a clear message telling them exactly what to do:

This command requires the 'audio' extra (Audio recording/playback).

Install with:
 uv tool install -p 3.13 "agent-cli\[audio]"
 # or
 agent-cli install-extras audio

If you prefer explicit control, there’s also a manual install-extras command:

agent-cli install-extras rag memory # Install specific extras
agent-cli install-extras --all # Install everything
agent-cli install-extras --list # Show available extras with descriptions

You can opt out via environment variable or config file:

export AGENT_CLI_NO_AUTO_INSTALL=1

# ~/.config/agent-cli/config.toml
[settings]
auto_install_extras = false

9. Keeping everything in sync #

The runtime system depends on two generated artifacts:

_extras.json: maps extras to descriptions and import names (for the find_spec check)
_requirements/*.txt: pinned dependency files for each extra (for the pip/uv install)

_extras.json is generated by sync_extras.py (run manually or in CI). The requirements files are generated by sync_requirements.py, which runs as a pre-commit hook whenever pyproject.toml or uv.lock changes. A check script enforces that _extras.json stays consistent with pyproject.toml—if you add an extra but forget to update the metadata, the commit fails with a clear error.

This means the runtime system never drifts from the source of truth. pyproject.toml is the canonical definition, everything else is derived.

10. Why uv made this practical #

I want to be clear: this system would be much harder without uv.

uv tool install with persistent extras in uv-receipt.toml—extras survive upgrades automatically
uvx for ephemeral execution with extras specified inline
uv pip install that’s fast enough for runtime auto-install to feel acceptable
uv export for generating pinned requirements files from uv.lock

With traditional pip, auto-installing at runtime would take 30-60 seconds and feel terrible. With uv, it typically takes a second. That’s the difference between “annoying” and “barely noticeable.”

11. Tradeoffs #

This approach works well, but it’s still a tradeoff:

There’s metadata to maintain. The package-name-to-import-name mapping requires a lookup table with special cases like piper-tts → piper and pydantic-ai-slim → pydantic_ai. I wish pyproject.toml had a field for this.
The check only verifies importability, not version compatibility. find_spec tells you a module exists, not whether it’s the right version. In practice this hasn’t been a problem because uv’s pinned requirements handle versions, but it’s a theoretical gap.
Auto-install can fail in restricted environments. Corporate machines, read-only containers, air-gapped systems—the opt-out exists for a reason.
Re-exec is a CLI pattern, not a library pattern. os.execvpe replaces the process, which means any shell hooks or signal handlers set up before the decorator runs are lost. Fine for CLI tools, would be problematic for libraries.

12. Not dependency injection #

I originally called this “dependency injection” in my head, but that’s the wrong term entirely. Dependency injection is about providing object dependencies through constructors or function parameters. This is something different.

The best names I’ve come up with:

runtime optional dependency resolution — accurate but a mouthful
on-demand extras installation — emphasizes the user experience
lazy extras — short, probably what I’d use in conversation

If there’s a standard term for this pattern, I haven’t found it.

13. Conclusion #

The underlying principle is simple: the tool should know what it needs, and it should get it. Users shouldn’t have to study a matrix of 13 optional extras to figure out which pip command to run. They should just type the command they want, and it should work.

uv’s speed and tooling model made this practical. A decorator made it ergonomic for command authors. Pre-commit hooks made it maintainable. And three environment-detection strategies made it work regardless of how the user installed the tool.

If you’re building a Python CLI with optional features that keep growing, consider this pattern. Your users will thank you, mostly by never having to ask “which extras do I need?”

The code discussed here is part of agent-cli, an open-source suite of AI-powered command-line tools. The implementation lives primarily in agent_cli/core/deps.py.

Confessions of an asymmetric hypocrite: on Python dependencies

Sun, 27 Apr 2025 00:00:00 +0000

Introduction: The Dependency Dilemma 🤔 #

I just listened to a podcast “Build software that lasts!” with Bert Hubert (a fellow Dutch), who prefers to reduce the number of dependencies to an absolute minimum (also see his related blog posts [1] and [2]). While I agree one shouldn’t use is-even as a dependency (is-odd is a better choice 😏), I think there is a lot of value in reducing the number of dependencies but also in using the right tool for the job and not continuously re-inventing the wheel. The podcast got me thinking about my own stance on dependencies and how I actively avoid introducing dependencies into my libraries while expecting (hoping 🤞) others to depend on my work.

Like many software developers, I’ve spent countless hours wrestling with dependencies. Dependency hell is easy to fall into, especially in dynamic languages like Python. Those moments where installing or updating one package breaks another, or where a tiny library pulls in hundreds of transitive dependencies, are frustratingly common. This frustration partly motivated me to create unidep, a tool aimed at simplifying dependency management across different Python tools like pip and conda.

But working on unidep and my other projects forces me to think about my own relationship with dependencies. I’ve realized I operate with what feels like an “asymmetric hypocrisy.” My standards for the libraries I publish are extremely strict regarding which external dependencies I consume. Yet, as an author of over 30 Python packages, I inherently produce dependencies, hoping others will adopt my work. And then, when building applications or CLIs, my caution often disappears entirely.

This post dives into that asymmetry. It’s about the different hats I wear: the minimalist library author (as a dependency consumer), the hopeful library author (as a dependency producer), and the pragmatic application developer. In this post, I’ll explain why I take these different approaches and why it’s really about the situation, not a contradiction.

Table of contents

1. The Library Author Hat 🎩 (as Dependency Consumer): Maximum Caution #

When I build and publish an open-source Python library – like Adaptive, pipefunc, or unidep – my primary focus shifts to the users who will depend on my code. My choices directly impact their projects. This responsibility means I have to be very careful about adding external dependencies:

Minimize and Scrutinize: Fewer dependencies mean fewer potential conflicts for users. I aggressively minimize the core runtime dependency list.
Favor Battle-Tested Dependencies: I strongly prefer dependencies that are already widely adopted and trusted by the community. (I have been called that person that always wants the latest and greatest, the truth is that I am a sucker for shiny new things but typically only adopt them once they are already quite popular.) Think numpy or cloudpickle. These are less likely to cause unexpected issues or disappear overnight.
Low Transitive Dependency Count: A key criterion is how many other packages a potential dependency pulls in. Ideally, I choose dependencies with zero or very few further dependencies. For example, in pipefunc, dependencies like numpy, networkx, and cloudpickle are good examples – they are popular and don’t rely on many other things.
Optional Dependencies for Heavy Features: If a feature really needs a large or less common dependency, I try making it a optional extra (project.optional-dependencies). Users opt-in only if they need that specific capability.
Avoid Pinning (Mostly): As mentioned before, hard version pins (==1.2.3) in libraries are usually harmful. Minimum versions (>=1.2) or safe upper bounds (<2.0) are preferred.
License Awareness: The license of a dependency matters immensely, especially regarding compatibility and obligations (GPL considerations).

The guiding principle is: Do not cause pain for your users. A library should integrate smoothly into diverse environments.

2. The Library Author Hat 🎩 (as Dependency Producer): The Paradox #

Here’s where the “hypocrisy” might seem to surface. While I’m cautious about consuming dependencies for my libraries, I am also the author of over 30 packages. I invest time and effort into these tools, naturally hoping they prove useful and that others will adopt them – making my libraries dependencies in their projects.

Is it hypocritical to be wary of adding dependencies while simultaneously creating them for others? Perhaps slightly, but it’s also the nature of contributing to an ecosystem. I try to handle this by strictly following the rules from Section 1 for my own libraries. If I expect others to rely on my code, I feel responsible to make that dependency as low-risk and stable as possible. I want my libraries to be the kind of well-behaved, minimal-friction dependencies that I myself would be comfortable adding.

3. The Application Developer / Contributor Hat 🧢: Practicality is Key #

When the context shifts to building an application – like a command-line tool, a web service, or a personal project – my rules become much less strict. The key differences are isolation (keeping things separate) and how far the effects reach.

Isolation Reduces Risk: Applications often run in controlled, isolated environments. Docker, conda/micromamba (Python Environment Jungle post), or simple venvs contain the dependencies. A conflict within the application’s environment doesn’t break the user’s entire system or other unrelated projects.
CLIs as a Prime Example: Many tools I’ve built are CLIs (like tuitorial or helpers within my dotfiles). These might have tens of dependencies. Because they are installed into isolated environments (often via uv, pipx, or within a dedicated conda env), this complexity is okay. The risk is contained.
Developer Tools Welcome: Productivity boosters like ruff, mypy, etc., are fair game as development dependencies (My Python Project Toolkit). They don’t impact the end-user runtime (if packaged properly), so I am much more lenient with dev tools.
Focus on Functionality: The goal is often to deliver features or solve a problem efficiently. Leveraging existing libraries, even those with dependencies, is usually faster and more robust than reimplementing complex logic. It usually makes more sense to use existing libraries.

In this context, the focus changes from minimizing external impact (libraries) to maximizing internal capability and maintainability within a controlled scope (applications).

4. Conclusion: Not Hypocrisy, but Context-Driven Strategy 🙏 #

My approach to dependencies is “it depends on the situation”. I’m stricter about dependencies for my libraries than for my applications or personal tools. I am cautious about adopting dependencies while actively producing them for others.

But I don’t see this as hypocrisy. It’s a practical approach based on different situations and goals:

Libraries (as Consumer): Focus on keeping the software community stable and making things easy for users. Be conservative.
Libraries (as Producer): Create software responsibly, following the same high standards you expect from others.
Applications: Focus on adding features and developing quickly within separate environments. Be pragmatic.

Understanding the different risks and goals for each situation helps make good decisions about dependencies. It’s about choosing the right trade-offs for the specific situation, whether you’re building a foundational block for the community or a specific tool for a controlled environment.

How do you handle dependencies across different types of projects? Do you find yourself navigating similar questions? Share your thoughts!

GPL for Python packaging ⚖️

Mon, 10 Feb 2025 00:00:00 +0000

Understanding GPL for Python Packaging #

Over the last 10 years–since I started working on and with open source projects–the GPL topic has frequently come up with friends and colleagues. I simply write this post as a reminder to myself and to have something concise and clear to link to later. Also, IANAL, so if you are relying on this post, please do your own research, I have included all the links I used to make this post.

The GNU General Public License (GPL) is widely used in the open-source world, but its implications can be unclear, especially regarding Python packaging. If you develop or distribute Python software, understanding GPL variants is crucial.

What is GPL? #

GPL is a strong copyleft license—meaning software using GPL-licensed code must also be released under the GPL. There are several key variants relevant to Python:

GPLv2 & GPLv3: These licenses require that software using GPL code (directly or indirectly) also be open-sourced under GPL upon distribution.(¹,²) If your Python application imports or depends on a GPL package, your code must adopt GPL licensing.
LGPL (Lesser GPL): LGPL is more permissive and is usually applied to libraries. It allows proprietary software to link or import the library without forcing the proprietary software itself to adopt LGPL.(³,⁴) You just need to ensure users can replace or modify the LGPL component.
AGPL (Affero GPL): AGPL is similar to GPLv3 but includes a critical extra clause: it applies even when users access the software remotely over a network (such as web services or APIs). If you use an AGPL-licensed Python library, your entire application must be open-sourced under AGPL, even if you’re only hosting it online.(⁵,⁶)

What Does “Linking” Mean in Python? #

GPL licenses often refer to “linking,” traditionally associated with compiled languages (e.g., C, C++). Python doesn’t link code in the traditional sense. However, it is generally agreed upon that importing a Python module counts as linking for licensing purposes.(¹,⁷) Therefore, directly importing a GPL-licensed Python package typically triggers the GPL’s copyleft requirements.

Why Does This Matter for Python Packaging? #

Python makes it easy to incorporate third-party libraries. But unintentionally using GPL-licensed libraries (especially GPL or AGPL) legally obligates you to open-source your proprietary code.

Direct Dependencies: If you directly import a GPL-licensed Python library, your software must be GPL-licensed upon distribution.(¹,²)
Indirect (Transitive) Dependencies: If any dependency in your Python project indirectly relies on GPL-licensed code, your entire software must also adopt GPL licensing upon distribution. Even a hidden, indirect GPL dependency triggers the GPL requirements.(⁸,⁹,¹⁰)

Recommendations #

Check Licenses Carefully: Always verify the licenses of both your direct and indirect dependencies to prevent unintended GPL licensing obligations.(¹¹,⁷)
Use Permissive or LGPL Licenses: When planning to keep your own software proprietary, prioritize libraries with permissive licenses (e.g., MIT, Apache, BSD) or LGPL-licensed libraries, as they allow for proprietary use without triggering GPL requirements.(³,⁴)
Separate GPL Code: If you must utilize GPL code, explicitly separate it from proprietary code. Using subprocess calls or isolated microservices is often recommended since it counts as “mere aggregation,” not linking, thus avoiding GPL obligations.(¹²,⁷)

By clearly understanding GPL licensing implications, you can confidently manage your Python project’s dependencies and safely distribute your software.

Python environments that stuck: micromamba and uv 🐍

Fri, 03 Jan 2025 00:00:00 +0000

1. Introduction 🚀 #

Like many Python developers, I’ve spent considerable time trying different environment management tools to find what works best for me.

After exploring various options from virtualenv to Conda, I found that Conda-based solutions fit my needs well - mainly because they handle non-Python dependencies without much hassle.

While Conda itself can be slow when setting up environments, tools like Mamba and Pixi provide the same functionality with better performance.

In this post, I’ll share my experience with different Python environment tools and provide practical commands for setting up Python environments.

Table of contents

2. The Alternatives 🌟 #

Before we start look at this comparison tables for the most popular Python environment management tools.

I had to split the tables into two because of the number of columns.

Table 1: Python Environment Management Tools

Tool	Key Features	Native/3rd Party	Non-Python Deps
Pyenv	Python version management	3rd Party	❌
Pipenv	Package & env management	3rd Party	❌
Conda	Package & env management	3rd Party	✅
Mamba	Fast Conda replacement	3rd Party	✅
Micromamba	Lightweight Mamba	3rd Party	✅
Docker	Containerization	3rd Party	✅
Venv	Built-in env management	Native	❌
Virtualenv	Env management (pre-3.3)	3rd Party	❌
Homebrew	macOS/Linux package manager	3rd Party	✅
Pixi	Fast Conda alternative	3rd Party	✅
uv	Fast pip and venv replacement	3rd Party	❌
System Package Managers	OS-based package management	Native	✅

Table 2: Python Environment Management Tools

Tool	Ease of Use	Isolation	Cross-Platform Support	Community & Support	Performance
Pyenv	Medium	✅	Windows, macOS, Linux	Good	Good
Pipenv	Easy	✅	Windows, macOS, Linux	Good	Good
Conda	Easy	✅	Windows, macOS, Linux	Excellent	Average
Mamba	Easy	✅	Windows, macOS, Linux	Growing	Fast
Micromamba	Medium	✅	Windows, macOS, Linux	Growing	Fast
Docker	Hard	✅	Windows, macOS, Linux	Excellent	Good
Venv	Easy	✅	Windows, macOS, Linux	Excellent	Good
Virtualenv	Medium	✅	Windows, macOS, Linux	Good	Good
Homebrew	Easy	❌	macOS, Linux	Excellent	Good
Pixi	Easy	✅	Windows, macOS, Linux	Growing	Fast
uv	Easy	✅	Windows, macOS, Linux	Growing	Very Fast
System Package Managers	Easy	❌	Depends on OS	Depends on OS	Good

Now that we’ve seen the alternatives, let’s dive into setting up environments with the help of Conda and Mamba.

🐍 Pyenv #

A popular tool that enables you to easily manage multiple Python versions on a single system. It allows you to switch between different Python versions without interfering with system-level installations.

Setup Instructions

Install pyenv:
- macOS: brew install pyenv
- Ubuntu: curl https://pyenv.run | bash
Add the following lines to your shell configuration file (e.g., ~/.bashrc, ~/.zshrc):
```
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
```
Restart your shell or run source ~/.bashrc (or source ~/.zshrc)
Install a specific Python version: pyenv install 3.8.5
Set the Python version for your project: pyenv local 3.8.5

📦 Pipenv #

A package management and virtual environment management tool that combines the functionality of pip and virtualenv. It streamlines the process of installing and managing packages within isolated environments.

Setup Instructions

Install pipenv: pip install --user pipenv
Change to your project directory: cd my_project
Initialize a new virtual environment: pipenv --python 3.8
Install packages: pipenv install <package_name>
Activate the environment: pipenv shell

🌐 Conda #

A cross-platform package manager and environment management tool, often used in conjunction with Anaconda and Miniconda. Conda simplifies the installation and management of packages and dependencies, especially for data science and machine learning projects.

Setup Instructions

Download and install Miniconda: https://github.com/conda-forge/miniforge#miniforge3
Create a new environment: conda create --name my_env python=3.8
Activate the environment: conda activate my_env
Install packages: conda install <package_name>

🚀 Mamba #

Mamba is a fast, drop-in replacement for Conda, designed to resolve and install packages more quickly. It uses the same Conda repositories and environment files, making it easy to switch between the two.

Setup Instructions

Install Mambaforge with Mamba pre-installed:

Download and install Mambaforge with Mamba for your platform: https://github.com/conda-forge/miniforge#mambaforge

Alternatively, you can install Mamba within an existing Conda environment:
```
conda install -c conda-forge mamba
```
Create a new environment with Mamba:
```
mamba create --name my_env python=3.8
```
Activate the environment:
```
mamba activate my_env
```

Install packages using Mamba:

mamba install <package_name> -c conda-forge

By using Mamba instead of Conda, you can significantly speed up environment resolution and package installation while still benefiting from the Conda ecosystem.

🌌 Micromamba #

Micromamba is a lightweight and fast alternative to the Mamba package manager. It is a statically linked C++ executable that doesn’t require a base environment or come with a default Python version. It supports a subset of Mamba and Conda commands and is designed to resolve and install packages quickly. For those who need Conda features, you can install Conda using Micromamba with micromamba install conda.

Setup Instructions

Download the appropriate Micromamba binary for your platform from the official repository.
Make the binary executable and move it to a directory in your PATH:
```
chmod +x micromamba mv micromamba /usr/local/bin/
```

Create a new environment with Micromamba:

micromamba create -n my_env python -c conda-forge

Activate the environment:
```
micromamba activate my_env
```

Install packages using Micromamba:

micromamba install <package_name> -c conda-forge

If you need Conda’s features, you can install Conda within your Micromamba environment:
```
micromamba install conda
```

For example, if you need nb_conda_kernels to use your Micromamba environment in JupyterLab:

micromamba install conda nb_conda_kernels

And you are set up!

Micromamba is the perfect solution for those who want a fast and lightweight package manager while still being able to access Conda’s features when needed.

🐳 Docker #

A containerization platform that allows you to create lightweight, portable environments called containers. With Docker, you can package your Python application along with its dependencies, ensuring consistent execution across different systems.

Setup Instructions

Install Docker: https://docs.docker.com/get-docker/

Create a Dockerfile in your project directory with the following contents:

FROM python:3.11
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "your_script.py"]

Build the Docker image: docker build -t my_project .
Run the Docker container: docker run my_project

🏞️ Venv #

A built-in Python module available since Python 3.3, which allows you to create lightweight virtual environments. Venv simplifies the process of creating isolated environments for different projects, ensuring that package dependencies don’t conflict with each other.

Setup Instructions

Create a new virtual environment: python3 -m venv my_env
Activate the environment:
- macOS/Linux: source my_env/bin/activate
- Windows: my_envScriptsactivate
Install packages: pip install <package_name>

🌐 Virtualenv #

Virtualenv is a third-party Python environment management tool that predates the built-in venv module. It allows you to create isolated environments for different projects, ensuring that package dependencies don’t conflict with each other. Virtualenv is compatible with Python 2.7 and later versions, whereas venv is only available since Python 3.3.

Setup Instructions

Install virtualenv: pip install virtualenv
Create a new virtual environment: virtualenv my_env
Activate the environment:
- macOS/Linux: source my_env/bin/activate
- Windows: my_env\Scripts\activate
Install packages: pip install <package_name>

🍺 Homebrew #

Homebrew is a popular package manager for macOS and Linux that simplifies the installation and management of software. Though it is not a dedicated Python environment management tool, it does offer support for installing and managing multiple Python versions alongside other software.

Homebrew can be useful for installing and managing Python versions for system-level usage, but it’s not the best choice for managing isolated project-specific dependencies. For project-specific dependency management, it’s recommended to use other tools like venv, virtualenv, or Conda in conjunction with Homebrew-installed Python.

Setup Instructions

Install Homebrew (if not already installed):
- macOS: Follow the instructions at https://brew.sh
- Linux: Follow the instructions at https://docs.brew.sh/Homebrew-on-Linux
Install a specific Python version:
```
brew install python@3.11
```
Add the installed Python version to your PATH by adding the following line to your shell configuration file (e.g., ~/.bashrc, ~/.zshrc):
```
export PATH="/usr/local/opt/python@3.11/bin:$PATH"
```
Restart your shell or run source ~/.bashrc (or source ~/.zshrc) to apply the changes.
Verify the Python version: python3.11 --version

Keep in mind that Homebrew is mainly for installing and managing software at the system level. To manage project-specific dependencies and create isolated environments, combine Homebrew with other tools like venv or virtualenv for better dependency management.

💻 System package managers #

Some operating systems have their own package managers (e.g., apt for Ubuntu, pacman for Arch Linux, or Homebrew for macOS) that can be used to install and manage Python environments. However, using system package managers may lead to conflicts with other system packages and is generally not recommended for managing project-specific dependencies.

I will skip the setup instructions for this one, as it is just not recommended to do this!

🌟 Pixi #

Pixi is a new package manager built in Rust that uses Conda-forge packages but aims to be significantly faster and simpler than Conda. It’s developed by prefix.dev (the team behind Mamba) and focuses on providing a more streamlined experience. It is inspired by tools like Cargo, nix, and npm and aims to provide a modern package management experience for Python developers.

Setup Instructions

Install pixi following instructions at prefix.dev
Initialize a new project:
```
pixi init my-project
cd my-project
```
Add dependencies:
```
pixi add python numpy pandas
```
Run Python in the environment:
```
pixi run python
```

Create and run tasks (defined in pixi.toml):

# Add a task
pixi task add hello "python hello_world.py"

# Run the task
pixi run hello

Start an interactive shell in the environment:
```
pixi shell
```

Pixi can also be used as a global installation tool:

# Install tools globally
pixi global install starship
pixi global install fish

# Create a named environment with multiple packages
pixi global install --environment data-science-env \
 --expose python --expose jupyter \
 python jupyter numpy pandas

Key features:

Fast package resolution and installation
Uses Conda-forge packages
Built-in task runner
Global package installation support
GitHub Actions integration with automatic caching
Simple and intuitive CLI

⚡ uv #

uv is an extremely fast Python package installer and resolver written in Rust. It can be used as a drop-in replacement for pip and can generate virtual environments. While it is quite new it is already being used by many developers due to its speed and ease of use. Besides being fast, it is awesome because it is a single binary that can be used to also bootstrap Python itself.

Setup Instructions

Install uv:

curl -LsSf https://astral.sh/uv/install.sh | sh

Create a new virtual environment:
```
uv venv
```
Activate the environment:
- Unix/macOS: source .venv/bin/activate
- Windows: .venv\Scripts\activate
Install packages (much faster than pip):
```
uv pip install numpy pandas
```

It will automatically maintain a uv.lock file to ensure deterministic builds. Install this lock file with uv sync.

Key features of uv:

Up to 10x faster than pip
Built-in virtual environment support
Compatible with pip’s CLI interface
Global package caching
Deterministic builds

There is much more to uv, so check out the documentation for more details.

3. Conclusion 🏁 #

The choice you make for managing your Python environments ultimately depends on your use case. If you don’t have to frequently deal with non-Python dependencies, I would recommend using uv for its speed and simplicity. However, because I often work with projects that require non-Python dependencies, I’ve found that Micromamba is the perfect fit for me.

That being said, each tool has its own strengths and weaknesses, and it’s important to choose the one that works best for your specific needs. I hope this journey through Python environment management has been helpful and informative. Good luck in finding your perfect Python environment workflow! 🐍

4. Further Reading 📚 #

Here are some resources for further exploration:

Pyenv: Pyenv GitHub repository and documentation
Pipenv: Pipenv documentation
Conda: Conda documentation
Mamba: Mamba documentation
Micromamba: Micromamba documentation
venv: venv documentation
virtualenv: virtualenv documentation
Anaconda: Anaconda homepage
Miniconda: Miniconda documentation
Homebrew: Homebrew homepage
Pixi: Pixi documentation
uv: uv documentation

UniDep 🧬

Fri, 13 Dec 2024 00:00:00 +0000

(See my intro post)

Ever tried to set up a Python project that needs both pip and conda packages? Then you know the pain of maintaining multiple requirement files. UniDep was born from years of frustration with this exact problem.

📖 Origin Story #

While working on large monorepos with hundreds of dependencies, I kept running into the same issue: maintaining separate files for conda (environment.yml), pip (requirements.txt), and Python packaging (pyproject.toml). Coming from languages like Rust, where package management is a joy, Python’s fragmented ecosystem can feel overwhelming, especially to newcomers (obligatory xkcd.com/1987).

What started as a simple tool to avoid duplicate dependency definitions turned into something much more powerful. In enterprise environments, it’s common to find monorepos with over 1000 dependencies spread across dozens of packages. Setting up these environments traditionally involves following lengthy setup guides, installing packages in the correct order, and hoping nothing breaks.

Then came the “aha” moment - what if we could install everything with a single command? After implementing UniDep, those complex setup guides became a one-liner: unidep install-all -e. It installs all dependencies and sets up every package in the monorepo in editable mode. What used to take hours now takes minutes.

The difficulty of managing pip and conda

🔧 Technical Highlights #

Single source of truth for all dependencies
Supports both requirements.yaml and pyproject.toml
Handles platform-specific dependencies
Generates conda-lock files both for entire monorepos and fully consistent lock files for its individual packages
Works with pip, conda, and mamba
Monorepo-friendly with local dependency support
Integrates with setuptools and hatchling
≥99% test coverage, fully-typed

📊 Impact #

221 GitHub stars
Featured on Python Bytes podcast
Saves countless hours of dependency management
Makes complex project setups accessible
Actively maintained with 70 releases

🎯 Challenges and Solutions #

Different package naming between pip and conda
Platform-specific dependencies
Cross-platform compatibility (Linux, Windows, MacOS)
Supporting pip-only workflows

💡 Lessons Learned #

Good developer experience is worth the effort
Sometimes you need to build a bridge between existing tools
The best tools make hard things easy
Dependency resolution is hard but can be simple for users

🔮 Future Plans #

Adding Pixi support - allowing seamless migration to Pixi without changing config files
Eventually, Pixi might make UniDep obsolete (and that’s okay!)
For now, UniDep bridges the gap between pip and conda, making life easier for developers

Want simpler dependency management? Check out UniDep on GitHub!

#OpenSource #Python #DevTools #Programming #PackageManagement #Conda

conda-forge Contributions 🐍

Wed, 11 Dec 2024 00:00:00 +0000

(See my intro post)

Unlike other projects in this advent calendar, I want to celebrate being part of something much bigger: the conda-forge community, where thousands of volunteers maintain the packages that power scientific Python.

📖 Origin Story #

I started with conda-forge in 2016 during my PhD work with Kwant, a quantum transport simulator. Installing it was a nightmare - it required system-level numerical libraries like MUMPS and Scotch which required a special config file to link. When I discovered conda-forge and its mission to make scientific software installation painless, I knew this was the solution!

The humble work of conda-forge maintainers

🔧 Technical Highlights #

Over the past 8 years, I’ve maintained over 40 build recipes, including:

Major scientific packages like VTK, MUMPS, and Scotch
Quantum physics tools like Qsim, Qcodes, and Kwant
Visualization libraries like HoloViews
Plus all the packages I’ve authored myself

The real magic of conda-forge is its infrastructure:

Automated builds across Linux, macOS, and Windows
Strict dependency version management
Comprehensive CI/CD pipelines
Community-driven quality control

📊 Impact #

What makes conda-forge truly special is how initial contributions blossom into collaborative efforts. I’ve experienced this firsthand: after creating the initial MUMPS feedstock, 25 other contributors joined in, each bringing their unique expertise.

A recent experience also illustrates this: I added CUDA support to both Google’s Qsim and IBM’s Cirq packages. Shortly after, Leo Fang, an engineer from NVIDIA who knows CUDA far better than I do, stepped up and made significant improvements. This is the beauty of open source - experts naturally gravitate to where they can make the biggest impact.

This collaborative approach solves a problem I’ve encountered at several companies: the tendency to maintain complicated build systems internally with custom hacks. Instead of each organization reinventing the wheel, conda-forge provides a shared platform where:

Maintenance burden is distributed
Expert knowledge flows freely
Build practices are standardized
Infrastructure is public and reusable

🎯 Challenges and Solutions #

Keeping up with upstream changes
Managing complex dependency trees
Cross-platform compatibility
Coordinating with upstream maintainers

💡 Lessons Learned #

Community effort beats solo work for infrastructure
Automation is crucial for reliability
Small contributions compound over time
Being part of something bigger is rewarding

Want to contribute to scientific Python? Check out conda-forge and the staged-recipes repository!

#OpenSource #Python #Scientific #Programming #Community

My favorite tools that keep my Python projects sane 🧰

Sat, 22 Apr 2023 00:00:00 +0000

🌟 1. Introduction #

As a passionate Python developer, I am always on the hunt for the best tools to streamline my development process and enhance the quality of my projects. Over time, I have discovered a set of tools that have become indispensable in my workflow, making my coding experience more efficient and enjoyable. Using the right developer tools for Python projects is crucial for maximizing productivity, ensuring code consistency, and maintaining high-quality code.

In this blog post, I will share these essential tools that I now use for every single new project. We will explore the benefits of each tool and discuss how they can help you supercharge your Python development. The tools covered in this post include pre-commit, Black, Ruff, Mypy, Versioningit, Pyproject.toml with PEP 621, Nox, pytest, GitHub Actions CI matrix, and Read the Docs, Sphinx, MyST-NB, and Jupytext notebooks for documentation. Let’s dive in and learn how to enhance your Python projects with these powerful tools!

Table of contents

🛠️ 2. Pre-commit #

One of the most common challenges developers face when working on a project, whether alone or in a team, is maintaining code quality and consistency. As projects grow, it becomes increasingly difficult to enforce coding standards and catch issues before they make their way into the codebase. This is where pre-commit comes in.

Pre-commit is a powerful tool that helps you manage and maintain code quality by automating checks before commits. It achieves this by setting up a series of hooks that run before each commit, ensuring that only code that meets your predefined standards is committed to the repository. By catching issues early, pre-commit helps you keep your code clean and consistent, making it easier to review and maintain.

One of the reasons pre-commit has gained widespread adoption is its flexibility and extensibility. You can choose from a vast range of built-in hooks, such as code formatters, linters, and security checkers, or even create your own custom hooks. This allows you to tailor pre-commit to the specific needs of your project, ensuring that it enforces the exact coding standards you require.

Using pre-commit is beneficial for both individual developers and teams. For solo developers, it helps maintain discipline and enforces coding standards, making it easier to onboard collaborators in the future. For teams, it streamlines the review process by catching issues before they are submitted for review, reducing the time spent on addressing code inconsistencies and style violations.

Let’s take a look at an example. Suppose your project follows the PEP 8 coding standard, and you want to ensure that all code committed to the repository adheres to these guidelines. You can configure pre-commit to run a PEP 8 linter, such as Flake8, before each commit. This way, any code that does not meet the PEP 8 standard will be flagged, and the commit will be blocked until the issues are resolved.

In summary, pre-commit is an invaluable tool for maintaining code quality and consistency in your Python projects. By automating checks before commits, it ensures that your codebase remains clean and adheres to your chosen standards, making your development process more efficient and enjoyable.

Adding pre-commit to your Python project is simple and straightforward. To get started, you’ll first need to install the pre-commit package using pip:

pip install pre-commit

Next, create a configuration file named .pre-commit-config.yaml in your project’s root directory. This file will define the hooks you want to run before each commit. For example, to use Black for code formatting and Flake8 for linting, your configuration file might look like this:

repos:
 - repo: https://github.com/psf/black
 rev: stable
 hooks:
 - id: black
 - repo: https://github.com/pycqa/flake8
 rev: master
 hooks:
 - id: flake8

Once you have set up the configuration file, run the following command to install the pre-commit hooks:

pre-commit install

Now, every time you attempt to make a commit, pre-commit will run the specified hooks. If any issues are found, the commit will be blocked, and you’ll be prompted to fix the problems before trying again. To run pre-commit manually on all files in your repository, use the following command:

pre-commit run --all-files

By incorporating pre-commit into your development process, you’ll be able to catch and fix issues before they make their way into your codebase, ensuring a consistent and high-quality Python project.

🎨 3. Black: auto-formatting for Python #

As a developer who has always been particular about coding style, I’ve had my fair share of discussions on the proper way to indent multiline statements and other formatting nuances. These conversations, although sometimes necessary, can take up valuable time that could be better spent on actual development. That’s where Black, the uncompromising code formatter, comes in and saves the day.

Adopting Black has made a significant impact on my development workflow. No more unnecessary discussions about formatting preferences; everyone on the team simply follows the rules set forth by Black. This consistency across the codebase not only makes the code more readable but also reduces the time spent on formatting debates.

One of the best aspects of Black is that you don’t need to worry about writing perfectly formatted code initially. You can write your code as quickly as you want, even if it’s poorly formatted, and then simply run Black to transform it into a beautifully formatted masterpiece. This can be a real time-saver, especially when working on large projects with many contributors.

To install Black, run the following command:

pip install black

You can then format your code by running Black in the terminal:

black your_file.py

Alternatively, you can use the VS Code command palette to run Black on the currently open file. However, the most efficient way to use Black is in combination with pre-commit, as mentioned in the previous section. This way, your code is automatically formatted every time you commit, ensuring a consistent codebase without any extra effort.

In summary, Black is an invaluable tool for enforcing consistent code formatting throughout your Python projects. By adopting it and integrating it with pre-commit, you can save time and focus on what truly matters: writing great code.

🚀 4. Ruff: the fastest Python linter and auto-formatter #

In the past, I’ve used several tools to maintain code quality, such as isort, pyupgrade, flake8, and pydocstyle. However, I’ve recently discovered a powerful new tool called Ruff), an extremely fast Python linter written in Rust. Ruff is not only capable of replacing all these tools but also executes much faster, which significantly improves the development workflow.

Ruff offers an impressive range of features and performance enhancements. Some of its standout features include:

10-100x faster than existing linters
Over 500 built-in rules
Autofix support for automatic error correction
Built-in caching to avoid re-analyzing unchanged files
First-party editor integrations for VS Code and more

Moreover, Ruff has gained widespread adoption in the Python community and is used in major open-source projects like Apache Airflow, FastAPI, Hugging Face, Pandas, and SciPy. The tool is backed by Astral, which provides active development and support.

One of the best aspects of Ruff is the native VS Code extension, which continuously scans your code and can fix problems as you write. This seamless integration makes it an indispensable part of any Python development workflow.

To install Ruff, simply run the following command:

pip install ruff

To use Ruff in your project, run the following command in the terminal:

ruff your_file.py

For the best experience, I highly recommend using Ruff in conjunction with pre-commit, as mentioned earlier. This ensures that your code is always linted and formatted before committing, maintaining a consistent and high-quality codebase.

In conclusion, Ruff is an incredibly fast and powerful Python linter that streamlines the development process by consolidating multiple tools into a single, efficient package. By adopting Ruff and integrating it with pre-commit, you can enjoy a more productive and enjoyable Python development experience.

🔍 5. Mypy: static type checking for Python (find bugs before they happen) #

As developers, we’re always looking for ways to improve the quality of our code and minimize the time spent on debugging. This is where Mypy comes in. Mypy is a powerful static type checker for Python, designed to catch potential runtime errors before they occur. By using Mypy, you can ensure that your code adheres to type hints and that it runs smoothly without unexpected issues.

Mypy offers several advantages to Python developers, including:

Early detection of potential type errors, preventing many common bugs
Improved code readability and self-documentation through type annotations
Easier refactoring and code maintenance, as type annotations provide clear contracts between functions and modules
Better collaboration and communication within teams, since type annotations make the code’s intent and expectations more explicit

Adding Mypy to an existing project may initially seem time-consuming and frustrating, as it might flag numerous issues that require attention. However, once you’ve addressed these concerns and properly set up Mypy, the benefits far outweigh the initial effort. In fact, the time you invest in fixing type-related issues will quickly be recovered as Mypy helps you prevent bugs and reduce debugging time.

For a long time, I too ignored Mypy and only typed my code without actually checking it with the tool. But after truly embracing Mypy and integrating it into my development process, I can confidently say that I will never go back. The value it brings to the table in terms of code quality, readability, and reliability is simply too great to ignore.

By proactively identifying and resolving type-related issues, Mypy not only saves you precious debugging time but also enhances the overall quality of your codebase. As you continue to rely on Mypy, you’ll find that it becomes an indispensable part of your Python development toolkit, ensuring that your projects remain robust and maintainable.

To start using Mypy in your project, first install it with the following command:

pip install mypy

Then, you can run Mypy on your Python files by executing the following command in your terminal:

mypy your_file.py

For even better integration, you can add Mypy as a pre-commit hook, ensuring that your code is always checked for type consistency before being committed. This further strengthens the code quality and helps prevent potential runtime errors.

In summary, incorporating Mypy into your Python development workflow adds an extra layer of safety and clarity, making it easier to write high-quality, reliable code. By combining Mypy with other tools like Ruff and pre-commit, you can create a robust development environment that promotes productivity and code quality.

🧪 6. Nox and pytest for testing: a powerful combination #

When developing software, it’s essential to ensure that your code is reliable and functions as expected. The key to achieving this goal is writing thorough tests, which not only help you identify issues early on but also prevent regressions in the future. While it’s often tempting to dive headfirst into coding, taking a step back and first writing tests for your solution can lead to more efficient and effective development. Even if this approach doesn’t come naturally, it’s crucial to write tests for every problem you encounter to keep your codebase stable and reliable.

In the realm of Python testing, Nox and pytest stand out as powerful tools that can greatly enhance your testing process. Nox is a versatile automation tool that simplifies running tests in multiple virtual environments, while pytest is a popular and feature-rich testing framework for Python. Together, these tools provide a formidable testing environment that simplifies test management and execution.

In the past, you may have used Tox, a tool that manages test environments with a text-based configuration file (e.g., in setup.cfg or pyproject.toml). While Tox has served developers well for many years, including myself, Nox offers a more flexible alternative. Instead of relying on a text-based configuration, Nox takes advantage of a Python configuration file, providing virtually limitless configurability. This added flexibility makes it easy to create complex test matrices with different versions and dependencies, and even accommodate special conditions using simple if statements. After happily using Tox for years, I discovered Nox and found that it elevated my testing workflow to a whole new level.

To get started with Nox and pytest, create a basic nox.py file and a sample test file for pytest. Here’s an example nox.py configuration:

import nox

@nox.session(python=["3.7", "3.8", "3.9"])
def tests(session):
 session.install("-r", "requirements.txt")
 session.install("-r", "test-requirements.txt")
 session.run("pytest")

And an example test file for pytest:

import pytest

def test_addition():
 assert 1 + 1 == 2

def test_subtraction():
 assert 3 - 1 == 2

Beyond basic testing functionality, pytest boasts powerful features such as fixtures, mocking and patching, and parameterization. Pytest fixtures enable you to create reusable resources for your tests, while mocking and patching allow you to replace parts of your system with fake objects, isolating your tests from external dependencies. Parameterization, implemented using pytest.mark.parametrize, enables running the same test with multiple sets of inputs and expected outputs.

In conclusion, leveraging Nox and pytest for testing offers numerous advantages for developers who strive to improve code reliability and automate their testing workflows. By harnessing Nox’s exceptional configurability and pytest’s extensive testing features, you’ll establish a robust testing environment that bolsters code quality and ensures optimal performance.

🏷️ 7. Versioningit: version management with Git tags #

(Optional, only related when making a Python package)

For developers seeking an automated and customizable package versioning solution, Versioningit offers a powerful yet flexible approach. It is a setuptools plugin that automatically determines your package’s version based on your version control repository’s tags. The real advantage of Versioningit lies in its ability to customize the version format and even override separate functions used for version extraction and calculation.

Versioningit makes your release workflow extremely simple, especially if you have set up an automated build and twine PyPI upload pipeline. To create a new release, you only need to tag a version, and then click on the “Release” button in GitHub.

One of the major advantages of Versioningit is its minimal boilerplate code compared to other tools like Versioneer. Additionally, the Python __version__ string works seamlessly with editable installs (pip install -e .), unlike setuptools_scm, which is a common issue faced by developers.

Over the years, I have explored many options for package versioning, including Versioneer, Miniver, setuptools_scm, bumpversion (or bump2version), and manually setting everything. Versioningit stands out due to its simplicity, customization options, and compatibility with various workflows.

Some key features of Versioningit include:

Support for Git, modern Git archives, and Mercurial
Customizable version formatting using template strings
Optional writing of the final version and other details to a file for loading at runtime
Custom setuptools commands for inserting the final version and other details into a source file at build time
Customizable functions for VCS querying, tag-to-version calculation, version bumping, version formatting, and writing the version to a file

To use Versioningit in your project, simply add it to your project’s pyproject.toml file in the build-system.requires key. Then, create a [tool.versioningit] table in your pyproject.toml file. You can get up and running with just the minimal configuration - an empty table.

Once you have a [tool.versioningit] table in your pyproject.toml and your repository has at least one tag, building your project with setuptools will automatically set your project’s version based on the latest tag in your Git repository. With Versioningit, you can customize the version format using placeholder strings, making it adaptable to your project’s specific needs.

In summary, Versioningit streamlines the package versioning process and simplifies the release workflow. By automating package versioning based on version control repository tags and providing extensive customization options, it helps make the management of package versions more efficient and less error-prone.

📝 8. Pyproject.toml and PEP 621: Configuration Consolidation #

The pyproject.toml file plays a crucial role in consolidating configurations and setup for various tools in a Python project, such as Black, Ruff, Mypy, Versioningit, Nox, and pytest. This adherence to PEP 621 standards streamlines the development process by providing a single source of truth for multiple tools, eliminating the need for additional configuration files like setup.cfg or setup.py.

By specifying packaging details directly in the pyproject.toml file, you can efficiently manage your project’s dependencies and metadata. For example, the Adaptive Scheduler package demonstrates this approach with a comprehensive pyproject.toml file containing all the necessary configurations.

(click to unfold):

[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools", "wheel", "versioningit"]

[project]
name = "adaptive_scheduler"
description = "Run many `adaptive.Learner`s on many cores (>10k) using `mpi4py.futures`, `ipyparallel`, `dask-mpi`, or `process-pool`."
requires-python = ">=3.8"
dynamic = ["version"]
maintainers = [{ name = "Bas Nijholt", email = "bas@nijho.lt" }]
license = { text = "BSD-3" }
classifiers = [
 "Development Status :: 2 - Pre-Alpha",
 "Intended Audience :: Developers",
 "Intended Audience :: Science/Research",
 "License :: OSI Approved :: BSD License",
 "Operating System :: OS Independent",
 "Programming Language :: Python",
 "Programming Language :: Python :: 3",
 "Programming Language :: Python :: 3.8",
 "Programming Language :: Python :: 3.9",
 "Programming Language :: Python :: 3.10",
 "Programming Language :: Python :: 3.11",
 "Topic :: Scientific/Engineering",
 "Topic :: System :: Distributed Computing",
]
dependencies = [
 "adaptive >= 0.14.1",
 "cloudpickle",
 "dill",
 "ipyparallel",
 "ipywidgets",
 "itables",
 "jinja2",
 "loky",
 "numpy",
 "pandas",
 "psutil",
 "pyarrow",
 "pyzmq",
 "rich",
 "structlog",
 "tinydb",
 "toolz",
 "tqdm",
 "versioningit",
]
[project.optional-dependencies]
all = ["dask-mpi", "mpi4py"]
test = ["pytest", "pytest-asyncio", "coverage", "pytest-cov"]
docs = [
 "myst-nb",
 "sphinx_fontawesome",
 "sphinx",
 "furo",
 "myst-parser",
 "emoji",
]
[project.urls]
homepage = "https://adaptive-scheduler.readthedocs.io/"
documentation = "https://adaptive-scheduler.readthedocs.io/"
repository = "https://github.com/basnijholt/adaptive-scheduler"

[project.readme]
content-type = "text/x-rst"
file = "README.rst"

[project.scripts]
adaptive-scheduler-launcher = "adaptive_scheduler._server_support.launcher:main"

[tool.versioningit]

[tool.versioningit.onbuild]
build-file = "adaptive_scheduler/_version.py"
source-file = "adaptive_scheduler/_version.py"

[tool.setuptools.packages.find]
include = ["adaptive_scheduler.*", "adaptive_scheduler"]

[tool.pytest.ini_options]
addopts = """
 -vvv
 --cov=adaptive_scheduler
 --cov-report term
 --cov-report html
 --cov-report xml
 --cov-fail-under=35
 --asyncio-mode=auto
"""

[tool.coverage.report]
exclude_lines = [
 "pragma: no cover",
 "raise NotImplementedError",
 "if TYPE_CHECKING:",
 "if __name__ == .__main__.:",
]

[tool.black]
line_length = 88

[tool.ruff]
line-length = 150
target-version = "py38"
select = ["ALL"]
ignore = [
 "T20", # flake8-print
 "ANN101", # Missing type annotation for {name} in method
 "S101", # Use of assert detected
 "PD901", # df is a bad variable name. Be kinder to your future self.
 "ANN401", # Dynamically typed expressions (typing.Any) are disallowed in {name}
 "D402", # First line should not be the function's signature
 "PLW0603", # Using the global statement to update `X` is discouraged
 "D401", # First line of docstring should be in imperative mood
 "SLF001", # Private member accessed
]

[tool.ruff.per-file-ignores]
"tests/*" = ["SLF001"]
"tests/test_examples.py" = ["E501"]
".github/*" = ["INP001"]

[tool.ruff.mccabe]
max-complexity = 18

[tool.mypy]
python_version = "3.8"

(from here)

As seen in the example, the pyproject.toml file includes details about the build system, project metadata, dependencies, optional dependencies, URLs, and configurations for various tools such as Versioningit, setuptools, pytest, coverage, Black, Ruff, and Mypy. By consolidating these configurations, you can maintain a more organized and maintainable project, making it easier to understand and update settings for different tools.

In summary, the pyproject.toml file and adherence to PEP 621 provide a unified approach to configuring and setting up your Python project, making it easier to manage dependencies, metadata, and tool-specific settings. This streamlined process ultimately leads to a more efficient and maintainable development workflow.

🌐 9. GitHub Actions CI Matrix: Testing on Multiple Platforms and Auto Publishing to PyPI #

GitHub Actions is an excellent tool for automating workflows and improving code quality through continuous integration (CI). By using a CI matrix, you can test your code on multiple platforms and Python versions, ensuring that your package remains compatible and stable across different environments.

To set up a .github/workflows/nox.yaml file for GitHub Actions, you can use the following simplified example:

name: nox

on:
 pull_request:
 push:
 branches: [main]

jobs:
 test:
 runs-on: ${{ matrix.platform }}
 strategy:
 fail-fast: false
 matrix:
 platform: [ubuntu-latest, macos-latest, windows-latest]
 python-version: ["3.8", "3.9", "3.10", "3.11"]

 steps:
 - uses: actions/checkout@v2
 - name: Set up Python ${{ matrix.python-version }}
 uses: actions/setup-python@v2
 with:
 python-version: ${{ matrix.python-version }}
 - name: Install dependencies
 run: pip install nox
 - name: Test with nox
 run: nox -e py${{ matrix.python-version }}

This configuration file specifies the platforms and Python versions for testing, as well as the steps to set up the environment, install dependencies, and run tests using nox. With this simple configuration, you set up tests on MacOS, Windows, Linux, and multiple Python versions, ensuring that your package remains compatible across different environments.

Another essential part of the CI/CD process is deploying your Python package. You can use the .github/workflows/publish-package.yaml file to automate the package deployment to PyPI whenever a new release is published:

name: Upload Python Package

on:
 release:
 types: [published]

jobs:
 deploy:

 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v2
 - name: Set up Python
 uses: actions/setup-python@v2
 with:
 python-version: '3.x'
 - name: Install dependencies
 run: |
 python -m pip install --upgrade pip
 pip install setuptools wheel twine build
 - name: Build and publish
 env:
 TWINE_USERNAME: __token__
 TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
 run: |
 python -m build
 twine upload dist/*

This workflow sets up Python, installs necessary dependencies, builds the package, and publishes it to PyPI using Twine. The process integrates seamlessly with versioningit, ensuring that the published package has the correct version number based on the git history.

In summary, leveraging GitHub Actions for continuous integration and automated workflows enhances code quality and reliability, while also simplifying the development and deployment processes. By testing across various platforms and Python versions, and automating package deployment, you can be confident that your package is well-maintained and accessible to users.

📚 10. Read the Docs, Sphinx, MyST-NB, and Jupytext notebooks for documentation #

Documentation is crucial for any software project, as it helps users understand how to use the package and provides guidance for contributors. Read the Docs, Sphinx, MyST-NB, and Jupytext notebooks are powerful tools that, when combined, facilitate the creation and maintenance of high-quality documentation.

tl;dr: check out the Adaptive documentation for an example of how to use these tools to create a comprehensive documentation website.

Read the Docs is a documentation hosting platform that automatically builds, version controls, and hosts your documentation for free. It integrates well with Sphinx and GitHub, allowing your documentation to be updated whenever you push changes to your repository.

Sphinx is a documentation generator that can transform reStructuredText files, MyST files, or Jupyter notebooks into various output formats, such as HTML, LaTeX, or PDF. It is highly extensible, supporting custom themes and plugins, and can automatically generate API documentation from your Python code.

MyST-NB is an extension to Sphinx that enables it to parse Jupyter notebooks and execute the code cells during the build process. It uses the MyST (short for Markedly Structured Text) markdown parser, which adds support for Sphinx roles and directives, making it a powerful alternative to reStructuredText.

Jupytext notebooks is a Jupyter extension that enables you to save Jupyter notebooks in Markdown format (or other text-based formats) instead of the default JSON format. By using Jupytext notebooks, you can store your code, explanations, and visualizations as plain text, which is easier to version control and can be seamlessly integrated with MyST-NB and Sphinx.

With MyST-NB and Jupytext, you can write your documentation using Jupyter notebooks in Markdown format. This approach combines the benefits of Jupyter notebooks, such as interactive code execution and rich output, with the simplicity of Markdown and the powerful features of Sphinx. The result is a highly readable, user-friendly, and maintainable documentation that caters to both users and developers.

To get started with MyST-NB, you’ll need to install it using pip:

pip install myst-nb

Next, configure your Sphinx project to use MyST-NB by adding the following lines to your conf.py file:

extensions = [
 "myst_nb",
]

Now, Sphinx will be able to parse and execute your Jupyter notebooks during the documentation build process, integrating them seamlessly with your existing documentation.

In summary, using Read the Docs, Sphinx, MyST-NB, and Jupytext notebooks for your documentation enables you to create high-quality, interactive, and easily maintainable documentation that benefits your project’s users and contributors. This combination of tools ensures that your documentation remains up-to-date, easily navigable, and engaging, ultimately enhancing the overall experience for your project’s audience.

For a full example, see how I setup Adaptive’s documentation using Sphinx, MyST-NB, and Jupytext notebooks. readthedocs.yaml, docs/source/conf.py, and here is an example notebook that is used in the documentation: docs/source/tutorial/tutorial.Learner1D.md

🎉 11. Conclusion #

In conclusion, incorporating these powerful tools into your Python development workflow can significantly enhance your productivity and code quality. By adopting pre-commit, Black, Ruff, Mypy, Versioningit, Pyproject.toml with PEP 621, Nox, pytest, GitHub Actions CI matrix, and Read the Docs, Sphinx, MyST-NB, and Jupytext notebooks for documentation, you create a streamlined, efficient, and reliable development environment.

These tools help maintain code consistency, catch issues early, automate package versioning, simplify testing, consolidate configurations, and create high-quality documentation. By integrating them into your projects, you can focus on writing great code while ensuring that your work remains robust, maintainable, and accessible to users and contributors alike.

Don’t hesitate to explore these tools and incorporate them into your Python development toolkit. They can transform your development experience, making it more enjoyable and efficient, ultimately leading to higher-quality projects and happier developers. Happy coding!

Packaging | Bas Nijholt

Auto-installing extras at runtime so users never have to think about dependencies

1. From one command to a dependency maze #

2. Why “just use extras” breaks down #

3. The decorator #

4. Checking without importing #

5. The metadata bridge #

6. Three environments, three strategies #

A) uv tool install — the persistent environment #

B) uvx — the ephemeral environment #

C) Regular virtualenv, pipx, or system install #

7. Preventing infinite re-execution loops #

8. What users actually see #

9. Keeping everything in sync #

10. Why uv made this practical #

11. Tradeoffs #

12. Not dependency injection #

13. Conclusion #

Confessions of an asymmetric hypocrite: on Python dependencies

Introduction: The Dependency Dilemma 🤔 #

1. The Library Author Hat 🎩 (as Dependency Consumer): Maximum Caution #

2. The Library Author Hat 🎩 (as Dependency Producer): The Paradox #

3. The Application Developer / Contributor Hat 🧢: Practicality is Key #

4. Conclusion: Not Hypocrisy, but Context-Driven Strategy 🙏 #

GPL for Python packaging ⚖️

Understanding GPL for Python Packaging #

What is GPL? #

What Does “Linking” Mean in Python? #

Why Does This Matter for Python Packaging? #

Recommendations #

Python environments that stuck: micromamba and uv 🐍

1. Introduction 🚀 #

2. The Alternatives 🌟 #

🐍 Pyenv #

📦 Pipenv #

🌐 Conda #

🚀 Mamba #

🌌 Micromamba #

🐳 Docker #

🏞️ Venv #

🌐 Virtualenv #

🍺 Homebrew #

💻 System package managers #

🌟 Pixi #

⚡ uv #

3. Conclusion 🏁 #

4. Further Reading 📚 #

UniDep 🧬

📖 Origin Story #

🔧 Technical Highlights #

📊 Impact #

🎯 Challenges and Solutions #

💡 Lessons Learned #

🔮 Future Plans #

conda-forge Contributions 🐍

📖 Origin Story #

🔧 Technical Highlights #

📊 Impact #

🎯 Challenges and Solutions #

💡 Lessons Learned #

My favorite tools that keep my Python projects sane 🧰

🌟 1. Introduction #

🛠️ 2. Pre-commit #

🎨 3. Black: auto-formatting for Python #

🚀 4. Ruff: the fastest Python linter and auto-formatter #

🔍 5. Mypy: static type checking for Python (find bugs before they happen) #

🧪 6. Nox and pytest for testing: a powerful combination #

🏷️ 7. Versioningit: version management with Git tags #

📝 8. Pyproject.toml and PEP 621: Configuration Consolidation #

🌐 9. GitHub Actions CI Matrix: Testing on Multiple Platforms and Auto Publishing to PyPI #

📚 10. Read the Docs, Sphinx, MyST-NB, and Jupytext notebooks for documentation #

🎉 11. Conclusion #

A) `uv tool install` — the persistent environment #

B) `uvx` — the ephemeral environment #

C) Regular virtualenv, `pipx`, or system install #