Scientific-Computing | Bas Nijholt

pfapack 🔢

Mon, 23 Dec 2024 00:00:00 +0000

Ever needed to calculate the square root of the determinant of a skew-symmetric matrix? Probably not, but if you’re in quantum physics, you might have! This is called the Pfaffian. Today’s project, pfapack, is a Python package that provides efficient routines for computing it. The underlying algorithms and original code implementations were developed by my PhD co-promotor Michael Wimmer. I created the pfapack package to make this powerful tool easily accessible.

📖 Origin Story #

Working in quantum physics, I frequently need to compute the Pfaffian. Michael had not only developed the algorithms but also wrote highly optimized C, FORTRAN, and Python implementations, distributing them as a tar file on his website. To make it easier for researchers working in Python, I packaged his existing Python code, added wrappers for his C and FORTRAN implementations, and set up automated builds for PyPI and conda-forge to make it easily installable.

🔧 Technical Highlights #

Python Wrapper for Efficient Code: User-friendly Python interface to Wimmer’s optimized C and FORTRAN.
Multiple Algorithms: Supports both the Parlett-Reid and Householder methods.
Optimized for Performance: Leverages low-level optimizations for speed.
Conda and Pip Installable: Easy installation via conda-forge and PyPI.
Automated Wheel Builds: Uses pypa/cibuildwheel for streamlined, automated builds and releases.
Cross-Platform Compatibility: Works on Linux, macOS, and Windows (with some caveats).

📊 Impact #

Makes specialized mathematical algorithms readily available to Python users.
Simplifies the use of highly optimized code.
15 GitHub stars.
Used in research projects that require efficient Pfaffian computations, particularly in quantum physics.

🎯 Challenges and Solutions #

Bridging Languages: Wrapping C and FORTRAN for Python integration.
Automated Builds: Setting up automated builds for multiple platforms was complex.
Windows Compatibility: As always Windows support is a PITA; it currently requires MSYS2 and MinGW64.

💡 Lessons Learned #

Making specialized tools accessible broadens their impact.
Wrapping low-level code in Python can improve usability.
Automated builds are essential for maintainability.
pypa/cibuildwheel is awesome for cross-platform builds.

🙏 Credits #

All the credits for the underlying algorithms and original code implementations go to Michael Wimmer. I simply made it easier to distribute and use in Python.

Want to compute Pfaffians efficiently in Python? Check out pfapack on GitHub or install it via pip install pfapack or conda install -c conda-forge pfapack!

#OpenSource #Python #ScientificComputing #Math #Pfaffian #FORTRAN #C

pipefunc 🕸️

Fri, 20 Dec 2024 00:00:00 +0000

(See my intro post)

Of all the projects I’m sharing this month, this one that started as a passion project excites me the most!

📖 Origin Story #

Sometimes the best projects are born from the most unexpected moments. More than 1½ year ago, while on parental leave with my twin boys, I found myself with small pockets of time during naps to work on a problem that had been bothering me for years: the tedious bookkeeping required in complex computational workflows. Every scientific computation project seemed to reinvent the same patterns - managing function dependencies, parameter sweeps, result caching, parallelization, and a lot of boilerplate to combine the resulting data. I wanted something that would let scientists focus on their science, not on pipeline management.

Example of a pipeline generated by PipeFunc

🔧 Technical Highlights #

Automatic DAG construction via very simple and lightweight syntax
N-dimensional parameter sweeps with automatic parallelization
Visual pipeline representation using NetworkX
Resource profiling (CPU, memory, time)
Type validation between pipeline stages
Ultra-fast: only 15 µs overhead per function
Flexible caching strategies (memory, disk, cloud)
Integration with scientific computing tools:
- NetworkX for graph algorithms
- NumPy for multi-dimensional arrays
- Xarray for labeled arrays
- Zarr for distributed storage
- Adaptive for parallel computing

📊 Impact #

230 GitHub stars
700+ tests with 100% coverage
Fully typed codebase
Comprehensive documentation
Tested on real workflow on SLURM cluster
Covered on Pycoder’s Weekly with >100k subscribers

🎯 Challenges and Solutions #

Balancing simplicity with power
Making complex workflows intuitive
Handling distributed computing edge cases
Ensuring type safety across the pipeline
Optimizing performance without sacrificing features

💡 Lessons Learned #

Sometimes the best time to code is during baby naps
Complex problems can have elegant solutions
Scientific computing needs better tooling
Good abstractions make hard things easy
Type hints and tests prevent headaches

🔮 Future Plans #

The journey is far from over. Plans include:

Enhanced cloud computing support
More interactive visualization options
Interactive pipeline debugging tools
Expanded parameter sweep capabilities

Want to simplify your computational workflows? Check out pipefunc on GitHub or read the documentation!

#OpenSource #Python #DataScience #ScientificComputing #Programming

Adaptive Scheduler 🚀

Tue, 17 Dec 2024 00:00:00 +0000

(See my intro post)

Ever tried to efficiently run 100,000 cores from a Jupyter notebook? No? Well, I have, and it led to some interesting discoveries about the limits of parallel computing tools.

📖 Origin Story #

During my 2018 internship at Microsoft Quantum, we faced a unique challenge: our adaptive sampling algorithms needed to run on massive computing clusters (50,000+ cores), but existing tools couldn’t handle it efficiently.

The problem was interesting: traditional parallel computing tools like Dask rely on a central scheduler process. Imagine 100,000 cores, each finishing a task every 10 seconds. That means every 10 microseconds, a core needs new work! With typical scheduler overhead being 1-50 milliseconds, workers would spend 99% of their time waiting rather than computing.

Usually, you’d solve this by batching work ahead of time. But with adaptive sampling (yesterday’s project), that’s impossible - you need the results of previous calculations to know what to compute next. It was like trying to conduct an orchestra where each musician needs personal instructions every few seconds!

🔧 Technical Highlights #

We created a solution that:

Acts as a meta-scheduler, launching multiple sub-schedulers (Dask, ipyparallel, mpi4py, etc.)
Minimizes communication between scheduling layers
Handles job management through a simple database
Supports automatic fault tolerance and data saving
Works with any cluster scheduler (SLURM, PBS)
Provides real-time progress tracking in Jupyter notebooks
Scales to 50,000+ cores without breaking a sweat

📊 Impact #

Used extensively for real quantum device simulations
26 GitHub stars (but don’t let that fool you - it’s highly specialized!)
Enables interactive supercomputing from a simple Jupyter notebook
Likely ran for many millions of core-hours across various projects
Actively maintained with 61 releases since 2018, proving its ongoing value

🎯 Challenges We Solved #

Scheduler overhead killing performance at scale
Data locality and fault tolerance
Real-time monitoring of massive parallel jobs
Making supercomputing accessible through notebooks
Automatic job recovery after crashes (e.g., when using spot VMs or Heisenbugs)

💡 Lessons Learned #

Sometimes you need to think outside the box
The best tools hide complexity behind simple interfaces
Real-time feedback is crucial, even for massive computations
When scaling up 1000x, you often need a completely different approach
Making things “just work” from a notebook is worth the effort

Want to run massive parallel computations efficiently? Check out Adaptive Scheduler on GitHub!

#OpenSource #Python #HPC #QuantumComputing #ParallelComputing

Adaptive 📈

Mon, 16 Dec 2024 00:00:00 +0000

(See my intro post)

During my PhD, I was running quantum transport simulations on our 800-core computational cluster. Traditional parameter sweeps using dense grids were hitting the cluster’s limits - wasting precious compute time on uninteresting regions while potentially missing important features. Most parameter spaces we explored were “boring” with occasional interesting regions that required higher sampling density.

📖 Origin Story #

Initially, in 2016, I had a collection of one-off adaptive sampling scripts. Together with Joseph Weston and the support of my PhD advisor Anton Akhmerov, we turned it into a full-fledged Python package “Adaptive”. What began as an internal tool to make our cluster usage more efficient slowly gained traction in the wider scientific computing community.

🔧 Technical Highlights #

Multiple adaptive sampling strategies (1D, 2D, ND, averaging stochastic functions)
Parallel execution support (local, MPI clusters, distributed systems)
Live plotting in Jupyter notebooks with real-time feedback
Customizable loss functions for different sampling priorities
Minimal dependencies

📊 Impact & Applications #

1164 GitHub stars and thousands of daily downloads
Cited in 20+ scientific publications
Dependency of many packages, most notably: Orange Quantum System’s Quantify software for controlling real quantum computers!

Used in cutting-edge research across fields:

Quantum Computing: Simulating and controlling superconducting qubits and trapped ion systems
Quantum Physics: Exploring topological quantum systems and novel materials
Materials Science: Studying graphene and other 2D materials
Engineering:
- Optimizing radar system performance
- Fluid dynamics simulations
Chemistry: Molecular modeling and vibrational spectroscopy
Most Unexpected Use: Found in a paper on predicting energy usage of wartime bungalows!

This range of applications shows how a tool built for quantum transport simulations can find surprising uses across disciplines, from fundamental physics to everyday buildings.

🎯 Challenges and Solutions #

Making parallel execution truly scalable while maintaining sampling intelligence
Creating intuitive visualization tools for complex parameter spaces
Handling stochastic functions and noisy data

💡 Lessons Learned #

Building a user base takes active effort - giving talks and engaging online made all the difference
Good documentation converts interested users into regular contributors
This was my first open source package and learned all about packaging, testing, and CI/CD

Want smarter parameter space exploration? Check out Adaptive on GitHub or try our interactive tutorial!

#OpenSource #Python #ScientificComputing #ParallelComputing #Programming

conda-forge Contributions 🐍

Wed, 11 Dec 2024 00:00:00 +0000

(See my intro post)

Unlike other projects in this advent calendar, I want to celebrate being part of something much bigger: the conda-forge community, where thousands of volunteers maintain the packages that power scientific Python.

📖 Origin Story #

I started with conda-forge in 2016 during my PhD work with Kwant, a quantum transport simulator. Installing it was a nightmare - it required system-level numerical libraries like MUMPS and Scotch which required a special config file to link. When I discovered conda-forge and its mission to make scientific software installation painless, I knew this was the solution!

The humble work of conda-forge maintainers

🔧 Technical Highlights #

Over the past 8 years, I’ve maintained over 40 build recipes, including:

Major scientific packages like VTK, MUMPS, and Scotch
Quantum physics tools like Qsim, Qcodes, and Kwant
Visualization libraries like HoloViews
Plus all the packages I’ve authored myself

The real magic of conda-forge is its infrastructure:

Automated builds across Linux, macOS, and Windows
Strict dependency version management
Comprehensive CI/CD pipelines
Community-driven quality control

📊 Impact #

What makes conda-forge truly special is how initial contributions blossom into collaborative efforts. I’ve experienced this firsthand: after creating the initial MUMPS feedstock, 25 other contributors joined in, each bringing their unique expertise.

A recent experience also illustrates this: I added CUDA support to both Google’s Qsim and IBM’s Cirq packages. Shortly after, Leo Fang, an engineer from NVIDIA who knows CUDA far better than I do, stepped up and made significant improvements. This is the beauty of open source - experts naturally gravitate to where they can make the biggest impact.

This collaborative approach solves a problem I’ve encountered at several companies: the tendency to maintain complicated build systems internally with custom hacks. Instead of each organization reinventing the wheel, conda-forge provides a shared platform where:

Maintenance burden is distributed
Expert knowledge flows freely
Build practices are standardized
Infrastructure is public and reusable

🎯 Challenges and Solutions #

Keeping up with upstream changes
Managing complex dependency trees
Cross-platform compatibility
Coordinating with upstream maintainers

💡 Lessons Learned #

Community effort beats solo work for infrastructure
Automation is crucial for reliability
Small contributions compound over time
Being part of something bigger is rewarding

Want to contribute to scientific Python? Check out conda-forge and the staged-recipes repository!

#OpenSource #Python #Scientific #Programming #Community

numthreads 🧵

Sat, 07 Dec 2024 00:00:00 +0000

(See my intro post)

Sometimes the smallest tools solve the most persistent problems. Today’s project is about taming automatic parallelization in scientific computing - a deceptively simple challenge that has cost countless CPU hours.

📖 Origin Story #

While working on high-performance computing clusters, I frequently encountered a counterintuitive problem: code running slower on multiple CPU cores than on a single one. Many scientific libraries (NumPy, SciPy) automatically parallelize operations, which sounds great but can actually harm performance when you’re already parallelizing at a higher level. After repeatedly explaining to colleagues why they needed to set various environment variables to disable this behavior, I created numthreads to solve it once and for all.

Overenthusiastic threading

🔧 Technical Highlights #

Controls thread count for major numerical libraries:
- OpenBLAS
- Intel’s Math Kernel Library (MKL)
- OpenMP
- NumExpr
- Accelerate
Zero dependencies
Tiny (≤7KB) yet solves a real problem
Both CLI and Python API

📊 Impact #

Over my scientific computing career, I’ve encountered this threading issue dozens of times. Since creating this package a year ago, I’ve helped at least 5 different researchers speed up their workflows, sometimes by orders of magnitude. The beauty lies in its simplicity - the fix is often as simple as:

eval $(numthreads 1)

💡 Lessons Learned #

The simplest solutions are often the best
More cores doesn’t always mean faster code
If you find yourself repeating the same instructions, write a tool
Even a 7KB package can save hours of computing time

Want better control over your numerical computations? Check out numthreads on GitHub!

#OpenSource #Python #HPC #ScientificComputing #Programming

Open Science Publications 🔬

Fri, 06 Dec 2024 00:00:00 +0000

(See my intro post)

Today’s post is about something I have strong opinions on: making scientific research reproducible. While many researchers talk about open science, actually making your work reproducible for anyone requires significant effort.

📖 Origin Story #

During my Ph.D. and subsequent research, I noticed a frustrating pattern in scientific publications: claims of “code available upon request” that often led nowhere, or code snippets that wouldn’t run without substantial modification. I made an effort that my publications would be different – each includes complete, runnable code that reproduces every figure and result.

🔧 Technical Highlights #

Multiple repositories covering quantum physics experiments and simulations:
- orbitalfield - “Orbital effect of magnetic field on the Majorana phase diagram” (94 citations)
- supercurrent-majorana-nanowire - “Supercurrent interference in few-mode nanowire Josephson junctions” (78 citations)
- zigzag-majoranas - “Enhanced proximity effect in zigzag-shaped Majorana Josephson junctions” (56 citations)
- azure-quantum-tgp - “Protocol to identify a topological superconducting phase in a three-terminal device” (52 citations)
- spin-orbit-nanowires - “Spin-orbit protection of induced superconductivity in Majorana nanowires” (81 citations)

🔄 Evolution of Best Practices #

What I consider best practices has evolved over time. Today, I would:

Use Pixi for universal lock files across operating systems and programming languages
Provide a minimal Docker container that just installs the pixi lock file
Create self-documenting Jupyter notebooks that reproduce every result
Include clear figure-to-code mapping for paper reproducibility

🎯 Challenges and Solutions #

Balancing code cleanliness with research deadlines
Managing large datasets efficiently
Ensuring long-term reproducibility
Making complex physics simulations accessible

💡 Lessons Learned #

Clean code takes time but saves more in the long run
Documentation is as crucial as the code itself
Lock files are essential for true reproducibility
Making code public improves its quality

Want to explore quantum physics simulations? Check out the repositories above, each linked to their corresponding papers with full reproduction instructions.

#OpenSource #OpenScience #Physics #QuantumComputing #Python

Crafting unique Ph.D. thesis covers with Python 🎨

Sat, 04 Apr 2020 13:40:00 +0000

Originally posted on quantumtinkerer.tudelft.nl/blog/thesis-cover on 2020-04-04 13:40.

I will hand out around a hundred copies of my thesis to my defense committee, colleagues, and my family and friends. Let’s be honest, most people will probably not get further than attempting to read the summary and appreciating the cover. Four years of work has gone into generating the content of the thesis, so I figured, at least some thought has to go into the design of its cover. Unfortunately, I am by no means an expert on a graphical design or even competent enough to attempt to use any kind of graphic design software. But luckily for me, I do consider myself an expert in Python , and why not make the thesis design a fun process?

Initial Design: Adaptive Sampling and Data Visualization #

My vague plan was to somehow incorporate a plot resulting from a simulation run with Adaptive [source ], the software I wrote during my Ph.D. to adaptively sample functions in parallel and my proudest work. I even had a potential simulation result in mind that I would use for it, specifically, one that has been on the background of my good friend’s phone for some years and one that has earned me quite some internet points on Reddit. This plot (below) is one of the results of Chapter 4 of my thesis [source ] and this paper.

Your browser does not support the video tag.

I went back to the code that generates the plot and started hacking away in Matplotlib. Quickly, I realized that not only the plot above was going to look cool, but that many other data sets that I had also resulted in beautiful patterns. I hit the paradox of choice: I had a total of 3671 unique data sets (all for Majorana devices with either different parameter sweeps or different physical effects that were included in the model) and no idea how I was going to make a selection.

This is an early attempt (and rather ugly in hindsight)

I decided to postpone the choice of which data to use and just focus on how to plot the data first. I experimented with different colormaps, fading colormaps that gradually reveal the underlying triangulation better, and more. On a weekend night, after telling a friend I was with about my plans, we (as usually happens) started to program some of the ideas we came up with. We settled on a design where the triangulation would slowly fade from the top to bottom and where each triangle simultaneously transitioned from a solid color (the average of the values at its vertices) to densely interpolated values. See this Jupyter notebook which generated the final version of the cover. After we also chose a nice colormap (by simply creating the cover for each possible colormap,) the following was suggested: “You know what would be even cooler?! To make every cover unique.” Ten minutes later, I had started 3671 jobs on our computational cluster. When the files slowly started to appear on my computer (I have my local files synced with the cluster), we could not have been more excited. There were simply so many cool looking alternatives. Too many to manually select now, to be honest—a problem for later I thought.

Overcoming the Paradox of Choice: Clustering and Neural Networks #

Four months later, with the rest of my thesis completely done and about a day left on the deadline to submit my thesis, I found myself facing the selection problem I had successfully ignored for so long. There must be a smarter way to select the covers, I thought. Many of them look roughly similar (and some of them are even quite boring or not ecstatically pleasing), so it does not make sense to scroll through all of them. I could cluster them by similarity and then I would just have to choose the nicely looking clusters while being able to instantly discard many.

Finally, a good application to use neural networks!

I still had the 3671 covers in pdf format in some folder, so using the wonderful GNU Parallel, I converted and rasterized all of them to jpeg with

find . -iname \*pdf -print0 | parallel -0 sips -s format jpeg {} --out {.}.jpg

Then I used a pre-trained convolutional neural network model (VGG16) to extract features in a lower-dimensional representation. I fed the feature arrays to a K-Means clustering algorithm that sorted them in 80 clusters. To procrastinate a little more, I created an interactive widget using ipywidgets (see video below) to facilitate the selection process.

Your browser does not support the video tag.

These are the clusters that I selected in a grid

The selection process was done using this Jupyter notebook.

Finally, I copied the selected clusters into different folders and used the OS’s file explorer to manually select a few covers from each cluster.

Final Touches: Fine-Tuning Colors and Adding Edition Numbers #

Then I printed an initial version of my thesis, however, I wasn’t really happy with the colors.

To choose the right color scheme I got help from a non-tech savvy friend who went to art school. Due to the COVID-19 crisis, we couldn’t sit side by side, so I created this Binder “app” with sliders.

Your browser does not support the video tag.

Additionally, since all of them were different anyway, I gave them an identifier and added

edition n of 120

to the back.

These are a few examples of the final result (click on one of the images to open gallery view)

A few days later, this was delivered to my doorstep

And finally I mailed them to my Ph.D. committee

Now it’s time to start preparing for my defense at May 11, 2020 at 15:00!

Notes #

The data and code to generate all printed covers is available at basnijholt/thesis-cover
The simulation code that generates the data is available at basnijholt/spin-orbit-nanowires
The video of the Reddit post is generated using this code