Data-Visualization | Bas Nijholt

tiller-streamlit 💰

Sun, 22 Dec 2024 00:00:00 +0000

Tracking personal finances is crucial, especially when working towards financial independence (see my WenFIRE post on Day 4). Today’s project is tiller-streamlit, a Streamlit app that visualizes your financial data from Tiller. I host it on my local network under money.local using Traefik for easy access.

📖 Origin Story #

Used to use Intuit Mint, but when they shut down, I needed an alternative with full data control.
Tiller automatically populates your financial data into a Google Sheet that you own forever.
With 20+ accounts, manual updates are tedious – Tiller automates it.
Combined Tiller’s data with Streamlit’s ease of use, creating a personal finance dashboard.

🔧 Technical Highlights #

Powered by Streamlit: Turns Python into a web app with minimal code.
Tiller Integration: Connects to Tiller’s Google Sheets for automated data retrieval.
Interactive Visualizations: Uses Plotly and Altair for dynamic charts.
Customizable Analysis: Filter by categories, time periods, and more.
Net Worth Tracking: Calculates and visualizes net worth over time.
Dockerized Deployment: Easy deployment with Docker, and served locally with Traefik.

📊 Impact #

Primarily for personal use, it has significantly improved my understanding of my finances.
Great way to learn Streamlit and explore its capabilities.
Visualizing spending patterns helps identify areas to cut back and make informed financial decisions on the path to financial independence.

🎯 Challenges and Solutions #

Learning Streamlit: Excellent documentation and community support made it a smooth process.
Data Wrangling: Transforming Tiller’s raw data into a format suitable for visualization.
Deployment: Docker and Traefik simplified local deployment, making it accessible on my home network.

💡 Lessons Learned #

Data Ownership Matters: Owning your financial data gives you control and flexibility.
Visualization is Key: Seeing your finances visually makes a huge difference.
Streamlit is Powerful: Simplifies web app development.
Automation is Essential: Tiller’s automatic data population makes the whole process sustainable.
Docker and Traefik Simplify Local Deployment: Making it easy to access the app on my home network.

🔮 Future Plans #

Budgeting Features: Integrate budget tracking and forecasting.
Goal Setting: Set financial goals and track progress.
Investment Tracking: Incorporate investment data for a complete picture.

Want to visualize your own finances with Tiller and Streamlit? Check out tiller-streamlit on GitHub!

#OpenSource #Python #Streamlit #PersonalFinance #DataVisualization #Tiller #Traefik #Docker

Interactive Graph Visualization 🕸️

Sat, 21 Dec 2024 00:00:00 +0000

(See my intro post)

After creating pipefunc (Day 20), I wanted a way to visualize its (possibly large and complex) computational graphs interactively in Jupyter notebooks. What started as a simple visualization PR evolved into two standalone packages: graphviz-anywidget (Python) and graphvizsvg (JavaScript) - which unexpectedly introduced me to JavaScript development!

📖 Origin Story #

While working on pipefunc’s visualization features, I discovered that existing solutions didn’t quite meet my needs. However, I found several components that would solve my issue:

AnyWidget (interactive GraphViz in Jupyter)
@hpcc-js/wasm-graphviz (GraphViz in WASM)
d3-graphviz (interactive GraphViz in D3)
jquery.graphviz.svg (GraphViz SVG manipulation)

AnyWidget would work with ESM modules, however, I had no clue what that even was. The jquery.graphviz.svg package wasn’t one and hadn’t been maintained in 8 years.

With AI as my JavaScript mentor, I first rewrote jquery.graphviz.svg into a modern ESM module (graphvizsvg), adding comprehensive tests that uncovered bugs in the original. Then, using AnyWidget, I created graphviz-anywidget to connect everything together - combining the WASM-powered d3-graphviz with the interactive features from graphvizsvg into a seamless Python package.

To my surprise, I found JavaScript’s development tooling remarkably sophisticated - especially npm, which handles everything from linting to testing to bundling to publishing. Something we’re still missing in Python!

🔧 Technical Highlights #

graphviz-anywidget:

Interactive Graphviz visualization in Jupyter
WASM-powered graph rendering
Directional graph traversal
Node and edge highlighting
Works in JupyterLab, Notebook, and VS Code

graphvizsvg:

Modern ESM module for interactive SVG visualization
Automatic color transitions
Bi-directional graph traversal
Comprehensive test coverage
My first npm package!

📊 Impact #

Enhances pipefunc’s visualization capabilities
Modernizes legacy jQuery code for modern web development
Proves that diving into new technologies can yield useful tools

🎯 Challenges and Solutions #

Learning JavaScript/TypeScript from scratch
Making WASM work in Jupyter
Getting it to work in VS Code

💡 Lessons Learned #

Don’t let preconceptions limit your tech choices
Modern JavaScript development is actually enjoyable
AI can be an excellent programming mentor
Good visualization makes complex data accessible
Breaking features into separate packages increases reusability
Test coverage helps find bugs in even well-used code

Want to visualize graphs interactively? Check out graphviz-anywidget and graphvizsvg!

#OpenSource #JavaScript #Python #DataVisualization #WebDev

PhD Thesis Cover Design 🎨

Sat, 14 Dec 2024 00:00:00 +0000

(See my intro post)

When most people think of open source, they think of software. But what about open-sourcing your PhD thesis and creating a generative art system for its cover?

📖 Origin Story #

With my PhD defense approaching and needing around 100 copies of my thesis, I faced a design challenge. As someone more comfortable with Python than Photoshop, I turned what started as a simple visualization idea into a full-blown generative art project with machine learning-based clustering.

Multiple printed copies of the thesis

🔧 Technical Highlights #

Generated 3,671 unique covers from quantum physics simulation data
Used VGG16 neural network to extract features from covers
Implemented K-means clustering to group similar designs
Created interactive IPywidgets for design selection
Built a Binder app for real-time color scheme adjustment
Added unique edition numbers (1/120, 2/120, etc.)
Made both thesis and cover design code fully open source

The project combines:

Adaptive sampling for physics simulations
Neural networks for image feature extraction
Interactive visualization tools
Parallel processing for batch generation
Version control for scientific writing

📊 Impact #

Thesis repository: 14 GitHub stars
Cover design repository: 36 GitHub stars
Real-world adoption: Dr. Tomas Pippia from TU Delft used the code to create his own thesis cover
~100 unique printed copies, each with its own generated cover and edition number

🎯 Challenges and Solutions #

Managing 3,671 unique PDF files and their conversions
Creating a clustering system for visually similar designs
Building interactive tools for non-technical collaborators
Ensuring consistent print quality across designs
Balancing artistic appeal with scientific relevance

💡 Lessons Learned #

Open source isn’t just about code - it’s about sharing knowledge
Python can be a powerful tool for creative projects
Machine learning can help solve unexpected problems
Good visualization requires both technical skill and artistic sense
Interactive tools make collaboration with non-programmers easier

📚 Fun Fact #

Due to COVID restrictions, I defended my thesis without any guests present. The silver lining? I still have about 100 beautifully unique copies at home, each with a different generated cover and edition number. If you’d like a physical copy of this piece of quantum physics meets generative art, just let me know!

Want to create your own thesis cover or see how I wrote mine? Check out the thesis cover project and thesis source!

#OpenSource #PhD #Python #MachineLearning #DataVisualization

Calendar of Life 📅

Sun, 01 Dec 2024 00:00:00 +0000

(See my intro post.)

Visualizing the finite weeks of our lives, because apparently, I needed another reminder of my mortality!

📖 Origin Story #

You know how sometimes you’re just casually browsing the internet, not looking to have an existential crisis? Well, that’s exactly what happened when I stumbled upon Tim Urban’s Wait But Why post “Your Life in Weeks” and saw Kurzgesagt’s “Calendar of Your Life” poster. Instead of just feeling mildly anxious about the finite nature of existence like a normal person, I decided to cope by writing some Python code. Because that’s what we do, right?

🔧 Technical Highlights #

The project uses matplotlib to create a grid where:

Each row represents one year (up to 80 years, optimistically speaking!)
Each column represents one week (52 weeks per year)
Different colors show different life stages (early childhood, school, university, etc.)
The current week blinks in the animated version, just in case you needed a more urgent reminder of time passing
Supports both dark and light modes, because existential dread comes in all themes

One interesting technical detail is handling the fact that 52 weeks isn’t exactly one year. The code uses a small correction factor (days_per_week = 365.25 / 52) to ensure the visualization stays accurate over many years. Yes, I made sure your existential crisis would be mathematically precise.

📊 Impact #

With its whopping 5 GitHub stars, this project clearly hasn’t taken the world by storm. But hey, sometimes the best tools are the ones we make for ourselves, right?

🎯 Challenges and Solutions #

The main challenges were (this was actually a very simple one):

Accurately mapping life events to week numbers (turns out life doesn’t fit neatly into boxes)
Creating smooth animations for the current week indicator (because a blinking dot really drives home the whole “time is passing” thing)
Handling the 52 vs 52.18 weeks per year discrepancy (thanks, Earth’s orbit)
Making the visualization work well in both dark and light modes (for viewing your life calendar at any time of day)

💡 Lessons Learned #

This project taught me that sometimes the most impactful tools are the simplest ones. Sure, it’s just a bunch of dots on a grid, but each dot represents a week of life - past, present, or future. It’s a humbling reminder that while we can’t add more dots to our grid, we can try to make each one count.

Want to contemplate your own mortality in a more organized way? Check out the project on GitHub!

#OpenSource #Python #DataVisualization #Programming

Crafting unique Ph.D. thesis covers with Python 🎨

Sat, 04 Apr 2020 13:40:00 +0000

Originally posted on quantumtinkerer.tudelft.nl/blog/thesis-cover on 2020-04-04 13:40.

I will hand out around a hundred copies of my thesis to my defense committee, colleagues, and my family and friends. Let’s be honest, most people will probably not get further than attempting to read the summary and appreciating the cover. Four years of work has gone into generating the content of the thesis, so I figured, at least some thought has to go into the design of its cover. Unfortunately, I am by no means an expert on a graphical design or even competent enough to attempt to use any kind of graphic design software. But luckily for me, I do consider myself an expert in Python , and why not make the thesis design a fun process?

Initial Design: Adaptive Sampling and Data Visualization #

My vague plan was to somehow incorporate a plot resulting from a simulation run with Adaptive [source ], the software I wrote during my Ph.D. to adaptively sample functions in parallel and my proudest work. I even had a potential simulation result in mind that I would use for it, specifically, one that has been on the background of my good friend’s phone for some years and one that has earned me quite some internet points on Reddit. This plot (below) is one of the results of Chapter 4 of my thesis [source ] and this paper.

Your browser does not support the video tag.

I went back to the code that generates the plot and started hacking away in Matplotlib. Quickly, I realized that not only the plot above was going to look cool, but that many other data sets that I had also resulted in beautiful patterns. I hit the paradox of choice: I had a total of 3671 unique data sets (all for Majorana devices with either different parameter sweeps or different physical effects that were included in the model) and no idea how I was going to make a selection.

This is an early attempt (and rather ugly in hindsight)

I decided to postpone the choice of which data to use and just focus on how to plot the data first. I experimented with different colormaps, fading colormaps that gradually reveal the underlying triangulation better, and more. On a weekend night, after telling a friend I was with about my plans, we (as usually happens) started to program some of the ideas we came up with. We settled on a design where the triangulation would slowly fade from the top to bottom and where each triangle simultaneously transitioned from a solid color (the average of the values at its vertices) to densely interpolated values. See this Jupyter notebook which generated the final version of the cover. After we also chose a nice colormap (by simply creating the cover for each possible colormap,) the following was suggested: “You know what would be even cooler?! To make every cover unique.” Ten minutes later, I had started 3671 jobs on our computational cluster. When the files slowly started to appear on my computer (I have my local files synced with the cluster), we could not have been more excited. There were simply so many cool looking alternatives. Too many to manually select now, to be honest—a problem for later I thought.

Overcoming the Paradox of Choice: Clustering and Neural Networks #

Four months later, with the rest of my thesis completely done and about a day left on the deadline to submit my thesis, I found myself facing the selection problem I had successfully ignored for so long. There must be a smarter way to select the covers, I thought. Many of them look roughly similar (and some of them are even quite boring or not ecstatically pleasing), so it does not make sense to scroll through all of them. I could cluster them by similarity and then I would just have to choose the nicely looking clusters while being able to instantly discard many.

Finally, a good application to use neural networks!

I still had the 3671 covers in pdf format in some folder, so using the wonderful GNU Parallel, I converted and rasterized all of them to jpeg with

find . -iname \*pdf -print0 | parallel -0 sips -s format jpeg {} --out {.}.jpg

Then I used a pre-trained convolutional neural network model (VGG16) to extract features in a lower-dimensional representation. I fed the feature arrays to a K-Means clustering algorithm that sorted them in 80 clusters. To procrastinate a little more, I created an interactive widget using ipywidgets (see video below) to facilitate the selection process.

Your browser does not support the video tag.

These are the clusters that I selected in a grid

The selection process was done using this Jupyter notebook.

Finally, I copied the selected clusters into different folders and used the OS’s file explorer to manually select a few covers from each cluster.

Final Touches: Fine-Tuning Colors and Adding Edition Numbers #

Then I printed an initial version of my thesis, however, I wasn’t really happy with the colors.

To choose the right color scheme I got help from a non-tech savvy friend who went to art school. Due to the COVID-19 crisis, we couldn’t sit side by side, so I created this Binder “app” with sliders.

Your browser does not support the video tag.

Additionally, since all of them were different anyway, I gave them an identifier and added

edition n of 120

to the back.

These are a few examples of the final result (click on one of the images to open gallery view)

A few days later, this was delivered to my doorstep

And finally I mailed them to my Ph.D. committee

Now it’s time to start preparing for my defense at May 11, 2020 at 15:00!

Notes #

The data and code to generate all printed covers is available at basnijholt/thesis-cover
The simulation code that generates the data is available at basnijholt/spin-orbit-nanowires
The video of the Reddit post is generated using this code