Ten years ago I gave my first talk at FOSDEM. Back then I was pushing for better tooling to inspect systems, to avoid configuration drift, and for universal system descriptions (USD) — a way to consistently define Linux systems (see this talk).
Three years ago, I came back with a completely different paradigm: make existing systems immutable and image-based, remove drift at the root, and do it the same way across major Linux distros to avoid lock-in (see this talk).
TL;DR#
This year, on February 1st, I’ll be back to present Hadron, our new image-based Linux distribution, at FOSDEM. If you want the backstory, read on.
Disclaimer#
These ideas and projects are the work of many people. I’m not the visionary here; I’m just sharing a path I believe in and the lessons I’ve picked up along the way. The credit belongs to the communities and teams behind them.
Configuration Drift#
In 2016 I was part of the Machinery team at SUSE. Machinery is a tool for system introspection. It showed the current state of a system: which packages were installed and how their configuration differed from the defaults. That made it easy to spot a server drifting from the desired configuration. It felt powerful, because it made visible something that is usually invisible: how systems drift.
One of the ambitions of Machinery was to inspect multiple Linux distributions, but anyone who has worked with them knows how much they can differ. We thought we could tackle this by introducing USD. It didn’t pick up, and maybe that’s for the best.
What the industry did pick up was the use of configuration management systems (CMS). These introduced the idea of a single source of truth in the form of recipes, cookbooks, and playbooks. But CMS have limits. A full blog post is needed to explain all of that, so for now let’s focus on the main issue. If a system can be changed during runtime, then it’s only a matter of time before it happens in a way you did not predict and did not want.
And that was the recurring pain we kept running into: configuration drift. CMS designers understand this and introduced features to re-run configuration of their systems periodically. At best this leaves the system briefly disconfigured; at worst it fails to address changes that are not included in the system description. And for Machinery, well, introspection is nice to have, but not useful enough to get traction. So Machinery eventually reached its natural end, and I took a hiatus from this problem.
Image-Based / Immutable Systems#
Fast forward to 2023, I joined the Kairos team. Kairos turns an existing Linux distribution into an image-based one. You don’t mutate the base system at runtime; you swap the full image, or you don’t. Part of the system is mounted read-only (immutable), so package manager changes during runtime are off the table. No more drift.
As a bonus, this approach comes with side effects. Your bill of materials (SBOM) can be trusted because your system doesn’t change unexpectedly. You can use measured boot for critical parts like the kernel, cmdline, and initramfs, store the measurements in a TPM, and ensure only systems signed with your keys can boot.
Now Kairos is not the only project offering image-based systems out there. There’s Google COS, Amazon Bottlerocket, Microsoft Flatcar, SUSE MicroOS, to name a few. And yes, I mention the ones backed by major platform vendors to make a point: this is a space getting serious investment and it will keep growing. But Kairos is unique in that it lets you choose your underlying Linux distro.
Enter Hadron#
That flexibility Kairos gives is an asset. Within one organization you might be paying licenses for RHEL in your datacenter but running Ubuntu on your NVIDIA Orin devices; Kairos has you covered. Or maybe your team prefers Alpine? No problem. But that flexibility comes at a cost. After building hundreds of artifacts across distributions and options, we noticed two things: it can be confusing for newcomers, and some systems need just the bare minimum.
Our solution is a new Linux distribution, Hadron. It will not compete against all other distros; it will be one more flavor to use Kairos, but it will be the default and the only one we ship. This addresses both issues, gives us a minimal, upstream-first base to build on, and gives us a single baseline to test, document, and ship. Anyone who prefers to use one of the other base distros can keep doing so; they just need to build the images themselves, but we also provide all the tooling you may need with the Kairos Factory, which lets you build images from the CLI, web UI, or even a GitHub Action.
Hadron is minimal and lean, upstream-first, and designed for the hard requirements of the edge while still running cloud-native workloads. It uses an uncommon combination of musl libc, systemd, vanilla kernel, and busybox to keep the base small and the attack surface lower while staying modern.
FOSDEM 2026#
Join us on February 1st at FOSDEM with my colleague Dimitris Karakasilis. We will share more insights into Hadron. We have a deep respect for this community, and that’s why we bring this talk to them. We want to present our work and gather feedback from those who have been building distributions for a while now.
Reply by Email