What shipped​

v4.2.0, on 18 August. The last thing holding it was two advisories the release scan was still ignoring (kairos#4332). Cleared, tagged, announced.

Reset works again on a non-UKI install. This is the one I would want fixed if I were running Kairos. A state reset on a non-UKI system could come back with the wrong sysroot labels and without its recovery extensions (immucore#611, immucore#613), shipped out through kairos-init#436. It is a bug you only meet on the day you actually needed reset to work. It now has a test covering it (hadron#565) and a lint that fails the build if someone adds a boot state and forgets to handle it (immucore#614).

A bad config_url no longer eats your cloud-init. If the kernel cmdline pointed at a config URL that could not be fetched, the agent skipped the local cloud-init files too, so a typo in one place silently dropped your whole configuration. Fixed in kairos-agent#1344.

Riscv64 is back to one build path. Hadron could already build for riscv64, that part isn't news. What hadn't happened was a release: a split native-plus-emulated pipeline for it was slowing down releases for the architectures that matter more right now. hadron#572 drops the native runner and goes back to plain emulation. Slower, but most of it caches, and it gets stability back. Maybe the next release finally includes a riscv64 image.

Fork pull requests get CI. Before this week, a contribution from a fork got no build at all, which is a bad first experience and a slow one. It builds now, behind a gate a maintainer has to press (kairos#4323, extended in kairos#4328).

Builds are off ttl.sh, and that's a security fix as much as a tidiness one. Images used to hop between CI jobs through a public throwaway registry. They now go to quay or move as tarballs between jobs (hadron#566, hadron#567, kairos-init#440), and the credential that publishes to the production registry no longer sits in a workflow that runs on code from a pull request, where anyone could put it there (kairos#4327). The approach follows what Dimitris Karakasilis (jimmykarily) had already worked out for kcrypt-discovery-challenger.

A reviewer that is not me. Last week I wrote I was leaning toward a separate agent doing a first pass. It exists. Every pull request opens as a draft, a reviewer subagent reads the diff, the ticket and the tests, and deliberately does not read the conversation that produced them, because a reviewer that inherits the author's reasoning is just the author wearing a hat. It sends the change back or moves it to ready for review. On kairos-io it may comment and nothing else. It never merges anything, anywhere. Merge is still mine, in every repository. I reviewed 25 of the 31 changes the agent opened against Kairos this week, compared with 13 of 18 last week. Too early to say if it helps. Ask me next week.

The numbers​

Same caveats as last time. These count both my own account and the agent's.

Two rows are worth a sentence. The board went from ninety-nine opened against thirty-three closed to a hundred and ten against ninety. Last week it was filling about three times faster than it drained. It is now roughly keeping pace, and I would rather have that than a smaller number in either column.

What went wrong​

Four postmortems, and the same failure runs through most of them.

A rule nobody checks does not get followed. A policy about which model to use for which kind of work, cheaper ones for grunt work, better ones for judgment calls, had been sitting in the agents' constitution for a month. Every session reads that file. Its application rate over the month was zero. Not low. Zero. It surfaced by accident, because a ticket happened to ask what model a session was running.

The failure mode is what makes it expensive. The rule does not get broken loudly. It reports success. For a month every session read that policy, understood it, and did not apply it. Nothing anywhere produced a signal. To anyone auditing this later, an unenforced rule in that file looks exactly like an enforced one.

Two smaller versions of the same failure landed the same day, each fixed the same way.

The fix in both cases was moving the text into the shared file every session reads, through a branch and a pull request. That is the whole difference between a preference and a mechanism, and it costs one paragraph. The model policy got a check at the moment a ticket is claimed. A poller that filed thirty-six junk tickets in a minute now refuses a first run over five items and prints the command to clear the backlog. None of the fixes were a better-worded rule.

Two things reported healthy while being dead. A user-level timer, the one that tells me when the agent is mentioned on GitHub, went dark for eight hours while is-active and is-enabled both said yes. Neither of those answers the question I care about, which is whether it will ever fire again. Separately, leftover VM disk images filled a shared disk quota on the machine all of this runs on, and every command in every concurrent session started failing with a bare exit code pointing nowhere near the cause. Both cost me more hours than the governance ones did, and both are the same lesson wearing different clothes. The check has to check the thing you care about, not the thing that is easy to report.

What I am doing about it. The fix is boring: an audit of every rule in that constitution, sorted by whether anything except a session reading it carefully makes it true. I already know that audit will find things. The model policy sat there for a month and I found it by accident.

Where that leaves me​

A better week on output and a more uncomfortable one on process. The system produces more every week, and the more it produces, the more expensive the gap gets between the rules I have written down and the rules anything actually enforces.

As before: this post was drafted by the system it describes, and I reviewed, edited and merged it myself. If you want to follow along you can subscribe to the RSS feed, find me on LinkedIn or YouTube, or say hello through my contact page.

What actually landed​

The one I care about most is a release gate.

Kairos releases were not gated on a vulnerability scan of the bundle we actually ship. It is fiddlier than it sounds, it touches release CI, and it is exactly the kind of ticket I put off until it hurts enough to force my hand.

It landed this week in kairos#4300, together with clearing the 23 fixable advisories it immediately surfaced. A monthly repository access audit landed too, in community#31.

The rest is dependency bumps, CI plumbing and version pins across nine repositories, and honestly that's fantastic. This is maintainer work. Keeping dependencies current, keeping the release pipeline green and keeping the CVE count down is the job, and it's exactly the kind of work you want to delegate. It's also the part that quietly eats the time I'd rather spend elsewhere.

One caveat: shipped is not released. 4.2.0 is still blocked on a separate problem as I write this.

There's another category I nearly left out, because it isn't code. A good chunk of this week was developer relations: conference admin, keeping track of what is worth writing or posting about, the small things that pile up. That work used to happen by hand, in the gaps between other things, and it was the first thing to slip when I got busy. Now it gets a ticket like everything else, and it mostly gets done without me.

The numbers​

Same caveats as last time. These count both my own account and the agent's.

The first row is the one I'd point at. It was zero when I published the last post, and I said then that anyone comparing tables should read that row first. It's 19 now: 18 pull requests across nine kairos-io repositories, plus one to edgevpn, which ships inside every Kairos release.

The last row is my own board, the private one where I file work, not the Kairos issue tracker. Ninety-nine issues opened against thirty-three closed, up from twenty and fifteen. Roughly: 23 about Kairos, 18 about the homelab, 11 about this website, and about 42 about the factory and its rules. So it's filling about three times faster than it drains. I'll come back to why that isn't purely bad.

What broke​

Halfway through the week I changed how sessions are organized. They used to be scoped by project: one for Kairos, one for my own repos, one for planning. Now a session is defined by who it acts as and which machine it runs on. They're lanes, and a lane picks up whatever ticket is labeled for it, Kairos one hour and this website the next.

The design is better and I'd do it again. But a lot of things that had been working quietly stopped working the moment it landed, and nobody warned me. The agents applied a migration to a running setup without surfacing what it would break. Then, when things did break, I got confident answers about the cause that turned out to be guesses rather than assessments.

A clean example, while writing this post. I asked which of my posts were published. The agent read a local checkout sitting on an old draft branch and gave me a date ten days off, without saying which branch it had read. There was nothing for me to catch it on. That's now a rule: fetch first, read the published branch, say which branch you read.

Old habits die hard​

I have a talk recording next week, and I handed part of the preparation to Claude. Not through the board, just me in a chat window, the old way. The analysis and the suggestions were good. The slides still need work.

None of it is measured, because none of it went through the system. It won't show up in any of the tables above, and I only noticed afterward.

This suits the way I think​

Something I didn't anticipate.

My brain jumps from topic to topic. That usually leaves me two options, both bad. I chase the tangent and end up yak shaving three levels deep in something unrelated to what I sat down to do. Or I write it down, and the note joins a backlog that grows until it's unmanageable and I stop opening it.

There's a third option now. I can write the ticket and have someone start on it. The tangent gets captured and worked without me following it.

That reframes the 99 against 33 above. Part of it is a board filling faster than it drains, which is a real problem. Part of it is that writing something down finally has a payoff instead of being deferred guilt.

Reviews, and the thing that makes them hard​

Everybody says reviewing AI output is the bottleneck, and I couldn't agree more. What I didn't expect is the specific way it fails.

It's about time, obviously. But not only time. When the reviews are long and there are many of them, I stop reading them at the same level of attention. It creeps. I catch myself skimming a diff I would have read line by line two weeks ago, not because I decided it was low risk, but because it's the eleventh one today. The gate is still there on paper. The attention behind it is not.

I have a data point on that. Of the 18 changes that reached kairos-io, I reviewed 13. One merged with no review from me at all: an external maintainer approved and merged it eight seconds apart, on a pull request the agent had opened without the draft flag that exists precisely to keep other maintainers out of the loop until I've looked. Nothing bad shipped, the change was correct, but the gate wasn't there.

Two things make it worse, and they need different fixes. The agents write a lot, and a long explanation is not a better one. I've started asking for concision, which I can do today. The other is throughput, and I don't have an answer: the better the system gets, the more there is to review, so every improvement makes this harder.

What I'm leaning toward is a separate agent doing a first pass, and finding a way to skim that doesn't quietly become rubber-stamping. Whatever I replace line-by-line reading with has to keep the slop out of a project other people depend on. That's the only thing the gate is for.

More time on my phone​

The last thing, and I'm genuinely of two minds.

I spend more time on my phone now, because that's where the board is. I can go downstairs, make coffee, unblock an agent from the kitchen and come back. That's good for me. Long stretches at my desk give me back pain and make me tense, and this week I got real work done while not sitting at it.

The other side is obvious. Things being on my phone means I'm more likely to look at them when I'd rather not. Reviewing a pull request while I'm with my family isn't a win just because it's efficient. I don't have a rule for this yet. I'm writing it down before it becomes a habit I stop noticing.

Where that leaves me​

A good week. The thing I said I'd start, started, and it produced work I actually care about instead of more scaffolding.

As before: this post was drafted by the system it describes, and I reviewed, edited and merged it myself. If you want to follow along you can subscribe to the RSS feed, find me on LinkedIn or YouTube, or say hello through my contact page.

These days I am not really writing code. It gets written by AI and I read it. That shift happened faster than I expected, and it has left me with a question I cannot settle.

Does the language still matter?

I do not mean it in the usual sense. Not typed or untyped, not compiled or interpreted, not fast or slow. I mean something narrower: does it matter whether a language was designed for readability and enjoyment?

Why these two​

Ruby and Go are not obvious rivals. Nobody is picking between them to write a Kubernetes controller this afternoon. But they bracket a shift I lived through, and they are the two languages I have written most of my code in.

A lot of the early cloud infrastructure was Ruby. Vagrant, BOSH, large parts of Cloud Foundry. I spent a couple of years on the BOSH OpenStack CPI myself. Then Docker and Kubernetes arrived with better answers to the same problems, and Go arrived with them. Cloud Foundry eventually began moving components to Go as well. So this is not an abstract pairing for me. It is the language I came from and the language I ended up in.

Two ends of the same table​

Go is, in my opinion, the opposite of Ruby. I do not think it was made for the programmer to enjoy. It was made to be a lingua franca, something many developers across many projects could pick up quickly. A good enough solution, on purpose.

I am not trying to downplay it. There is real elegance in a lot of Go's design, it is extremely effective, and there are good reasons it became the default in the cloud native space. Its explicitness is a feature, and I rely on it every day. But it feels more like an industrial solution than an artistic expression. Those two languages sit at opposite ends of the same table.

Here is roughly what that looks like. Say you want the names of the nodes that are ready, sorted.

def ready_node_names(client)
  client.list_nodes
        .select(&:ready?)
        .map(&:name)
        .sort
end

func ReadyNodeNames(ctx context.Context, c Client) ([]string, error) {
	nodes, err := c.ListNodes(ctx)
	if err != nil {
		return nil, fmt.Errorf("listing nodes: %w", err)
	}

	names := make([]string, 0, len(nodes))
	for _, n := range nodes {
		if n.Ready {
			names = append(names, n.Name)
		}
	}
	sort.Strings(names)

	return names, nil
}

I want to be fair about what that comparison shows, because it is easy to rig this kind of example and prove nothing.

The Go version is not unclear. Every line in it is obvious, which is exactly what Go is going for. It also does something the Ruby version does not: it tells you what happens when listing the nodes fails, and it wraps the error with enough context that you will know where it came from. In the Ruby version that failure is invisible. An exception propagates up through code you cannot see from here. If I am reviewing the Go function I know how it behaves when the API is down. If I am reviewing the Ruby one, I have to go and find out.

That is a real trade, and Go's side of it is not boilerplate. It is the part I actually want to read.

The rest is. The make with a capacity hint, the range loop, the append, the second return value threaded through every caller. I have read those exact lines thousands of times. They carry no information specific to this function.

Why I think it matters more now, not less​

Reading that boilerplate was never fun. But it was survivable, because there was only so much code a human could produce in a day. That is no longer the constraint. We can now generate far more code than we used to, and the reading has not gotten any faster.

I do not think I need to control every line anymore, or hold a whole codebase in my head the way I once did. I have made my peace with that. But I do want to be able to maintain software over the long term, and for that, review is still the thing that matters.

I am not claiming it always will be. Maybe we end up with other ways to judge whether a system is maintainable, and then none of this matters and I will happily stop reading diffs. But that is not where we are. Today, when a model gets something wrong, a person still has to go in and fix it, and that person has to read it first.

So while that holds, how a language reads matters more than ever. It does not have to be Ruby's expressiveness specifically. It has to be that the signal-to-noise ratio of a diff is high enough that I am reading decisions instead of scanning past ceremony to find them.

What I would want​

I would love to see new languages designed for this: not for the joy of writing them, but for making code review easy and effective.

I have a rough idea of the shape. As little boilerplate as possible. Concise. Pleasant to read. And no special cases you have to master or get bitten by, because the reviewer is now often the person who did not write the code, and every subtlety is a trap for them specifically.

Maybe the code does not even need to be written in that language. Maybe it gets transpiled into it, and the readable form is the one you review.

I am well aware that models can review code too. I use that, and it catches things. But I have yet to see a fully lights-off factory that delivers maintainable code, the kind a human opens six months later without wanting to tear their eyes out. That is the bar, and nothing I have run clears it yet.

And while we are at it: could we please also come up with something better than YAML?

Anyway. What is your favorite language to review code in, and why?

The agents have been running since Monday 20 July, the day I flew out, so that is two and a half weeks ago. This post is the first report back, and I want to be precise about what it is and is not. Nothing from this experiment has shipped to Kairos yet. Not one line. Everything the system has produced so far has gone into the scaffolding that runs it, plus my own homelab. So this is not a success story, and it is definitely not a "look what AI did for me" post. It is a write-up of what it took to get the thing running, and of the five separate ways it went wrong while I was doing that.

What would you do with six months of Claude Max 20x?​

Not long ago I applied to Anthropic's Claude for Open Source program, and I got accepted. The benefit is a free six-month Claude Max 20x subscription, the top tier, which normally runs about $200 a month, so roughly $1,200 over the six months. When something like that lands in your inbox unplanned, you have to make the best of it.

The email that started it, on 17 July 2026.

I settled on two answers.

The first is what to spend it on: getting as far through the Kairos backlog as I can, and doing it as an agent-led factory rather than by pairing my way through tickets one at a time. Kairos is where my open-source time goes, and its backlog is long enough that the difference between "me, in the evenings" and "me plus a factory" should actually be visible.

The second is to document it. Six months of the top tier is an unusual thing to be handed, and the interesting part is not that I have it, it is what comes out of it. So I am writing down what I do with these months and what happens as a result, including the parts that do not work. This post is the first installment, and going by the ratio so far, the parts that do not work are going to be the bulk of the material.

Pairing with models already works. Now what?​

Here is the thing that made me want to try this at all. A lot of my code over the last several months already came out of working with a model, in a chat window, one turn at a time. So for me the question is not whether a model can write useful code. I already have my answer to that, and it is yes, as long as a human is sitting right next to it. Two pieces of my own recent work convinced me.

The first is kairos-lab. I am honest with myself about this one: I would not have built it if it were not for AI. Not because I could not have written the code, but because I would never have found the time. Pairing with a model lowered the cost of starting enough that a project I had been putting off for months finally happened. That is a real change, and it is easy to underrate. Most of the ideas we never ship do not die because they are too hard. They die because we never find the free evening to begin.

The second is the riscv64 work on Kairos, and it taught me something different. Supporting a new architecture pushed me down to a lower level than I usually work at, into territory where I am not fluent and would normally hesitate before even asking a question out loud. Working alongside a model changed that. I felt confident enough to poke at things I would otherwise have left to someone with more experience, to ask the naive question, and to keep going when the first answer was wrong. It did not just help me write code faster. It widened the set of problems I felt allowed to take on.

Put those two together and the pattern is clear. Pairing with a model helps me start things I would have skipped, and it helps me reach into areas I would have avoided. For me, that result is already in.

So the honest next question is: now what? There is a lot of talk right now about agent factories, teams of models turning out software while their owners sleep. I am wary of the hype, and at the same time genuinely curious. Is a factory like that something only a company or a well staffed team can build, or can one person stand up a small version that actually works? That is a big part of what I want to find out. The specific step I want to test is delegation.

Can I move from pair-programming to delegation?

Calling it pairing might sound a little off, since there is no human on the other side. But I have done a lot of pair programming over my career, some of it full extreme programming, and working with a model honestly feels similar to me. Not the same: you lose the visual cues and the easy back and forth you get from another person. Even so, the shape of it, thinking out loud, taking turns, correcting each other, is close enough that pairing is the word that fits. In that sense, pairing means I am in the loop on every step. Delegation means the agents do the work on their own, and my judgment moves to three gates instead of every keystroke:

1. Issue acceptance. Deciding what is worth doing, and writing it down clearly enough that it can be handed off.
2. PR review. Reading what comes back and deciding whether it is correct, whether it fits, and whether it is honest about its own limitations.
3. Final testing. Actually running the thing before it goes anywhere near users.

I kept these gates on purpose. It would be tempting to aim for a lights-out factory, one that merges its own work while I sleep, but almost everything I have read says that road leads to real problems with code quality over time. As appealing as it is to become a 10x developer overnight, I care more about being able to maintain what gets written. If the honest price of that is being a 2x developer instead, or whatever the real number turns out to be, I will take it. Maintainable beats fast.

The setup, briefly​

The architecture is deliberately modest, and the hardware bill is the least interesting part of this post, which is exactly the point. One always-on mini PC, a Beelink SER5 with a Ryzen and 32 GB of RAM, sits in my homelab running the sessions. No GPU, no rack, no cloud instance. The 32 GB is more than a setup like this needs for most projects, since the sessions themselves are not memory hungry, but Kairos work means booting VMs to test images and upgrade paths, sometimes several at once, and that is where the headroom earns its place. Everything else is a GitHub project board: issues go in, agents pick them up, pull requests come out. And every one of those pull requests passes through me before it lands.

The board, checked from a departure gate before the flight.

That review gate is not hypothetical, and it is not tied to my desk. The first pull requests this system produced, I read and merged from my phone while standing at a departure gate. It was a one-file change to a node's boot documentation, checked and squashed between boarding announcements. I mention it not as a flex but because it is the honest shape of the thing. If staying in the loop required me to be sitting at my keyboard, this would not be delegation. It would just be a slower way of doing the work myself.

Reviewing that first change from my seat: the diff to a node's boot documentation.

One decision I did not expect to enjoy as much as I did: the agents commit under their own GitHub identity, @mauro-agent. Authorship stays mine, so the work still shows up as mine, but the committer is the agent. That means anyone reading the history can see exactly which commits came out of the factory and which ones I typed myself. If I am going to publish results in six months, I want that line to be visible in the git metadata rather than in a claim I make afterwards.

It matters in the other direction too. I have years of history under mauromorales, and I am not willing to hand that over to a bot account or watch it thin out because a machine is doing the typing. The work is still mine to answer for: if something the agent wrote turns out to be wrong, it went out under my name, and that is on me, not on the tooling. Author mine, committer the agent's, is the best balance I have found. I keep the history, I keep the responsibility, and anyone who wants to know how a given commit came to exist can just look.

The split itself is not new to me. Back when I was pairing with humans, we did the same thing in git: whoever had their hands on the keyboard was the committer, and the other one went in as author, so a single commit carried both people. The only difference here is that we never swap seats. The agent is always the one committing, and I am always the one signing for it.

The avatar I gave it. It is supposed to be me.

I gave it an avatar, because of course I did, and the avatar is supposed to be me. My girlfriend took one look at it and recognized me straight away, then could not stop laughing at how geeky I am, which was genuinely one of the better moments of the week. It is a strange time to be working in this field. I am building a small factory of software agents in my spare evenings, and the funniest part of it is that the agent doing the work is wearing my face in an eighties anime flight suit.

What actually broke​

Now the part that matters. In two and a half weeks I have written five postmortems. None of them are serious. Nothing shipped broken, nothing was unrecoverable, nothing reached a user. But I want them in public, because a list of five small failures in seventeen days is a much more useful data point than any number of merged pull requests.

A pull request that said "Merged" and was not. One change was branched on top of another change that had not landed yet. The base merged first, then the stacked one merged too, and GitHub cheerfully reported both as merged. Except the second one had merged into a branch, not into the main line. Its contents, an architecture decision record and a rule change to the agents' own instructions, silently never arrived. I only found out the next day when a rule that should have existed was not there. The status was technically true and completely misleading, which is a category of bug I now watch for.

A rule that was too absolute the first time it met reality. I had written "never use --no-verify" into the agents' instructions, meaning never bypass the local git hooks. Then a task required proving that the server-side branch protection actually rejects a direct push to main, and the only way to test that was to get past the local hook so the server could be the thing doing the rejecting. The session did it, disclosed it, and was right to. The rule is now "never use --no-verify to land changes; permitted only for negative tests where rejection is the expected outcome, and it must be stated every time." A rule with no exception is not stricter, it is just wrong more often.

A session that nearly wrote into the wrong repository. A session scoped to planning work got most of the way through drafting a blog post directly into my website's repository before I interrupted it. There was, in fact, another session whose whole job is that repository. Nothing about the environment signaled that boundary. The clone was right there. What stopped it was me, reading the conversation at that exact moment, by luck. I have since made the routing explicit, but the deeper issue stands: the scoping rules are prose, and prose does not enforce anything. The real fix is credentials that cannot push where the session is not supposed to push, and I have not built that yet.

A session that blocked silently with no way for me to know. A session tried to open a link and hit a browser extension flow that needed a click to continue. It just stopped and waited. Nothing reached my phone. I found out only because I happened to open the session on the server directly. The whole design of running this while I am away rests on GitHub being the channel back to me, and GitHub had nothing to show, because the session never got as far as writing anything down. The mitigation is a rule that anything potentially blocking gets announced somewhere I can see it before it is attempted. The underlying gap, no reliable way to be told "your agent is stuck", is still open.

An environment quirk that ate real hours. Certain configuration paths in this sandbox appear as device stubs rather than files, and ordinary git operations that need to write there fail with "Device or resource busy". Disabling the sandbox does not help, because the mechanism sits below it. It cost me the same debugging session five separate times before it got written down properly. Worse, the cleanup step after one of those failures assumed a commit had succeeded when it had not, and rolled a real commit off my local branch. Nothing was pushed, so no harm done, but "assume the previous step worked" is exactly how a safe cleanup becomes a destructive one.

Read as a set, none of these are arguments against working this way. Every one of them is an argument for keeping a human at the gates. Dex Horthy made an argument recently about why software factories fail that has stuck with me: models are trained against tests that pass or fail, and nothing in that loop penalizes making the codebase worse to live in. So a lights-off factory degrades quietly, because nobody is reading the diffs. My five postmortems are small precisely because somebody was reading them. That is the whole finding.

The thing I did not plan for​

On 29 July, Claude went down.

29 July 2026. Nothing to do but wait.

There is nothing to do in that situation. The factory stops. And it is worth saying plainly that this is a real drawback of building this way, not a footnote. It is the same shape of dependency as GitHub going down, which would also stop everything, since the board, the issues and the pull requests all live there. I have built a system with two single points of failure and neither of them is mine.

What I would want is a local model good enough to keep doing basic tasks, plus a local copy of the tickets and artefacts so I could keep making progress offline. I am not going to get there in this experiment, because a machine that runs a decent model locally costs considerably more than the mini PC this whole thing runs on, and I am not making that investment to answer this question. The minimal hardware footprint is a feature of the experiment, and I would rather report the limitation honestly than spend my way around it.

The other option is subscribing to several of these services at once so you can fail over between them, but then the cost climbs fast. What I would actually want, and I do not know whether it exists, is a service that sells you a frontier model without guaranteeing which one. If you are strongly attached to a particular model you would hate that. I would take it over being stuck. It reminds me of that xkcd about compiling: the outage is not a break, it is just a period where you cannot do the thing and cannot honestly do anything else either.

The numbers so far​

An experiment needs a baseline, otherwise in six months I will just tell myself a nice story. So before I started, I measured my own output across the Kairos repositories for the six months leading up to this:

I am not putting these here to brag, and I do not want to oversell what they measure. Merged pull requests are close to a vanity metric, in the same way lines of code are. Plenty of those 142 are one-line fixes and release chores, and a count like this treats them the same as a week of real work. But I have to base the comparison on something, and this is the least dishonest number I could find.

What the table cannot show is how the time actually splits. My Kairos work is part advocacy and part maintainership, and the advocacy side is invisible in a table of pull requests. In the same six-month window I gave ten talks across seven events, among them FOSDEM, CfgMgmtCamp and KubeCon EU. A talk is not an hour of work. It is designing it, writing the CFP, building the deck, practicing, traveling and being there. I do not have a number for those hours, but there are a lot of them, and I would not give them up.

That is the honest label on the "before" column: I am not satisfied with the amount of technical work I get to ship. I would like to be. That, more than any count, is what I want out of these six months, and if it works it feeds the other side too, because more work shipped is more work worth proposing a talk about.

Maybe two and a half weeks for this does not sound like much. Worth knowing that I was on vacation for most of it, so I was really only playing with this for an hour in the evenings, if that. Put another way, the whole thing took less than twenty-four hours to build.

For the period since, counted the same way and with the same caveats, here is what the system has actually produced in two and a half weeks:

That last row is the honest headline. Almost all of this output is the factory building and documenting itself. That is not a disappointment, it is what the first phase was for, but it does mean anyone reading a comparison of these two tables should read the last row first.

What is next​

Now that I am back from vacation, the Kairos phase starts. Up to this point the agents have only had write access to my own repositories, which is a low-stakes place to find out that a rule is too absolute or that a merged pull request never merged. Working against a project other people depend on is a different proposition, and it starts with a narrow scope, fork-and-pull-request only, and nothing landing without my explicit approval on each specific change.

I have no results to report there yet. The next post on this will have some, and if what it has instead is another list of postmortems, I will publish that.

One last honest note​

This post was drafted by the system I am describing. I then reviewed it, edited it, and merged it, the same way I intend to handle every other pull request it produces. Some people think writing this way is stupid, that if a machine drafted the words they do not really count as yours. I see it more like having a ghost writer. The ideas, the judgment, and the final say are mine; the first draft is help. If that feels a little recursive, it is, and I think that is the most honest way to start. You are reading the first output of the experiment, held to exactly the standard I am promising to hold the rest of it to.

I will report back as I go, not just once at the end but at points along the way, with the failures included. If you want to follow along, you can subscribe to this blog's RSS feed, find me on LinkedIn or YouTube, or reach out through my contact page.

But it's not like this was my first attempt at public speaking. Back in 2015 and 2016 I gave it my first shot with some minor success. I was even at FOSDEM and LinuxCon back then, which was huge for me at the time, but then I stopped.

The reason I used to tell myself and others was that I had stopped working in the open. And there is some truth in that. It is much easier to give a talk when the work is public, when people can see the code, the issues, the discussions, the decisions, and all the rough edges. Open source gives you a shared body of work to point to, not just a story people have to take your word for. But if I'm honest with myself, that was not the full reason. Working on proprietary code has never stopped people from giving good technical talks.

The real reason was impostor syndrome.

I felt like maybe I could convince the people selecting talks, and maybe I could even convince the right kind of audience, but I could not convince myself. And when you cannot convince yourself, it becomes hard to keep going.

The person who originally inspired me to start speaking was Cornelius Schumacher. He has been active in open source for a long time, and one of the things I learned from him was not to sit too comfortably, but also not to drown myself in a challenge that was too big. The useful place was somewhere in the middle: uncomfortable enough to grow, but not so overwhelming that you cannot move. That lesson stayed with me, even if for a while I was not really acting on it.

What changed with Kairos was not that I suddenly became fearless. I didn't. There is still stage fright. There is still the moment where I look at a topic and think: "Do I really understand this well enough to explain it to other people?" There is still the discomfort of being visible, of being judged, and of maybe being publicly wrong. Being back in open source definitely made it easier, but what really made it click was how I think about teamwork. We all have our skills, but we also all care about certain things more than others. I thrive when I find a spot where I can help and nobody else can, or nobody else wants to. Promoting Kairos was that spot. Someone needed to fill it, and morphing into what a gap needs so that others can keep doing what they are good at is, I think, the thing I am actually good at.

So I decided to step up and take that role, without leaving my maintainer duties. Not because I thought I was the perfect person for it. Not because I think I'm an excellent speaker. And definitely not because my goal is to be in front of people just for the sake of being in front of people. I did it because the project needed it, because I care about the work, and because it forced me to confront an unresolved experience I had been carrying for years.

One moment made this very clear to me. In 2023 I was part of a panel at KubeCon Paris, sitting next to people from Microsoft, Google, and AWS. I walked in feeling like the underdog in the room. But at some point during the conversation I noticed I was not just keeping up, I belonged in it. Nobody handed me that feeling. I had to notice it myself.

I believe all this public speaking has helped people discover Kairos, which is the main goal. Preparing a talk forces me to understand things better. Explaining an idea in public exposes the weak parts of my own thinking. Questions from the audience show me where the story is unclear. Crafting a demo helps expose bugs and UX issues in the system. And doing this repeatedly has made me a better engineer, not just a better communicator. But there's also one aspect that I think can be improved. Talks tend to be one-sided communication, except for the few questions you get at the end. So I want to try more ways to promote the project that allow me to engage further with people.

So here's to those 30 times on stage. For me personally, I think I can close a chapter. The question I'm asking myself now is: what is the next thing that sits just outside my comfort zone, but not so far outside it that I cannot grow into it? I think the answer is exploring other formats like video or online workshops. If this sounds interesting to you, please follow me on LinkedIn or YouTube, where I've started publishing some conversations with the Kairos team. And if you would like to collaborate, please reach out through my contact page!

But those two signals, even taken together, don't tell you much about who actually controls the project, who gets to decide its direction, or what happens if the company behind it changes its mind. That's governance, and it's the signal most evaluations skip - not because people don't care, but because it's harder to check than licensing or commit activity, and nobody has really taught them where to look.

This post focuses on the CNCF and friends - the CNCF specifically, the Linux Foundation umbrellas nearby (LF Edge, NeoNephos), and a couple of LF projects that share governance DNA. That's the neighborhood I'm invested in at the moment. There are plenty of good governance stories in other foundations and in projects outside any foundation at all, but a full tour would make this post much longer than I want it to be. The questions I end up with transfer; apply them wherever you're evaluating.

What governance actually is, and why it matters now more than ever​

Governance is the set of rules determining who makes decisions about a project. Who merges code. Who decides the roadmap. What happens if maintainers disagree, or if the company behind the project is acquired or changes direction. These procedural questions are what determine whether software you depend on will still be serving you in five years.

Distributed governance - multi-vendor, auditable, with decisions made in public - has real benefits. The project outlives any one company's interest. Decisions get made by people with different perspectives, which generally produces better technical outcomes. You get a legible, predictable path for raising issues and having them addressed. And you get something harder to name: the quiet confidence that you're not betting your platform on one CEO's roadmap.

Concentrated governance - typically single-vendor, often implicit, usually fine right now - has its own benefits. Decisions are faster. Direction is clearer. There's no committee. A small team that knows what it's doing can move a project forward quickly, and plenty of excellent software has been built this way. The trade-off is that you're betting on that team's continued interest, alignment, and financial health. When those align with your own interests, it's a great bet. When they don't, you have limited recourse.

Neither are right or wrong. They are just different models that have different pros and cons in this context.

What has changed recently is that the volume of code available is about to increase dramatically. AI-assisted development is going to produce a lot more code - and a lot of that code will be published as open source by individuals or small groups who haven't put any governance structure in place, because there hasn't been time to. At the same time, existing open source projects are going to see more contributions flowing in than their maintainers can realistically review. In both cases, governance is the thing that helps. For new projects, governance is what distinguishes "published as open source" from "maintained as open source." For existing projects, the right governance structures are how you absorb a surge of contributions without losing the quality bar.

That shift is what makes governance checking more load-bearing now than it was a few years ago. "It's open source" used to carry an implicit governance signal - if something was open source, there was probably a person accountable to a community somewhere. As the total pool of open source code grows faster than the pool of accountable maintainers, that implicit signal weakens. Foundations are one of the more reliable ways to tell the difference between "open source with real stewardship" and "open source with whoever is around."

That's the context this post is set in.

CNCF project governance, explained​

The CNCF hosts projects at one of three maturity levels: Sandbox, Incubating, and Graduated. All three carry the same "CNCF project" label, but they represent different stages of governance development, and understanding the difference is most of what governance evaluation at this foundation comes down to.

Let me describe what each of them represents, less as a CNCF process description and more as what it means for you if you're adopting.

Sandbox is the entry rung. Getting into Sandbox is not nothing. You donate your code and IP to a vendor-neutral foundation. You accept the CoC, the trademark policy, the whole framework. That's a real signal - the project is saying, in public, "we want to play the governance game." Most projects that start private never make it this far, and most that get to Sandbox are genuinely trying.

What Sandbox doesn't tell you is whether the hard work has happened yet. And the hard work is getting other companies involved - people from different places with merge rights and accountability, actually shaping the project's direction. That work is harder than it sounds. It's always easier for a company to consume something and trust it'll stay maintained than to put engineering time into maintaining it themselves. The famous xkcd 2347 about modern infrastructure depending on some random project that one person has been maintaining for years describes this perfectly, and the dynamic that produces it is the same dynamic that makes multi-vendor maintainership expensive.

So when you adopt a Sandbox project, in practice you're in bed with the company maintaining it. There's nothing intrinsically wrong with that - it's a bit like the benevolent dictator model, and plenty of excellent software has been built that way. But you should be clear with yourself that it is that model. The maintaining company's priorities are going to shape the project. If they pivot, so does it.

Incubation is where that picture starts to change. The TOC runs a real due diligence process. Adopters get interviewed. Governance gets reviewed. A project doesn't reach Incubation without showing that multiple organizations care about it enough to contribute maintainers and use it in production. This is the level where "single-vendor" starts to become "single-vendor by habit rather than by structure" - the scaffolding is there for more than one company to matter.

Graduated is where the picture has fully changed. Third-party security audit, formal governance review, sustained production adoption across diverse organizations, full TOC due diligence. Kubernetes is graduated. Kyverno graduated in March 2026, as announced by the CNCF. When you see a project at this level, somebody who doesn't work for the sponsoring company has checked the work.

None of the three levels are wrong, and neither is adopting at any of them. What matters is knowing which model you're getting, because they have genuinely different trade-offs. The question isn't "is this project mature enough" - it's "which model of governance am I willing to get in bed with, and what am I trading off in exchange for what."

And then there's the landscape​

It gets more confusing than that. The CNCF Landscape is a separate thing from the list of CNCF projects. The landscape is a categorical map of the cloud-native ecosystem, and it includes CNCF-hosted projects and vendor products and other open-source projects that are not hosted by the CNCF but have a place on the map.

To the CNCF's credit, the landscape does encode the distinction visually. CNCF-hosted projects appear in large boxes with colored frames indicating their maturity level, non-hosted open source projects appear in small white boxes, and proprietary products appear in gray boxes. The landscape guide explains all of this.

The problem isn't that the distinction doesn't exist. The problem is that most practitioners never read the guide. They see a familiar logo on a page that says "CNCF" at the top, and they assume everything on that page is under CNCF governance in some form. It's not. Box size and frame color are visual cues that require prior knowledge to decode, and the unaware visitor doesn't even realize there's a distinction to decode.

The landscape is trying to serve two audiences - people looking for CNCF projects specifically, and people wanting a broader ecosystem map. A more prominent label, a filter on by default, or separate URLs would help the first group.

Why this matters at the OS layer, specifically​

When we say "Linux," we actually mean two different things. There's the kernel, which is genuinely the same everywhere - the same code that Linus merges is the same code that eventually lands on your servers, regardless of distribution. Then there's the distribution, which is a set of package choices layered on top of that kernel: which init system, which libc, which package manager, which userspace utilities, which defaults, which patches. The kernel is a technical artifact. The distribution is a governance artifact. And the distribution is what you actually run.

Historically, the distinction was small enough to gloss over. Debian, Fedora, openSUSE, Arch - opinionated on packaging, but all in the mix-and-match tradition. Components were swappable, the init system was replaceable, and the distribution was opinionated without being load-bearing.

That's not true for all modern OSes. Some are more like Android than Linux. Android runs on the Linux kernel, but the parts that matter in practice - Google Play Services, the proprietary framework - are vendor-owned and close to impossible to remove. "Android is built on Linux" is technically true and practically misleading. You're running Google's OS on top of a kernel that happens to be open.

The same pattern shows up in cloud-native operating systems. Some projects genuinely inherit the mix-and-match tradition - you can replace components, swap the orchestrator, change the base distribution. Others bundle vendor-specific choices so tightly that the open-source kernel underneath is almost decorative. Both patterns can be fine depending on what you're trying to do, but they represent fundamentally different bets. In the first case, you're adopting a piece of open infrastructure that happens to have a company maintaining it. In the second case, you're adopting a company's product that happens to have an open kernel underneath. Those are not the same thing, and governance is the lens that tells you which one you're actually looking at.

This matters more at the OS layer than at higher layers because Android-style bundling is hardest to see at the OS layer. A policy engine is bounded - you know what it does, you can swap it. An OS spans thousands of packages, boot paths, and defaults. The cumulative effect of one vendor's choices can reshape "using this OS" in non-obvious ways.

And on top of that, there's an empirical observation: OS evaluation is underpracticed. Most platform teams reach into policy engines, observability stacks, ingress controllers - they've built pattern-matching instincts for those layers. Fewer teams reach into the OS, so when they do, they tend to fall back on the shallowest signals more heavily than they would for a layer they evaluate regularly.

Those two things together - the Android-vs-Linux ambiguity at the distribution level, and underpracticed evaluation - are what make governance checking at the OS layer matter more in practice than at most other layers, even if it isn't uniquely important in theory.

Where cloud-native is going, and why that amplifies the governance gap​

There's one more thing worth naming, because it changes the stakes of this whole conversation.

Cloud-native, until now, has mostly been a public-cloud, single-cloud story. The CNCF landscape grew out of an environment where most people were deploying into one hyperscaler at a time, and many of the projects we rely on carry assumptions that quietly reflect that environment. This is why people who try to run cloud-native stacks in a sovereign cloud, or at the edge, or across multiple providers, keep running into seams - pieces that work beautifully on the happy path and awkwardly off it.

That's going to change. Sovereign cloud pressure is real and accelerating, driven by geopolitics and regulation. Edge is going to expand dramatically, driven by AI workloads moving closer to where the data is and by latency requirements that can't be served from a faraway region. Cloud-native is going to stop being "running on one cloud" and start being "running across a federation of environments that didn't necessarily coordinate with each other." That's the direction of travel, and the signals are already visible.

In that future, distributed governance isn't just philosophically preferable - it's structurally aligned with where the workloads are going. A project whose maintainers come from multiple organizations, deploying into multiple environments, will accommodate that diversity naturally because it reflects their own reality. A project whose decisions live inside one vendor will, over time, optimize for that vendor's environment - not out of malice, but because the people making decisions are solving their own problems. The exception is a vendor whose commercial posture is itself distributed, and those vendors exist but are rare. For everyone else, concentrated governance means the project is going to fit the single-cloud past better than the federated future.

The operating systems currently in the CNCF​

At the time I'm writing this, there are three operating systems hosted by the CNCF. Three.

A note before I list them: I'm a Kairos maintainer, so I'm biased. What follows is my personal view, not the view of my employer or of the Kairos team - I'm speaking for myself. I'll try to be fair about all three projects, and where I can't be neutral I'll say so.

I'm listing them in the order they joined the CNCF, because one of the things we're going to talk about checking is intention, and intention is partly revealed by how early a project committed to this path in the first place.

• Kairos - Sandbox (accepted April 13, 2024), and the first operating system project accepted into the CNCF, as listed on the CNCF project page. Kairos is the underdog of this lineup - it's backed by a smaller company rather than a hyperscaler. Big-tech backing cuts both ways: it brings resources and influence, but those companies also retire or deprioritize projects when strategy shifts, and their weight on a project's direction is proportional to their size. An underdog project that's still here and still growing is telling you something about the conviction of the people maintaining it. Kairos is showing several of the signals the TOC looks for at Incubation; the public writeup of the current state is in the Become a Kairos adopter for CNCF Incubation post.
• Flatcar Container Linux - Incubating (accepted October 2024). Flatcar is the first operating system distribution to reach CNCF Incubation in the foundation's history, according to the CNCF announcement. Microsoft is the primary steward via the Kinvolk acquisition, with contributions from Cisco, Equinix, Wipro, and others. Flatcar cleared the Incubation bar, which means adopter interviews happened, governance got reviewed, and the project held up to external scrutiny.
• bootc - Sandbox (accepted January 2025). A Red Hat-led project for bootable OSes delivered as OCI images, building on the ostree and composefs machinery already used by Fedora CoreOS and RHCOS. Its current status is reflected on the CNCF project page. At Sandbox, the governance trajectory beyond Red Hat hasn't yet been externally validated.

All three of these projects are in the special-purpose operating system space, and I have genuine admiration for the technical work in all three. I'm looking narrowly at what their current CNCF status says about how far the multi-vendor governance process has developed. One of them has been through Incubation due diligence. Two of them haven't. That's what the shared CNCF label can flatten if you don't look past it.

The ones just outside, and what their situations teach​

Part of what makes the CNCF list short at the OS layer is that several of the OSes practitioners encounter live somewhere else. Looking at the broader landscape is useful because it shows the range of governance patterns that an OS project can end up in.

• Garden Linux is built by SAP as the base for the Gardener ecosystem. Gardener was donated to the NeoNephos Foundation, a Linux Foundation initiative aligned with European cloud-sovereignty efforts, as described on the Gardener site. Being in NeoNephos is a positive signal - there's real intent there. My honest expectation is that anything claiming to be sovereign should eventually have distributed governance, because concentrated sovereignty from a single vendor is a contradiction in terms. That takes time, and that's fine; the point is that the structural direction is set up to support it.
• EVE-OS sits under LF Edge, a Linux Foundation umbrella for edge projects, as shown on the LF Edge project page. The foundation wrapper is real, but in operational practice the maintainer roster is heavily Zededa-affiliated - which is the company that donated the project. This is a common LF Edge pattern and it's not a knock on the project; it's a reminder that foundation membership at the Stage level EVE is in is a different kind of signal than CNCF Incubation.
• Talos - Sidero Labs' OS - is listed in the CNCF Landscape as a non-hosted project, in the same category of entry where commercial offerings also appear. This is exactly the situation the landscape section above describes: present on the landscape page, not under CNCF governance. The distinction is easy enough to see once you know to look for it, and easy to miss if you don't. Talos Linux itself is MPL 2.0 open source; its companion management plane Omni is source-available under BSL 1.1.
• Bottlerocket is AWS-maintained, Apache 2.0 licensed, not in any foundation. My read is that it's essentially designed for internal AWS use - which is fine; it does that job well - but the posture is closer to "internal OS with open source code" than "open source project with external contributors." It's similar in shape to Google COS, just with better visibility: a website, docs, a public repository, a changelog.
• Google COS is Google-maintained, proprietary, used inside Google Cloud across multiple services (including as the default for GKE, but not limited to it). The extent to which it's used beyond what's publicly documented is hard to say from outside Google, which is itself a governance observation. When a widely-deployed OS has almost no public surface, the governance conversation can't really start.

One observation about this whole list: neither LF Edge nor NeoNephos publishes a maturity model equivalent to the CNCF's Sandbox/Incubating/Graduated tiers. That's unfortunate, because it means the governance state of projects under those umbrellas is harder to read at a glance - you have to do the work yourself, case by case. Whatever criticism the CNCF deserves about the landscape, the maturity tiers themselves are a real contribution, and it would help the whole ecosystem if other foundations adopted something similar.

None of these are wrong. Several are serving real users in production. But when you line them up next to the three CNCF OSes, the spectrum of governance patterns becomes visible - Incubating at the CNCF, Sandbox at the CNCF, inside an LF umbrella at the NeoNephos or LF Edge level, pure single-vendor open source, single-vendor proprietary. All of them are available to you as a platform team. The question is which one you want under your workloads, for which reasons, with what expectations about what happens in five years.

What to actually look for​

If you're evaluating any OS - CNCF-hosted or not - these are the signals I'd check after the license:

• Adoption. Who is actually using this in production, and are those adopters diverse enough that the project isn't serving one company's internal needs? If the only serious users are the company that maintains it, the project might be great - but you're essentially adopting their internal tool. That's a different bet.
• Supporting companies and individuals. How many organizations are contributing meaningfully, and how many people are in the maintainer role? A project with maintainers from one company isn't necessarily worse than one with five - but the structural fragility is real, and you should know which you're getting.
• Public records, not just code. It's not enough for the code to be open. Decisions, governance changes, maintainer onboarding, votes - the auditable parts of how the project is run should be visible too. GitVote is one mechanism. Public maintainer meetings with published minutes are another. The point is that an interested person can reconstruct how decisions got made without having to ask anyone privately.
• Intention to keep growing. Maturity isn't a static state; it's a direction. A Sandbox project with a dated, public trajectory toward Incubation is in a different place than one that's been at Sandbox for three years without visible movement. Ask projects about their next governance milestone - not just their next feature.

Apply those to Kairos if you're evaluating Kairos. Apply them to Flatcar if you're evaluating Flatcar. Apply them to Bottlerocket or Talos or Garden Linux with context-appropriate adjustments. The signals are the same; only the scenery changes.

I'm one of the maintainers of Kairos, an operating system built for cloud-native workloads. I also served a term as co-chair of what was then the CNCF Special Purpose Operating System Working Group under TAG Runtime. Both of those shape how I see this space. These are my personal views, not the official views of my employer or of the Kairos project. I'll try to be fair to other projects and honest about where my perspective is partial.

GPOS and SPOS​

Most people, when they say "operating system," are thinking of a general-purpose operating system (GPOS). Linux, macOS, Windows, the BSDs. The defining property of a GPOS is exactly what it sounds like: it's designed to run a wide range of workloads on a wide range of hardware, configurable in a wide range of ways, for a wide range of users. GPOSes are the right answer when you don't know in advance what the machine is going to do, when the machine is going to do several things at once, or when a human is going to log in and use the machine interactively.

A special-purpose operating system (SPOS) is the opposite. It's designed around a specific operational model, workload, or hardware profile. You give up generality in exchange for fit: the OS matches its purpose more precisely than a general-purpose OS can, because it doesn't have to accommodate every other purpose too.

SPOS is an umbrella term, and the umbrella is wider than most cloud-native conversations acknowledge. It covers:

• Mobile operating systems like Android and iOS, purpose-built for touch-first, battery-constrained, app-store-distributed devices
• Real-time operating systems (RTOS) used in industrial control, avionics, and automotive systems, where deterministic timing is the defining constraint
• Embedded operating systems for routers, appliances, IoT devices, and other devices with fixed functions
• Unikernels: single-application images that collapse the OS/app boundary entirely
• Cloud-native operating systems: the subcategory I'll spend the rest of this post on

These are all SPOSes, and they are not interchangeable. An RTOS would be a bad fit for a phone, Android would be a bad fit for an industrial controller, and a cloud-native OS would be a bad fit for either. The "special" in "special-purpose" means something different in each case.

The SPOS Working Group, and why "cloud-native OS" is a better name for this subcategory​

For a while, the CNCF had a working group specifically focused on this space. It was the Special Purpose Operating System Working Group, it sat under TAG Runtime, and it was the home where practitioners from Bottlerocket, Flatcar, Talos, Kairos, openSUSE MicroOS, and others compared notes. In the CNCF TAG restructure that happened in 2025, TAG Runtime and several adjacent TAGs were consolidated into a new TAG called Workloads Foundation, and the old working groups, including SPOS, were dissolved as part of that restructure. Kairos, Flatcar, and bootc sit under TAG Workloads Foundation.

The dissolution of the SPOS WG is actually useful for a naming conversation I've been wanting to have for a while. "SPOS" was always a slightly awkward fit for the specific category that working group covered: it's accurate but ambiguous, since SPOS includes things that have nothing to do with cloud-native. The original WG charter scoped itself further to "container OSes," which was more specific but still named the category by its current workload rather than its architecture. Containers are what these systems run today. WASM is what some of them will run tomorrow. Whatever comes after is what they'll run in ten years. Defining the category by the workload makes the definition brittle as workloads evolve.

What I think works better, and what I'll use for the rest of this post and a series that follows it, is cloud-native OS. The term names the design goal rather than the implementation, which makes it robust to how workloads and techniques evolve. And now that the SPOS WG no longer exists as an active body, there's no institutional name to displace. The conversation is free to pick up a clearer term.

What makes an OS "cloud-native"?​

Here's where I want to be careful, because it's easy to define cloud-native OS the wrong way.

The tempting definition is to list features: image-based delivery, immutable root filesystem, atomic upgrades, declarative configuration, OCI-artifact transport, trusted boot. These are the properties most of the projects in this space have, and they're the properties I'll spend most of the rest of the series talking about.

But features aren't the definition. Features are how you achieve the definition. If we say a cloud-native OS is one that has these five features, then any project that adopts those features, even if it doesn't serve cloud-native goals, would qualify, and any project that serves cloud-native goals through different techniques wouldn't. Both of those conclusions seem wrong to me.

A better framing starts with what cloud-native itself is about. The CNCF defines cloud-native as a set of practices for building and running scalable applications in modern, dynamic environments. The original context was public cloud, but the same goals have since been applied on-prem, at the edge, in air-gapped environments, and across sovereign clouds. The goals are:

• Scalability: systems that scale up and down with demand without requiring architectural redesign
• Reliability: systems that tolerate the failure of individual components without taking everything down
• Portability: workloads that can move between environments without being rewritten
• Manageability at scale: fleets that can be operated declaratively rather than node-by-node
• Observability: systems whose state is legible to automated monitoring and human operators

Cloud-native techniques, containers, immutable infrastructure, declarative APIs, orchestration, are how the industry has been achieving those goals. They're not the goals themselves.

So here's the definition I'd propose: a cloud-native OS is an operating system designed to serve cloud-native goals. It's scalable, reliable, portable, manageable at scale, and observable, at the OS layer, for the fleet it's part of. The features most of these projects share (image-based delivery, immutability, atomic upgrades, declarative config) are the current best-known techniques for achieving those goals, but they're not the only possible techniques, and future cloud-native OSes might achieve the same goals in different ways.

I think this framing is better than a features-list definition for three reasons. First, it's honest about what we're actually trying to do: the techniques serve the goals, the goals don't serve the techniques. Second, it's robust: if someone invents a better technique for achieving cloud-native goals at the OS layer ten years from now, we don't have to redefine the category. And third, it makes the evaluation question sharper: instead of asking "does this OS have image-based upgrades," you ask "does this OS actually serve my cloud-native goals in my environment," which is the question you needed answered anyway.

Did the OS stop mattering?​

At some point in the cloud-computing timeline, the industry started acting like the OS didn't matter anymore. Platforms-as-a-Service abstracted everything below the application. You wrote code, shipped it, and the platform handled the rest. Kubernetes went a step further: the OS became the thing Kubernetes ran on, and for many teams the answer to "which OS?" was "whichever one the cloud provider picked for us."

That abstraction is genuinely useful for a lot of teams. If you're a startup moving fast, you probably shouldn't be spending time on OS choices. The platform is your operating environment, and that's fine.

But there's a cost to taking the abstraction too far. When the OS is invisible, you lose the ability to reason about things the OS is in the best position to answer: how do I know what's actually running on this machine, how do I verify nothing has been tampered with between build and boot, how do I upgrade a fleet predictably, what happens when a node behaves strangely, what's my recovery story if a physical device is compromised, how do I prove to an auditor that my compliance claims are backed by measurable facts?

These questions don't stop existing because the OS got abstracted. They just stop having good answers. And as platform teams take more direct ownership of edge deployments, sovereign clouds, air-gapped environments, and Kubernetes infrastructure they don't rent from a hyperscaler, the questions come back. Cloud-native OSes are what you get when you take those questions seriously and design the OS layer around them, rather than inheriting a general-purpose distribution and hoping for the best.

Examples in the current landscape​

A few projects in this space, as of now:

• Bottlerocket: AWS's purpose-built OS for container workloads, Apache 2.0 licensed, not in any foundation
• Flatcar Container Linux: CNCF Incubating, Microsoft-stewarded via the Kinvolk acquisition, the first OS to reach CNCF Incubation
• Talos Linux: Sidero Labs', API-driven with no shell access, MPL 2.0 for the OS itself
• Kairos: CNCF Sandbox, first operating system accepted into the CNCF, distribution-agnostic and Kubernetes-first
• bootc: CNCF Sandbox, Red Hat-led, bootable OS images using ostree and composefs
• EVE OS: LF Edge project, created by ZEDEDA for distributed edge and fleet-managed deployments
• Garden Linux: SAP-built, donated to the NeoNephos Foundation under Linux Foundation Europe
• Google Container-Optimized OS: Google's internal OS for GCP, not open source in any meaningful sense
• openSUSE MicroOS: a transactional-update variant of openSUSE aimed at container workloads

Each of these makes different trade-offs, and I'll go deeper on several of them in other posts. The point of listing them here is just to make clear that the cloud-native OS category isn't hypothetical. It has real projects, real adopters, and a real body of practice that has accumulated over the last several years.

What this series will cover​

This post is the on-ramp to a series I'm starting on cloud-native operating systems: what they are, what they promise, what they actually deliver, how to evaluate them, and where the category is going. Upcoming posts will cover:

• Why image-based delivery is the foundational technical shift, and why most cloud-native-OS properties depend on it
• How to think about governance at the OS layer, since the people who control the OS project end up controlling part of your platform
• What trusted boot and measured boot actually prove, and why the "platform contract" framing matters more than the pure security framing
• How to evaluate security claims in this space without being taken in by marketing that oversells attack-surface arguments
• The honest trade-offs: what you give up moving from a general-purpose OS to a cloud-native one, and when those trade-offs aren't worth it
• A deeper survey of the landscape above, with governance, licensing, and adoption context for each project

If you're evaluating any of these systems for real, or just trying to make sense of the category, I hope the series is useful.

Let's continue the conversation​

If you're working on this space too, or just thinking through these ideas, feel free to reach out through my contact page.

Why it exists​

Quickstarts fall apart when people get stuck on local virtualization. On Apple Silicon, VirtualBox is a non-starter. On Linux vs macOS, QEMU setup and packages differ just enough to slow people down. That detour is not the point. The goal of the quickstart is to teach the Kairos lifecycle and the idea of immutable systems, not to turn someone into a host-virtualization expert.

kairos-lab simplifies this path. It gives newcomers one consistent set of steps across macOS and Linux so they can focus on learning Kairos, not wrestling with host-specific setup.

Who it's for (and who it's not)​

This is for people who rarely touch VMs and just want to try Kairos. It's also for workshops, where I want everyone starting from the same baseline so we can spend time on the actual content.

Power users should absolutely keep using their preferred virtualization software. kairos-lab is not a replacement for full-featured VM tooling. It's the on-ramp.

Getting started​

Install via Homebrew or download a binary from the releases page. That's it.

My first vibe-coded work project​

This is the first project I'm fully vibe coding for work. I didn't write the code myself; I acted as a tester and product owner. That might sound like a bad thing, but I see it as a positive in this case.

This is not production infrastructure. It's a helper tool for onboarding and workshops. I would not have had the time to build this on top of my day-to-day work without vibecoding it. And in my opinion the value it brings (removing onboarding friction) is higher than the cost of maintaining it, especially because maintenance will probably be vibemaintenance.

Side note: I used ChatGPT 5.3 codex (via open-code) for the alpha version, which worked well on macOS. To fix some bridged networking issues on Linux, I used Opus 4.5 (via cursor-agent).

This is also a follow-up to "Coding with AI: From Blank Canvas to Working App in a Day", which I wrote exactly 10 months ago. The level of improvement since then is insane. I can clearly see this becoming a de facto way of building many projects, and with the right guardrails, also for production code.

I wasn't surprised that building these images natively on ARM would be faster than doing it from an x86 machine using emulation. That part was expected.

What surprised me was how much faster it was.

Seeing a Raspberry Pi 5 consistently outperform a 3-year-old high-end Intel laptop by almost 2x forced me to re-evaluate some assumptions I had about build performance, hardware specs, and what "powerful enough" actually means when you're building ARM software.

The setup​

I ran the same Docker-based build targeting linux/arm64 on two machines.

On paper, the Intel laptop should dominate: it has about 3.5x more CPU cores (14 vs 4) with much higher boost clocks, roughly 8x the RAM (62 GB vs ~8 GB), and an NVMe drive that is typically much faster than a USB SSD in both throughput and latency.

On the Intel system, ARM execution is handled through Docker BuildKit with binfmt_misc enabled using tonistiigi/binfmt, which registers qemu-aarch64 at the kernel level. No static QEMU binary is copied into the image; emulation happens transparently during the build.

The Dockerfile​

The Dockerfile is not compiling application code. It performs a lot of OS-level work:

• apt-get update
• installing many packages
• running post-install scripts
• enabling system services
• generating initramfs/dracut bits
• pulling NVIDIA repositories

In other words: this is much closer to assembling a small Linux distribution than building a typical container image.

If you want to see the exact build steps, the Dockerfile is here.

Results​

Using the exact same Dockerfile and target architecture:

• Raspberry Pi 5: real 7m35s
• Intel laptop: real 14m31s

Despite the Intel machine having dramatically more CPU power, memory, and faster storage, it still took almost twice as long. For transparency, I ran the build once on the Raspberry Pi 5 and a couple of times on the Intel machine; the Intel timings were consistent enough that cold vs warm effects did not change the overall outcome.

Why this happens​

The decisive difference isn't raw performance, it's architecture alignment.

On the Raspberry Pi, everything runs natively. ARM binaries execute directly on an ARM CPU.

On the Intel machine, all ARM binaries run through QEMU user-mode emulation. That means every instruction has to be translated before it can execute.

This build workload is particularly unfriendly to emulation:

• heavy apt / dpkg usage
• many short-lived processes
• lots of filesystem operations
• syscall-heavy post-install scripts
• very little meaningful parallelism

This isn't a compute-bound workload where faster clocks and more cores help. It's dominated by overhead, and emulation multiplies that cost.

As a result, even a much more powerful x86 system can lose badly to a modest ARM system when the workload is OS-heavy.

The part that really matters: cost vs time​

A Raspberry Pi 5 costs roughly EUR 80-100, depending on the RAM configuration and availability.

That's not "cheap" in an absolute sense, but in practice it's a bargain.

If that machine saves me even a few minutes per build -- multiplied across many iterations while working on images -- it pays for itself very quickly. Not in hardware terms, but in focus, iteration speed, and reduced friction.

Instead of waiting 15 minutes for a build to finish on my laptop under emulation, I can let a small ARM box do the work in half the time, quietly, in the background.

For the type of work I'm doing, that trade-off is an easy decision.

This applies even more to CI pipelines​

The same logic applies to pipelines, arguably even more so.

For Kairos, we run our ARM builds on native ARM runners, which GitHub provides for free. Moving away from emulated ARM builds has significantly reduced how long our pipelines run.

The impact is very noticeable:

• faster feedback loops
• less wasted CI time
• fewer flaky or timing-sensitive failures
• lower overall pipeline cost

When your builds are dominated by package installation, system initialization, and OS-level steps, native execution isn't an optimization, it's the correct architectural choice.

Side note: about Docker "cache" confusion​

While running these experiments, I repeatedly saw output like:

CACHED FROM ubuntu:22.04@sha256:...

even when using --no-cache and --pull.

What's happening here is subtle:

• --no-cache disables Docker build step caching
• it does not clear BuildKit's internal content store
• base image layers and remote ADD blobs can still be reused by digest

If you really want to start from a clean slate, you need to clear BuildKit's cache explicitly:

docker builder prune -a --force

This removes cached content stored by BuildKit itself, not just images visible via docker images.

It's a blunt tool, but useful when you're trying to reason about cold-build performance.

What I took away from this​

I didn't learn that native ARM is faster -- I already knew that.

What I learned is that architecture alignment matters far more than raw hardware specs for certain workloads.

When your build process is dominated by package managers, system initialization, and distribution-level tooling, native execution can easily outperform much stronger hardware running under emulation.

The Raspberry Pi 5 didn't win because it's fast.

It won because it speaks the right language.

Three years ago, I came back with a completely different paradigm: make existing systems immutable and image-based, remove drift at the root, and do it the same way across major Linux distros to avoid lock-in (see this talk).

TL;DR​

This year, on February 1st, I'll be back to present Hadron, our new image-based Linux distribution, at FOSDEM. If you want the backstory, read on.

Disclaimer​

These ideas and projects are the work of many people. I’m not the visionary here; I’m just sharing a path I believe in and the lessons I’ve picked up along the way. The credit belongs to the communities and teams behind them.

Configuration Drift​

In 2016 I was part of the Machinery team at SUSE. Machinery is a tool for system introspection. It showed the current state of a system: which packages were installed and how their configuration differed from the defaults. That made it easy to spot a server drifting from the desired configuration. It felt powerful, because it made visible something that is usually invisible: how systems drift.

One of the ambitions of Machinery was to inspect multiple Linux distributions, but anyone who has worked with them knows how much they can differ. We thought we could tackle this by introducing USD. It didn’t pick up, and maybe that’s for the best.

What the industry did pick up was the use of configuration management systems (CMS). These introduced the idea of a single source of truth in the form of recipes, cookbooks, and playbooks. But CMS have limits. A full blog post is needed to explain all of that, so for now let’s focus on the main issue. If a system can be changed during runtime, then it’s only a matter of time before it happens in a way you did not predict and did not want.

And that was the recurring pain we kept running into: configuration drift. CMS designers understand this and introduced features to re-run configuration of their systems periodically. At best this leaves the system briefly disconfigured; at worst it fails to address changes that are not included in the system description. And for Machinery, well, introspection is nice to have, but not useful enough to get traction. So Machinery eventually reached its natural end, and I took a hiatus from this problem.

Image-Based / Immutable Systems​

Fast forward to 2023, I joined the Kairos team. Kairos turns an existing Linux distribution into an image-based one. You don’t mutate the base system at runtime; you swap the full image, or you don’t. Part of the system is mounted read-only (immutable), so package manager changes during runtime are off the table. No more drift.

As a bonus, this approach comes with side effects. Your bill of materials (SBOM) can be trusted because your system doesn’t change unexpectedly. You can use measured boot for critical parts like the kernel, cmdline, and initramfs, store the measurements in a TPM, and ensure only systems signed with your keys can boot.

Now Kairos is not the only project offering image-based systems out there. There's Google COS, Amazon Bottlerocket, Microsoft Flatcar, SUSE MicroOS, to name a few. And yes, I mention the ones backed by major platform vendors to make a point: this is a space getting serious investment and it will keep growing. But Kairos is unique in that it lets you choose your underlying Linux distro.

Enter Hadron​

That flexibility Kairos gives is an asset. Within one organization you might be paying licenses for RHEL in your datacenter but running Ubuntu on your NVIDIA Orin devices; Kairos has you covered. Or maybe your team prefers Alpine? No problem. But that flexibility comes at a cost. After building hundreds of artifacts across distributions and options, we noticed two things: it can be confusing for newcomers, and some systems need just the bare minimum.

Our solution is a new Linux distribution, Hadron. It will not compete against all other distros; it will be one more flavor to use Kairos, but it will be the default and the only one we ship. This addresses both issues, gives us a minimal, upstream-first base to build on, and gives us a single baseline to test, document, and ship. Anyone who prefers to use one of the other base distros can keep doing so; they just need to build the images themselves, but we also provide all the tooling you may need with the Kairos Factory, which lets you build images from the CLI, web UI, or even a GitHub Action.

Hadron is minimal and lean, upstream-first, and designed for the hard requirements of the edge while still running cloud-native workloads. It uses an uncommon combination of musl libc, systemd, vanilla kernel, and busybox to keep the base small and the attack surface lower while staying modern.

FOSDEM 2026​

Join us on February 1st at FOSDEM with my colleague Dimitris Karakasilis. We will share more insights into Hadron. We have a deep respect for this community, and that’s why we bring this talk to them. We want to present our work and gather feedback from those who have been building distributions for a while now.

I’ll never forget this year, less for the work and more for my son’s birth. While I want to keep that personal, it’s too big to ignore. This new role as a parent is teaching me different ways to prioritize and work, and I’m so excited about it.

Update (2025-12-30): Added CNCF blog posts.

20 Years Working in Tech​

This year marked my 20th year in the tech industry. Over that time, I’ve worked on a lot of interesting projects and at some well-known tech companies. I earned a promotion to Staff Engineer at Spectro Cloud, and it feels like an important next step and a continuation of growth. It’s especially meaningful because I’m working on a project I find both interesting and impactful, alongside a team I truly admire, so I feel very proud of this accomplishment.

The Cloud Native OS​

That project is Kairos. It’s the most challenging tech I’ve worked on because it demands programming knowledge and a deep understanding of Linux and its components: init systems, boot mechanisms, and more. It’s a lot of fun for my inner nerd, but it spans so many areas that are still new to me. And as if that wasn’t challenging enough, a big portion of my work is also about promoting the project and improving the user experience. I’m learning that on the go; I only started doing DevRel work last year. Here’s what mattered most:

• k0s integration: Kairos had first-class k3s support and a patchwork of community providers. I brought k0s in as a first-class option using the same Kairos-native cloud-init configuration, which means the same YAML-driven workflow, EdgeVPN-based P2P cluster formation, and a much smoother onboarding path. It turns Kairos into a Kubernetes meta-distribution, with native support for both k3s and k0s while keeping the immutable, secure-by-default experience intact.
• Factory Web UI: To help people try Kairos and understand how the factory works, I made solid improvements to the Factory Web UI. That work led directly to my Pok8s talk on image-based systems and curated Kubernetes OS images, where I frame the problem space with playful metaphors and show why Kairos is a strong balance between convenience and control.
• kairos-factory-action: Once the Factory Web UI makes the workflow tangible, the next step is to run it inside pipelines. I built the kairos-factory-action so teams can generate images quickly by passing a few arguments in GitHub Actions. It’s now what we use to test, build, and release Kairos.

My colleagues also shipped an amazing feature: a new Linux distribution specifically designed for image-based systems. On my side, I produced a Quick Start series for it. I’m proud of how it came out. It’s pushing me out of my comfort zone and giving me new ways to reach people and share Kairos, Hadron, and what I now see as the Cloud Native OS.

This year I delivered 11 presentations, mostly at CNCF-related events, and every stage was a chance to share what we’re building and listen to what users need. I’m also excited that 4 events are already scheduled for 2026. See the full list on my public speaking page.

CNCF Blog Posts​

• Building secure Kubernetes Edge images with Kairos and k0s: A joint post with William Rizzo on why Kairos + k0s make a secure, immutable, and low-ops edge stack, and how the integration simplifies real-world edge deployments.
• How to integrate Kairos architecturally into an edge AI platform: A field story with Jordan Karapanagiotis from Aurea Imaging on running K3s with Kairos on NVIDIA Jetson devices, using immutable upgrades and A/B rollbacks to keep fleets consistent and maintainable.

CNCF Groups and Meetups​

I continued co-hosting the CNCF Belgian Chapter and ran four well-attended meetups, around 30 people each, bringing companies and practitioners together. It’s been rewarding to see Belgium’s Cloud Native scene get stronger, and I’m excited to keep at it into 2026.

The CNCF removed working groups, which closed the Special-Purpose Operating System Work Group. I was bummed about not being a co-chair anymore, but it feels like a closing chapter. I’m now participating in the TAG Workloads Foundation, and I’m looking to reconnect with other OS makers to see if we can put something together.

Gratitude and Acknowledgments​

Like last year, thanks to my team for being so supportive and hardworking, you guys rock! To the many people in the Cloud Native space who have been present, you know who you are, thank you. And a very special thanks to William Rizzo. Together we worked on the k0s integration, delivered 4 of those talks, and he even presented at the very first CNCF Community meetup I organized this year. William is very knowledgeable about Kubernetes and CNCF, but most importantly I’m very happy to have made a friend along the way.

What I Missed (And Why That’s Okay)​

I didn’t hit everything I set out to do. I wanted deeper Kubernetes study (a homelab was supposed to help) and a content workflow that turns maintainer work into regular posts. I did level up my AI tooling, but the homelab is still on the backlog, and my content pipeline isn’t where I want it. That gap is a reminder to keep the goals a little out of reach so there’s room to grow.

Looking Ahead​

Heading into 2026, I’m very excited about the possibilities with Kairos and Hadron. Talking with our users and learning about the different projects where they’re using Kairos, I see all the things we’re doing well. Seeing what other projects are doing, I feel like we have a very stable foundation and more flexibility than most. So if the stars align, my goal is to keep promoting the project and introduce it as the Cloud Native OS.

To achieve that, I plan to make Hadron the default in Kairos docs while still keeping all other flavors as first-class options. I also want to publish at least one technical video each month in 2026.

I plan to submit a CNCF Ambassador application and aim for acceptance or clear feedback. I believe in Cloud Native technology, and part of that is helping people see how relevant the OS layer still is in the Kubernetes world.

I've never used Fedora before, but I've used openSUSE, and to be honest, most major distros are so similar that for the majority of my needs all I have to do is understand which package manager they use and the underlying init system. In the case of Fedora Silverblue those are rpm-ostree and systemd. For the latter, there are no surprises; it's the same one I've used for many years now in openSUSE and Ubuntu, the two major distros I use. For the former, it's a bit less obvious because here we're comparing apples to pears: while openSUSE has zypper and Ubuntu has apt, Fedora has dnf, but we can't make use of it since the system is immutable. rpm-ostree is more of a hybrid between a package manager and an image manager. So yes, it knows about packages but it doesn't install them on your running system. Maybe first we need to quickly introduce what an immutable system is, and then we can dig into how you maintain such systems.

Disclaimer: I'm not an expert in Fedora Silverblue. If there's anything in this article that is wrong or misleading, please send me an email (bottom of the article) so I can correct it and learn from my mistake.

Immutable OS​

An immutable OS is not completely immutable, otherwise you wouldn't be able to run anything. What most operating systems out there mean when they say they are immutable is that there is a layer mounted read-only and therefore it cannot be modified. This is meant as a positive thing because it gives you a predictable system while reducing the attack surface, or at least making it harder to get hacked. There's a lot more under this term, but let's use this definition for this post. This means that by design you cannot install, remove, or upgrade packages in these systems.

System Modifications​

Obviously if you couldn't install, remove, or upgrade anything on your Linux desktop, it would be pretty useless. Hence we need mechanisms for modifying the system. There are two major approaches to achieve this. The first is to make changes in the areas of the system that aren't mounted read-only. That's partially true (there are more nuances), but for the sake of keeping things simple we're going to say that's the case. For example, within my home directory I can do whatever I want. This gives me lots of freedom without exposing the more sensitive areas of the system. We can achieve this via Flatpaks and toolboxes; you can think of them as two ways to containerize applications. There are obviously more ways to extend the system and I'll dig into them a bit later, but I'm mentioning these first because Fedora Silverblue seems to promote them.

The other way to modify the system is not to change the running system at all, but to create a new version of the immutable image based on the current image plus our desired changes. Fedora's solution for this is rpm-ostree. I haven't dug enough into Fedora's ecosystem to tell if there are more ways to generate images, so I'll focus on this one for now.

Flatpaks​

Flatpaks are very user-friendly. You can install them through the system's Store, which is not really a store because you don't pay for anything, but it's named like that for users coming from the Apple, Microsoft, or Android ecosystems. Installing applications is very simple: just look for what you want and click install.

Linux being Linux, you can also search, install, update, and remove Flatpaks via the command line. You can read more about it in their documentation but a quick example would be:

flatpak install gimp

This is how I've installed Slack and Visual Studio Code.

Not everything is perfect though. For example, Visual Studio Code throws a messages after installation about missing permissions, and giving me only a hint about how to solve the problem is IMO bad UX. Instead of just offering a close button, it should let me jump straight into the settings where I can make the needed changes.

Toolboxes​

From what I understand, a toolbox is like a sandbox. Imagine spinning up a container that already mounts local directories and the network, making it perfect to install everything you need for specific development projects. Normally I'd use tools like goenv or rbenv to have multiple versions of programming languages. With toolboxes I can do that for everything, not just the language runtime. Plus, it's better isolated! Not only are they cool, they're well integrated with the terminal. For example, if you're already inside a toolbox, opening a new tab drops you in the same toolbox.

These toolboxes run on top of Podman, Red Hat's container runtime. I'm more of a Docker user myself and while the experience of using Podman was ok for the most part, they aren't one-to-one compatible. I had to jump through some hoops to get some of the commands on the Kairos documentation running properly on Podman.

This is how I've installed Vim and npm.

So far the experience with toolboxes has been great. The only complaint I have is that my muscle memory keeps trying to open Vim on the host system when I know it's only installed in the toolbox. I'd like to make the toolbox the default for the terminal and switch to the host system, or a different toolbox, only when I need to. I still have to check if there's a way to configure that.

Update 2025-11-23

I've been pointed out to Distrobox, which seems to run on different hosts and allows you to spin up contianers of different distributions. This resonates better with me and it's also a good fit for Kairos.

RPM-OStree​

Okay, so far we've covered how to install software on the running system, but at some point you'll notice that there are packages you cannot easily install via Flatpaks or within a toolbox. In that case, you want to create a new image with those packages. In Fedora Silverblue land this process is called layering.

The tool used to layer your system is rpm-ostree. Think of it as a package manager that can peek across images. Within its database it keeps the currently installed packages and tracks the packages you want to add to your system. I say "tracks" because it doesn't apply the changes to the running system. To see the changes you must restart, and then you get a new image, most likely the original base with your layers on top, though I still need to read more about the exact architecture.

In the example below, you can see that my system's previous image included the cursor package, which has been removed in my current one, denoted by the ● symbol, and there's a new version of the base image with 43 upgraded packages.

mauro@workstation:~$ rpm-ostree status
State: idle
Deployments:
  fedora:fedora/42/x86_64/silverblue
                  Version: 42.20251027.0 (2025-10-27T00:31:19Z)
               BaseCommit: 220e7462ba4ef4389f88a7d1de72c266676ef317225f8d003e82a9c3b37dd45a
             GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944
                     Diff: 43 upgraded
          LayeredPackages: cockpit cockpit-machines libvirt-daemon-config-network libvirt-daemon-kvm qemu-kvm tailscale virt-install virt-viewer

● fedora:fedora/42/x86_64/silverblue
                  Version: 42.20251023.0 (2025-10-23T00:38:32Z)
               BaseCommit: 71f6dfcd9d4b76bb831f9e8a82cf244fc698d74e763de63c44cb20b0ca4d205a
             GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944
          LayeredPackages: cockpit cockpit-machines libvirt-daemon-config-network libvirt-daemon-kvm qemu-kvm tailscale virt-install virt-viewer

  fedora:fedora/42/x86_64/silverblue
                  Version: 42.20251023.0 (2025-10-23T00:38:32Z)
               BaseCommit: 71f6dfcd9d4b76bb831f9e8a82cf244fc698d74e763de63c44cb20b0ca4d205a
             GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944
          LayeredPackages: cockpit cockpit-machines libvirt-daemon-config-network libvirt-daemon-kvm qemu-kvm tailscale virt-install virt-viewer
            LocalPackages: cursor-1.7.33-1759462983.el8.x86_64

I see this as a very interesting transition from traditional mutable systems, where you interact with the package manager to install, remove, and upgrade packages in your current system. rpm-ostree gives you a similar feeling, with the major difference that the system you're interacting with is the upcoming one. The other project that comes to mind is ABRoots, but I don't know enough about it to compare them.

I can also diff the database to see how those 43 packages upgrade (I'm only showing three here for the sake of readability):

mauro@workstation:~/Personal$ rpm-ostree db diff
ostree diff commit from: booted deployment (df21230f0d8c842212b5530a492eaec12d9735aa259a9da3f81abccc6d67d29a)
ostree diff commit to:   pending deployment (ad6195067ab96b2b8f9d776b9b5b5346fa2575818efab36cbf6c4b314440303a)
Upgraded:
  ...
  intel-gpu-firmware 20251011-1.fc42 -> 20251021-1.fc42
  ...
  python3 3.13.7-1.fc42 -> 3.13.9-1.fc42
  ...

This is how I've installed libvirt and Tailscale:

For libvirt it was as simple as running:

sudo rpm-ostree install libvirt

For Tailscale, you need to add its sources:

# add tailscale package to the root filesystem
$ sudo ostree remote add tailscale https://pkgs.tailscale.com/stable/fedora/tailscale.repo
$ sudo rpm-ostree install tailscale

# reboot
$ sudo reboot

# now register the device into the tailnet
$ sudo systemctl enable --now tailscaled
$ sudo tailscale up

# there will be a URL displayed, use it to add the device to the tailnet.
# after you've registered it, it will automatically connect every time
# you reboot.

Taken from Andreas Happe's blog

So far I haven't encountered any issues layering my images, but I did notice on some videos and forums that people tend to avoid layering at all costs. I don't know if this is Fedora Silverblue's official suggestion or just a community thing, but I don't think it's good advice. Having built so many immutable systems now with Kairos, I don't see the problem with diverging from the upstream system as long as it's a deliberate design decision. In my example, the packages are meant to run in kernel space, so I don't even know if there would be a way to add them without layering, so there should be no shame in doing that.

Other ways to extend the system​

As I mentioned before, there are more ways to extend the system outside these possibilities; you can lean on systemd's system extensions, ship custom binaries, and what not. So far, I've used an AppImage and a binary to install Cursor's IDE and CLI respectively. The problem with this approach is that you're on your own, so it's up to you or the app developer to make sure you're getting updates with the necessary security patches. In the case of Cursor it does self-update, but that won't be the case for all apps.

Talking about Cursor, on their website they recommend installing their rpm/deb package instead of the AppImage, so first I tried installing it within the toolbox. That resulted in Cursor complaining about the keyring and what not, so I ended up installing it with rpm-ostree, which worked well but required me to repeat the process whenever I wanted to upgrade, and that's a bit too much for my personal taste. So I decided to test the AppImage, and it hasn't given me any trouble during the time I've been using it.

Conclusion​

I've tried to stay objective in the previous sections. My goal with this post is not to praise or trash Fedora Silverblue, but simply to share the thoughts that come to mind. These thoughts are obviously biased by my experience and personal preference.

Traditional and immutable systems are completely different approaches to managing operating systems, and both are valid. We just happen to be more familiar with traditional systems when we talk about desktop environments. If we look at smartphones, we'd realize we've been using "immutable" systems for a long time already.

Immutable systems are a different beast, and some aspects of their architecture might get you scratching your head and thinking, "why do they make things so difficult? I just want to install a package." I think that's the wrong way to look at them. While adding packages can feel harder, things like reproducibility and security often feel easier. That's exactly where I'm a bit confused about Silverblue's value proposition. It does make it easier to manage packages between images, but there's still a learning curve for doing everything in the immutable way. At the same time, I don't see it offering Trusted Boot for enhanced security or, more importantly, a workflow that lets me start from container images when I'm dealing with servers or appliances. Maybe that's why there's so much focus on bootc right now, but I'm not savvy enough with Fedora Silverblue, rpm-ostree, or bootc to say if that's correct. Bottom line, for me it's mostly useful to learn their technologies.

Let me elaborate on that a bit because I don't want to sound unfair. My main goal in running Linux on my workstation is development. There's nothing, IMO, that Fedora Silverblue offers me today that is crucial to get my work done; instead, I end up learning its quirks and being part of a niche community. And the only reason I'm using it is because I want to learn more about how other immutable systems operate. For example, I'd also like to test NixOS at some point. On that front, rpm-ostree, Podman, and toolboxes are really cool. On the other hand, for servers or appliances it's even a "downgrade" in terms of experience compared to Kairos.

Is there someone I think this is a good option for? Yes. If you're someone who does most things in the browser or apps you can find in the Store, this is actually a great system because you don't have to worry about the OS at all, just like you don't care about the OS on your phone.

Also, in case you're wondering why I try Fedora Silverblue instead of just running Kairos, there's a good reason for that. Kairos official images are aimed at "server" use cases, like when you download the server version of Ubuntu or Fedora, not the desktop one. And sure, I could build a desktop myself, like Jacques Landru does in his Polimorphic OS abcd, which I might do at some point, but precisely because I haven't managed such a system, I'm interested in learning how other immutable systems do it to see what I'd have to deal with.

So will I keep using Fedora Silverblue as my daily driver? Yes, but I'm not sure for how long; unless something blocks my work at Kairos, or another distro tempts me, I'll stick with it. If you're already using Silverblue, send any tips my way. Would you try an immutable desktop like Silverblue yourself? What's still blocking you from putting immutable systems on your servers?

Update 2025-11-23

When working on Kairos, I need to be able to build a container on a different architecture than the host system. It's been a few weeks since I tried this and I don't have the issues in mind anymore, but let's just say it was a terrible experience. So, I'm not using Silverblue any more.

One of my favorite lessons as a software developer was the SOLID principles from object-oriented programming. They were never canonical rules about how to write code, but they made me a better Rubyist, not because they told me what to do, but because they made me think why I was doing it.

Today, I find myself thinking in the same terms, but at a deeper layer of the stack: the operating system.

Some of the original SOLID names have been adapted to fit this context, and, just like before, these are not laws, only principles I personally find valuable when building robust, secure, and reproducible systems.

SRP - Single Responsibility Principle​

A Single-Purpose Operating System (SPOS) has one responsibility, in contrast to general-purpose OSes like the one I'm using to write this post, which I also use for software development, video calls, entertainment, online shopping, and so on.

A SPOS is more like your ISP's router or a smart appliance. Historically, these systems only existed in embedded environments, but we no longer have to limit ourselves to that model thanks to the advancements in containerization.

To achieve this, we try to slim down the image to only that which is necessary, the benefits: the attack surface shrinks, audits become easeir, and updates risk less unintended side effects.

Smell: You maintain a single generic image that gets heavily configured at runtime for different roles. Instead, build slim, purpose-built images or extend them cleanly (see OCP).

OCP - Open for Extension, Closed for Modification​

Once your base system has a single responsibility, it should remain closed for modification but open for extension.

Avoid touching the system definition and instead, extend the system through mechanisms like system extensions, bundles, or other. Ideally pick those that can be verified or measured.

Smell: Two machines serve the same purpose but require different firmware, so you bake all firmware into the base image. Instead, keep the base sealed and extend only the firmware required for that specific system.

LSP - Lifecycle Stability Principle​

Every SOLID system should maintain a stable lifecycle: creation, deployment, upgrade, and rollback should all be predictable and reversible. By no means, should the system fail to boot after a lifecycle operation, or if it does it should automatically try the last running system.

In addition to those, there should be a factory reset mechanism to restore the base image and clearing runtime state without re-provisioning the machine from scratch. This operation is not reversible by design. And a recovery mode to debug, but keep in mind that this last one should be used like an airplane's black box, it's something that you know is there just for the very worst case scenarios.

When those guarantees hold, you can trust the system even in an edge location where you cannot easily go in and investigate the state of the system.

Smell: You constantly find yourself accessing the recovery mode of the system, because upgrades leave the system in a broken sate.

ISP - Immutable System Principle​

To achieve LSP, provisioning and upgrading need to be atomic operations, either they are done or not, there should not be an inbetween state. This is hard to achieve with GPOS where the system can fail finishig upgrading a list of packages and yet still boots. While it might sound like a positive thing, it is not because all the sudden you find yourself in an unknown state. Whether your applications work with this set of packages is a matter of luck and not something you have previously tested. You are likekly to have this kind of problem even if you're using a configuration management sytems.

Talking about CMS, we also need to address the elephant in the room. While these systems are meant to follow a recipe that will leave every system looking the same, they have a design flaw, they depend on the upstream source not changing, which we know it's not the case. In practical terms, this means that having defined that you want a certain package present in your system is not SOLID because if you run this process at different moments you might get different results. Even if you pin the version number, the dependencies could change at the source. The worst part of it is how they have been sold as the final solution for managing "cattle" leaving companies thinking they are secure and auditable, while these are just smoke and mirrors.

The only way out of this rat race is immutability.

Smell: After provisioning a fleet, you find that at least one of the machines differes even if just slightly from the others

DIP - Dependency Inversion Principle​

The immutable core should depend on stable abstractions, not on external, changing details.
A change at the outer layer, say, configuration, shouldn't require rebuilding the entire image.

Immutability isn't about freezing everything, it's about controlling where change is allowed to happen. When a system's base layer is immutable, changes are isolated to clearly defined areas (data partitions, overlays, or extensions). Overlay partitions can even reset drift automatically. Any unauthorized change disappears on reboot because the base image is reapplied. And because updates land through fresh images, frequent reboots become an asset: there is no prize for the longest-running, unpatched node when every restart brings you back to a known-good state. This separation keeps the core reproducible and secure while still allowing flexibility where it's needed.

The system layout should reflect that principle: rebuilds only for base components changes (kernel, security), while higher-level ones happen through configuration or declarative overlays.

Smell: You tweak a config file and suddenly need to rebuild the whole OS image. That's an inversion failure, the dependency direction is wrong.

Conclusion​

By reinterpreting SOLID through this lens, we can design systems that are stable at their core yet flexible at their edges, systems we can automate through our pipelines just like we already do with code, and that remain robust enough to trust in production, whether in a datacenter or at the edge.

To put the principles into practice:

• Start by isolating the single purpose of each image, even if it means producing more artifacts.
• Lock the base layer, then expose controlled extension points for firmware, drivers, or workloads.
• Make lifecycle guarantees explicit: document the upgrade path, rollback trigger, and recovery tool.
• Keep mutability quarantined to known surfaces so audits and drift detection stay tractable.
• Route environment-specific details through declarative inputs instead of image rebuilds.

Just like SOLID once made our codebases easier to maintain, these principles can do the same for our systems, helping us build a world of software and infrastructure that's truly SOLID, in name and execution.

One thing I’ve been meaning to do for a while is level up my Kubernetes game.

That might sound odd. I work on Kairos, a Linux-based OS for running Kubernetes, but in practice, most of my day-to-day work is in Linux, containers, and Go. I rarely need to interact with Kubernetes itself. It reminds me of Matz, the creator of Ruby, saying he’s actually a C developer. I guess it's not that uncommon in our field. Still, I want to get better at Kubernetes because the better I understand it, the better I can support Kairos users. And the best way I learn? By building stuff.

The setup​

The week before Hackweek, I went on eBay to buy some used machines for a home cluster. That plan flopped. The seller completely botched the delivery, and I had to request a refund. Thankfully, I found a local shop instead: tweedehandslaptop.com. Not the cheapest option, but totally worth it. Fast service, even during the Belgian summer slowdown. For now, I only bought one node to start small and grow organically.

The node is an HP ProDesk 600 G4 Mini with an Intel Core i5 8500T (8th gen, 2.1 GHz), Intel UHD Graphics 630, 16 GB of RAM, and a 256 GB SSD. I got it for €275. I can already hear the fan even when it’s running without a load, so I’m wondering how bad that will get in the future, but my inner geek thinks it’s worth the trouble. The plan is to eventually add an amd64 worker node that also handles storage, plus a Raspberry Pi 4 to run arm64 workloads and connect to physical peripherals like a doorbell and a camera.

I also picked up a copy of The Book of Kubernetes, which I’ve been enjoying. The exercises are hands-on and solid, but it reminded me why I enjoy working on Kairos.

Configuration management systems like Ansible feel so much heavier compared to working with immutable systems, where you start with the final image. And it’s not just about complexity. It’s kind of wild that everyone has to rerun the configuration of their system from scratch. With an immutable image, you save time and resources by building it once and distributing it. It's more ecological too, you’re not pointlessly running the same setup hundreds of times across different machines and users. Imagine if we didn’t use caching in other areas of computing. It’d be madness! And yet, somehow, our industry is still stuck in the CMS world. But we’re changing that. 😉

Special shoutout to abcd by Jacques Landru, who uses Kairos to teach computer networking. That’s a great example of how infrastructure should be built for learning environments.

Building the image​

One of the first things you need to get used to with immutable systems is the need to rebuild the image for any change you make. While you're still learning, it's tempting to rely on a GUI, but sooner or later, that workflow needs to move into a pipeline.

At Kairos, we’ve worked hard to make that pipeline as smooth as possible. Tools like AuroraBoot offer both a web UI and a CLI, and you can plug the latter straight into your automation. Still, I wanted something even simpler, especially for day-one users.

That’s why my main Hackweek project was creating the kairos-factory-action: a GitHub Action that makes it ridiculously easy to produce Kairos images from a repository. You don’t need to write a Dockerfile or understand the internals of AuroraBoot to get started. Just drop in a workflow like this:

name: Docker Build
uses: kairos-io/kairos-factory-action/.github/workflows/reusable-factory.yaml@main
secrets:
  registry_username: ${{ secrets.REGISTRY_USERNAME }}
  registry_password: ${{ secrets.REGISTRY_PASSWORD }}
with:
  base_image: ubuntu:24.04
  model: "generic"
  arch: "amd64"
  version: "auto"
  iso: true
  grype: true
  registry_domain: "quay.io"
  registry_namespace: "mauromorales"
  summary_artifacts: true
  kubernetes_distro: "k0s"
  kubernetes_version: "v1.33.3+k0s.0"

This snippet declares the base image, Kubernetes distribution, version, and output format (an ISO, in my case). No boilerplate needed.

Boot, browser, boom

Once the GitHub Action finishes, you download the ISO, flash it to a USB stick, and boot up the machine. Open a browser at IP:8080, paste your config, and boom, single-node cluster running.

The configuration at this point is very simple, I mostly followed Kairos' example for setting up a manual single-node cluster with k0s with one small tweak: instead of using the --single flag, I passed --enable-worker and --no-taints. This lets me run it as a single-node cluster for now, while keeping the door open to add more nodes later.

k0s:
  enabled: true
  args:
    - --enable-worker
    - --no-taints

If I want to change something, I just commit it to my repo. The action builds a new image, I point the node to it, and upgrade. No manual builds. It honestly feels a bit like updating an Apple device: the OS just shows up ready to go.

It’s a small cluster (so far), but seeing it boot for the first time felt like flipping the switch on a personal cloud. Can’t wait to see what else I can build with it.

Next steps​

First thing I wanted to deploy? My doorbell app. But then I realized I wanted to tweak a couple things.

That led to a new project: Mowa, short for MacOS Web API. It’ll let my cluster talk to my Mac Mini programmatically. More on that in a follow-up post.

If you’ve got tips or stories of your own, I’d love to hear them.

The experiment​

I wanted to build something simple: a web app that visualizes your life in weeks. Each week is a box. Past weeks are dark, future ones are light. You choose your birthdate, and it instantly shows you how many weeks have passed, how many are left, and what percentage of your (approximate) 4000-week lifespan you've lived.

I used ChatGPT to help craft the initial prompt, iterate on mockups, and step in for debugging when Cursor got stuck in a loop. Most of the development happened in the Cursor IDE.

You can try it here: 4000.mauromorales.com. Code’s on GitHub.

Why I Start with a Prompt for the Prompt​

In my experience, I get better results when I first ask ChatGPT to generate a prompt based on my own rough notes or ideas, rather than jumping straight into writing one myself. You might wonder: if I already know what I want, why not just write the full prompt directly?

For me, it’s a bit like brainstorming. I might have a clear goal in mind, or just a few scattered thoughts, and by handing those to ChatGPT, I get back a structured version of what I’m trying to say. Reading that output helps me spot gaps, clarify assumptions, and refine the ask. It’s like having someone else paraphrase your request so you can check: “Did you get what I meant?” That loop has helped me avoid misunderstandings and sharpen the initial direction before any code gets written.

Here’s the initial prompt I gave ChatGPT:

Help me write a prompt to build one single page app that has 4000 boxes each representing a week of a persons life. Every week that has passed will be painted in black or a dark color every coming week in white or a light color. Ideally all black and white. There is a date picker to select your birthdate, once elected it will automatically calculate how many weeks from the 4000 have passed and paint them, and how many are pending and keep them clear. A percentage is also displayed next to the row where the black and white boxes meet.

From there, we riffed on mockups (see image below) and iterated until it felt right. I asked ChatGPT to generate a prompt for Cursor (my AI coding assistant), restricting it to Tailwind only, no other dependencies. In hindsight, that constraint backfired: it kept defaulting to raw CSS, which didn’t play well with Tailwind’s utility-first mindset.

Coding with AI: What it was like​

The first version worked. It looked close enough to the mockup. But it didn’t hold up on mobile, and dark/light mode was glitchy. I replaced the native date picker with one from Flowbite, which broke everything. The AI couldn’t fix it, so I had to step in and debug manually.

Other rough edges:

• The static app needed fingerprinting for CSS/JS assets to avoid caching issues. When I asked, the AI only did CSS. I had to request JS separately. It worked in the end, but required hand-holding.
• I requested a small feature to load a date via the URL: nailed on the first try.
• I asked for a randomizer mode (simulate life expectancy between 0 and 5000 weeks): almost worked, just needed tweaks.

All in all, it took a couple of hours from idea to delivery.

Pairing with AI isn’t like pairing with a person, and that matters​

So far, I see AI like a pairing partner. It’s good at some things, but not at everything.

One big difference: it’s always confident. A human pair might say “I’m not sure,” or you’d pick up on hesitation in their tone or body language. AI doesn’t do that. I’d love if these tools could express uncertainty in their language, using phrases like “I think”, “maybe”, or “this tends to work” when the solution isn’t guaranteed.

Another loss: speed. The AI writes code instantly, which leaves no room for the natural pauses that happen when pairing with a human. You miss the chance to say: “What command did you use just now?”, “Why that approach?”, or “Oh, this reminds me of…” Those micro-reflections spark understanding, and sometimes, new ideas. Without them, the process becomes transactional, and I think that’s a missed opportunity.

Maybe it’s my fault for expecting it to feel like real pairing. Pairing isn’t just about building. It’s about learning together, spreading context, and sharing reasoning. Maybe I need a better analogy. And maybe finding that analogy will help me use AI more effectively.

What I learned​

1. Clean code isn’t the goal anymore​

I was talking to a friend at a big tech company, and we ended up discussing how clean code isn’t always a priority, especially at scale. In a monolith, you care about maintainability. You can’t just throw everything away each time you want to make a change. But with microservices, it’s different. They’re small, isolated, and often disposable. Sometimes, rewriting is cheaper than refactoring.

That’s where AI starts to shine. It’s often faster to generate a scoped, clean version of a feature than to surgically patch bloated legacy code.

This app wasn’t meant to be a microservice, but it’s small enough that the same principle applies. I haven’t read the full codebase, and honestly, I don’t feel the need to. Some decisions along the way already feel like prototype-quality work, not something I’d want to maintain long-term. And maybe that’s okay. If the cost of rewriting is near-zero, maybe “clean” doesn’t matter in the same way it used to.

2. Debugging feels like context-switching into someone else’s project​

Even when I technically wrote the code, the AI rewrote pieces so often that I’d revisit "done" sections to make sure nothing silently broke. It felt like jumping into a fresh repo each time.

3. AI doesn’t validate its own work​

It happily rewrote working code without checking whether the change actually functioned. For a simple app, I could play QA. But for anything more complex, this is risky, especially for people who don’t know what testing even looks like. At a minimum, AI tools should surface test coverage, suggest test cases, or provide validation options.

4. The right words matter​

I wasted time with vague phrasing like “the styles aren’t updating.” When I got specific, e.g., asking how to fingerprint the style.css to avoid cache issues, the AI nailed it.

Lesson: technical vocabulary = better results.

5. Screenshots helped but were not enough.​

Combining screenshots with text worked better than text alone, especially when describing visual bugs or layout issues. But layout was a challenge. Sometimes it wasn’t enough to describe what I wanted visually, even with screenshots. In some cases, I had to explicitly tell the AI: “Put this <div> inside that one, add this class, give it this attribute.” The specificity mattered. The vaguer I was, the more it hallucinated or improvised in the wrong direction.

6. It never pushed back​

The AI treated my prompts as gospel, even when better options existed. It never suggested fingerprinting early, or warned that it preferred raw CSS over Tailwind. It didn’t offer alternatives, opinions, or guardrails. I want tools that challenge my assumptions, especially when it comes to things like security.

7. AI-generated Git commits: hit or miss​

I experimented with letting AI write my Git commit messages. Sometimes they were great. Other times they just echoed obvious file changes, which I’d rather avoid.

8. Delegating front-end work felt great​

I’m more of a back-end dev, so I was happy to hand off the UI. I still reviewed and tweaked things, but I didn’t get bogged down in unfamiliar territory. If tools like these keep improving, maybe we’ll all become full-stack by default, not because we master every layer, but because we can delegate what we don’t specialize in.

That said, deep expertise still matters. Especially for architecture, scaling, or debugging hard problems.

9. Abstractions raise the floor, but also the ceiling​

Some worry that using AI makes your coding muscles weak. I get that. But it feels more like moving to a higher-level language. You still need to understand what’s happening under the hood to use it well. It’s not a shortcut past skill. It’s a shift in where skill gets applied.

Wrapping up​

This project reminded me that good prompts matter, but so does good judgment. AI helped me ship something I might’ve abandoned otherwise. It’s not perfect, but it’s done. That’s a win.

If you’ve built something with AI, or you’re experimenting with it in your workflow, I’d love to hear how it went. I’m especially curious about how others are navigating the mix of delegation, debugging, and discovery. What did it help with, and where did it fall short?

Update: What if the app didn’t need to be an app?​

After building and deploying the site, a funny thought hit me: this thing is so small. Do I even need a web app?

So I asked ChatGPT to generate a GPT that is the app.

Functionality-wise, it nailed it on the first try. The visual output? Awful. No matter how much we tweaked it, the graphical layout never felt right. But then we landed on a workaround: a Python script that generates an ASCII representation of the graph. Surprisingly satisfying.

What makes this version more interesting is the interaction model. You don’t need to pick a date from a UI, you can just say “I’m 22 years old,” or “I was born in Ecuador in 1990,” or “I’m of Asian descent.” It adapts the calculation accordingly. You can also configure the GPT to only respond about the life-weeks concept and nothing else.

Is it perfect? No. But for 30 minutes of experimentation, I think it’s a success, and maybe even more accessible than the web version for some people.

You can try it here: Your Life in Weeks GPT

Embracing Community at Kairos​

Since 2023, I've been contributing to Kairos as a maintainer. This role has been a mix of technical problem-solving and building something larger than code: a community. In mid-2024, I decided to focus more on developer relations to strengthen Kairos' ecosystem.

Here's what this shift looked like:

• Conference Engagement: Over the year, I participated in 10 conferences across Europe and the US, including delivering a workshop and speaking on a panel. These opportunities allowed me to share Kairos' story, introduce its features, and, most importantly, connect with the Cloud Native community.
• Community Growth: By fostering conversations at these events, I witnessed an increase in community engagement, feedback, and contributions to Kairos. It's a reminder that great technology thrives when supported by an active and inclusive community.

This work reaffirmed my belief that building developer relationships is not just about promoting a product; it's about creating an ecosystem where developers feel heard, valued, and empowered.

Progressing in the Cloud Native Space​

2024 has also been a pivotal year for my involvement in the Cloud Native Computing Foundation (CNCF):

• I became a co-host of the CNCF Belgian Chapter, collaborating with local developers to strengthen the Cloud Native community in Belgium.
• I was honored to be appointed co-chair of the CNCF Special-Purpose Operating System Work Group. This role allows me to help shape the future of operating systems in cloud-native environments, ensuring they are secure, efficient, and innovative.

These roles represent an evolution in my career, not just as a contributor but as a leader who helps drive meaningful change in the ecosystem.

Gratitude and Acknowledgments​

None of these achievements would have been possible without the incredible support system I'm fortunate to have. I want to express my heartfelt thanks to my boss and my team. Your hard work and dedication make it possible for me to focus on community-building and technical advocacy. I'm equally grateful to my employer for supporting my growth and enabling me to take on these opportunities.

A special thank-you also goes out to the amazing people I've connected with this year, you know who you are. Whether through conferences, collaborations, or chance encounters, I've been genuinely delighted to meet and learn from so many talented individuals. These connections remind me that the Cloud Native community is built on the collective passion and generosity of its members.

Why These Achievements Matter To Me​

For most of my career, I saw myself as a tech person rather than a people person. I was comfortable focusing on code, systems, and solutions, but the idea of engaging with communities or building relationships felt outside my realm. This year, by stepping out of my comfort zone and putting in the hard work, I've realized that this is not only an important area but also one I genuinely enjoy.

Through these experiences, I've discovered that fostering connections and growing communities is just as rewarding as solving technical challenges. It's helped me break my own paradigms and embrace the fact that building relationships is a skill worth cultivating, one that complements and enhances my work as a technologist.

Looking ahead, I aim to build on this foundation by continuing to contribute to open-source projects, share knowledge, and connect with like-minded technologists. If there's one takeaway from 2024, it's that growth happens when you step out of your comfort zone and embrace collaboration.

Here's to another year of learning, building, and connecting!

Step 1: Install the Nextcloud Command-Line Client​

First, you'll need to install the Nextcloud command-line client to manage the synchronization:

sudo apt install nextcloud-desktop-cmd

Ensure the package installs successfully so that you can proceed with setting up automated sync tasks.

Configure Credentials​

To enable authentication for nextcloudcmd, add your credentials to the ~/.netrc file with the following format:

machine your-next-cloud-domain.org
login your-username
password your-password

Replace your-username and your-password with your actual Nextcloud login details. This file ensures that the command-line client can authenticate without manual input.

Step 2: Choose Directories to Sync​

For simplicity, I prefer syncing only the top-level directories that are important to me, such as src and similar folders. This approach keeps the sync process efficient and focused.

Step 3: Create the systemd Service File​

To automate the syncing process, create a systemd service file that defines how the sync should be executed. Save the following content as ~/.config/systemd/user/nextcloud-sync-src.service:

[Unit]
Description=Nextcloud Sync Src

[Service]
Type=simple
ExecStart=/usr/bin/nextcloudcmd -n --non-interactive --path /src /home/mauro/src https://your-next-cloud-domain.org
TimeoutStartSec=300
Slice=nextcloud-sync.slice

[Install]
WantedBy=default.target

Step 4: Create the systemd Timer File​

The systemd timer will trigger the service at regular intervals. Save the following as ~/.config/systemd/user/nextcloud-sync-src.timer:

[Unit]
Description=Run Nextcloud Sync Src every 5 minutes
Requires=nextcloud-sync-src.service

[Timer]
Unit=nextcloud-sync-src.service
OnUnitInactiveSec=5m
RandomizedDelaySec=5m
AccuracySec=1s

[Install]
WantedBy=timers.target

This configuration ensures that the service runs every 5 minutes with a randomized delay for slight variability.

Step 5: Create a systemd Slice File​

To limit resource usage, create a slice file to manage CPU and memory allocation. Save this content as ~/.config/systemd/user/nextcloud-sync.slice:

[Unit]
Description=Nextcloud Sync Slice
DefaultDependencies=no
Before=slices.target

[Slice]
CPUQuota=30%
MemoryMax=2G

This configuration will limit the sync process to 30% of the CPU and a maximum of 2 GB of memory.

Step 6: Load and Enable the Timer​

To activate the service and timer, reload the systemd user daemon and enable the timer:

systemctl --user daemon-reload
systemctl --user enable nextcloud-sync-src.timer
systemctl --user start nextcloud-sync-src.timer

Step 7: Monitor the Service and Timer​

To check the logs for the service, use:

journalctl --user -u nextcloud-sync-src

To list active timers and see when they last ran or are scheduled to run next, use:

systemctl --user list-timers

Conclusion​

With this setup, your headless server will automatically sync the specified directories with your Nextcloud server every 5 minutes. This approach keeps your essential files up-to-date without manual intervention, optimizing the workflow for your headless environment.

For this article, I will be making reference to the Ubuntu 24.04 server image because its configuration is easier to understand than the Raspbian one, which uses implicit defaults. I mounted the image as a loop device, hence the dev/loop44 in the examples, but if you burned it to an SSD or SD card, you could get the same results from /dev/sdX and /dev/mmcblkY.

root@zeno:~# lsblk -f /dev/loop44
NAME       FSTYPE FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop44
├─loop44p1 vfat   FAT32 system-boot F526-0340                             419.3M    17% /media/mauro/system-boot
└─loop44p2 ext4   1.0   writable    1305c13b-200a-49e8-8083-80cd01552617  781.9M    66% /media/mauro/writable

From the labels, we can assume the system-boot partition will be the one booting the system, but how does the system know this is the case. From the documentation, I was able to find this:

Partition numbers start at 1 and the MBR partitions are 1 to 4. Specifying partition 0 means boot from the default partition which is the first bootable FAT partition.

Bootable partitions must be formatted as FAT12, FAT16 or FAT32 and contain a start.elf file (or config.txt file on Raspberry Pi 5) in order to be classed as be bootable by the bootloader

Looking at the output of the previous command, only the system-boot partition has the right format, so let’s look into that one first.

# ls -1 /media/mauro/system-boot/
README
bcm2710-rpi-2-b.dtb
bcm2710-rpi-3-b-plus.dtb
bcm2710-rpi-3-b.dtb
bcm2710-rpi-cm3.dtb
bcm2710-rpi-zero-2-w.dtb
bcm2710-rpi-zero-2.dtb
bcm2711-rpi-4-b.dtb
bcm2711-rpi-400.dtb
bcm2711-rpi-cm4.dtb
bcm2711-rpi-cm4s.dtb
bcm2712-rpi-5-b.dtb
bcm2712-rpi-cm5-cm4io.dtb
bcm2712-rpi-cm5-cm5io.dtb
bcm2712d0-rpi-5-b.dtb
boot.scr
bootcode.bin
cmdline.txt
config.txt
fixup.dat
fixup4.dat
fixup4cd.dat
fixup4db.dat
fixup4x.dat
fixup_cd.dat
fixup_db.dat
fixup_x.dat
hat_map.dtb
initrd.img
meta-data
network-config
overlays
start.elf
start4.elf
start4cd.elf
start4db.elf
start4x.elf
start_cd.elf
start_db.elf
start_x.elf
uboot_rpi_3.bin
uboot_rpi_4.bin
uboot_rpi_arm64.bin
user-data
vmlinuz

We can see that the expected config.txt. Let’s take a look at its contents.

root@zeno:~# cat /media/mauro/system-boot/config.txt
[all]
kernel=vmlinuz
cmdline=cmdline.txt
initramfs initrd.img followkernel

[pi4]
max_framebuffers=2
arm_boost=1

[all]
# Enable the audio output, I2C and SPI interfaces on the GPIO header. As these
# parameters related to the base device-tree they must appear *before* any
# other dtoverlay= specification
dtparam=audio=on
dtparam=i2c_arm=on
dtparam=spi=on

# Comment out the following line if the edges of the desktop appear outside
# the edges of your display
disable_overscan=1

# If you have issues with audio, you may try uncommenting the following line
# which forces the HDMI output into HDMI mode instead of DVI (which doesn't
# support audio output)
#hdmi_drive=2

# Enable the serial pins
enable_uart=1

# Autoload overlays for any recognized cameras or displays that are attached
# to the CSI/DSI ports. Please note this is for libcamera support, *not* for
# the legacy camera stack
camera_auto_detect=1
display_auto_detect=1

# Config settings specific to arm64
arm_64bit=1
dtoverlay=dwc2

# Enable the KMS ("full" KMS) graphics overlay, leaving GPU memory as the
# default (the kernel is in control of graphics memory with full KMS)
dtoverlay=vc4-kms-v3d
disable_fw_kms_setup=1

[pi3+]
# Use a smaller contiguous memory area, specifically on the 3A+ to avoid an
# OOM oops on boot. The 3B+ is also affected by this section, but it shouldn't
# cause any issues on that board
dtoverlay=vc4-kms-v3d,cma-128

[pi02]
# The Zero 2W is another 512MB board which is occasionally affected by the same
# OOM oops on boot.
dtoverlay=vc4-kms-v3d,cma-128

[all]

[cm4]
# Enable the USB2 outputs on the IO board (assuming your CM4 is plugged into
# such a board)
dtoverlay=dwc2,dr_mode=host

[all]

I’m only interested in the first 5 lines.

• [all]: makes reference to the board, or in this case, all boards
• kernel: defines the kernel file to be read, in this case vmlinuz which was present on the files list
• cmdline: defines the file with the cmdline used to boot the kernel. In this case cmdline.txt which is also there
• initramfs: defines the initrd file to be loaded, in this case initrd.img, also there. And the followkernel stanza loads the initrd file in memory right after the kernel. Pay attention that this instruction, different from all others, doesn't use the assignment =.

Now we can take a look into cmdline.txt

# cat /media/mauro/system-boot/cmdline.txt
console=serial0,115200 multipath=off dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc

This tells us that the root of the system is a partition with the label writable. This matches with the output of our very first command. Listing everything in writable we find:

root@zeno:~# ls -1 /media/mauro/writable/
bin
bin.usr-is-merged
boot
dev
etc
home
lib
lib.usr-is-merged
lost+found
media
mnt
opt
proc
root
run
sbin
sbin.usr-is-merged
snap
srv
sys
tmp
usr
var

This looks like a common root directory for an Ubuntu system, so I will not go deeper into it.

From a Linux installation on a PC the bootloader, but it turns out that for the Pi 5 it is already part of the EEPROM. So we can trust that it’s present and following the instructions from config.txt.

An important part of this process is the Device Tree (.dtb files), which is also read by the bootloader. The Device Tree describes the hardware present on the board, ensuring that the kernel knows how to interact with all connected peripherals.

To summarize it all. When the Raspberry Pi 5 powers up, the EEPROM will look for the first bootable partition, where it will read the config.txt file. The configuration file tells the bootloader which kernel, initramfs and cmdline params to load. After that, it’s the kernel’s job to decide how to proceed, in this case after the kernel and initramfs are running in memory, it will pivot to the system living in the writable partition. Last and out of the scope of this article, the init system will finalize the system.

Raspbian​

Keep in mind that the Raspbian image doesn't define all these details, since it uses defaults:

The Raspberry Pi 5 firmware defaults to loading kernel_2712.img because this image contains optimizations specific to Raspberry Pi 5 (e.g. 16K page-size). If this file is not present, then the common 64-bit kernel (kernel8.img) will be loaded instead.

And I assume that for initramfs, if not defined, it will also look within the directory and load either initramfs8 or initramfs_2712 by default since those files are present in the raspbian image.

Disclaimer: Epomaker contacted me to write this review as an exchange for keeping the keyboard. They didn’t have a say on what I wrote, except for the two links to their web and Amazon shop.

First Impressions​

The first thing I noticed is the weight of this keyboard, it is thick! Immediately then, I knew this was a solid product (pun intended 🥁). The sound is a delight. And overall, it is a very good-looking keyboard. On the other hand, the knob feels a bit loose, and I’m not a fan of the font type they used for the caps.

But let’s not judge this book only by the cover! I spent almost two weeks with it as my daily driver to find out if this one is a keeper. My main interest in such a device is to write code and prose, so sorry gamers, I don’t have much for you. Here are the 4 areas which I assessed this keyboard on:

1. Comfort with extended use
2. Base functionality. I don’t want to be interrupted while I’m working because something is not connecting properly or other similar issues
3. Extra customizations. Being able to adapt the keyboard to my needs is a great plus for me
4. Aesthetics. I will be looking at this piece of equipment for at least 8 hours every day

Comfort​

The distribution of the keys is superb. I have small hands, but it doesn’t feel like an effort to do any key combination. I’ve experienced RSI in the past, and I will invest in a good keyboard rather than experience pain. I spend hours at a time typing, and so far, I haven’t felt any discomfort with the Tide75.

As a macOS user, the main issue I have with the layout is that it is missing the Cmd key on the right. Thankfully, this can be customized (more on this later)!

I really enjoy how the keyboard will not move a millimeter while I’m typing. I compared this to my Reform keyboard, which is the closest one in size I have, and the Epomaker Tide75 is significantly better. This boy is so heavy that even my DasKeyboard 4 Professional, which is a full-sized keyboard, couldn’t compete. Now this, of course, is a double-sided sword because it means I would never consider carrying it around in a backpack.

One thing I still have to get used to, is the minimum touch the keys require to react, but I like it, it just means I have to change some old bad habits, like resting my thumb on the space bar.

As for the sound, I’m a big fan of loud keyboards (Cherry MX Blue Switches), but you cannot take such a keyboard to an office unless you want to become an outcast, this is why, on my daily drivers I have (Cherry MX Silent Red Switches) with dampers. The sound of the Epomaker Lemon Switch, sits somewhere in between these two, which I would recommend for a single person office, or one that is ok with this level of noise. I tried putting dampers on the keycaps, but it didn’t really have an effect.

Functionality​

The keyboard can be connected using a cable, a dongle, or via Bluetooth. When connected via cable, the keyboard also charges the 4000mAh battery. I attest to the quality of the charge since I had to use the cable during the different times that I configured the keyboard, I’ll make sure to report that in a later update.

Connecting with the dongle worked like a charm. The only complaint I have is that the little housing where it stays put feels a bit loose, maybe it’s not so bad but for me, it’s another reason not to put it inside a bag pack.

Connecting via Bluetooth is super simple. You have 3 presets when pressing Fn + Q/W/R. Switching between them works just fine, and the right preset wakes up my computer as expected. This was very handy since I have multiple devices on my desk and I could switch between them without having to move around the desk as I used to. At some point when testing, I started experiencing disconnections, and it turned out to be caused by low battery. A battery indicator could be a great addition. It could even be done with the existing RGB LEDs, I think that would be pretty sick!

Media keys for macOS can be accessed with the Fn + digit, they are even in the same position as the Apple keyboard, which is a great design choice so you don’t have to learn a new layout. The only minor issue I found was in the manual where it says that Fn + S is Mac mode and Fn + A is Win mode, but for me is the other way around (at least for mac, I don’t have a Windows machine to test).

Customizations​

If I have a keyboard that is comfortable and the basic functionality fits my needs, then I can be happy with it, but being able to customize it, is what can make it a joy to type with a certain keyboard.

The manual says I can configure the layout by downloading an app from this repository, and there is a homebrew formula for it too. Unfortunately, I wasn't able to make it work. The app loads, but it never finds the keyboard. Thankfully, the online version did work. In addition to this app, you also need to download the keyboard’s definition file. The link on the manual doesn't work, but this one does.

Ok, with everything ready, you can go to the “design” tab and, unlike me, make sure to turn OFF “Use V2 definitions (deprecated)” despite it being a step in the manual 🤦‍♂️. Hit the load button, and select the definition file. If things work correctly, you will see the keyboard’s layout on the page. I have the “Long Spaces” version, which is nice, but the “Split Spaces” must be better in my opinion. Now you can move to the “configure” tab, and you should see the layout with all key definitions.

All changes you do to the keyboard, take effect in real time, which is pretty cool for playing with the RGB colors and such configurations. For configuring the layout, I’m still not sure if I like it because I like to make all my changes and change them at once. But it’s just a different way of working, and maybe I won’t even notice it after using it a couple of times.

There are 5 customizable layouts, which means lots of room for making this keyboard your own. For now, I will customize the default layer to have the right Cmd key, and layer 2 for having easier access to the arrow keys on the hjkl keys (yes, Vim has messed up this bad with my head 🙈) and the numbers. This is what it looks like:

If you like my setup, then feel free to grab it and play with it

Aesthetics​

Overall, I think the keyboard looks great. The black pitch of the aluminum mixed with the copper borders and the RGB LEDs on look fantastic! Even the bottom of the keyboard looks pretty nice. This is a product that you enjoy touching, I get a similar vibe when interacting with my MNT Reform 2 Laptop.

There is just one mistake Epomaker made here. The knob border is platinized, which in my opinion doesn't mix well with the copper. I’m also not a fan of the type used, for most letters it is ok, but 4$ reads more like 45. Words like PGDN give the impression that the painting went wrong, but it's just the font type they chose for it. Not a deal-breaker, but hopefully Epomaker will move away from this font in the future, check the difference with the font used in the MNT Keyboard, which is such a delight to read.

Size Comparison​

Pricing​

The Epomaker Tide75 has a $139.99 price tag, plus taxes. Since I didn’t pay for it, I asked ChatGPT to calculate it for me, and it thinks it would be an extra 26.65 EUR for taxes, making it a total of approximately 153 EUR.

Deciding whether a piece of equipment is worth buying is for you to decide, but compared to these other devices, the price seems just about right to me:

• Magic keyboard is 149 EUR. It comes with the T chip, but it doesn’t have the mechanical keyboard experience, replaceable switches, and neither the extensible configuration.
• DasKeyboard Pro is 179 EUR. It is a mechanical keyboard including the Numpad, but lacks RGB, replaceable switches and it’s not configurable.
• Vortex Core is 99 EUR. It comes with the RGBs, and it is configurable, but it’s only a 40% which might be challenging to adapt for many people.

Conclusion​

Overall, I think this is a great product and anyone looking for a mechanical keyboard will not be disappointed. It happens to be that my daily driver, the Ergodox EZ just broke a few months back, and the Tide75 will take its place as my number 1.

The issues I have with the keyboard are minor, but in my opinion, could take the Epomaker from a good choice, to a great one. Having an up-to-date manual, including extra blank keycaps for customization and most important, choosing a more readable font and matching the knob color with the rest of the keyboard.

Thanks Epomaker for this remarkable gift, I hope that they feel proud to see their little boy is helping build open-source.

The following links were given to me by Epomaker. I don’t make any commission on them. If you’d rather not use them, type Epomaker Tide75 in your preferred search engine, and you should find them, but you might get a different product than the one I reviewed here.

Interested in grabbing yours? Check out the Epomaker Online Store, or the Epomaker Amazon Store.

For this article, an edge device refers to a headless computer system (without direct human interface) deployed in remote locations like coffee shops, gas stations, or warehouses.

The Security Challenge at the Edge​

Contrary to popular belief, Linux systems lack certain critical security features found in Windows (Trusted Boot) and macOS (Startup Security). While Linux supports Secure Boot and full-disk encryption, these measures alone are insufficient for edge environments where devices are physically accessible to untrusted parties.

The primary security goals for edge devices are:

1. Preventing unauthorized access to data if the device is stolen.
2. Ensuring the device does not boot if tampered with.

Protecting Your Data with Encryption​

Encrypting the disk keeps your data safe when the device is powered off, addressing the first security goal. However, this protection is compromised if the device is tampered with, leading us to the second goal.

Protecting Your Device from Tampering​

Understanding the Linux boot process is crucial for securing a device against tampering. Upon powering on, a modern computer runs the UEFI firmware, which hands control to a bootloader. The bootloader initiates the operating system, which then decrypts your data and starts your application.

Secure Boot helps secure the initial stage by only allowing execution of digitally signed bootloaders. However, the problem lies in the next stage: most Linux distributions’ bootloaders do not verify the signatures of the Kernel or Initrd, nor do they measure the integrity of these components. This oversight allows potential tampering to go unnoticed.

Measuring for Integrity​

Measuring involves calculating a hash for artifacts like the Linux Kernel. Any change in these artifacts alters the hash. Utilizing Trusted Platform Module (TPM) chips, we can establish a validation system that only proceeds with booting if the measurements match the expected values.

The Unified System Image (USI)​

One effective solution is creating a Unified System Image (USI). This combines the Kernel, cmdline parameters, and Initrd into a single, immutable image. By measuring this single image, we ensure the integrity of the entire system. There’s no need to encrypt this image since it contains no sensitive data, which resides in the encrypted area. The system configuration and valuable data remain secure, and the image is mounted read-only to prevent changes.

For more detailed information on this process, refer to the UAPI Group’s page and Lennart Poettering’s article, Brave New Trusted Boot World.

Kairos: Simplifying Trusted Boot​

Implementing a USI with Trusted Boot can be complex. Kairos aims to simplify this process. Visit the Trusted Boot Installation instructions to try it out, or delve into the Trusted Boot Architecture documentation for a deeper understanding of how Kairos enhances security in untrusted environments.