Week 3 of Building a Software Factory: Shipping Kairos 4.2.0
Last week ended on a version that had shipped but not released, and a review load I was not keeping up with. Both moved. 4.2.0 is out, reset works again on non-UKI installs, fork pull requests finally get CI, and there is now a reviewer agent reading every change before I do. 34 changes landed against Kairos this week, up from 19.
What shipped
v4.2.0, on 18 August. The last thing holding it was two advisories the release scan was still ignoring (kairos#4332). Cleared, tagged, announced.
Reset works again on a non-UKI install. This is the one I would want fixed if I were running Kairos. A state reset on a non-UKI system could come back with the wrong sysroot labels and without its recovery extensions (immucore#611, immucore#613), shipped out through kairos-init#436. It is a bug you only meet on the day you actually needed reset to work. It now has a test covering it (hadron#565) and a lint that fails the build if someone adds a boot state and forgets to handle it (immucore#614).
A bad config_url no longer eats your cloud-init. If the kernel cmdline pointed at a config URL that could not be fetched, the agent skipped the local cloud-init files too, so a typo in one place silently dropped your whole configuration. Fixed in kairos-agent#1344.
Riscv64 is back to one build path. Hadron could already build for riscv64, that part isn't news. What hadn't happened was a release: a split native-plus-emulated pipeline for it was slowing down releases for the architectures that matter more right now. hadron#572 drops the native runner and goes back to plain emulation. Slower, but most of it caches, and it gets stability back. Maybe the next release finally includes a riscv64 image.
Fork pull requests get CI. Before this week, a contribution from a fork got no build at all, which is a bad first experience and a slow one. It builds now, behind a gate a maintainer has to press (kairos#4323, extended in kairos#4328).
Builds are off ttl.sh, and that's a security fix as much as a tidiness one. Images used to hop between CI jobs through a public throwaway registry. They now go to quay or move as tarballs between jobs (hadron#566, hadron#567, kairos-init#440), and the credential that publishes to the production registry no longer sits in a workflow that runs on code from a pull request, where anyone could put it there (kairos#4327). The approach follows what Dimitris Karakasilis (jimmykarily) had already worked out for kcrypt-discovery-challenger.
A reviewer that is not me. Last week I wrote I was leaning toward a separate agent doing a first pass. It exists. Every pull request opens as a draft, a reviewer subagent reads the diff, the ticket and the tests, and deliberately does not read the conversation that produced them, because a reviewer that inherits the author's reasoning is just the author wearing a hat. It sends the change back or moves it to ready for review. On kairos-io it may comment and nothing else. It never merges anything, anywhere. Merge is still mine, in every repository. I reviewed 25 of the 31 changes the agent opened against Kairos this week, compared with 13 of 18 last week. Too early to say if it helps. Ask me next week.
The numbers
Same caveats as last time. These count both my own account and the agent's.
| Activity | 20 Jul – 6 Aug | 7 – 15 Aug | 16 – 21 Aug |
|---|---|---|---|
| Changes shipped to Kairos | 0 | 19 | 34 |
| Changes to the agent factory itself | 22 | 26 | 40 |
| Changes to this website and the homelab | 9 | 20 | 6 |
| Architecture decision records written | 6 | 11 | 4 |
| Postmortems filed | 5 | 2 | 4 |
| Internal board: issues opened / closed | 20 / 15 | 99 / 33 | 110 / 90 |
Two rows are worth a sentence. The board went from ninety-nine opened against thirty-three closed to a hundred and ten against ninety. Last week it was filling about three times faster than it drained. It is now roughly keeping pace, and I would rather have that than a smaller number in either column.
What went wrong
Four postmortems, and the same failure runs through most of them.
A rule nobody checks does not get followed. A policy about which model to use for which kind of work, cheaper ones for grunt work, better ones for judgment calls, had been sitting in the agents' constitution for a month. Every session reads that file. Its application rate over the month was zero. Not low. Zero. It surfaced by accident, because a ticket happened to ask what model a session was running.
The failure mode is what makes it expensive. The rule does not get broken loudly. It reports success. For a month every session read that policy, understood it, and did not apply it. Nothing anywhere produced a signal. To anyone auditing this later, an unenforced rule in that file looks exactly like an enforced one.
Two smaller versions of the same failure landed the same day, each fixed the same way.
The fix in both cases was moving the text into the shared file every session reads, through a branch and a pull request. That is the whole difference between a preference and a mechanism, and it costs one paragraph. The model policy got a check at the moment a ticket is claimed. A poller that filed thirty-six junk tickets in a minute now refuses a first run over five items and prints the command to clear the backlog. None of the fixes were a better-worded rule.
Two things reported healthy while being dead. A user-level timer, the one that tells me when the agent is mentioned on GitHub, went dark for eight hours while is-active and is-enabled both said yes. Neither of those answers the question I care about, which is whether it will ever fire again. Separately, leftover VM disk images filled a shared disk quota on the machine all of this runs on, and every command in every concurrent session started failing with a bare exit code pointing nowhere near the cause. Both cost me more hours than the governance ones did, and both are the same lesson wearing different clothes. The check has to check the thing you care about, not the thing that is easy to report.
What I am doing about it. The fix is boring: an audit of every rule in that constitution, sorted by whether anything except a session reading it carefully makes it true. I already know that audit will find things. The model policy sat there for a month and I found it by accident.
Where that leaves me
A better week on output and a more uncomfortable one on process. The system produces more every week, and the more it produces, the more expensive the gap gets between the rules I have written down and the rules anything actually enforces.
As before: this post was drafted by the system it describes, and I reviewed, edited and merged it myself. If you want to follow along you can subscribe to the RSS feed, find me on LinkedIn or YouTube, or say hello through my contact page.