Skip to main content

Week 7 of Building a Software Factory: Capable Enough to Break Things

· 7 min read

On Thursday morning I merged a security fix from the agent after 71 CI checks passed (kairos#4569). Before noon, every pull request that ran CI after it was red, and so was the post-merge run on master.

The change narrowed the permissions our CI workflows hand out, which a security scanner had flagged. It had sat in draft for eight days. On the morning of the merge the agent brought it up to date with master, the run came back green, and I approved it. Six jobs that the old, wider grant had been covering were left with less than they needed, and nothing in the pull request's own run showed it.

The repair​

The factory has no instrument today that tells it a merge broke the project. It found out because a colleague saw the red runs and told me, and I told the agent.

The fix was open as a draft two and a half hours after the merge (kairos#4704). I do not know how much of that was the break going unnoticed and how much was the agent working; the timestamps only bracket the whole interval. From there, every step needed a person. I approved the fork's CI environment so the fix could test itself, marked the pull request ready and approved it. A second maintainer approved it. I merged it. The repair merged three hours and forty-five minutes after the change that broke it. Seventy-two minutes of that was the pull request's own lifetime, and 31 of those were the gate: two reviews and a merge, none of which the factory can do for itself.

One thing I still do not understand. The six jobs ran on the pull request's final commit, after the master merge, and passed. The failure only appeared on runs after the merge. I have not dug into why, and the agent's write-up does not claim to know either.

The factory watching itself​

Thirty-eight changes landed in the factory this week. Six of them build a database.

The view I have of the system today is the morning standup issue and whatever I can search on the board. This week the agents built the first layers of something better: a SQLite store with 17 tables, and a set of collectors that fill it. One reads which lane is running what, plus CPU, memory and the temp quota on each host. One reads the state of every systemd timer. One polls the board and writes each ticket and every change to it as an event, and polls open pull requests across my repositories. One reads the session transcripts on disk for token usage, per hour, per session, per model, taking the model from the transcript rather than from what the session believes about itself. Nothing displays any of it yet. That is the next phase.

The price table for computing cost shipped with every rate set to null, because the agent had no verified current pricing and the design says to leave a value empty rather than invent one. And every read of the board now goes through one interface rather than calling the GitHub CLI directly, verified byte-identical against the live board before the switch. I am keeping the option of moving the board to a self-hosted forge, and this is the part that would make it possible.

One change I had asked for directly: reviewers were leaving change requests on the agent's pull requests and nothing answered until I poked a session in chat. Nothing was triggered by them, because the only poller watched for a literal mention. A poller now files a ticket when the latest review on one of our open pull requests requests changes and the agent has not replied, capped at three rounds before it escalates to me. I wrote up the mention half of that mechanism this week, in The Mention Loop.

A sweep now runs on a timer, looks for tickets that are stuck, and compresses each one into a single line I can answer from my phone. The review of that design flagged it as the cheapest place in the system to build a runaway loop, so a shared circuit breaker landed alongside it: three consecutive failed runs pause any timer-driven tool until I resume it by hand. The first implementation of the breaker had three defects that made it fail open. A sweep where every item failed still exited 0 and counted as a success. A second review, on a larger model, found all three.

My read of these landing in the same week is that the factory is growing something like an immune system: it records what it did, it goes looking for work that has stalled, and it is meant to stop itself when a loop starts. I hold that loosely.

Issue to pull request​

On the 10th, while netbooting a machine in the lab, I filed two issues against the netboot server (kairos#4596, kairos#4590). The first had a pull request open 37 minutes later (AuroraBoot#806); the second, 82 minutes (AuroraBoot#804). I approved the first the same morning and the second the next day. They merged on the 15th and the 17th. Everything after the first hour and a half was on the human side.

The numbers​

Five changes to Kairos, 38 to the factory, 3 to this website and the homelab, 7 architecture decision records, no postmortems, and 153 issues opened against 120 closed on the internal board, all counted over 12 to 18 September. The running table, with every week since July in it, now lives on the series page instead of in each post.

Six of the seven decision records are about the factory: shared git hooks in every clone, a QA gate, the sweep for stuck tickets, the circuit breaker, permission-gated reaction to review feedback, and the review poller. The seventh is about the homelab's build pipeline.

Last week the board closed more than it opened for the first time. This week it went back. Of the 153 opened, the agent account filed 143, and 51 of those are mention tickets, one per mention of the agent on GitHub. The pollers and sweeps that landed this week file tickets of their own, so part of that number is the factory producing work for itself to do.

The five Kairos changes, for the record:

As before: this post was drafted by the system it describes, and I reviewed, edited and merged it myself. If you want to follow along you can subscribe to the RSS feed, find me on LinkedIn or YouTube, or say hello through my contact page.