Week 6 of Building a Software Factory: Hands on the Hardware
The netboot server for my lab moved from my Mac to a Linux box this week. The agents did the migration, with a few back-and-forths, and the one thing I did myself was point DNS at the new machine. Then I opened the dashboard and everything was already there: my data, my login, the same certificate. Realizing I had not had to interact with either machine directly was a little crazy.
Moving the netboot server
The reason for the move: I have been trying to netboot hegemon, an HP ProDesk 600 G4 Mini that is meant to become the control plane of my Kubernetes cluster. The server side is AuroraBoot, which ran on my Mac in a Docker Desktop container, and PXE boot from it never worked properly. During the move the agents found out why: Docker Desktop on macOS does not give a container real L2 broadcast visibility, so DHCP-proxy netboot could not have worked from that machine, independent of how AuroraBoot was configured. On the new host, midnight (a Beelink SER5, always on, 64 GB of RAM), the container runs directly on the host network and the problem went away outright.
The migration carried over the database, the artifacts, the admin password and the registration token, reused the TLS certificate, repointed DNS, and opened the host firewall for the dashboard and netboot ports. My existing browser session kept working.
There was one outage earlier in the week that I first blamed on the service. It was DNS: the LAN resolver answered fleet.home.arpa correctly, but Tailscale's MagicDNS sat first in the resolver list and had no route for home.arpa, so the browser got ERR_NAME_NOT_RESOLVED. A Split DNS entry fixed it.
The netboot work also surfaced bugs I am glad I did not have to find myself. One netboot failure came down to a script using bash [[ ]] syntax while running under POSIX sh (kairos#4599). Another: the netboot path legitimately passes an empty cloud-config argument, and AuroraBoot's validation rejected it (AuroraBoot#803), and the test for that argument was hanging CI (AuroraBoot#816). The agents found and fixed all three. That would have cost me a lot of debugging time, and I see little value in a person being the one to check for that class of error.
Kairos on a RISC-V board
Two weeks ago I published an open invitation asking anyone with real RISC-V hardware to test the riscv64 image, which had only ever been verified under QEMU. Spectro Cloud provided a StarFive VisionFive 2 (JH7110, 8 GB), so the first person to answer the invitation turned out to be me.
I flashed the raw image from the v0.1.3 release to a microSD card. The board boots the card on its own: U-Boot's EFI loader hands off, and GRUB 2.12 renders its menu and loads a kernel. That is the part nobody had confirmed before, GRUB on real silicon. It did not boot Kairos, though. The kernel handoff traps inside OpenSBI, the machine-mode firmware. The board's factory firmware is U-Boot 2021.10, built in February 2023, and Ubuntu's own VisionFive 2 documentation says that vendor U-Boot is not EBBR compatible and cannot boot Ubuntu without manual changes. The next step is flashing current firmware to SPI. There is a longer write-up of the whole session in draft; this is the short version.
What shipped
Supply chain and CI. Workflows that declared no permissions now run with least-privilege permission blocks (kairos#4503), and the FIPS example base images are pinned by digest (kairos#4505).
Fuzz testing. Config validation got a native Go fuzz test (kairos#4504), and FilterKeys got the same treatment (kairos#4540).
riscv64 releases. Two weeks ago I wrote that a single failing architecture was quietly blocking the whole release manifest from publishing, and that the fix was still open. It is merged now, and riscv64 releases are re-enabled (AuroraBoot#758).
Fleet dashboard. Cloning a Hadron device now pre-populates the kairos-init image (AuroraBoot#802), and the cloud config preview shows the real phonehome values instead of placeholders (AuroraBoot#805).
Dependencies. Nine pending Renovate branches were consolidated into one update, along with kairos-init v4.3.0 (AuroraBoot#784).
CNCF Incubation. Imec is added to ADOPTERS.md, and a claim in that file's header that only applied to production users is fixed (community#42). ADOPTERS.md is the last open artifact on the Incubation checklist.
Homelab. Five changes merged, mostly hegemon's control-plane node configuration and teaching the lab machines to trust the netboot server's self-signed certificate.
The numbers
| Activity | 29 Aug – 4 Sep | 5–11 Sep |
|---|---|---|
| Changes shipped to Kairos | 36 | 12 |
| Changes to the agent factory itself | 14 | 26 |
| Changes to this website and the homelab | 1 | 11 |
| Architecture decision records written | 1 | 2 |
| Postmortems filed | 0 | 3 |
| Internal board: issues opened / closed | 114 / 101 | 77 / 81 |
I counted with GitHub search over the 5 to 11 September window, and 11 September is not over as I write this, so this week's column is missing part of a day. The board closed more issues than it opened, 81 against 77. That is the first time in this series.
The three postmortems cover two incidents, and both are the same failure: references to my internal tickets ending up in public pull request descriptions, where they do not belong. The first recorded a leak and the cleanup. After it was written, one batch of work leaked seven more references, two of them into commit messages that were squash merged and cannot be cleaned up. Rules telling the agents to be careful did not hold. What did: a check script now runs over the drafted title, body and commit messages before any public pull request is created or edited.
Three postmortems for two incidents is its own small failure. Two lanes picked up the second incident independently and each wrote it up, four minutes apart, colliding on the same postmortem number. One of the two had to be renumbered afterwards.
Hands on the hardware
A lot of this week was manual work: putting an ethernet cable together, flashing microSD cards, watching a boot menu on a screen attached to a RISC-V board. While I did that, the agents kept opening the pull requests listed above. It is not a choice between one kind of work and the other. Both happen at the same time.
As before: this post was drafted by the system it describes, and I reviewed, edited and merged it myself. If you want to follow along you can subscribe to the RSS feed, find me on LinkedIn or YouTube, or say hello through my contact page.