Devlog
Blog
Notes from the lab — problems encountered, solutions found, lessons learned the hard way.
Web · Integration
Four small lies the APIs told me
Every panel on the portal that took longer than expected took longer for the same reason: the data was not shaped like the documentation implied.
Web · SEO
The site that existed but could not be found
Every page loaded correctly in a browser and none of them were in the index. The crawler was being told something different.
Linux · Operations
Rebuilding a machine, and the list I should have had first
The reinstall took an hour. Remembering everything the machine had quietly been doing took considerably longer.
AI · Local models
Six models on one card, and why not just the biggest one
Running local models means choosing constantly. The choice is not which model is best, it is which one fits the task and the memory.
Operations · Planning
The project I decided not to start yet
Camera based print failure detection is the obvious next thing to build. It is parked, on purpose, behind work that has to come first.
3D Printing · Infrastructure
Printing the rack the servers live in
A ten inch rack made of printed parts sounds flimsy until you look at what it is actually holding and where the loads go.
Kubernetes · Architecture
From one node to eight, and why three of them do nothing useful
The cluster grew a node at a time. The most important decision was giving three machines a job that produces no output.
Monitoring · Design
A dashboard nobody looks at is not monitoring
It is easy to build panels for everything that can be measured. The useful question is which of them would ever change what you do.
Kubernetes · Storage
Two storage classes, and knowing which workload gets which
Shared network storage and replicated block storage solve different problems. Putting a workload on the wrong one works right up until it does not.
Hardware · Lifecycle
The firewall that became a television box
Replacing a machine does not mean it is finished. This one was demoted, moved to another house, and is more useful there.
Networking · Design
Ten gigabit between two machines, one gigabit everywhere else
Upgrading the whole network to 10G would have been simpler to explain and considerably worse value.
Networking · Remote access
Remote access when you have no public address at all
Port forwarding assumes the internet can reach you. Behind carrier grade NAT it cannot, and the usual advice stops applying.
Kubernetes · Storage
Snapshots that quietly ate the disk they were protecting
Automatic snapshots are only half a policy. The other half is deleting them, and deleted files inside a volume do not free themselves.
3D Printing · Monitoring
Printers that tell you when they are wasting your filament
A print that fails in the first ten minutes and a print that fails at hour nine cost very different amounts. The difference is whether anything told you.
Media · Self-hosting
Cancelling the music subscription and what it actually cost
The self-hosted replacement plays everything I own and nothing I do not. That is both the feature and the drawback.
Kubernetes · DNS
Two resolvers, one address, and a controller that kept taking port 53
Making DNS survive a node reboot was the easy half. The hard half was a built-in controller quietly competing for the same port.
Web · Architecture
Removing the database made the portal more reliable
The family portal used to run on a backend with a database behind it. The rewrite has neither, and it broke less.
Storage · Monitoring
The disk told me months before it would have mattered
A counter that had been zero for years stopped being zero. Nothing was failing yet, which is exactly when to act.
Media · Hardware limits
Making the server work harder so the old client would not have to
The playback machine could not decode the format it was being handed. The fix was in a server setting, not on the client.
Operations · Scope
Two services deleted, and the lab got better
Both were working. Neither was being used. Keeping them running was costing more than the features were worth.
Networking · Architecture
The failover that was never going to fail over
A redundant firewall pair protects against one thing failing. It does nothing about the thing that actually fails.
Storage · Hardware
The migration that was about shelf depth
The old NAS was not slow. It just did not fit in the rack that the rest of the lab was being built around.
3D Printing · Infrastructure
Taking the print farm off the cloud
The printers now talk to the local network and nothing else. That removed a dependency and added some homework.
Linux · Packaging
The flatpak that installs fine and then refuses to work
Two separate traps on the same afternoon: an install that needs a desktop session to authorize, and a sandbox that cannot see the certificate it needs.
Automation · Reliability
Wake-on-LAN that survives an interface rename
The script kept working right up until a reinstall renamed a network interface, and then it failed quietly, which is worse.
Kubernetes · Config
A 256KB ceiling that kubectl apply never mentions
A ConfigMap update stopped working with no error that pointed anywhere useful. The limit was not on the object, it was on an annotation.
Kubernetes · Storage
The rolling update that can never finish
A restart left one pod pending forever. The deployment was doing exactly what it was told - the volume just could not be in two places at once.
3D Printing · Tooling
The slicer that stays on an old build on purpose
Every other package on this machine gets updated. The slicer is pinned to a specific build, and there is a note next to it explaining why.
AI · GPU
Vulkan instead of ROCm, and the desktop stayed alive
The supported path for GPU inference on this card kept taking the display down with it. The unsupported one just worked.
Networking · Security
From watching to blocking, months later
The IDS ran in alert-only mode for months before anything was allowed to drop a packet. That waiting period was the actual work.
Kubernetes · Incident
Two power outages, one etcd quorum loss
When all three control plane nodes cold-boot at once, etcd doesn't come back on its own.
Storage · Backup
The backup that was silently failing for a week
A daily cron job had been failing quietly since a storage migration — and nothing was watching it closely enough to notice.
Kubernetes · Networking
The DaemonSet that wouldn't stay dead
k3s's built-in service load balancer kept regenerating a broken DNS DaemonSet after every restart — until it got its own watchdog.
Networking · Monitoring
When a firmware upgrade breaks your monitoring
An OPNsense firmware upgrade silently broke firewall state metrics — and every future GUI update was set to break the fix again.
Storage · Planning
The RAIDZ expansion that avoided a bigger cage
Growing a ZFS pool used to mean building a second pool and migrating data across. A newer feature made that unnecessary.