I build the infrastructure that serves AI in production, and cut what it costs to run.

A race with the device plugin. The node was schedulable before the GPU was actually claimable, so Kubernetes placed work on a card that was not there yet.

Verification logic held the GPU while the workload tried to start. Two processes, one card, and the check always won because it ran first.

Firewall state outlived the machine it belonged to. The rules referenced an address the replacement no longer had, and nothing reported an error.

A dependency pointing at a host decommissioned weeks earlier. A dead host does not refuse a connection, it swallows it, so every request sat until the timeout expired and fell back. The suspiciously round number was the timeout, not the work.

The search path re-read all 25,631 product rows and 206 MB of vectors on every single message. Not a network fault. The query was simply too greedy to finish.

A cron job that exited non-zero into a void. A backup that never runs looks exactly like a quiet night, which is why the absence of alerts is not the same as the presence of health.