Your cluster is running on muscle memory

If you had to power cycle your cluster today, would your apps restart? Are you sure?

If you had to power cycle your cluster today, would your applications restart? Are you sure? Really sure?

The answer may be NO, and it’s likely you have no idea this is the case (until it happens). A running container has no live relationship with the registry it came from, because the layers are already unpacked on the node and the kubelet never rechecks anything for a container it isn’t being asked to create. Delete the image upstream and nothing about that pod changes. It carries on serving traffic on layers it fetched once, months ago, that nobody has verified since.

An imagePullPolicy of IfNotPresent hides this further, since it resolves against the node’s local store and never troubles a registry at all. That holds until the pod lands on a node which has never seen the image, and that happens on every node image upgrade, every autoscaler event and every spot reclamation. Staying put helps less than you’d think, because kubelet garbage collection clears images once the filesystem gets tight and the ones you haven’t touched in a while go first.

Bitnami is the live example everyone is currently dealing with. A chunk of the free catalog went to a legacy archive, the images came off the AWS public gallery in June, and clusters that had inherited Redis, Postgres and a pile of exporters from Helm chart defaults started failing one node replacement at a time. Any registry that goes away does exactly the same damage. You get an identical outage from a retention policy that reaps anything untouched for 90 days, a cleanup script that decides untagged manifests are garbage, or an expired card on a cloud account nobody remembers owning.

The registry you run yourself is worse, because it holds the only copy. Everything currently running stays up, so the first hour feels survivable, and then somebody opens the recovery runbook and finds that step one is to pull an image from the registry that is down. Registries also habitually run on the very cluster they serve. Go and read your restore procedure, then check whether it depends on the thing you would be restoring.

The reference that catches you is never your own application image. It is the init container, the metrics exporter, the sidecar added during an incident three years ago, or the default value in a Helm chart nobody has opened since installing it.

The strongest operational recommendation I can possibly make is simple... trigger a regular redeploy of your running applications, schedule a regular cluster power-cycle, and confirm everything comes back... dont assume, EVER. Do this before its too late, and you get your own Bitnami issue.

For those of you now facing the issue of apps failing to start due to the reliance on the now non-existent Bitnami images, good luck, and I hope you have copies of the images.