Three months ago, a Talos upgrade on my homelab ate my weekend. Eleven CloudNativePG clusters went into partial-replica state, five were corrupted beyond graceful recovery, and I spent the next two days restoring from Barman backups while my self-hosted services — Authentik, Forgejo, Paperless, the whole stack — sat half-dead. Today I ran the same kind of upgrade. No pager. No restores. No half-day spent sweating a half-promoted Postgres primary back into shape. The whole thing took about twenty-five minutes and I spent most of it watching a terminal in one window and writing this article in another.
I want to tell you exactly what changed, because I think the dominant lesson is counterintuitive, and the honest answer for what made the difference is not the one I would have picked before I sat down to write this.
I spent the morning running structured probes against both clusters before the upgrade, and I sat beside the machine as it rolled through each node. Five factors changed between the failed upgrade and the smooth one. One of them dominates by a wide margin, and it is not a patched bug. It is capacity headroom. Let me walk you through what I found.
The first factor — and by far the largest one — is fleet reduction. Before this cycle, I was running eleven live CloudNativePG Postgres clusters across three control-plane nodes on a cluster I call Mimir. Eleven clusters means roughly twenty-four Postgres pods, eleven Longhorn-backed persistent volumes, eleven load-balanced services, eleven external-DNS records, eleven Barman backup targets, and eleven sets of scheduled backups all competing for the same etcd, the same API server, and the same three nodes’ RAM. During a drain, Kubernetes evicts pods on a cordoned node. CloudNativePG responds by triggering a planned switchover — promote a replica, fence the old primary, update the Service endpoints. Each of those operations hits the API server. Each switchover takes somewhere between forty and eighty seconds under normal load. The default kubelet shutdown grace period, meanwhile, is twenty seconds. If the switchover does not finish inside that window, kubelet force-kills the old primary pod mid-promotion. You do not get a clean failover — you get a Postgres cluster that does not know who the primary is, and the only way out is a restore from backup.
At eleven clusters, a single node drain was triggering eleven switchovers in parallel against an operator whose reconcile queue was already falling behind. At five clusters, the same operator sailed through. The reconcile queue never backed up, the switchovers completed inside twenty seconds each, and the force-kills that did happen were cosmetic — they terminated pods whose work was already finished.
How did we go from eleven to five. Over the last month I migrated six CloudNativePG clusters to Supabase as per-app schemas — miniflux, miniflux-ingest, linkding, wallabag, input, and n8n. Another, home-iot, I archived. The remaining five — Authentik, Forgejo, Paperless, Quantified Self, and the still-live home-iot — are the ones that genuinely benefit from local, three-replica, blast-resistant Postgres. These are the dependency-loop clusters, where an outage would lock me out of the system that fixes the outage. For everything else, the operational tax of running dedicated three-node Postgres was always too high for the value.
The second factor is a plugin upgrade I landed two weeks ago. The plugin-barman-cloud plugin went from version zero point five to zero point six, which fixed a parallel-WAL-restore race and a handful of RBAC reconciliation bugs. Under single-cluster load these bugs rarely bit. Under eleven-cluster parallel drain load, they were almost certainly compounding the problem. I saw it briefly today — during the Mimir upgrade the single plugin pod went not-ready for about forty seconds while it was rescheduled onto a freshly-rebooted node, and all five CloudNativePG clusters reported an error status during that window. At five clusters, the forty-second stall was recoverable noise. At eleven, that stall would have consumed my entire grace budget, and the reconcile queue would have blown up behind it.
The third factor is external-secrets. I upgraded from version one to version two point three last week. The v1 series had a webhook race that, under cluster-wide secret reconciliation load, could stall the entire secret pipeline. Because every image pull secret in my cluster is sourced through external-secrets, a stall there would cascade into ImagePullBackOff errors on every pod trying to restart during drain. The v2 architecture separates the webhook path cleanly and eliminates that race.
The fourth factor is the kube-prometheus-stack upgrade to version eighty-three. Before this cycle, my Mimir Prometheus pod had a two-hundred-and-fifty-four-restart flap — a pre-existing churn pattern unrelated to any workload, but consuming API server and etcd budget continuously. Monitoring should be a passive observer, not a noisy neighbor. The chart upgrade stabilized the restart loop, which freed API bandwidth for the operators that actually needed it during drain.
The fifth factor is a piece of infrastructure I did nothing to — CloudNativePG’s planned-switchover logic itself. When a node is cordoned, the operator detects it, promotes a healthy replica to primary, updates the read-write Service endpoint, and only then lets the old primary terminate. The critical insight is that the fence-and-promote happens in milliseconds. The old primary pod getting force-killed at the twenty-second grace mark is, in every scenario I watched today, a cosmetic event — the new primary is already serving writes. The old pod’s death is just kubelet closing the door behind a tenant who has already moved out. This only works when the operator is responsive enough to fire the switchover in the first few seconds of the drain. Under overload, it isn’t.
So here is the picture if you add it up. Fleet reduction halved the reconcile load. Plugin, secrets, and monitoring upgrades removed three separate latency tails from that same reconcile path. And CloudNativePG’s own design handled the rest, as long as the operator had enough room to breathe. Today, during the Mimir upgrade, the cluster ran at roughly half its memory capacity and twenty-six percent CPU. That number is the whole story in one glance. It is the ratio of headroom to work. It is what went wrong last time, and it is what went right today.
There is a version of this story where I would write the rest of the article telling you which five tools to install to make your Kubernetes upgrades smooth. I am not going to write that version, because I think it would be wrong, and I want to stage a pivot right here toward what actually made the difference.
The thing that nags at me about today’s smooth upgrade is this — the shutdownGracePeriod is still twenty seconds. I did not patch kubelet. I did not change a single Talos setting. The CloudNativePG switchover still takes forty to eighty seconds in the worst case. The arithmetic that broke the cluster three months ago is still the same arithmetic. So why did the math work this time. The only honest answer is that I was operating with enough capacity margin that the stochastic distribution of switchover times mostly stayed inside the twenty-second window. Not every switchover did — I saw some old-primary pods get force-killed today as expected — but the ones that went over were the ones where the new primary was already promoted and the force-kill was benign.
That is a different statement than “the system is robust now.” The system is not robust. The system is operating inside its budget. If I tripled the cluster count tomorrow, every one of the five fixes I just listed would still be in place, and the upgrade would still fail. Because the dominant variable is not bug-fix-count — it is load-relative-to-capacity.
This is worth sitting with for a moment. As homelab operators and platform engineers we love tracking version bumps and changelogs. We maintain Renovate dashboards. We read release notes at breakfast. And the truth is that most of the reliability improvements in our systems come not from the patches we landed but from the load we never added, or the load we removed. Subtraction is hard to celebrate and therefore hard to prioritize. But subtraction is what actually bought me the margin today.
So, here is the honest playbook, ranked by leverage.
First, count your databases. Not your applications, not your namespaces, your databases. Open a terminal, run kubectl get cluster across all namespaces, and look at the number. Five is a number you can reason about during an incident. Fifteen is a number you can’t. If you are above eight, you are past the point where a routine upgrade is routine.
Second, ask which databases actually need to be three-replica, locally-hosted, disaster-resilient installations. The honest answer for most people is two or three. Your identity provider. Your source-of-truth for code and artifacts. Anything that, if it is down, prevents you from fixing the thing that is down. Everything else is a candidate for a managed service. Supabase as a per-app schema is extremely cheap for small data volumes. The operational burden you save is far greater than the monthly cost of a managed Postgres row.
Third, measure your drain-peak load before you upgrade. Look at memory utilization on your nodes during a normal workload, and then do the mental math of what happens when one of your N nodes becomes temporarily unavailable. If the survivors cannot absorb the load without pushing past ninety percent, your upgrade will fail at drain-peak. Not might fail. Will fail. Reduce load, or add capacity, before you upgrade.
Fourth, take an on-demand backup before every destructive operation. Not the scheduled backup that ran four hours ago. A fresh one, named for the operation, with the timestamp in the resource name. I ran five Barman backups this morning before I let Omni touch the first node. All five completed in under thirty-four seconds. They are my undo button. I will never skip this step again.
Fifth, watch the single-replica infrastructure pods during the upgrade. The barman-cloud plugin runs as one pod on my cluster. The CloudNativePG operator runs as two replicas. When either of those pods lands on a cordoned node, you have a blind window until it reschedules. During that window, every CloudNativePG cluster in your system is effectively paused. Know this. Plan for it. Consider whether you want to scale some of these infrastructure pods to multiple replicas before a major upgrade, even if the cost per-day in normal operation is a slight waste.
Sixth, upgrade node-by-node if your provisioning tool allows it, and let each node fully reconcile before the next. Omni drove my upgrade today as a rolling operation across all three control-plane nodes, and it worked, but I would have preferred explicit node-by-node gating so I could promote CloudNativePG primaries off the draining node by hand before the cordon hit. The fact that it worked automatically does not mean automation is the right policy in every case. Automation is the right policy at five clusters. It is probably the wrong policy at fifteen.
And here is what to avoid. Do not assume a patched release notes page makes your system safe at the same scale. The patches help at the margin. They do not change the arithmetic of the reconcile queue at load. Do not skip the capacity audit because last upgrade worked. Last upgrade worked because you had margin. The margin can shrink. Do not delete old Barman backups the day after a successful upgrade. Soak for a week. Data corruption is sometimes latent, and you may not notice for days. Do not confuse speed of recovery with robustness of design. A system that can be restored from backup in an hour is not the same as a system that does not need to be restored. And do not underestimate how much your ancillary controllers — monitoring, secrets, ingress — compete with your primary workloads for the same etcd budget. A flappy monitoring stack is not a cosmetic problem on a busy cluster. It is a reliability tax.
Here is the bottom line. Kubernetes upgrades are a capacity test dressed up as a software test. The version numbers, the changelogs, the patches — those are the paperwork. The real exam is whether your cluster is operating with enough headroom that a one-third capacity reduction during a node drain still leaves the operators, the reconcile queues, and the stateful workloads enough room to complete their work inside the grace windows the platform gives them. The question is not “is my software up to date.” The question is “am I running close to the edge.” If you are, no amount of patching will save you. If you aren’t, the upgrade will probably be boring — and that is the exact outcome you want.
Count your databases this week. Move the non-essential ones to a managed service next week. Measure your drain-peak memory the week after. In ninety days you will have a homelab that upgrades the way mine did today — quietly, in the background, while you are writing about it. That is the move.