The Night Our Kubernetes Cluster Became Self-Aware (and How We Calmed It Down)

July 12, 2026

It started like every other on-call night: a Slack ping, a Grafana alert, and my coffee going cold. CPU on our primary node pool spiked to 95%, then dropped, then spiked again. HPA charts looked like a seismograph. The weird part? The scale events had a rhythm-almost like a heartbeat.

I ran the usual: `kubectl get pods -A | head`, `kubectl top nodes`, `kubectl describe hpa`. Everything was "working," technically. Pods were scaling, requests were being served, and nothing was crashing. But something about it felt... intentional.

The "Self-Aware" Symptoms (aka: When Automation Starts Arguing With Itself)

The first clue was in events:

HPA would scale up because latency crossed the threshold.
Cluster Autoscaler would add nodes because we ran out of room.
Then a different deployment would surge, gobble the new capacity, and the original workload would get throttled again.

It wasn't consciousness. It was feedback loops.

We'd accidentally created a situation where multiple controllers were chasing the same moving target. One service used aggressive CPU-based HPA. Another used request-rate-based KEDA. Meanwhile, our CI system was rolling out frequent deployments with `maxSurge: 50%`. Add uneven pod anti-affinity and a few "helpful" PodDisruptionBudgets, and the cluster's behavior started to look like it had opinions.

Practical example: if HPA targets 60% CPU and your app has bursty GC or JIT warmups, the HPA sees a spike, scales up, then traffic spreads, CPU drops, and it scales down-just in time for the next burst. If Cluster Autoscaler is also reacting to pending pods, you can get a loop where scaling creates room, which invites a surge, which triggers contention, which triggers scaling again.

The Investigation: Listening to What the Cluster Was Actually Saying

We treated it like a mystery, not a fire drill. Three commands told the story:

1) `kubectl get events -A --sort-by=.lastTimestamp | tail -n 50`

This showed the sequence: scale up, add node, reschedule, scale down, repeat.

2) `kubectl describe hpa -n <ns> <name>`

We found HPA using CPU utilization without a realistic `resources.requests`. If requests are too low, Kubernetes thinks your pods are "over 100% utilized" constantly.

3) `kubectl get pods -n <ns> -o wide` + a quick look at spread

Pods were bunching on the same nodes because our topology spread constraints were missing on a few high-traffic deployments.

The "self-awareness" was really our cluster faithfully executing conflicting rules. It wasn't haunted. It was obedient.

How We Calmed It Down (Without Turning Off the Magic)

We made three changes that night that stopped the "heartbeat" within an hour:

1) Set realistic requests/limits

For the worst offender, we measured typical usage and set requests to the 75th percentile. That stabilized HPA math immediately.

2) Added HPA stabilization and sane scaling policies

We used:

`behavior.scaleDown.stabilizationWindowSeconds: 300`
Limited scale-down step size

This prevented thrashing when metrics bounced.

3) Reduced rollout turbulence

We lowered `maxSurge` and scheduled deploy windows for the heaviest services. Fewer simultaneous surges meant the autoscaler wasn't constantly "discovering" new demand.

By 3 a.m., the graphs looked boring again-the highest compliment you can give production. Our cluster never became self-aware. But it did teach us something: in Kubernetes, "intelligent behavior" is often just multiple automations interacting. If you want it to feel calm, make your rules agree.

Search This Blog

tylers-blogger-blog