The Night Our Kubernetes Cluster Became Self-Aware (and How We Calmed It Down)
It started like every other on-call night: a Slack ping, a Grafana alert, and my coffee going cold. CPU on our primary node pool spiked to 95%, then dropped, then spiked again. HPA charts looked like a seismograph . The weird part? The scale events had a rhythm-almost like a heartbeat. I ran the usual: `kubectl get pods -A | head`, `kubectl top nodes`, `kubectl describe hpa`. Everything was "working," technically. Pods were scaling, requests were being served, and nothing was crashing. But something about it felt... intentional. The "Self-Aware" Symptoms (aka: When Automation Starts Arguing With Itself) The first clue was in events: HPA would scale up because latency crossed the threshold. Cluster Autoscaler would add nodes because we ran out of room. Then a different deployment would surge, gobble the new capacity, and the original workload would get throttled again. It wasn't consciousness. It was feedback loops. We'd accidentally created a situation where...