• kubectl rollout restart deployment/my-app and eat the disruption.
• Install Reloader so it restarts your pods for you when a Secret changes.
• Mount the Secret as a volume and cross your fingers that your app re-reads the file (spoiler: most apps read it once at boot).

All three share the same flaw: the way you "pick up" a config change is to throw away the process and start a new one. For a rotating database password, that means dropping every in-flight request, cold connection pools, and a latency spike — all to change one string.

I've stared at that status more times than I'd like to admit, usually at the worst possible hour, usually while someone in a chat window is asking why traffic isn't going through. Eleven minutes is about how long it took once, on a cluster that had close to 40% CPU sitting free the entire time. That gap between "there's capacity somewhere" and "this pod actually got scheduled" is what this article is really about.

The incident occurred in the production backend infrastructure of a mobile application. External HTTP traffic was handled by Traefik, deployed on Docker Swarm manager nodes. Traefik ran in swarmMode, used providers.docker, was configured to use the mgp_default network, and had lbswarm disabled so that routing used the required overlay network rather than the Swarm VIP.

One of the critical services was mgp_router-app. It was an entry-point service with two replicas deployed to preserve availability if one of them failed.

This time, I want to take that same workflow off the laptop.

Every engineering organization that I have worked with eventually faces the same issue, which is that each team ships services differently. One team used Helm, another wrote raw manifests, and a third would have built a custom Bash script. As these different approaches accumulate, the supporting deployment steps often end up scattered across multiple Wiki pages that quickly go stale. New engineers then spend their first two weeks copying configuration values from an old repository and hoping they still work.

Kubernetes environments can drift, accumulate one-off fixes, and diverge across teams until a routine deploy breaks or a cost spike forces a review. This checklist gives platform, SRE, and engineering teams a way to keep clusters, deployments, and automation manageable as Kubernetes operations scale and teams grow. It covers standards, observability, releases, access, drift, and cost. Review it before promoting a service to production and revisit it as your environments shift.

There are two things that make a sidecar. First is that they share a network namespace, so they can reach each other over localhost rather than a network address. Second, they share a lifecycle. This means that they are created together, scaled together, and by default torn down together. Neither container has an existence independent of the other.

That is the gap I wanted to measure.

I spent years treating Spring Boot startup time as a fact of life, the way you treat weather. Then I found out the JVM has had a fix for a big chunk of it since Java 12; it works beautifully inside Docker, and almost nobody bakes it into their images. It is called Class Data Sharing, CDS for short, and this article shows you how to make your Docker build do the work

The repository's demo reports 30x+ actor-to-worker oversubscription for that sample workload, with sub-second activation. It is a demonstration, not a production capacity guarantee. 

What first got you interested in digging into complex infrastructure and systems problems?

A Familiar Routine

If you build data pipelines for a living, you've lived this story. Your PySpark job runs perfectly in a cloud notebook. You productionize it, push it through CI, deploy it to the cluster, and it fails. A dependency mismatch. A different Spark minor version. A Delta Lake protocol feature your local wheel doesn't know about. A timezone default nobody set.

Three weeks into a new ML platform job, I asked a coworker why the 'getting started' doc had a section called 'If conda breaks, try the alternative.' He laughed in a way that told me it wasn't a joke. Every new hire spent their first two days fighting Python versions, CUDA driver mismatches, and a vector database that someone had installed locally in 2022 and nobody dared touch. We had four individuals on the team, each with distinct working setups, and "it works on my machine" was no longer a mere punchline; it had become a regular agenda item during our daily standup meetings. 

Pushing that to our registry took eleven minutes on a good connection. Pulling it onto a fresh node during an autoscale event took even longer, and by the time the pod was ready, the traffic spike it was supposed to handle had already passed. That's the moment I stopped treating LLM containers like regular application containers, because they are not the same animal at all.

On June 22, 2025, this exact scenario occurred in the production cluster of the backend infrastructure for a socially significant public transportation mobile application. The system serves about 2 million users, several tens of thousands of daily active users, and a total load of around 1000–1600 RPS, so even partial degradation at a single entry point affected a high-load segment of traffic and could have had a noticeable impact on SLA metrics if it had not been localized in time.

The meeting that changed the platform team's direction was not a technical one. It was a conversation with a product engineer who had been at the company for eight months and had never successfully deployed to production without help from someone on the platform team. Not because she lacked skill. 

She was smart, experienced, and had successfully launched production systems at two previous jobs, but getting a working service into production meant dealing with fifteen different configuration files across four repositories, figuring out how Helm values files and Kustomize overlays worked together, and knowing which of the three CI pipeline templates to use based on whether the service needed a sidecar, a job scheduler, or neither.

Sponsored By: Nutanix The following is sponsored content. It may not reflect the views of our editorial staff.

The Kubernetes scaling problem nobody talks about

Enterprise platform teams encounter the same pattern repeatedly: a Kubernetes platform works well enough that nobody wants to change it.

This happens gradually as teams make reasonable technology choices: selecting different ingress controllers, secrets management tools, CD platforms, or observability software. Individually, none of these decisions is a problem. Months later, however, they’ve created a Kubernetes environment that only a handful of people understand. As soon as that one person gets sick or leaves the company, maintaining or improving the platform becomes much more difficult.

It covers:

Building agentic AI systems fundamentally changes how we handle application security. We are no longer just securing our own code. We are securing our infrastructure against code written dynamically by an LLM and executed on the fly. When building a multi-tenant AI platform, allowing an agent to run arbitrary scripts is a massive escape vector waiting to happen.

Google recently made the GKE Agent Sandbox generally available on their custom Arm-based Axion N4A instances. This gives us a highly efficient, hardware-optimized path to run untrusted code safely. Under the hood, this relies on gVisor to intercept application kernel calls and run them in a heavily restricted user-space kernel.