Kubernetes in Production: 10 Things Most Teams Miss

Most Kubernetes clusters that land on an SRE’s desk for the first time share a common pattern: the basic things work, but the safety nets are missing. Here are the gaps I see most often.

1. No resource requests or limits

If a pod has no requests, the scheduler has no basis to make a good placement decision. If it has no limits, a memory leak will bring down the node — and everything else running on it.

Every container needs both. Use a LimitRange to set sensible defaults so workloads without explicit values don’t slip through.

2. Liveness and readiness probes doing the same thing

A liveness probe that’s too aggressive restarts healthy pods under load. A readiness probe that’s too lenient routes traffic to pods that aren’t ready yet.

The distinction matters: liveness answers “should this pod be restarted?”, readiness answers “should this pod receive traffic?” They should check different things and use different thresholds.

3. No PodDisruptionBudget

Without a PDB, a kubectl drain for node maintenance can take down all replicas of a deployment simultaneously. A two-minute maintenance window becomes an outage.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: backend

One manifest. No reason to skip it.

4. Secrets stored in ConfigMaps

ConfigMaps are not encrypted at rest by default. Database passwords, API keys, and tokens belong in Secrets — and ideally in an external secrets manager (Vault, Azure Key Vault, AWS Secrets Manager) synced via External Secrets Operator.

I’ve seen production clusters where database credentials were base64-encoded in a ConfigMap committed to source control. The encoding is not encryption.

5. No Network Policy

By default, every pod in a Kubernetes cluster can reach every other pod. There is no automatic network isolation between namespaces. A compromised pod can probe the entire cluster.

Define a default-deny policy per namespace and whitelist only the connections that actually need to exist.

6. cluster-admin bound to service accounts

Service accounts with cluster-admin privileges exist in many clusters for “convenience.” This is the Kubernetes equivalent of running everything as root.

Audit your ClusterRoleBindings:

kubectl get clusterrolebindings \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.roleRef.name}{"\n"}{end}' \
  | grep cluster-admin

You’ll probably find things that shouldn’t be there.

7. imagePullPolicy: Always in production

This means every pod restart triggers a registry pull. In a partial network outage, pod restarts fail even when the image is already cached on the node.

Use IfNotPresent with explicitly tagged images — never latest. Pin to a digest for full reproducibility.

8. No Horizontal Pod Autoscaler

Static replica counts mean you’re either over-provisioned at normal load or under-provisioned during spikes. An HPA with a reasonable CPU/memory target is straightforward to configure and prevents a whole category of incidents.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: backend-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: backend
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

9. No multi-zone spread

Deploying all replicas to the same availability zone negates the redundancy that replication is supposed to provide. Use topologySpreadConstraints or pod anti-affinity to distribute workloads across zones and nodes.

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: backend

10. No runbook for common failure modes

When a pod is crash-looping at 2 AM, the on-call engineer should have a documented path for: how to check logs, how to rollback, how to manually scale if HPA isn’t responding, who owns this service. If that doesn’t exist, you’re paying for incidents you could have avoided.

---

This is a starting point, not a complete list. Production readiness is a continuous process.

If you want an independent assessment of your cluster’s production posture, book a free 30-minute technical review.