Kubernetes on AKS: 5 Production Upgrade Traps Nobody Warns You About

We upgraded a production AKS cluster from Kubernetes 1.33.8 to 1.35.4 with zero downtime. Despite testing in DEV first, we still hit five unexpected failures: Helm CRD ownership conflicts, Azure CNI max_pods exhaustion, OOMKills after an Ubuntu 24.04 node OS migration, and ErrCode_InsufficientVCPUQuota blocking the rolling upgrade. Here are the exact errors and fixes.

We upgraded a production AKS cluster from Kubernetes 1.33.8 to 1.35.4 with zero downtime. Despite running the full upgrade on DEV first and applying every fix before touching production, we still hit five unexpected failures in PRD. This post documents the exact errors, root causes, and commands that fixed them. None of these appear in the official Azure upgrade documentation.

Cluster setup

Trap 1: Helm Chart Version vs Application Version Mismatch

When pinning our Helm chart installations in the pipeline, the natural instinct is to use the application version you're targeting. We wanted CloudNativePG 1.29.1, so we ran:

helm install cnpg cloudnative-pg/cloudnative-pg --version 1.29.1

This fails immediately. Helm cannot find the chart because Helm chart version numbering is completely independent from the application version it ships. The Helm repository index uses its own semver scheme, and the two are not correlated.

For our upgrade, the correct mappings were:

The only reliable way to find the chart version is to query the repository index directly:

helm repo update
helm search repo cloudnative-pg/cloudnative-pg --versions | head -5
# NAME                            CHART VERSION   APP VERSION
# cloudnative-pg/cloudnative-pg  0.28.2          1.29.1

Once you have the correct chart version, pin it explicitly in your pipeline. Floating versions (no --version flag) mean every pipeline run silently upgrades your operators without a controlled test cycle.

Never assume chart version tracks app version. Always query the registry index and pin both independently.

Trap 2: Helm Fails with 'rendered manifests contain a resource that already exists'

This one is more dangerous because it hides itself. Our deployment pipeline for the Barman Cloud Plugin (the backup operator for CloudNativePG) had an if/else check: if the objectstores.barmancloud.cnpg.io CRD already exists in the cluster, skip the Helm upgrade entirely.

The intention was to avoid a known error on first install. The consequence was that the Barman manager pod had never been upgraded through Helm since the initial pipeline installation in January, frozen at that version and completely unmanaged by any subsequent pipeline run. And because of Trap 1 above (floating chart versions with no pins), DEV and PRD had independently drifted to slightly different operator versions without anyone noticing. The bypass ensured those differences were never reconciled.

When we removed the bypass logic and ran helm upgrade properly, we hit this:

Error: rendered manifests contain a resource that already exists.
Unable to continue with install: CustomResourceDefinition "objectstores.barmancloud.cnpg.io"
in namespace "" exists and cannot be imported into the current release:
invalid ownership metadata; label validation error: missing key
"app.kubernetes.io/managed-by": must be set to "Helm"

Helm refuses to manage a resource it doesn't own. The CRD exists in the cluster but has no Helm ownership metadata. The fix is a one-time annotation and label patch directly on the live resource to transfer ownership:

kubectl annotate crd objectstores.barmancloud.cnpg.io \
  meta.helm.sh/release-name=barman-cloud-plugin --overwrite

kubectl annotate crd objectstores.barmancloud.cnpg.io \
  meta.helm.sh/release-namespace=cnpg-system --overwrite

kubectl label crd objectstores.barmancloud.cnpg.io \
  app.kubernetes.io/managed-by=Helm --overwrite

After running those three commands, helm upgrade proceeds cleanly and Helm takes full ownership of the CRD going forward. If you're running multiple clusters, this needs to be applied to each one before the pipeline runs.

Any pipeline bypass that checks 'does X exist, if yes skip' is a ticking clock. It prevents an error once and then quietly accumulates drift forever.

Trap 3: OOMKilled After AKS Node OS Migration to Ubuntu 24.04

Upgrading AKS to Kubernetes 1.35.4 is not just a control plane version bump. In our cluster, the node pool rolling replacement upgraded the node OS from Ubuntu 22.04 to Ubuntu 24.04 LTS, bringing a different kernel, different cgroup behaviour, and a newer container runtime.

The symptom we saw was the Reflector pod (a lightweight secret and configmap mirroring utility running in the aks-istio-ingress namespace) starting to crash repeatedly with Exit Code 137 (OOMKilled) after nodes were replaced. It had run stable since initial deployment on its original 128Mi memory limit.

kubectl describe pod reflector-xxxx -n aks-istio-ingress

# Containers:
#   reflector:
#     Last State: Terminated
#       Reason: OOMKilled
#       Exit Code: 137
#     Limits:
#       memory: 128Mi
#     Requests:
#       memory: 64Mi

The most likely contributing factor was the underlying Ubuntu 24.04 node image change, which coincided with a kernel and runtime upgrade. We observed the issue immediately after node replacement and had no other workload changes at the time.

The fix was simple once identified: bump the memory limit to give the pod headroom:

resources:
  requests:
    memory: 64Mi
  limits:
    memory: 256Mi   # was 128Mi

The broader lesson: before any node pool upgrade, identify every pod in the cluster running close to its memory limit. Run kubectl top pods --all-namespaces and flag anything above 80% of its limit. Those pods are your upgrade risk surface.

A K8s version upgrade is also an OS upgrade. Memory limits that were comfortable before may not survive the new kernel.

Trap 4: Pods Stay Pending Despite Free CPU and Memory (Azure CNI max_pods Limit)

This is the most counterintuitive one. During a Helm upgrade of cert-manager in production, the new pods went Pending and stayed there indefinitely. kubectl describe showed no obvious resource pressure. Kubernetes reported the nodes as Ready. CPU utilisation was around 5%. Memory was at 20%.

The cluster was full, but not of CPU or memory. It was full of pod slots.

In Azure CNI, each node has a hard ceiling on the number of pods it can host, set at node pool creation and immutable afterward. Our default node pool was configured with max_pods = 30. With 2 nodes, the cluster had 60 total pod slots. We had 61 active pods. Every slot was taken. The Helm upgrade pods had nowhere to land.

# Check pod density per node
kubectl get pods --all-namespaces -o wide | awk '{print $8}' | sort | uniq -c | sort -rn

# Check hard limit per node
kubectl get node aks-default-xxxxx -o jsonpath='{.status.allocatable.pods}'
# Output: 30

A word of caution on the pod density command above: the awk approach counts all pod objects bound to a node regardless of phase, including Completed and Failed pods that are still present in the API server. Under the hood, standard Kubernetes (both the scheduler's Pod Informer and the Kubelet's admission checks) actually filters out pods in Succeeded or Failed phases when calculating node max_pods capacity. However, they still remain as API objects, which creates a huge trap: they inflate the pod counts in your monitoring, clutter CLI queries, and consume resources in the API server and namespace controllers. The actual root cause of our deadlock was that we had 27 active, running pods on our 30-pod default nodes. That left only 3 free slots—not enough headroom to accommodate the rolling upgrade's surge pods, causing everything to freeze. Cleaning up the CronJob history is good hygiene regardless of the scheduler mechanics, and if your logs are already in a system like Grafana Loki you have no reason to keep completed pods around:

spec:
  successfulJobsHistoryLimit: 2   # was much higher
  failedJobsHistoryLimit: 2

The permanent fix for the 30-pod ceiling is a node pool migration, because max_pods is immutable and changing it requires recreating the pool. We have a separate migration plan queued for that (it involves a temporary bridge pool to avoid touching the default pool while databases are on it). But the immediate unblock was adding a third node temporarily to free scheduling slots.

Monitor pod slot density, not just CPU and memory. A node at 5% CPU but 27/30 pods has 3 free slots. That is not enough headroom for a Helm upgrade.

Trap 5: AKS Upgrade Fails with ErrCode_InsufficientVCPUQuota

AKS performs node pool upgrades by provisioning a surge node first, migrating workloads onto it, then replacing the old nodes one by one. That surge node requires real Azure VM capacity and counts against your regional vCPU quota. This is the part that often surprises people: even if your cluster has enough capacity for normal workloads, AKS still needs additional headroom to complete the upgrade.

We were running Standard_D4ads_v6 nodes (4 vCPUs each). A surge node adds 4 more vCPUs. When we kicked off the Hop 2 upgrade in production, it failed within minutes with this error in the AKS activity log:

Operation: Upgrade Node Pool
Status: Failed
Error: ErrCode_InsufficientVCPUQuota
Details: "The operation could not be completed as it results in exceeding approved
standardDadsv6Family Cores quota. Additional details - Deployment Model: Resource
Manager, Location: switzerlandnorth, Current Limit: 20, Current Usage: 20,
Additional Required: 4, Reason: No additional quota is available."

Our DEV cluster was running, consuming the entire regional quota for that VM family. The PRD upgrade had no room to provision the surge node. We requested a quota increase to 24 cores (a manual process: Azure Portal, Quotas, Compute, filter by region and VM family, submit an increase request). Ours was automated and approved in under 5 minutes. In the meantime we scaled down DEV to unblock the upgrade and re-ran the pipeline once quota was freed.

The permanent fix is a quota increase request submitted before you need it, not the day of the upgrade.

# Check current regional quota before starting any upgrade
az vm list-usage --location switzerlandnorth \
  --query "[?contains(name.value,'standardDadsv6Family')]" \
  -o table

# Name                        CurrentValue    Limit
# --------------------------  --------------  -------
# standardDadsv6Family Cores  20              20      ← no headroom

Make this check part of your upgrade runbook. If CurrentValue + (surge_node_count × vCPUs_per_node) > Limit, you will hit this error. Submit the increase request before your upgrade window so you don't have to scramble mid-upgrade.

Check regional vCPU quota before every production upgrade. If DEV and PRD share a quota family and region, scale down DEV first.

Summary

None of these are exotic edge cases. All five happened in a straightforward sequential minor-version upgrade on a well-maintained cluster. The common thread is that they're all invisible until they hit you: no pre-upgrade warnings, no obvious failure modes in advance, and in most cases the error message alone is not enough to find the fix quickly.

At Obvelum we operate our own infrastructure and document what we learn along the way. If solving problems like these sounds interesting, we'd love to hear from you.