Why Terraform Lies About max_pods (And the Bridge Pool Pattern That Actually Works)

We needed to increase max_pods from 30 to 50 on a production AKS cluster. Terraform made it look like a one-line change. It is not. max_pods is immutable, zone-pinned Premium SSD v2 PVCs cannot reschedule freely, and naïve rotation would have taken both CloudNativePG primaries offline simultaneously. Here is the six-phase bridge pool pattern we used to do it safely.

We needed to increase max_pods from 30 to 50 on a production AKS default node pool. Terraform showed it as a one-line change. It is not. The following is a record of why it is not, what actually happened, and the six-phase procedure that let us complete the rotation without touching either PostgreSQL primary.

The incident that triggered this

After a routine cluster restart, both PostgreSQL primaries stayed Pending. No crashloop, no OOM - just unschedulable. The zone that held both primaries' PVCs had exactly one default node, and it was full: 30 pods out of 30 allocated slots, all taken by system daemons, Istio sidecars, Grafana agents, and application pods competing on cold boot. The fix was manual: delete two Grafana pods to free two slots. Both primaries scheduled within seconds.

This is a configuration smell, not an incident. A saturated node pool has no slack for restarts - even when the nodes have CPU and memory free in abundance. The permanent fix is raising max_pods.

Why max_pods is immutable

max_pods is set at VM provisioning time. Azure CNI pre-allocates (max_pods + 1) IP addresses per node from the subnet when the VM boots. This IP budget cannot be changed on a live VM - the subnet allocation is fixed at creation.

The AzureRM provider enforces this: temporary_name_for_rotation is required when changing max_pods. Attempting to apply without it will fail. This is documented explicitly in the azurerm_kubernetes_cluster resource reference:

The only correct approach is to replace all nodes in the pool using the temporary_name_for_rotation mechanism:

default_node_pool {
  name                        = "default"
  temporary_name_for_rotation = "tempdefault"   # triggers VMSS replacement
  max_pods                    = 50
  # ... rest unchanged
}

With this set, Terraform creates a new VMSS (tempdefault) with max_pods=50, drains all pods off the old VMSS, deletes the old VMSS, and renames tempdefault → default. After apply completes, remove temporary_name_for_rotation and apply once more to reconcile state.

temporary_name_for_rotation must be added at the same time as the max_pods change. Adding the attribute alone - with no other diff on the default pool - does not trigger rotation.

The zone-pinning complication

On production, CloudNativePG persistent volumes use Premium SSD v2 managed disks. Azure pins each disk to a specific availability zone at creation time, reflected in the PV's nodeAffinity rules. A CNPG pod whose PVC is pinned to Zone 2 can only schedule on a Zone 2 node.

# Check which zone each PVC is pinned to
kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values[*]}{"\n"}{end}'

This makes a naïve "drain everything and apply" rotation unsafe. The new VMSS may not land nodes in every zone - Azure controls zone assignment. If no new node ends up in Zone 2 and the old Zone 2 nodes are gone, both the CNPG primary and standby for that zone go Pending simultaneously - with no primary available.

DEV is not affected: standard managed-csi PVCs show zone: "" and can attach to any node in any zone.

Before cordoning or draining any node, check which CNPG PVCs are pinned to that node's zone and confirm a schedulable node exists in that zone. This is mandatory on production.

The bridge pool pattern

The solution: add a temporary User node pool before touching the default pool, evacuate all CNPG pods onto it, rotate the default pool, return CNPG, then remove the bridge. CNPG pods have a guaranteed schedulable target at all times.

Phase 1 - Add bridge pool

resource "azurerm_kubernetes_cluster_node_pool" "bridge" {
  name                  = "bridge"
  kubernetes_cluster_id = module.core.aks_cluster_id
  vm_size               = "Standard_D4ads_v6"
  node_count            = 2
  max_pods              = 50
  vnet_subnet_id        = module.core.aks_subnet_id
  zones                 = ["2", "3"]   # pin to exactly the zones your CNPG PVCs live in
  orchestrator_version  = var.kubernetes_version
}

Set zones to exactly the zones your CNPG PVCs are pinned to, with node_count matching the number of zones. With zones = ["1", "2", "3"] and node_count = 2, Azure picks 2 out of 3 at random - you could easily miss the one zone you actually needed.

Apply and wait for both bridge nodes to be Ready. Verify zone placement before proceeding:

kubectl get nodes -L topology.kubernetes.io/zone,agentpool

Phase 2 - Evacuate CNPG to bridge pool

First, cordon all default pool nodes so CNPG reschedules onto bridge and not back onto default:

kubectl cordon <default-node-1>
kubectl cordon <default-node-2>

Then for each CNPG cluster, repeat this procedure:

# Check replication status
kubectl cnpg status <cluster-name> -n <namespace>

# Verify target nodes
kubectl get pods -n <namespace> -o wide

Phase 3 - Rotate default pool

Add temporary_name_for_rotation and update max_pods in the same commit. Apply. Terraform creates the new VMSS, drains old nodes, deletes old VMSS, renames. Takes 15–25 minutes on a 2-node pool.

# Monitor rotation progress
kubectl get nodes -w
# Expect: old nodes SchedulingDisabled → deleted, new nodes appear with new VMSS ID

Phase 3.5 - Reconcile Terraform state

Remove temporary_name_for_rotation and apply again. This produces Plan: 0 to add, 1 to change, 0 to destroy - only the attribute removal. The cluster does not rotate again.

Phase 4 - Return CNPG to default pool

Cordon the bridge nodes so CNPG reschedules back onto default and not onto another bridge node:

kubectl cordon <bridge-node-1>
kubectl cordon <bridge-node-2>

Then repeat the same pod-by-pod procedure as Phase 2, in reverse: delete the standby first, wait for it to land on a default node, verify replication lag = 0 twice at 30s intervals, delete the primary, confirm both pods are Running on default nodes.

Phases 5 & 6 - Delete bridge pool

With CNPG back on default and bridge nodes cordoned, remove the bridge resource from Terraform and apply. Azure deletes the VMs. Plan: 0 to add, 0 to change, 1 to destroy.

Subnet IP budget

Azure CNI pre-allocates (max_pods + 1) IPs per node at VM creation. During rotation peak, old VMSS, new VMSS, and bridge all exist simultaneously:

A /24 subnet has 254 usable addresses. At rotation peak we needed 266 - it would have failed mid-rotation with IP allocation errors. We expanded the subnet to /23 (510 usable) first, which required stopping the cluster because Azure rejects in-place expansion when active IP allocations exist.

Run the IP budget calculation before you start: (old_nodes × (old_max_pods + 1)) + (new_nodes × (new_max_pods + 1)) + (bridge_nodes × (bridge_max_pods + 1)) must fit inside the subnet. A /24 is likely too tight if you have more than 2 nodes.

vCPU quota planning

Azure subscriptions have a regional vCPU quota per VM family - a hard ceiling on how many cores can be allocated simultaneously in a given region. It is not the same as what your cluster is currently using; it is the maximum Azure will let you provision. We started at a quota of 16 vCPUs. That was enough for normal operation but nowhere near enough for this rotation.

During temporary_name_for_rotation, both old and new VMSS exist simultaneously alongside the bridge pool. All three sets of VMs are allocated at the same time, which means peak quota is:

(old_node_count + new_node_count + bridge_nodes) × vCPUs_per_vm

# Our PROD: Standard_D4ads_v6 = 4 vCPUs, 2 nodes per pool
# Peak: (2 + 2 + 2) × 4 = 24 vCPUs

We increased the quota incrementally from 16 to 24 before starting. You can request a quota increase in the Azure portal under Quotas - Compute, filter by region and VM family, and submit an increase. In our experience it was approved automatically in under 5 minutes.

If DEV runs on the same subscription and VM family, stop it before starting PROD rotation. Stopping DEV releases its vCPU allocation immediately and is often enough to avoid a quota increase entirely.

# Check quota before starting
az vm list-usage --location switzerlandnorth \
  --query "[?contains(name.value,'standardDadsv6Family')]" \
  -o table

az aks stop --name <dev-cluster> --resource-group <dev-rg>

Bonus: cert-manager Cloudflare DNS-01 cleanup bug

While running this rotation we noticed a production TLS certificate that had been in Ready: False for a couple of days. Root cause: a Cloudflare DNS-01 cleanup bug in cert-manager v1.16.2. During challenge cleanup, the Cloudflare zone ID was empty in the DELETE call:

DELETE /zones//dns_records/<record-id>   ← empty zone ID → 400 from Cloudflare

cert-manager interpreted the 400 as a cleanup failure, marked the challenge as failed, and never retried issuance. The certificate renewal was silently broken with no alerting.

Fix: upgrade to v1.16.4 (patched), then delete the stuck challenges manually. cert-manager immediately re-issues. Our certificate went from False to True in approximately 15 seconds.

kubectl get challenges -A
kubectl delete challenge <stuck-challenge-name> -n <namespace>
If you use cert-manager with Cloudflare DNS-01 and are on v1.16.2 or earlier, check your certificates now. Run kubectl get certificate -A and look for any Ready: False that has been stuck for more than a few minutes.

Bonus: cold-boot Zitadel race condition

With more pod headroom, a separate problem became more visible: on every cold cluster start, all backend services launch before Zitadel is ready and log auth errors until manually restarted. The only fix was a manual rollout restart of every service after confirming Zitadel was healthy - every single time the cluster started.

The correct fix is an init container on each backend Deployment that polls Zitadel's internal health endpoint before the application container starts:

initContainers:
- name: wait-for-zitadel
  image: alpine:3.20
  command:
  - sh
  - -c
  - 'until wget -qO- http://zitadel.auth-system.svc.cluster.local:8080/debug/ready; do sleep 3; done'
containers:
- name: your-service
  # ...

Operational checklist

At Obvelum we run our own infrastructure and document what we learn along the way. If solving problems like these sounds like your kind of work, we would love to hear from you.