Controllers, informers and reconcile loops

Key points:

• A controller moves the world one step toward spec. All built-in behaviour works this way.
• Reconcile on current state, not events, and keep it idempotent.
• Informer reads come from a cache that can lag your own writes.
• Finalizers are for cleanup outside the cluster; owner references handle children inside it.
• Stuck in Terminating almost always means a finalizer whose controller is gone or failing.

Desired state vs actual state

spec is what you want, status is what’s true now. A controller loops to close the gap:

loop forever:
    desired = read spec
    actual  = observe the world
    if actual != desired:
        make one step toward desired
    write status

Deployments, ReplicaSets, Jobs, Services, PVCs - all built-in behaviour is controllers. An operator is a controller for your own custom resource (a CRD).

Level-triggered, not edge-triggered

Act on current state, never on “what changed”.

• Edge-triggered: “a pod was deleted, so create one.” Miss the event (a restart, a dropped connection) and the pod is never replaced.
• Level-triggered: “I want 3 pods, I see 2, so create 1.” Missed or duplicated events don’t matter; the next run fixes it.

This is how controllers survive crashes, restarts and network blips. Make reconcile idempotent: running it twice in a row must be safe. Same outside Kubernetes - a cron job asking “is anything overdue?” survives a skipped run; one asking “is this exactly my minute?” silently misses work.

Informers: watching without hammering the API

Controllers don’t poll - they use an informer.

flowchart LR
    API[API server] -->|"1. list, then watch"| INF[Informer]
    INF -->|"2. update"| CACHE[(Local cache)]
    INF -->|"3. enqueue key<br/>namespace/name"| Q[Work queue]
    Q -->|"4. take key"| REC[Reconcile]
    REC -->|"5. read from"| CACHE
    REC -->|"6. write changes"| API

1. List once, then watch from that resourceVersion.
2. Keep an in-memory cache; reads hit it, not the API server.
3. On change, enqueue the object’s key on a work queue. Ten changes to one object collapse into one key.
4. Workers reconcile each key; on error, re-queue with exponential backoff.
5. A periodic resync re-queues everything as a safety net.

If a watch breaks and its resourceVersion expired (410 Gone), the informer lists again.

Owner references and garbage collection

Delete the owner and the garbage collector deletes the children. The link is an ownerReference in the child: a Deployment owns ReplicaSets, a ReplicaSet owns pods.

Deletion mode	Behaviour
Background (default)	Owner deleted immediately; children cleaned up afterwards
Foreground	Children deleted first; owner disappears last
Orphan	Owner deleted; children left running with no owner

Use kubectl delete deployment my-app --cascade=orphan to hand pods over to a new controller: the Deployment goes, the pods stay.

Finalizers

Use finalizers only for things outside the cluster. Owner references already handle in-cluster children. A finalizer is a string in metadata.finalizers: “clean up before this object goes away”.

1. Delete the object. The API server only sets deletionTimestamp; the object stays.
2. The finalizer’s controller cleans up (deletes a cloud load balancer, drops a database), then removes its finalizer.
3. Once the list is empty, the object is deleted.

Leader election

Run several replicas; let only one act. Two active reconcilers would fight. They elect a leader via a Lease object (coordination.k8s.io) that the leader keeps renewing; if it stops, another replica takes over once the lease expires.

kubectl get lease -n kube-system   # see who leads kube-scheduler and kube-controller-manager

Status that tells the truth

• generation vs observedGeneration. metadata.generation bumps on every spec change; the controller sets status.observedGeneration once it has processed it. If they differ, status is stale.
• Conditions (Ready, Progressing, Degraded), each with reason and message, let ArgoCD and kubectl wait judge health.
• Events for failures a human must see.

Gotchas

• Stuck in Terminating. A finalizer’s controller is gone or failing, so the finalizer never clears. Find out why before removing it by hand, since that skips cleanup and can orphan cloud resources.
• Reading your own write. Right after an update, the informer cache may still hold the old version. Write code that’s correct when the cache is a moment behind.
• Update conflicts are normal. Writing a stale object returns 409 Conflict. Re-read and retry, or use server-side apply.
• Two controllers, one field. An HPA and a GitOps tool both setting replicas fight forever. Assign one owner per field.
• Status can lie. A controller that never checks the real world reports success anyway - a Gateway showed Programmed: True while its load balancer update had failed (the security group gotcha). Surface external failures in conditions and events.