Cilium vs kube-proxy

Key points:

• Default to kube-proxy in nftables mode for hundreds of Services; IPVS is deprecated in 1.35 and removed in 1.43.
• Move to Cilium for thousands of Services, high connection churn, identity- or DNS-aware policy, or slow drop debugging.
• On GKE Dataplane V2 and AKS Azure CNI powered by Cilium, you already run Cilium.
• With kube-proxy replacement, always set k8sServiceHost and clean up leftover kube-proxy rules.
• Learn cilium-dbg and hubble observe before the first incident.

What kube-proxy does

kube-proxy runs on every node, watches Services and EndpointSlices, and programs the kernel to DNAT packets for a ClusterIP (virtual, owned by nothing) to a real pod IP. Background in pod networking and CNI.

Mode	How it works	Status
iptables	Chains of rules; the Service lookup walks them in order, so cost grows with the number of Services	The default
nftables	The newer kernel firewall API; Service lookups use maps, so they stay fast at scale	GA in Kubernetes 1.33; planned to become the default
IPVS	The kernel’s built-in load balancer	Deprecated in 1.35, off by default from 1.40, removed in 1.43

kube-proxy only load-balances Services. NetworkPolicy, encryption and visibility come from elsewhere.

What Cilium does instead

Cilium is a CNI built on eBPF (small kernel programs run on network events). With kube-proxy replacement, it handles Services itself:

• Hash-map lookups instead of rule chains. 10 Services or 10,000 cost the same per new connection.
• Socket-level load balancing. On connect() to a ClusterIP, Cilium swaps in a backend pod’s address; no per-packet rewriting afterwards.
• NodePort and LoadBalancer in eBPF, with optional XDP acceleration (processing at the network driver), Maglev consistent hashing, and direct server return (DSR) so replies skip the extra hop.

flowchart TB
    subgraph KP["kube-proxy (iptables)"]
        A1[Pod connects to<br/>ClusterIP 10.96.0.10] --> A2[First packet walks<br/>KUBE-SERVICES chains]
        A2 --> A3[DNAT to a pod IP;<br/>later packets follow conntrack]
        A3 --> A4[Backend pod]
    end
    subgraph CI["Cilium (eBPF)"]
        B1[Pod connects to<br/>ClusterIP 10.96.0.10] --> B2["connect() rewritten once<br/>via eBPF map lookup"]
        B2 --> B4[Backend pod]
    end

Beyond Services, Cilium adds what kube-proxy never did:

• Identity-based network policy. Policy applies to a numeric identity derived from labels, not IP, so it survives IP changes. Supports L3/L4, DNS-name egress rules (toFQDNs), and L7 HTTP rules.
• Hubble - flow logs and a service map: who talked to whom, and which policy dropped what.
• Encryption between nodes with WireGuard or IPsec.
• BGP, egress gateways with fixed source IPs, a Gateway API implementation, and ClusterMesh for connecting clusters.

Side by side

	kube-proxy + a basic CNI	Cilium with kube-proxy replacement
Service lookup cost	Grows with Service count (iptables)	Constant
Rule updates	Rewrites rule sets; slow with thousands of Services	Updates map entries
NetworkPolicy	Depends on the CNI	Built in, L3-L7, DNS-aware
Visibility	iptables-save and conntrack	Hubble flows, drop reasons
Encryption	Separate tool	WireGuard or IPsec built in
Kernel needs	Anything modern	5.10+ (or RHEL 8.10’s 4.18)
Debugging skill	Widely known	eBPF tooling to learn
Moving parts	Small	A larger system to run and upgrade

On managed Kubernetes

• GKE Dataplane V2 and AKS Azure CNI powered by Cilium are managed Cilium - you already have it.
• EKS ships the VPC CNI and kube-proxy. Install Cilium yourself, replacing the VPC CNI (ENI mode) or chained on top; chaining keeps VPC networking but not every Cilium feature.

When kube-proxy is fine

Default: kube-proxy, ideally in nftables mode - it is simpler for hundreds of Services, no L7 policy or flow visibility needs, and a team that knows iptables.

Use Cilium when: thousands of Services, high connection churn, identity-based or DNS-aware policy, or “why was this connection dropped?” takes too long to answer.

Turning on kube-proxy replacement

Tested with helm template against Cilium 1.20.3:

kubeProxyReplacement: true
# Without kube-proxy, nothing makes the `kubernetes` Service IP work,
# so Cilium must be told the API server's real address
k8sServiceHost: api.my-cluster.example.com
k8sServicePort: 443
hubble:
  relay:
    enabled: true
  ui:
    enabled: true

Then remove kube-proxy and its leftover rules:

kubectl -n kube-system delete daemonset kube-proxy
kubectl -n kube-system delete configmap kube-proxy
# On every node (or just replace the nodes):
iptables-save | grep -v KUBE | iptables-restore

# Check Cilium took over
kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep KubeProxyReplacement
kubectl -n kube-system exec ds/cilium -- cilium-dbg service list

Gotchas

• Forgetting k8sServiceHost. Without kube-proxy, Cilium can’t reach the API server via the ClusterIP and networking never comes up. Set k8sServiceHost/k8sServicePort to the real API address.
• Leftover kube-proxy rules. Old rules still match traffic and fight Cilium. Clean them up or replace nodes; on EKS, remove the kube-proxy managed add-on or it comes back.
• Debugging habits stop working. iptables-save shows nothing useful. Learn cilium-dbg service list, cilium-dbg monitor --type drop, and hubble observe --verdict DROPPED before the first incident, not during it.
• Istio sidecars clash with socket-level LB. Cilium resolves the Service before Envoy sees it, bypassing Istio routing. Set socketLB.hostNamespaceOnly: true when running Istio sidecars.
• Policy turns on default-deny. Once a policy selects a pod, anything not allowed is dropped, DNS included. Always allow egress to kube-dns.
• High-churn labels explode identities. Each unique label set is a new identity, so commit-hash or timestamp labels cause constant churn. Limit which labels count towards identity.
• L7 policy goes through a proxy. HTTP-aware rules send traffic through a node-local Envoy. Expect added latency and no visibility into TLS-encrypted traffic.
• Bad upgrades take the cluster down. Agent restarts keep existing traffic flowing, but a bad CNI upgrade can break all cluster networking. Go one minor version at a time, run the pre-flight check, read the upgrade notes.
• Kernel versions matter. Some features need newer kernels than the 5.10 minimum. Check your node images before enabling them.