Operating at Scale: The volte.io ePDG in Production

Part 1 was about authentication. Part 2 covered bearer establishment. Part 3 followed a packet through the data path. By the end of those three articles, the picture was complete: a single ePDG pod, working correctly, handling VoWiFi subscribers from the first IKE packet to encrypted RTP.

This article is about the gap between one pod working correctly and a fleet of pods running production traffic. That gap is filled with operational concerns – load distribution, rolling upgrades, observability, graceful failure handling – and one technical reality that deserves to be addressed honestly: an IKE gateway is fundamentally pod-anchored, no matter how cloud-native the rest of the deployment looks. Understanding why, and what to build around it, is most of what this article covers.

1. What Shared State Is, and What It Is Not

The volte.io ePDG runs as a multi-replica Kubernetes deployment behind a shared LoadBalancer, with a Redis-backed shared state layer accessible to every pod. This is a common pattern for cloud-native applications, but it deserves careful description in the ePDG case because the temptation is to read more into it than is actually true.

1.1 What Lives in Redis

The shared state contains operational metadata about every active subscriber session across the entire deployment:

• The subscriber’s IMSI and current NAI.
• Which pod is currently handling the subscriber’s IKE SA.
• The bearer parameters – APN, allocated IP address, P-CSCF, F-TEIDs.
• Authentication metadata – when the session was established, which AAA returned the vector, when the next re-authentication is due.
• Charging counters – uplink and downlink bytes, session duration, accumulated quota.
• The S6b authorization state – whether an Abort-Session has been received, whether re-auth is pending.

Any pod can read or update any of these fields. This is what makes the deployment behave as a single logical ePDG: a charging report covers all subscribers, a subscriber lookup answers correctly regardless of which pod the subscriber is attached to, and an operations team has a single coherent view of the deployment.

1.2 What Does Not Live in Redis

The IKE Security Association – the cryptographic state that defines an active tunnel – does not live in Redis. It cannot, for several reasons that are worth stating clearly because they are sometimes papered over in vendor documentation.

The IKE SA’s encryption keys, integrity keys, message ID counters, and replay windows are negotiated between exactly two parties: the UE and the specific pod that completed IKE_AUTH. Those keys are derived from a Diffie-Hellman exchange that is unique to that exchange. Replicating them to other pods is technically possible but cryptographically pointless, because the IPsec Security Associations established from those keys are programmed into the Linux kernel of the specific node where that pod runs, with sequence number counters and replay windows that are kernel-managed and not exposed to userspace in any practical way.

A pod cannot meaningfully take over another pod’s IKE SA. The cryptographic state can be read, but the kernel state cannot be migrated; the kernel state can be re-installed elsewhere, but the sequence-number and replay-window state is by then stale; and even if all of that were resolved, MOBIKE (RFC 4555) – the only standardized mechanism for moving an IKE SA to a new endpoint – requires explicit cooperation from the UE, which most VoWiFi handsets do not implement.

The honest answer is that when a pod fails, the UEs attached to it have to re-establish their tunnels. This takes a few seconds: the UE detects loss of liveness through DPD (Dead Peer Detection) timeouts, then runs IKE_SA_INIT and IKE_AUTH against the LoadBalancer, which routes to a healthy pod, which re-authenticates against the AAA and re-establishes the bearer. The subscriber experiences a brief gap; an active call may drop; a registered IMS session typically survives if the UE re-attaches within the SIP registration’s retransmission window.

1.3 Why This Is Still a Useful Architecture

The point of a multi-pod deployment is not magical session migration – it is containment. When one pod fails, only the subscribers on that pod are affected. The remaining pods keep serving their subscribers without interruption. New subscriber attaches succeed against the surviving pods within seconds. Compared to a single-instance deployment, where a crash takes down the entire ePDG until the process restarts, this is a dramatic improvement in availability – even though no individual session was rescued.

This is the right framing for cloud-native ePDG operations: the deployment is highly available, even though individual sessions are not.

2. LoadBalancer Behavior and Session Affinity

Every IKE message in a session needs to reach the pod that holds the SA state. This is where LoadBalancer configuration matters more than it does for most workloads.

2.1 Why Session Affinity Is Non-Negotiable

A typical Kubernetes LoadBalancer distributes incoming connections across backend pods using some hashing function – often based on a 5-tuple of source IP, source port, destination IP, destination port, and protocol. For most workloads this is fine; each connection is independent.

For IKE, this is fatal. IKE_SA_INIT and IKE_AUTH are separate UDP packets, often arriving from different source ports if the UE’s NAT remaps them. If the LoadBalancer routes them to different pods, IKE_AUTH lands at a pod with no record of the IKE_SA_INIT it is supposed to follow, and the session fails.

The volte.io ePDG deployment configures sessionAffinity: ClientIP on the LoadBalancer service, with a generous timeout (typically three hours, matching the IKE SA lifetime). This pins each UE – identified by source IP – to a single backend pod for the duration of its session. New UEs are distributed across pods normally; existing UEs return to their pod regardless of source-port changes.

2.2 The MOBIKE Complication

MOBIKE (RFC 4555) is the IKE extension that lets a UE move its tunnel endpoint to a new IP address – for example, when transitioning from home WiFi to a mobile hotspot. From the LoadBalancer’s perspective, this looks like a new client IP, which would be hashed to a (possibly different) backend pod, which would not have the session.

The volte.io ePDG handles MOBIKE address changes by leveraging the Redis-backed session metadata: when an IKE message arrives at a pod that has no local SA matching the IKE SPI, the pod consults Redis, identifies which pod holds the SA, and either forwards the message internally or instructs the UE to re-establish. The first option preserves the session at the cost of an internal hop; the second is simpler but visible to the subscriber as a brief reconnect.

In practice, MOBIKE is rarely exercised by handset implementations – most UEs simply re-attach when their IP changes. We support it because the specification calls for it, but the operational pattern in the field is overwhelmingly “tear down, rebuild” rather than “migrate”.

2.3 Health Probes and LoadBalancer Integration

Each ePDG pod exposes both a Kubernetes liveness probe and a more specific readiness probe. Liveness asks “is the process running and responsive?” – answered by a simple HTTP endpoint that returns 200 if the Erlang VM is healthy. Readiness asks “is this pod ready to accept new sessions?” – a more nuanced question.

A pod is considered ready when it has loaded its configuration, registered its presence in Redis, established Diameter connectivity to the AAA, and confirmed GTP-C reachability to the configured PGWs. A pod that is up but cannot reach the AAA does not advertise readiness, which keeps the LoadBalancer from routing new sessions to it. Existing sessions on that pod continue to function as long as their already-established Diameter and GTP-C peers remain reachable.

This separation matters during partial outages. If the AAA in one zone goes down, the readiness probes on pods that depend on that AAA flip to not-ready, the LoadBalancer routes new traffic to other zones, and existing sessions survive on whatever connectivity they already have. The deployment degrades gracefully rather than failing all at once.

3. Rolling Upgrades and Connection Draining

A working production deployment needs to be upgraded – for security patches, for protocol fixes, for new features – without taking VoWiFi service down for the operator’s subscribers. This is what Kubernetes rolling upgrades are designed for, but they need cooperation from the application.

3.1 The Shutdown Sequence

When Kubernetes decides to terminate a pod – whether for a rolling upgrade, a node drain, or a scale-down – it sends SIGTERM to the pod’s main process and waits up to a configurable grace period (default 30 seconds, which we extend significantly for the ePDG) before sending SIGKILL.

The volte.io ePDG’s reaction to SIGTERM is deliberate:

1. Mark the pod as not ready in its readiness probe response. The LoadBalancer notices this within a few seconds and stops routing new sessions to this pod.
2. Continue serving existing sessions normally. The pod’s IKE SAs are still valid, the kernel XFRM state is still installed, packets still flow.
3. Stop accepting new IKE_SA_INIT requests locally. Any that slip through during the LoadBalancer’s update window are silently dropped, prompting the UE to retry – which is then routed to a healthy pod.
4. Wait until either (a) all sessions terminate naturally, or (b) the configured drain timeout expires.
5. On drain timeout, signal teardown to remaining sessions via IKE Delete payloads, releasing GTP sessions on the PGW and S6b sessions on the AAA cleanly.
6. Exit.

The drain timeout is configurable per deployment. For a leisurely overnight upgrade with a small subscriber base, several hours is reasonable – most VoWiFi sessions do not last that long, and the upgrade completes when natural session turnover empties the pod. For a deployment with many long-lived sessions or a tighter upgrade window, a shorter timeout accepts that some subscribers will see a re-attach during the upgrade.

3.2 What the Operator Sees

A well-configured rolling upgrade looks like this from the operator’s perspective: existing subscribers continue to use the network, new subscribers attach to upgraded pods, and a small number of subscribers – those whose pods are the last to be drained – see a brief re-attach. Total VoWiFi availability stays above 99.9% during the upgrade window, even though no individual session was preserved across the pod replacement.

This is, again, the difference between high availability of the deployment and high availability of individual sessions. The architecture optimizes for the former because the latter is technically infeasible without UE cooperation that does not exist in practice.

4. Observability

A production ePDG without observability is operationally invisible. The volte.io ePDG exports three categories of telemetry, all following Prometheus and OpenTelemetry conventions consistent with the rest of our stack.

5. Closing the Series

Across four articles, we have walked the entire VoWiFi attach: from the first IKE_SA_INIT packet hitting the ePDG’s public IP, through EAP-AKA / EAP-AKA′ authentication against the HSS, through S6b authorization and S2b bearer establishment with the PGW, through the data path that moves packets between IPsec ESP and GTP-U, and finally through the operational concerns of running it all at scale in Kubernetes.

The pattern that runs through all four parts is the same one the rest of the volte.io stack follows: standards-compliant where the specification matters, pragmatic where the deployment matters, and honest about the limits of what cloud-native architecture can and cannot do. An ePDG is a stateful protocol gateway with deep entanglement with the kernel; pretending otherwise produces brittle systems. Acknowledging the constraint and engineering around it produces systems that actually work.