Following a Packet: The User Plane in the volte.io ePDG

Part 1 covered authentication. Part 2 covered bearer establishment. By the end of those exchanges, the UE has a tunnel, an IP address, and somewhere to register. It can now use the network.

This article is about what happens during that use – the moment a SIP REGISTER, an RTP packet, or a TCP segment crosses the ePDG. The architecture is interesting because the ePDG sits at the boundary between two very different worlds: an IPsec ESP tunnel terminated on one side, a GTP-U tunnel originating on the other. Translating between them – fast and at scale – is where the design choices in the volte.io ePDG do most of their work.

1. The Two Tunnels Meeting at the ePDG

Before following a packet, it helps to picture what the ePDG looks like from a packet’s perspective.

2. The Uplink Path: From the UE to the PGW

A subscriber places a SIP REGISTER. The handset’s IP stack hands the packet to its IPsec implementation, which wraps it in ESP, which UDP-encapsulates it on port 4500, which sends it out the WiFi interface and across the internet to the ePDG’s public IP address.

What happens when that UDP packet arrives at the ePDG host?

2.1 Reception and ESP Decryption

The packet hits the host’s NIC. If the NIC supports IPsec inline offload – as the NVIDIA/Mellanox ConnectX cards we use for our IMS deployments do – the NIC recognizes the UDP-encapsulated ESP, looks up the matching Security Association in its hardware table, decrypts the payload, verifies integrity, and presents the cleartext inner IP packet to the kernel as if it had arrived in the clear.

If the NIC does not offload, the same work happens in software via the kernel’s XFRM subsystem. AES-NI on any modern x86 CPU makes this fast enough for most deployments – typically a few cycles per byte for AES-GCM, dominated more by memory bandwidth than CPU.

Either way, the cleartext packet that emerges has the UE’s allocated IP address as its source, and some destination – for our SIP REGISTER, that destination is the P-CSCF address the UE learned during IKE_AUTH.

2.2 Routing Decision

Now the kernel asks the same question it asks of every packet: where does this go?

This is where the architecture gets interesting. The cleartext packet’s destination is the P-CSCF, which sits inside the operator’s IMS network. It cannot just be routed through the host’s main routing table, because the host has no direct connectivity to the IMS – only the PGW does.

The volte.io ePDG handles this with policy-based routing. The packet has just emerged from the UE’s IPsec tunnel; the kernel marks it with metadata that identifies which tunnel it came from. A policy rule directs all packets carrying that mark into a per-subscriber routing table whose default route points into a GTP-U tunnel.

The relevant Linux primitives are:

• ip rule for matching the mark and selecting the routing table.
• gtp kernel module (in mainline since Linux 4.7) for representing the GTP-U tunnel as a network device.
• fou for the UDP encapsulation, where appropriate.

The result is that the kernel’s standard forwarding path – the same path used for any other routed traffic – moves the packet from “just decrypted from ESP” to “about to be encapsulated in GTP-U” without any userspace involvement.

2.3 GTP-U Encapsulation

The GTP-U device receives the packet, prepends an 8-byte GTP-U header containing the F-TEID the PGW expects (learned during the Create Session Response in Part 2), and hands the result to the kernel’s UDP stack with the destination set to the PGW’s GTP-U endpoint on port 2152.

The kernel sends the UDP packet out through whatever interface routes to the PGW – typically a private interface into the operator’s mobile core. The PGW receives it, strips the GTP-U header, looks up the session by F-TEID, and routes the inner IP packet onward to its destination – for our SIP REGISTER, that means delivering it to the P-CSCF.

2.4 What the ePDG Userspace Did During All This

Nothing.

That is the architectural point. The userspace Erlang process that handles IKE signaling, that manages session state, that talks to Redis and Prometheus – it was not involved in any of the steps above. The packet went NIC → kernel XFRM → routing → kernel GTP-U → NIC, with userspace only ever having programmed the state up front. This is what allows the data path to scale linearly with hardware, independent of how many control-plane events the ePDG is handling at the same time.

3. The Downlink Path: From the PGW to the UE

The reply to the SIP REGISTER eventually comes back through the same path in reverse – a 200 OK from the IMS, routed through the PGW, encapsulated in GTP-U, and arriving at the ePDG.

3.1 GTP-U Reception and Decapsulation

The packet arrives on the ePDG’s mobile-core-facing interface as UDP on port 2152. The kernel’s GTP-U module looks up the F-TEID in the GTP header, identifies the session, and delivers the decapsulated inner IP packet – a packet whose destination is the UE’s allocated IP address – to the kernel’s IP layer.

3.2 The Routing Trick

This is where careful policy configuration matters. The kernel’s main routing table, asked where to send a packet destined for the UE, would not have a useful answer – the UE’s IP address is in the operator’s address pool, not in any subnet directly attached to the host. Without help, the packet would either be dropped or sent out the default route, which would be wrong.

The help comes from a routing rule that catches packets emerging from a GTP-U device and directs them into a routing table whose default route points into the matching IPsec SA. The kernel sees a packet whose destination matches the outbound XFRM policy installed at the end of the IKE_AUTH exchange, and the policy directs the packet through the outbound Security Association.

3.3 ESP Encryption and NAT Traversal

The XFRM subsystem (or the NIC’s hardware offload, if available) encrypts the packet with the outbound key, prepends the ESP header with the next sequence number, computes the integrity check value, and produces an ESP packet ready to send.

For NAT-traversed sessions, the ESP packet is then UDP-encapsulated on port 4500 per RFC 3948. The destination is the public IP and port from which the UE’s IKE messages arrived – which may well be a NAT’d address, where the operator’s network sees the carrier-grade NAT or the home router’s external interface, not the UE’s internal address.

The packet leaves the public interface, traverses the internet, and arrives at the UE, where the handset’s IPsec stack reverses the process.

3.4 Sequence Numbers and the Replay Window

One detail worth noting on the downlink: ESP includes a 32-bit sequence number that increments with every packet. The receiver maintains a replay window – typically 64 packets wide – and rejects any packet whose sequence number has already been seen, or whose sequence number is too far ahead of the window’s leading edge.

This works fine for a stable, in-order link. It can cause unexpected packet drops on a link with significant reordering, which Wi-Fi sometimes is. The volte.io ePDG configures a wide replay window (1024 packets) on its outbound SAs by default, and exposes the parameter for tuning. This is not a frequent operational concern, but when it does come up – typically with mobile devices on flaky public Wi-Fi – having the parameter exposed avoids a frustrating diagnostic exercise.

4. Why Kernel XFRM and Not Userspace ESP

It is worth addressing a question that comes up regularly in design discussions: why kernel XFRM rather than a userspace ESP implementation, the way some ePDG implementations handle it?

The kernel XFRM subsystem has been in mainline Linux since 2002. It has been used by every Linux-based VPN product, every IPsec gateway, and most Android phones for over two decades. It supports hardware offload through a clean abstraction. It handles edge cases – fragment handling, MTU discovery, NAT-T keepalives, replay protection across SA rekeying – that any userspace implementation would need to reinvent.

A userspace ESP implementation, by contrast, requires every packet to traverse the kernel-userspace boundary at least twice (once for ingress, once for egress), with the associated context switches and memory copies. It does not benefit from hardware offload unless the NIC exposes a userspace-friendly interface. And it has to reimplement, slowly and probably with more bugs, what the kernel already does well.

There are reasons to choose userspace ESP – primarily flexibility for specialized environments where the kernel’s IPsec is not available – but for a Linux-based ePDG running in Kubernetes on commodity or accelerated hardware, kernel XFRM is the correct choice. It is the same conclusion every serious Linux-based IPsec product has reached, and we see no reason to disagree.

5. The Same Picture, in Numbers

To make the architecture concrete, here is what a single uplink RTP packet (a 20-millisecond audio frame, around 200 bytes) costs in compute on a modern x86 server:

• NIC reception and DMA – nanoseconds, hardware-bound.
• NAT-T UDP decapsulation – a few cycles, kernel UDP code path.
• ESP decryption (AES-GCM, AES-NI) – roughly 1 cycle per byte, so a few hundred cycles for the payload.
• Routing lookup and policy application – a few cycles, dominated by cache effects.
• GTP-U encapsulation – tens of cycles, dominated by header construction.
• Kernel UDP transmit – tens of cycles.
• NIC transmission and DMA – nanoseconds, hardware-bound.

The total budget is on the order of 1000 CPU cycles per packet, or roughly 300 nanoseconds on a 3 GHz core. With one core dedicated to softirq processing, that comfortably handles several million packets per second – far more than a single ePDG instance is ever likely to see in practice. With NIC offload of the ESP step, the cycle budget per packet drops further, and CPU stops being the bottleneck before the NIC does.

This is what lets a VoWiFi deployment actually run on Kubernetes. The data plane is in the kernel; the control plane is in pods. The two scale independently, and the data plane scales linearly with the hardware available to it.

Coming Up in Part 4

The data path is fast, the control plane is clean, and a single ePDG instance can handle the traffic of a small operator. The remaining question – and the subject of Part 4 – is how this scales horizontally across multiple ePDG replicas.

We will look at what Redis-backed shared state actually does (and, importantly, does not do) for an IKE-based gateway. We will examine LoadBalancer behavior and session affinity, and what happens when a UE’s session needs to be reconstructed elsewhere. We will discuss Connection Draining during rolling upgrades, and the operational primitives – health probes, observability, graceful shutdown – that turn a single working pod into a fleet that can be operated like any other Kubernetes workload.