Linux IPsec fixes AES-GCM nonce reuse in ESP offload paths

Jérémy Jean has posted a seven-patch series to the Linux kernel networking list that closes several AES-GCM nonce reuse bugs and Extended Sequence Number (ESN) authentication failures in the IPv4 and IPv6 ESP offload paths.

The defects sit in how the xfrm ESP stack and related drivers assign and consume packet sequence numbers when large (GSO) frames are split for software encryption or NIC offload. In multiple cases the stack advanced or shared the sequence counter before building the GCM IV or associated data, so later packets could reuse a prior nonce, or a sender could authenticate with high-order ESN bits the receiver would never reconstruct. One path affected Mellanox mlx5 IV generation; another mishandled untrusted GSO packets from TUN or virtio-net before sequence allocation. A final overflow rollback edge case is essentially unreachable in practice (on the order of 2^64 packets under one key) but still corrected for correctness.

Nonce reuse under a fixed GCM key is a severe cryptographic failure: it can expose the XOR of plaintexts and leak the authentication key, allowing forged ciphertexts without recovering the AES key. The changes target both software and hardware offload, keep the shared sequence counter consistent across segments, and are marked for stable kernels. Jean’s series consolidates and extends two earlier individual reports on the same subsystem.