Kernel lib fixes target maple-tree crash and rhashtable deadlock

Liam R. Howlett of Google has posted library fixes aimed at Linux 7.3 that close a maple-tree crash under RCU and a deadlock in rhashtable growth, the latter exploitable by an unprivileged user via a network bridge.

In the maple tree, an RCU reader that restarts after its node is gone can dereference a null pointer when the tree has been emptied or holds only a single root entry. Howlett’s series corrects the reader path and adds a regression test for emptying the tree while an RCU walk is in progress. A separate change, prompted by Lorenzo Stoakes, makes maple-tree debug warnings and bugs go through the kernel’s normal WARN and BUG paths after dumping tree state, so automated fuzzers such as syzbot actually see them.

The more operationally serious fix is in rhashtable. Chris Mason reports that filling a bridge forwarding database (FDB) in a user namespace can deadlock CPUs. When an insert in atomic context finds the table too full, rehash allocates a larger table with GFP_ATOMIC via kvmalloc. Since a change in 6.19, kvmalloc may fall back to vmalloc even for non-blocking flags once kmalloc cannot satisfy the size. vmalloc takes locks that are unsafe under the bridge’s hash lock and softirq receive path, producing a classic ABBA hang: one CPU holds the bridge lock and waits inside vmalloc while another holds the vmalloc lock and the bridge softirq waits on the bridge lock. Flooding a learning port with new source MACs, combined with local vmalloc activity, is enough to trigger it; other CPUs that later call vmalloc or vfree can hang as well.

Mason’s patch keeps non-blocking rehash on kmalloc or a nested table instead of taking the vmalloc path. Herbert Xu called the underlying kvmalloc non-blocking fallback a fundamental problem and asked whether it should simply be removed so non-blocking callers always stay on kmalloc. Memory-management developers replied that similar lock-order risks exist for any GFP_ATOMIC work that takes sleeping-unsafe locks, and that atomic flags guarantee no sleep, not freedom from ABBA deadlocks.

The maple-tree null-deref fix and the rhashtable atomic-rehash change are the pieces operators and stable trees will care about first; the wider kvmalloc question remains open on the lists.