Race found in Linux HugeTLB vmemmap optimization path

Linux memory-management developers have confirmed a real race in HugeTLB Vmemmap Optimization (HVO) that can turn a speculative page reference get into a write fault on read-only struct-page memory.

The problem followed work by Kiryl Shutsemau that removed fake-head pages and the scaffolding around them, including an RCU read-side critical section on the lockless refcount bump used by page-cache lookups and other speculative walkers. David Hildenbrand reported that the removal was unsafe. A getter can observe a non-zero refcount, then be rescheduled while the page is freed and reused as a hugetlb tail. HVO then remaps that tail's vmemmap entry to a shared page marked read-only. When the atomic compare-exchange resumes it still attempts the store. On x86, even a failing locked cmpxchg writes, so the access faults.

HVO exists to reclaim most of the memmap backing a huge folio by aliasing tail struct pages onto one shared page. The zero-refcount check is what makes a late get fail cleanly after reuse; without a grace period before the remap, nothing stops the race above. Shutsemau sketched a fix that lets the atomic fault and recovers via an exception-table entry, so the get simply fails and the fast path stays free of RCU. Peter Zijlstra rejected that direction as overly invasive without a clearer justification.

Other options on the table include mapping the shared tail writable (a failed bump writes the same value back, so the race becomes harmless, at the cost of turning any stray store into silent corruption of every HVO folio of that order in the zone) and limiting the read-only mapping to debug builds. Muchun Song opposed the debug-only approach, noting that production RO mappings at ByteDance had already caught memory-failure bugs that would otherwise have corrupted shared tail state, and that under the newer shared-tail scheme such a write damages all optimized folios rather than one.

HVO is implemented on x86, RISC-V, and LoongArch. No consensus fix has landed yet.