Kernel mm series kills VM_SPECIAL, clarifies VMA flag rules

Lorenzo Stoakes has posted a large memory-management series that tears down the Linux kernel’s long-standing VM_SPECIAL / VMA_SPECIAL_FLAGS mask and replaces it with predicates that say what a VMA actually is allowed to do.

The old mask mixed several unrelated ideas: whether core mm owns the mapping, whether it can expand or merge, whether mlock or GUP must stay away, and whether a driver is mapping MMIO or kernel pages. Drivers abused flags such as VMA_IO, hugetlb sat awkwardly beside the special path, and different subsystems each invented their own notion of “special” (VDSO, THP, zapping, and so on). Stoakes calls the result “all rather a mess.”

The series establishes hard invariants. Mappings managed by core mm may not set the I/O flag or clear may-write in an mmap hook; the kernel now validates VMA state after every mmap and mmap_prepare. Kernel memory mappings must set the mixed-map bit. Call sites that once tested the special mask (userfaultfd move, MADV_DOFORK, mlock support, KSM compatibility, migration setup, reclaim look-around, and others) switch to named checks such as whether the VMA is mm-managed, mm-backed, fixed, or hugetlb.

Several in-tree users are updated to match: usbmon and scatterlist move to mmap_prepare, hfi1 and the ALSA PCM status page map pages eagerly, and defio, cmt_speech, uprobes, and the BPF arena correctly mark kernel pages. Out-of-tree drivers that still set the I/O bit on an mm-managed mapping will warn and fail the mmap.

David Hildenbrand is reviewing the series and has been acking individual patches while pressing for clearer wording around temporary checks and the evolving meaning of mixed-map (historically “refcounted and non-refcounted,” increasingly “kernel-backed pages”). No functional change is intended for corrected in-tree code; the goal is to stop flag abuse and make VMA behaviour self-documenting for driver writers and future mm work.