Kernel fix breaks vmalloc livelock under memory pressure
A circular lock dependency in the vmap purge path could stall the whole system when reclaim and purge work collided.
A deadlock in the Linux kernel's vmalloc layer could freeze the system under memory pressure, and a fix from Ye Liu replaces blocking lock takes with trylocks on the two reclaim-reachable paths that caused it.
The purge lock can be held for a long time while the lazy purge path waits for background workers. Those workers can enter direct reclaim and, via the vmap pool shrinker, try to take the same lock. The result is a circular wait: the lock holder never finishes because the workers never run, and reclaim never progresses.
The shrinker only decays the vmap pool and does not free pages, so skipping a cycle when the lock is busy is safe. The overflow reclaim path that forces a purge can likewise skip if another thread already holds the lock; the allocator retries and a notifier fallback remains if space is still short. Either failure breaks the cycle so the lock holder's wait can complete.
The bug was introduced when a shrinker was added to drain vmap pools. Uladzislau Rezki and Dev Jain suggested the trylock approach. Andrew Morton has taken the change for review.