llama.cpp MoE GPU cache nearly doubles decode on dual Vulkan cards

llama.cpp is gaining a GPU-resident LRU cache for Mixture-of-Experts (MoE) weights that normally live in host RAM, so only the experts a decode step actually needs stay on the device.

Early numbers from a dual-GPU Vulkan setup make the point. On two Radeon RX 6950 XT 16 GB cards linked over OCuLink from a Ryzen 7 7840U handheld (64 GB system RAM), running a Qwen3.8-Flash-Next UD-Q2_K_XL build with experts forced to CPU and layers on GPU, warm-cache decode rose from 8.2 tokens/s with the cache disabled to 12-13 tok/s at 128 slots (~85% hit rate), 17.4-17.8 at 256 slots (~92%), and 18.9-19.1 at 280 slots (~93%). Slots accumulate across cards. Pushing further spilled allocations back to host and collapsed throughput.

At full 256K context, a smaller micro-batch cut per-card compute buffers sharply, freeing tens of extra cache slots and improving both decode and prefill locality when experts remain in host memory. A deployed layout around 265 slots delivered roughly 18-20 tok/s.

The same report flagged two practical gaps: the MoE debug log path did not link under MSVC without the Windows file-lock calls, and a single-token gate blocked the cache on small speculative-verification batches, turning multi-token drafting into a loss on a weak host. An opt-in token limit restores cache use for those batches without changing the default.