llama.cpp MoE cache work and related ML tooling

llama.cpp drew most of the day's activity with proposals on MoE weight caching, quantization types, and serving offload options. A separate note on Hugging Face Transformers offered work on Mamba fine-tuning support.

GPU LRU cache for host-offloaded MoE experts

Users reported speedups from a GPU-resident LRU cache for host-offloaded MoE expert weights in llama.cpp pull request 27861. Participants extended the implementation to Vulkan and to Gemma4. The work targets faster inference when expert weights live primarily in host memory.

gguf_writer method additions

A small pull request thread added methods to gguf_writer in llama.cpp. One participant extended the change for prefix-LM use and reported matching evaluation numbers.

Maintainer objects to IQ2_NL and IQ3_NL types

A single maintainer objected to the pull request adding IQ2_NL and IQ3_NL quantization types for CPU in llama.cpp. The objection cited maintenance burden and the lack of an evaluation process.

Early RDMA work for disaggregated serving

One post shared early driver work for RoCE v2 RDMA intended to support disaggregated prefill and decode in llama.cpp. The effort is preliminary support for disaggregated LLM serving.

Band selection request for CPU FFN offload

A user requested an optional band-selection mode for the existing --n-cpu-ffn offloading path in llama.cpp. The aim is to prioritize the largest FFNs when choosing what to keep on CPU.

Offer of Mamba fine-tuning support

A volunteer offered to implement first-class fine-tuning support for Mamba and Mamba-2 SSMs through a pull request on an open Hugging Face Transformers issue. The note states the architecture is production-ready while the training path in Transformers is not.