llama.cpp prefill gains and PyTorch CUDA plans

llama.cpp saw multiple performance and packaging proposals, led by large prefill speedups and MoE cache work. PyTorch discussions covered complex tensor compile support and a CUDA 13.2 default for the 2.14 release matrix.

Direct PLE reads speed llama.cpp prefill

A llama.cpp proposal reports more than 2x cold-cache prefill gains on GB10 from direct reads of the lazy PLE table, with related memory-headroom benefits on Apple Silicon. The change targets Windows and other platforms where cached PLE access had been a bottleneck. Readers tracking local inference throughput should watch the PR for broader backend impact.

SemVer Docker tags for llama.cpp releases

Maintainers are discussing fixes so release commits receive SemVer-tagged Docker images on GHCR rather than depending on daily cron builds. The thread addresses missing release builds and inconsistent tagging. Stable image tags matter for reproducible deployments of llama.cpp servers and tooling.

GPU-resident LRU cache for MoE offload

An early user report with benchmarks describes a GPU-resident LRU cache for host-offloaded MoE expert weights in llama.cpp. The design aims to keep hot experts on device while still allowing large models to spill to host memory. MoE users on constrained GPUs may gain lower latency without full expert residency.

Experimental SM120 CUTLASS MoE prefill kernels

A PR adds experimental SM120 CUTLASS MoE prefill kernels for MXFP4 and NVFP4 formats in llama.cpp. Early discussion covers repack timing and MMVQ refactoring around the new paths. The work targets newer NVIDIA hardware for quantized MoE prefill performance.

torch.compile path for complex tensors

PyTorch posted experimental torch.compile support for complex-valued tensors by routing them through real-valued decomposition. The change extends compile coverage to workloads that mix complex arithmetic with graph optimization. Scientific and signal-processing users gain a clearer path to compiled complex ops.

RFC to default CUDA 13.2 in PyTorch 2.14

PyTorch opened an RFC to promote CUDA 13.2 to the default and stable PyPI build for release 2.14 while retaining 13.0 and 12.6. The matrix update would shift what most pip users get without extra index selection. Downstream packagers and extension authors should track the chosen defaults.

Multimodal embeddings for Qwen3-VL in llama.cpp server

An opening PR adds image-plus-text input support to llama.cpp embedding and rerank endpoints for Qwen3-VL models. The server change aligns the HTTP API with vision-language embedding workflows. Teams building retrieval over mixed media can test the new endpoint shape once merged.

Inconsistent get_device_capability return types

A PyTorch report flags non-uniform return types from get_device_capability across CUDA and XPU backends, which breaks shared heuristic metadata. The single post calls out the type mismatch as a portability hazard for code that assumes one shape. Backend-agnostic device heuristics need a consistent contract.