llama.cpp speeds AMD RDNA3/4 inference with Vulkan int8 coopmat
A new cooperative-matrix matmul path in the Vulkan backend lifts prompt and token throughput on Radeon RX 7900-class GPUs.
llama.cpp has landed a Vulkan backend path that implements int8 matrix multiply using cooperative matrices aimed at AMD RDNA3 and RDNA4 GPUs.
The change targets the matrix-heavy work in quantized LLM inference. On hardware that exposes Vulkan cooperative-matrix support, the backend can keep more of that work on the GPU’s matrix units instead of falling back to slower general-purpose paths. That matters for local inference on consumer Radeon cards, where prompt processing and token generation are often limited by how fast quantized matmuls run.
Early testing on an AMD Radeon RX 7900 XTX with a 26B-class Gemma build (Q4_0) reported clear gains versus prior Vulkan master builds across both prompt-processing and text-generation benchmarks, including longer contexts. Device capability reporting also picked up broader fp16/bf16 matrix features once the new path was active.
The work is specific to the Vulkan backend and to AMD GPUs that advertise the relevant cooperative-matrix extensions; other backends and vendors are unchanged.