freenode
AI & ML

Transformers adds FP8 backend for QuantizedCache

Native PyTorch FP8 concat replaces a defensive workaround, with layout fixes aimed at future kernels.

Hugging Face Transformers is adding an FP8 backend to QuantizedCache so key-value caches can stay in 8-bit floating point through generation, cutting memory for long-context inference.

Contributor jiqing-feng dropped a uint8-view workaround after confirming that torch.cat already handles FP8 natively. Checks across recent PyTorch builds on CPU and CUDA show the path works from 2.6 up, which matches the floor already on main; 2.5 does not.

A contiguous copy on prefill stays. Prefill states arrive from a transpose and later tokens from concatenation, so without that copy the buffer would carry mixed strides for its lifetime. Dequantized updates hide the mismatch today, but any follow-on path that feeds the buffer straight into an FP8 kernel would not. The cost is one small copy per prefill per layer. Tests now build states the way attention does, including non-contiguous layouts, so that case is covered.