Transformers gains DeepSeek-V4.1-Flash text backbone

Hugging Face Transformers is adding support for DeepSeek-V4.1-Flash, the text backbone of DeepSeek’s V4 family, so the model can be loaded and run through the usual pretrained and generate paths.

The release sits on top of the existing DeepSeek V4 stack and adds three notable pieces. CSA2 is a ratio-scheduled compressed sparse attention scheme with KV sharing: layers that share a compression ratio share one compressed KV cache and indexer, with a two-level indexer that pre-filters blocks and a short sliding window of raw tokens plus a fixed top-k of compressed entries per query. Engram injects n-gram hash-table memory into the residual stream at two layers, using large fp8 tables built from a tokenizer-derived compressed vocabulary. Quantization-aware training is baked into the forward pass for window and compressed KV paths, alongside a single-pass variant of V4’s hyper-connections.

Weights and reference inference code are already public on the Hub (on the order of 510 GB in fp8 / packed-fp4 form). The Transformers work covers the causal language-model backbone first, including expert parallelism and long-context use. A vision tower, aligner, and processor are planned as a follow-up; multi-token prediction draft layers ship in the checkpoint but are ignored on load for now, matching the earlier V4 handling.

For users, the practical result is another large open DeepSeek checkpoint that can be driven from Transformers without a one-off stack, once the remaining multimodal pieces land.