freenode

← digests

llama.cpp speculative decoding and PyTorch AOT kernels

AI & ML2026-10-28

Work in llama.cpp and PyTorch advanced inference and kernel compilation paths. Two prototype-oriented pull requests targeted speculative decoding throughput and ahead-of-time kernel registration.

Optional p_min skip in llama.cpp speculative decoding

A pull request in ggml-org/llama.cpp proposes skipping the minimum speculative decoding probability check for sampled MTP and DFlash2 drafts. The change makes the p_min skip optional in that path and reports 20 to 40 percent tokens-per-second gains in the included benchmarks. Readers tracking local LLM serving will care because the option directly affects draft acceptance overhead on those draft models.

PyTorch prototype for AOTI kernel registration via TVM-FFI

A prototype pull request in pytorch/pytorch adds torch.library.register_aot_impl to compile DSL kernels into AOTI through TVM-FFI. The API is aimed at registering such implementations for ahead-of-time use. Developers building custom operators should watch it as an early hook into that compilation route.