freenode
AI & ML

llama.cpp auto thread count now prefers physical cores

When users pass -t -1, the runtime sizes worker threads to physical cores rather than SMT logical cores.

llama.cpp now treats the automatic thread setting (-t -1) as one worker per physical CPU core, not per logical core. That changes the default on machines with simultaneous multithreading (SMT or hyperthreading), where each physical core appears as two logical CPUs.

The rationale is workload shape, not raw thread count. LLM inference is largely memory-bound: weight loads and KV-cache traffic dominate, and sibling threads share L1/L2/L3 caches and memory controllers. On quantization, matmul, and attention kernels that already aim for full core use, a second SMT thread often adds contention without throughput gains and can regress latency.

That default matches common practice in other memory-sensitive stacks. OpenMP-style runtimes, Intel MKL, oneDNN, and PyTorch frequently prefer physical cores for similar work. Contributors argued that -t -1 should mean “optimal,” not “maximum threads,” and the project already had physical-core detection used elsewhere; the auto path now follows the same policy.

Users who still want logical-core oversubscription can set an explicit thread count.