PyTorch pipelining RFCs target activation memory
Two RFCs opened on PyTorch's distributed pipelining path, both aimed at how activations and gradients are kept during pipeline schedules. Neither is a merged change; each is an initial proposal with a single discussion thread.
Cross-rank activation offloading and prefetching
An RFC proposes cross-rank activation offloading and prefetching in torch.distributed.pipelining. The stated goal is to ease memory pressure in interleaved pipeline schedules by moving activations off a rank and bringing them back ahead of use. Developers running memory-tight pipeline parallelism would be the ones affected if the design lands.
Rank-local cache for shared pipeline values
A second RFC proposes shared pipeline values backed by a rank-local activation cache, with gradients completed by summing. It sits in the same torch.distributed.pipelining area and would change how those shared values and their gradients are stored per rank. The proposal is early and has not been adopted.