Transformers begins FSDP2 and expert parallelism work

Hugging Face Transformers has opened work to enable Fully Sharded Data Parallel 2 together with expert parallelism using a two-dimensional device mesh along FSDP and tensor-parallel axes.

The target is large mixture-of-experts training. Those runs already shard parameters across data-parallel ranks while splitting experts and tensor dimensions across GPUs. A native mesh would give both strategies one shared process layout inside the library instead of forcing users to wire them together outside it.

The integration is still early. Continuous integration against the change was still failing, with multiple test failures in a large suite, so the feature is not ready for general use.