Hugging Face seeks MXFP4 backward kernels for GPT-OSS training

Hugging Face is asking experienced kernel developers to write MXFP4 backward passes so the GPT-OSS model can be trained without leaving its compact 4-bit weight format.

GPT-OSS landed in Transformers with MXFP4 weights that run efficiently on H100, B100, and 50-series GPUs, but only forward kernels exist today. Training therefore requires converting weights to bfloat16, which multiplies memory use by roughly four and slows the job substantially. Attention weights already stay in bfloat16; the missing piece is a backward path for the mixture-of-experts matmul that dominates the model.

Arthur Zucker framed the request as a call for native MXFP4 training, ideally with kernels that still reduce memory on GPUs lacking FP4 hardware by falling back to FP8 or bfloat16 compute. The work targets the community Triton kernels package that Transformers loads at runtime. Zucker warned that the task is not beginner-friendly: the kernels are written in Triton and demand prior CUDA or Triton experience. Once a solid backward kernel lands, the Transformers team says it will handle integration.