Mixtral

Pure Triton MoE Kernel Beats Megablocks on Mixtral at Batch Sizes Under 512

A fused Triton kernel cuts MoE forward pass from 24+ launches to 5, beating Megablocks by 31% at batch size 128.