Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models

Zihan Qiu | Zeyu Huang | Bo Zheng | Kaiyue Wen | Zekun Wang | Rui Men | Ivan Titov | Dayiheng Liu | Jingren Zhou | Junyang Lin |