HMoE: Heterogeneous Mixture of Experts for Language Modeling

An Wang | Xingwu Sun | Ruobing Xie | Shuaipeng Li | Jiaqi Zhu | Zhen Yang | Pinxue Zhao | Weidong Han | Zhanhui Kang | Di Wang | Naoaki Okazaki | Cheng-zhong Xu |