HMoE: Heterogeneous Mixture of Experts for Language Modeling EMNLP

An Wang | Xingwu Sun | Ruobing Xie | Shuaipeng Li | Jiaqi Zhu | Zhen Yang | Pinxue Zhao | Weidong Han | Zhanhui Kang | Di Wang | Naoaki Okazaki | Cheng-zhong Xu |

Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization EMNLP

Guanghui Song | Dongping Liao | Yiren Zhao | Kejiang Ye | Cheng-zhong Xu | Xitong Gao |