Transformer-based large language models often suffer from inter-layer parameter redundancy, where functional transformations are redundantly learned across network depths.
We propose CS-MoE, a novel Transformer architecture featuring cross-layer expert sharing to address this inefficiency.
Deviating from the widely used Mixture-of-Experts (MoE) architecture that terminates each Transformer block with layer-isolated experts, CS-MoE combines layer-independent experts with concurrent access to a centralized, globally shared expert pool.
This Global Experts Sharing mechanism enables elastic control over token-level parameter activation and computational consumption (FLOPs).
Experiments demonstrate that CS-MoE achieves lower perplexity than equal-scale dense Transformers while activating only 55% of parameters.
Furthermore, its performance scales monotonically with an increased number of activated experts and approaches MoE counterparts that consume more FLOPs by expanding the shared pool with a fixed FLOPs budget.
CS-MoE also establishes a flexible Pareto frontier between computational cost and model capacity, offering an efficient alternative for computation-constrained environments.