Introduction to UniMoE-Audio

UniMoE-Audio

UniMoE-Audio is a unified framework that seamlessly combines speech and music generation. Powered by a novel dynamic-capacity Mixture-of-Experts design, it adapts intelligently to input complexity, enabling high-fidelity voice and expressive music within a single model.

UniMoE-Audio introduces a dynamic-capacity routing mechanism based on Top-P sampling for adaptive expert allocation, together with a hybrid expert design that separates domain-specific computation (dynamic experts) from universal representations (shared experts). To address data imbalance and task conflicts, UniMoE-Audio adopts a structured three-stage training curriculum. From voice cloning and text-to-speech (TTS) to text-to-music (T2M) and video-text to music (VT2M), UniMoE-Audio supports diverse creative workflows. Extensive experiments confirm its state-of-the-art performance and superior cross-task synergy, paving the way toward universal audio generation.

On the left of Fig. 1, comparison against specialized baselines reveals the failure of naive joint training, which causes a clear performance degradation on speech generation and more significant decline on music generation. The root of this problem lies in the divergent objectives of speech and music tasks as well as severe data imbalances, which make naive joint training incapable of capturing their inherent complexity.

In contrast, UniMoE-Audio yields synergistic gains across both tasks. On radar charts in Fig. 1, UniMoE-Audio is shown achieving competitive performance against leading models on a wide array of speech (a) and music (b) metrics.

A Novel Dynamic-capacity MoE for Mitigating Task Conflict

To provide adaptive allocation of computational resources between simple and complex tasks, the model's architectural core is a Transformer enhanced with Dynamic-Capacity Mixture-of-Experts (MoE) layers. The following Fig. 2 shows the overall architecture of UniMoE-Audio.

Experiment Results

We evaluate UniMoE-Audio against state-of-the-art specialized models for speech and music generation. The results, summarized in Table 1, Table 2 and Table 3, demonstrate that our single, unified model not only avoids the performance degradation typical of naive joint training but achieves competitive or even superior results in both domains.

Speech Synthesis

Speech Synthesis (TTS) converts text into natural and fluent speech, focusing on clarity, naturalness, and speaker characteristics.

Across multilingual benchmarks including SeedTTS, LibriSpeech, and AISHELL-3, UniMoE-Audio leads across all metrics, with WER as low as 0.019 and UTMOS up to 4.36 (3.86 for Chinese), outperforming UniAudio, Qwen-Omni, and others. This indicates superior speech clarity, naturalness, and speaker consistency.

Text to Music Generation

Text-to-Music Generation (T2M) transforms text descriptions into music clips, evaluating the model's ability to capture melody, rhythm, and atmosphere.

On both MusicCap and V2M-bench, UniMoE-Audio achieves the best performance across key metrics such as PC, PQ, and CE (e.g., PC=6.00, PQ=7.77, CE=7.34 on MusicCap), outperforming strong baselines. This shows its ability to generate music with clear melody, coherent rhythm, and high perceptual quality.

Video-Text to Music Generation

Video-to-Music Generation (VT2M) generates music aligned with video content, testing cross-modal alignment and emotional expression.

On V2M-bench, UniMoE-Audio significantly outperforms AudioX, reaching PC=5.88, PQ=7.62, and CE=6.96, while remaining competitive on diversity metrics like FAD. This demonstrates stronger capability in aligning video content with contextually fitting music, highlighting its cross-modal generation strength.