AMD unveils 'Instella-MoE,' a proprietary AI model trained on its own GPUs, offering higher performance than the Gemma-4-E4B in a smaller form factor.
AMD has released ' Instella-MoE ,' an AI model developed using its own GPUs and software. Instella-MoE is available for free in various variations, including pre-trained and partially trained models, and the code and framework used for training are also publicly available.
Introducing Instella-MoE: A State-of-the-Art Fully Open Mixture-of-Experts Language Model — ROCm Blogs
Instella-MoE is an AI model developed using AMD's AI development GPUs, 'Instinct MI300X' and 'Instinct MI325X'. It is developed as a MoE model that activates only some parameters during inference, with a total of 16 billion parameters and 2.8 billion active parameters.
Instella-MoE has released the following six models:
Instella-MoE-16B-A3B-Pretrain: A pre-trained model trained on a dataset of 7.1 trillion tokens.
Instella-MoE-16B-A3B-Midtrain: Intermediately trained model with enhanced mathematical, coding, and thinking abilities.
Instella-MoE-16B-A3B-Base: A base model with an extended context window to 64k tokens.
Instella-MoE-16B-A3B-SFT: A model that has undergone supervised fine-tuning on the base model.
Instella-MoE-16B-A3B-DPO: A model with direct priority optimization applied to the SFT model.
Instella-MoE-16B-A3B-Think: A model that applies reinforcement learning to a DPO model.
The following graph compares the performance of Instella-MoE-16B-A3B-Base with various AI models. The horizontal axis shows the number of active parameters, and the vertical axis shows the average score of the benchmark test. Instella-MoE-16B-A3B-Base scores lower than Qwen3.5-4B-Base, but scores higher than models of comparable size.
The following is a performance comparison between the Instella-MoE-16B-A3B-Think and competing models. It achieves a higher score with fewer active parameters than the Gemma-4-E4B-it.
Each model in Instella-MoE is available at the following link.
Instella-MoE ✨ - a amd Collection
https://huggingface.co/collections/amd/instella-moe
Instella-MoE is trained using AMD's training framework ' Primus ' and MoE architecture ' FarSkip-Collective '. The codebase used to train Instella-MoE is also publicly available at the following link.
GitHub - AMD-AGI/Instella-MoE: Instella MoE is a fully open state-of-the-art Mixture-of-Experts language model, trained end-to-end on AMD Instinct™ GPUs. · GitHub
https://github.com/AMD-AGI/Instella-MoE
Related Posts:
in AI, Posted by log1o_hf