AntAngelMed is a medical-domain language model with 103 billion total parameters, but it does not activate all of those parameters during inference. Instead, it uses a Mixture-of-Experts (MoE) architecture with a 1\/32 activation ratio, meaning only 6.1 billion parameters are active at any given time when processing a query.<\/p>\n
It helps to know how MoE architectures work. In a standard dense model, every parameter participates in processing every token. In an MoE model, the network is divided into many \u2018expert\u2019 sub-networks, and a routing mechanism selects only a small subset of them to handle each input. This allows you to have a very large total parameter count \u2014 which typically correlates with strong knowledge capacity \u2014 while keeping the actual compute cost of inference proportional to the smaller active parameter count.<\/p>\n
AntAngelMed inherits this design from Ling-flash-2.0, a base model developed by inclusionAI and guided by what the team calls Ling Scaling Laws. The specific optimizations layered on top include: refined expert granularity, a tuned shared expert ratio, attention balance mechanisms, sigmoid routing without auxiliary loss, an MTP (Multi-Token Prediction) layer, QK-Norm, and Partial-RoPE (Rotary Position Embedding applied to a subset of attention heads rather than all of them). According to the research team, these design choices together allow small-activation MoE models to deliver up to 7\u00d7 efficiency compared to similarly sized dense architectures which means with only 6.1B activated parameters, AntAngelMed can match roughly 40B dense model performance. Separately, as output length grows during inference, the relative speed advantage can also reach 7\u00d7 or more over dense models of comparable size.<\/p>\n
![\"\"](\"https:\/\/www.marktechpost.com\/wp-content\/uploads\/2026\/05\/Screenshot-2026-05-12-at-2.21.07-PM-1.png\")
Training Pipeline<\/strong><\/h2>\nAntAngelMed uses a three-stage training process<\/strong> designed to layer general language understanding on top of deep medical domain adaptation.<\/p>\nThe first stage<\/strong> is continual pre-training on large-scale medical corpora, including encyclopedias, web text, and academic publications. This phase is built on top of the Ling-flash-2.0 checkpoint, giving the model a strong general reasoning foundation before medical specialization begins.<\/p>\nThe second stage<\/strong> is Supervised Fine-Tuning (SFT), where the model is trained on a multi-source instruction dataset. This dataset mixes general reasoning tasks \u2014 math, programming, logic \u2014 to preserve chain-of-thought capabilities, alongside medical scenarios such as doctor\u2013patient Q&A, diagnostic reasoning, and safety and ethics cases.<\/p>\n
The first stage<\/strong> is continual pre-training on large-scale medical corpora, including encyclopedias, web text, and academic publications. This phase is built on top of the Ling-flash-2.0 checkpoint, giving the model a strong general reasoning foundation before medical specialization begins.<\/p>\n The second stage<\/strong> is Supervised Fine-Tuning (SFT), where the model is trained on a multi-source instruction dataset. This dataset mixes general reasoning tasks \u2014 math, programming, logic \u2014 to preserve chain-of-thought capabilities, alongside medical scenarios such as doctor\u2013patient Q&A, diagnostic reasoning, and safety and ethics cases.<\/p>\n