Research Papers Topics
Sparsely activating subsets of parameters so model capacity grows without proportional compute.
Long Context · Kimi K3 vs DeepSeek V4 vs Qwen3.8-Next vs Nemotron 3 Ultra
Four 2026 frontier open models target 1M context four ways: Kimi K3 runs 3:1 delta attention, DeepSeek V4 compresses sparse KV, Qwen3.8-Next runs 3:1 GDN, Nemotron 3 Ultra leans on Mamba-2.
Mixture of Experts · Mixture of Experts vs Switch Transformer vs Mixtral vs DeepSeek-V3
MoE replaces one big feed-forward layer with many experts plus a router, so a model stores far more parameters than it spends per token. Switch Transformer, Mixtral 8x7B and DeepSeek-V3 mark the arc.
Mixture of Experts · Renmin University of China
MPI redesigns MoE routers by aligning router rows with expert weight directions. On 11B MoE, average benchmark accuracy rises from 40.92 to 42.76 with only 0.2% training slowdown.
Mixture of Experts · National University of Singapore
dMoE aligns token-level MoE routing with block-parallel decoding in diffusion LLMs. On LLaDA2.0-mini it cuts unique experts per block from 69.5 to 14.6, keeps 99.11% accuracy, and frees 76-80% of expert memory.
Mellum 2 is JetBrains' open-weight 12B Mixture-of-Experts code model that activates only 2.5B parameters per token, matching dense 4B-14B baselines on software tasks at a fraction of the per-token compute.
Fine-Tuning & Adaptation · Mind Lab
MinT keeps one frontier base model resident and swaps only LoRA adapters, cutting the model-handoff step by 18.3x on a 4B dense model and 2.85x on a 30B MoE, while addressing million-scale adapter catalogs.
Mixtral 8x7B routes each token to 2 of 8 experts per layer, so it holds 47B parameters but uses only ~13B per token — and matches or beats Llama 2 70B and GPT-3.5 under Apache 2.0.
SenseNova-U1 puts image understanding and image generation in one network with shared attention. Its A3B variant hits 80.55 on MMMU and 0.91 on GenEval — a single model that reads and draws.
Mixture of Experts · Google Research
Switch Transformer simplifies Mixture-of-Experts by routing each token to a single expert, hitting up to 7x faster T5 pretraining at fixed compute and scaling to 1.6 trillion parameters with bfloat16 training.