Research Papers Compare
Architecture and method explainers grounded in the source papers, with the key numbers.
Interpretability · Sparse Autoencoders vs SAEs vs GemmaScope
LLM neurons are polysemantic: one cell fires for French text, DNA and HTTP at once. A sparse autoencoder unpacks that superposition into single-meaning features, and GemmaScope scaled it to 400+ open SAEs.
Mixture of Experts · Mixture of Experts vs Switch Transformer vs Mixtral vs DeepSeek-V3
MoE replaces one big feed-forward layer with many experts plus a router, so a model stores far more parameters than it spends per token. Switch Transformer, Mixtral 8x7B and DeepSeek-V3 mark the arc.
Fine-Tuning & Adaptation · on-policy distillation vs OPD vs knowledge distillation
On-policy distillation trains a small student on its own rollouts, scored by a teacher. Five 2026 papers show how to stabilize it, prune it to 5% of tokens, and stretch it to speculative decoding and image models.
Language Models · DeepSeek V4
DeepSeek V4 keeps V3's MoE and multi-token prediction, rebuilds attention as a compress-then-sparsify hybrid, widens the residual stream under a doubly stochastic constraint, and trains most matrices with Muon.
Language Models · Kimi K3
Kimi K3 uses three delta-rule linear-attention layers per global MLA layer, attention over earlier layers for depth, and 16 of 896 experts through a half-width latent, for 2.5x K2's scaling efficiency.
Long Context · KV cache compression
The KV cache, not the weights, runs out of memory at long context. Four 2026 routes attack it: 2-bit quantization for long decodes, lookahead offloading, 1:16 soft-token compression and fixed-size memory.
Language Models · Nemotron 3
Nemotron 3 Ultra keeps most token mixing in Mamba-2 layers, uses attention with only 2 KV heads, routes tokens to 22 of 512 experts, and trained in NVFP4; that is why it decodes up to 5.9x faster than GLM-5.1.
Language Models · Qwen3.8-Next
Qwen3.8-Flash-Next activates 6B of 125B parameters plus 51B host-memory n-gram tables; a 3:1 GDN hybrid, sparse attention, a gated four-branch residual and Muon give Qwen3.7-Plus quality at a ninth of the FLOPs.
Efficient AI · Speculative decoding
A small drafter proposes several tokens, the big model verifies them in one pass, and rejection sampling keeps the output exact. The whole game is how many draft tokens survive, and 2026's gains come from the drafter.