Tag: #moe

← All posts

• Text LLM Technical Reports: What to Read, and How to Read Design Decisions Instead of Rankings

  Nine text-LLM technical reports, each verified by opening the arXiv abstract page directly. What DeepSeek-V3, DeepSeek-R1, Qwen3, Gemma 3, Olmo 3, Kimi K2, MiniMax-01, Mellum2 and s1 each did that was new, and which limi

  2026-08-12 · 7 min read #ai-papers#paper-review#technical-report#llm#moe

• MoE Routing — How an Expert Gets Picked

  Reading the config fields of a mixture-of-experts layer against real models. Comparing Mixtral 2-of-8, Qwen3 8-of-128, the 256 routed experts plus a shared expert in DeepSeek-V3, and the sparsity of 48 in Kimi K2, then w

  2026-08-12 · 6 min read #ai-papers#model-internals#mixture-of-experts#moe#routing

• How These Models Were Actually Built — Dissecting the 2026 Open-Weight Pipeline

  Reading the cards and technical reports of the open-weight models sitting near the top of Hugging Face as of August 2026, this post lays out the production pipeline in order, from data collection to quantized deployment.

  2026-08-02 · 17 min read #llm#pretraining#moe#post-training#quantization

• How a 26B Model Runs in 2GB of RAM — Resident Memory and Working Set Are Not the Same Number

  TurboFieldfare, posted to Show HN on July 29, 2026, claims to run Gemma 4 26B-A4B on M-series Macs in about 2GB of RAM. 14.3GB is installed on disk, the only thing resident in RAM is a 1.35GB shared core, and the expert

  2026-07-31 · 12 min read #ai#llm#quantization#apple-silicon#moe

• What DeepSeek V4 Flash Actually Changes — A Cost-per-Capability Question, Not a Leaderboard One

  On July 31, 2026, DeepSeek moved its V4-Flash API into public beta. The architecture is identical to April's preview — 284B total parameters, 13B active MoE, 1M context — and the only thing that changed is post-training.

  2026-07-31 · 11 min read #ai#llm#deepseek#inference-cost#moe

• Tencent Hy3: reading a 295B open-weight MoE without the hype

  On July 6, 2026, Tencent released Hy3 under Apache 2.0: a Mixture-of-Experts reasoning and agent model with 295B total parameters but only 21B active per token. Here is what is genuinely new, where it sits in the Chinese

  2026-07-11 · 5 min read #hy3#tencent#hunyuan#open-weights#moe

• Running GLM-5.2 on a slow computer — how colibrì streams a 744B model from disk

  colibrì is a ~1,300-line pure-C inference engine that runs GLM-5.2, a 744B-parameter MoE model, on a consumer PC with 25GB of RAM. The trick is MoE sparsity plus disk streaming: only ~9.9GB of dense layers stay resident,

  2026-07-11 · 5 min read #ai#llm#local-inference#moe#quantization

• Top LLM Papers 2024-2026 - Llama, DeepSeek, Qwen, Mistral, Phi, RLHF, DPO, CoT, RAG, FlashAttention, vLLM Reading List ♪ Listenable

  A curated reading list of 30+ must-read LLM papers for engineers building with LLMs in 2024-2026. Covers foundation models (Llama 3/4, DeepSeek-V3/R1, Qwen3, Mistral, Phi-4, Gemma 3), training innovations (MoE, MLA, GQA)

  2026-05-16 · 19 min read #llm#papers#llama#deepseek#qwen

• Foundation Model Architectures 2026 — Beyond the Transformer / Mamba 2 / Hyena / RWKV / RetNet / Griffin / Jamba / xLSTM / TTT / DiT / MoE / Flash Attention 3 Deep Dive ♪ Listenable

  In 2026 the foundation-model world is no longer Transformer-only. Vaswani 2017 "Attention is All You Need" remains the standard, but next to it stand state-space models (Mamba, Mamba 2), the linear-RNN renaissance (RWKV,

  2026-05-16 · 22 min read #foundation-models#transformer#attention-is-all-you-need#vaswani#mamba

• 2025 Open Source AI Models Showdown: DeepSeek R1 vs Llama 4 vs Qwen 3 vs Mistral ♪ Listenable

  DeepSeek R1 (671B/37B), Llama 4 Scout/Maverick, Qwen 3 (235B MoE), Mistral 8x22B — complete comparison of the 2025 open-source AI model leaders with benchmarks, licenses, deployment guides, and cost analysis.

  2026-03-22 · 20 min read #open-source#ai#llm#deepseek#llama

• 2025 AI Research Trends: Top HuggingFace Papers and 10 Defining Research Directions

  A developer-focused review of HuggingFace trending papers and the 10 defining AI research trends of 2025. DeepSeek-R1 pure RL reasoning, Nemotron-Cascade 30B/3B MoE, GRPO, PagedAttention, million-token context limitation

  2026-03-21 · 15 min read #ai-research#papers#huggingface#reasoning#moe

• Mixture of Experts (MoE) Architecture Paper Deep Analysis: From GShard to DeepSeek-MoE ♪ Listenable

  Analyzes core papers on Mixture of Experts architecture, comparing routing strategies and training stability techniques across GShard, Switch Transformer, Mixtral, and DeepSeek-MoE.

  2026-03-14 · 34 min read #ai-papers#moe#transformer#deepseek

• Mixture of Experts (MoE) Architecture Deep Analysis: Evolution from Switch Transformer to Mixtral and Efficient Scaling Strategies

  Deep analysis from the core principles of MoE architecture to Switch Transformer single expert routing, Mixtral 8x7B Sparse MoE, and DeepSeek-MoE fine-grained strategy. Covers routing mechanisms, load balancing loss, tra

  2026-03-11 · 17 min read #ai-papers#moe#switch-transformer#mixtral#model-architecture

• Deep Dive into Mixture of Experts (MoE) Architecture: From Switch Transformer to Mixtral and DeepSeek

  A comprehensive analysis of Mixture of Experts (MoE) architecture. Covers the mathematical foundations of Sparse MoE, routing strategies in Switch Transformer, Mixtral 8x7B, and DeepSeek-V3, training stability techniques

  2026-03-10 · 15 min read #ai-papers#moe#transformer#mixtral#deepseek

• Deep Dive into Sparse Mixture of Experts (MoE) Architecture: From Design Principles to DeepSeek-V3 and Qwen3 ♪ Listenable

  Analyzing the mathematical principles, routing strategies, and load balancing techniques of Sparse MoE architecture, covering the design choices and practical training/inference optimization of modern MoE models from Swi

  2026-03-06 · 22 min read #ai-papers#moe#sparse-model#deepseek#2026-03

• Mixture of Experts (MoE) Architecture: A Complete Analysis ♪ Listenable

  A complete analysis of MoE architectures, from the principles of Sparse MoE to the MoE implementations in Mixtral and DeepSeek-V3, routing strategies, and load balancing.

  2026-03-03 · 6 min read #ai-papers#moe#mixtral#deepseek#2026-03