Tag: #model-internals
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 8 posts
8 posts are tagged #model-internals; the most recent was published on 2026-08-12.
Often tagged together: #ai-papers 8 #long-context 2 #attention 1 #benchmarks 1 #bpe 1
Most read in the last 90 days:
The Training Recipe — From Pre-training to Post-training, and What Reports Write Down
Comparing, exactly as written in the reports, the three stages and six context extensions of Llama 3, the three-stage pre-training of Qwen3, the learning-rate schedule and two-phase YaRN extension of DeepSeek-V3, and the
2026-08-12 · 6 min read #ai-papers#model-internals#pretraining#training-recipe#data-mixturePositional Encoding — RoPE and the Price of Context Extension
Showing through wavelength arithmetic how a single ropetheta value acts on context length, then comparing the 500000 of Llama 3, the ABF step in Qwen3, the YaRN settings of DeepSeek-V3 and Kimi K2, and the partial rotati
2026-08-12 · 6 min read #ai-papers#model-internals#rope#positional-encoding#long-contextAnatomy of config.json — Reading a Model From One Settings File
How every field in config.json — hiddensize, numhiddenlayers, numattentionheads versus numkeyvalueheads, headdim, intermediatesize, ropetheta, vocabsize, tiewordembeddings — shows up in memory and speed, and a hand count
2026-08-12 · 6 min read #ai-papers#model-internals#config-json#transformer#llm-architectureReading a Technical Report Critically — What Gets Written Down and What Goes Missing
How to separate the verifiable from the unverifiable in a model technical report. Covers the limits of self-reported benchmarks, the places where a config and a report disagree, values that could not be read from gated r
2026-08-12 · 7 min read #ai-papers#model-internals#tech-report#benchmarks#evaluationMoE Routing — How an Expert Gets Picked
Reading the config fields of a mixture-of-experts layer against real models. Comparing Mixtral 2-of-8, Qwen3 8-of-128, the 256 routed experts plus a shared expert in DeepSeek-V3, and the sparsity of 48 in Kimi K2, then w
2026-08-12 · 6 min read #ai-papers#model-internals#mixture-of-experts#moe#routingInside the Tokenizer — Why Korean Costs More Tokens, and What That Costs
How byte-level BPE works, then downloading the actual tokenizer files of Qwen3, DeepSeek-V3, and Mixtral to tokenize the same English and Korean text and compare. Covers the vocabulary-size tradeoff, why the config vocab
2026-08-12 · 6 min read #ai-papers#model-internals#tokenizer#bpe#korean-nlpNormalization and Activation — Keeping Training From Falling Apart
Confirming from config values why RMSNorm, pre-norm, and SwiGLU became the defaults, then walking through the newer devices that stop attention logits from exploding — the QK-Norm of Qwen3 and the QK-Clip of Kimi K2 — wi
2026-08-12 · 6 min read #ai-papers#model-internals#rmsnorm#swiglu#training-stabilityAttention Variants — From MHA to MLA, and How the KV Cache Shrinks
Comparing MHA, MQA, GQA, and MLA using real config values. Working out with formulas and numbers how the grouped-query attention in Qwen3, Mixtral, and GLM-4.5 and the latent attention in DeepSeek-V3 and Kimi K2 reduce p
2026-08-12 · 6 min read #ai-papers#model-internals#attention#gqa#mla