Tag: #transformer

← All posts

• AI for Everyone, Part 4 — Captioning Images With 1.37M Parameters, and Why Part 3 Bug Was Absent Here

  We joined a CNN encoder to a transformer decoder and captioned Fashion-MNIST images. With 1.37 million parameters and ten minutes of training, the label hit rate reached 91%. This part looks at what cross-attention does

  2026-08-22 · 7 min read #ai#captioning#multimodal#transformer#pytorch

• AI for Everyone, Part 1 — Training a 16M-Parameter Language Model From Scratch in 15 Minutes

  Training a language model from scratch on a single GPU. With the TinyStories dataset and a 16-million-parameter decoder-only transformer, we produced readable English fairy tales in 15 minutes. We look at why the causal

  2026-08-19 · 9 min read #ai#llm#transformer#pytorch#hands-on

• Korean Dev Blog Curation 3 — AI and ML in Practice, 14 Posts I Opened and Checked

  Fourteen Korean-language posts on applying AI and ML in production, selected for being concrete and reproducible. They cover an end-to-end LangChain RAG pipeline, how semantic search actually works through embeddings and

  2026-08-12 · 12 min read #curation#큐레이션#ai#llm#rag

• Anatomy of config.json — Reading a Model From One Settings File

  How every field in config.json — hiddensize, numhiddenlayers, numattentionheads versus numkeyvalueheads, headdim, intermediatesize, ropetheta, vocabsize, tiewordembeddings — shows up in memory and speed, and a hand count

  2026-08-12 · 6 min read #ai-papers#model-internals#config-json#transformer#llm-architecture

• Understanding Positional Encoding — From Sine Waves to RoPE

  Starting from why Transformers need positional information, this article explains sinusoidal, learned, and relative positional encodings step by step, then RoPE and ALiBi. It connects length extrapolation and context ext

  2026-06-26 · 17 min read #llm#positional-encoding#rope#alibi#long-context

• Dissecting the Transformer — From Attention to KV Cache

  A from-scratch breakdown of the Transformer: self-attention, multi-head, positional encoding, the FFN, and residual connections with normalization. It connects tensor shapes and parameter counts, causal masking, encoder/

  2026-06-26 · 16 min read #llm#transformer#attention#positional-encoding#kv-cache

• The KV Cache and PagedAttention — Everything About Inference Memory

  A deep dive into the KV cache, the single biggest consumer of memory in LLM inference. We cover what the KV cache is and why it eats memory, the memory arithmetic and fragmentation problem, PagedAttention block managemen

  2026-06-26 · 12 min read #kv-cache#paged-attention#inference#gpu-memory#quantization

• Building an LLM from Scratch — A Stanford CS336 Style Learning Roadmap

  Stanford CS336 (Language Modeling from Scratch) keeps hitting the Hacker News front page, and from-scratch LLM building is hot again. This post dissects the full curriculum — tokenizers, attention, distributed training,

  2026-06-12 · 15 min read #llm#transformer#cs336#deep-learning#tokenizer

• LLM Landmark Papers Roundup 2026 - Transformer / Scaling Laws / Flash Attention / Mamba / DeepSeek-R1 / Titans Deep Dive ♪ Listenable

  From the 2017 Attention Is All You Need paper to 2026 Titans and DeepSeek-R1, a thematic roundup of the 50+ landmark papers that built the LLM era. Transformer, BERT, GPT 1-3, Scaling Laws, Chinchilla, InstructGPT, PaLM,

  2026-05-16 · 21 min read #llm-papers#transformer#scaling-laws#flash-attention#mamba

• Foundation Model Architectures 2026 — Beyond the Transformer / Mamba 2 / Hyena / RWKV / RetNet / Griffin / Jamba / xLSTM / TTT / DiT / MoE / Flash Attention 3 Deep Dive ♪ Listenable

  In 2026 the foundation-model world is no longer Transformer-only. Vaswani 2017 "Attention is All You Need" remains the standard, but next to it stand state-space models (Mamba, Mamba 2), the linear-RNN renaissance (RWKV,

  2026-05-16 · 22 min read #foundation-models#transformer#attention-is-all-you-need#vaswani#mamba

• LLM Landmark Papers Guide — From Attention to GPT, LLaMA, DeepSeek, o1, and Claude (with References, 2026)

  Where do the real shifts in LLMs come from? From Attention is All You Need in 2017 to the reasoning models of 2026, this guide organizes the 20-odd landmark papers you must know, by era and theme. Each paper is compresse

  2026-05-14 · 15 min read #llm#research-papers#transformer#gpt#llama

• NPU Deep Dive: How Transformer Architecture Runs Directly on Silicon ♪ Listenable

  A complete technical breakdown of how NPUs differ from CPUs and GPUs, how every transformer operation maps to hardware, and why LLM inference is memory-bound — not compute-bound. From Apple ANE to Qualcomm Hexagon, Groq

  2026-03-18 · 22 min read #npu#transformer#ai-hardware#quantization#kv-cache

• Building LLM from Scratch: Complete Guide to Understanding GPT through Code ♪ Listenable

  A complete guide to building and understanding Large Language Models from scratch. Implement a full GPT architecture in PyTorch — from tokenizer to Transformer, pretraining, and fine-tuning.

  2026-03-17 · 21 min read #llm#gpt#transformer#from-scratch#deep-learning

• Transformer Architecture Complete Analysis: From Attention to Modern LLMs ♪ Listenable

  A complete guide to understanding Transformer architecture from scratch. Covers Self-Attention, Multi-Head Attention, Positional Encoding, and Encoder-Decoder structure through Flash Attention, RoPE, and GQA with formula

  2026-03-17 · 22 min read #transformer#attention#deep-learning#nlp#ai

• Deep Learning Time Series Analysis Complete Guide: LSTM, Transformer, PatchTST, TimesFM

  A comprehensive guide to mastering time series data with deep learning. From preprocessing and ARIMA to LSTM, Temporal Fusion Transformer, PatchTST, Mamba, and the latest foundation models like TimesFM — all with practic

  2026-03-17 · 17 min read #time-series#lstm#transformer#forecasting#deep-learning

• Natural Language Processing Complete Guide: Zero to Hero - From Text Processing to LLMs ♪ Listenable

  A comprehensive guide from NLP fundamentals to the latest LLMs. Learn text preprocessing, Word2Vec, RNN/LSTM, Attention, Transformer, BERT, and GPT step by step with practical code examples.

  2026-03-17 · 33 min read #nlp#natural-language-processing#transformer#bert#gpt

• Mixture of Experts (MoE) Architecture Paper Deep Analysis: From GShard to DeepSeek-MoE ♪ Listenable

  Analyzes core papers on Mixture of Experts architecture, comparing routing strategies and training stability techniques across GShard, Switch Transformer, Mixtral, and DeepSeek-MoE.

  2026-03-14 · 34 min read #ai-papers#moe#transformer#deepseek

• KV Cache Optimization Deep Dive: GQA, MLA, and MHA Attention Mechanisms with Memory Efficiency Strategies

  From Transformer Self-Attention KV Cache fundamentals through MHA, MQA, GQA (Llama 2/3), and MLA (DeepSeek-V2/V3) mechanism memory analysis and comparison, KV Cache compression techniques (quantization, eviction policies

  2026-03-11 · 17 min read #ai-papers#kv-cache#attention-mechanism#gqa#mla

• Deep Dive into Mixture of Experts (MoE) Architecture: From Switch Transformer to Mixtral and DeepSeek

  A comprehensive analysis of Mixture of Experts (MoE) architecture. Covers the mathematical foundations of Sparse MoE, routing strategies in Switch Transformer, Mixtral 8x7B, and DeepSeek-V3, training stability techniques

  2026-03-10 · 15 min read #ai-papers#moe#transformer#mixtral#deepseek

• FlashAttention Paper Analysis: Revolutionizing Transformer Training and Inference with IO-Aware Exact Attention ♪ Listenable

  An in-depth analysis of the FlashAttention series (v1 through v3). Covers the tiling strategy of the IO-aware algorithm, GPU SRAM/HBM memory hierarchy utilization, backward pass recomputation, parallelism improvements in

  2026-03-09 · 22 min read #ai-papers#flash-attention#transformer#gpu-optimization#attention-mechanism