Tag: #quantization

← All posts

• The Hybrid Attention in Qwen3.8-27B — Only 16 of 64 Layers Grow a KV Cache ♪ Listenable

  The reason a 27B model can carry a 262K-token context on a laptop is not its parameter count but its layer layout. Qwen3.8-27B places 48 of its 64 layers as linear attention (Gated DeltaNet) and only 16 as ordinary atten

  2026-08-14 · 8 min read #llm#inference#quantization#local-llm#attention

• Licenses and Deployment: Reading the Types and Verifying Quantized Builds

  The short identifier in an open model license field is a table of contents; the actual terms live in the full text. This post organizes the license types actually encountered on 2026-08-12, what community licenses and no

  2026-08-12 · 7 min read #ai#llm#huggingface#open-source-llm#license

• Inside vLLM (7) — Deployment Tuning, Common Pitfalls, and OOM Triage

  A practical order for tuning a vLLM deployment. Covers what gpumemoryutilization actually sets, when to use tensor parallelism versus pipeline parallelism, how to choose quantization and a KV cache data type, and a diagn

  2026-08-12 · 10 min read #vllm#gpu#quantization#tensor-parallel#llm

• How These Models Were Actually Built — Dissecting the 2026 Open-Weight Pipeline

  Reading the cards and technical reports of the open-weight models sitting near the top of Hugging Face as of August 2026, this post lays out the production pipeline in order, from data collection to quantized deployment.

  2026-08-02 · 17 min read #llm#pretraining#moe#post-training#quantization

• What's Trending on Hugging Face Right Now — A Map of August 2026

  As of August 2, 2026, I went through the Hugging Face trending list directly and organized models that are actually usable by purpose. In order — general LLMs, coding, embeddings and rerankers, vision, speech, image/vide

  2026-08-02 · 18 min read #llm#huggingface#open-weights#model-selection#quantization

• How a 26B Model Runs in 2GB of RAM — Resident Memory and Working Set Are Not the Same Number

  TurboFieldfare, posted to Show HN on July 29, 2026, claims to run Gemma 4 26B-A4B on M-series Macs in about 2GB of RAM. 14.3GB is installed on disk, the only thing resident in RAM is a 1.35GB shared core, and the expert

  2026-07-31 · 12 min read #ai#llm#quantization#apple-silicon#moe

• LLM Inference VRAM Math — the KV Cache Blows Up Before the Weights Do

  How to answer "will this model fit on our GPU" with a few multiplications. Weight memory is just parameter count times bytes per parameter, but the thing that actually blocks a deployment is the KV cache, which grows wit

  2026-07-26 · 12 min read #llm#inference#vram#kv-cache#quantization

• How Much VRAM to Run an LLM Locally — Compute It From Formulas, Not Tables ♪ Listenable

  The answer to "how many GB does an 8B model need?" is not a table but two formulas. Weights are parameter count times bpw divided by 8; the KV cache is 2 times layers times KV heads times headdim times bytes times tokens

  2026-07-17 · 24 min read #llm#quantization#kv-cache#local-llm#gpu

• Why the Default Vector Index Moved from HNSW to Disk — the Tradeoffs of Elasticsearch bbq_disk ♪ Listenable

  As of Elasticsearch 9.4 (2026-05-05), the default index type for float vectors changed to bbqdisk — a disk-based IVF index. In under a year the default moved from int8hnsw to bbqhnsw and then, a second time, to a disk in

  2026-07-16 · 23 min read #vector-database#elasticsearch#ann-index#quantization#hnsw

• Quantizing the KV Cache to 4 Bits — SAW-INT4 and System-Aware INT4

  In long-context LLM serving, the real memory bottleneck is not the weights but the KV cache. Quantize it naively to INT4 and accuracy collapses; the April 2026 preprint SAW-INT4 pairs token-wise INT4 quantization with a

  2026-07-11 · 6 min read #ai#llm#quantization#inference#kv-cache

• Running GLM-5.2 on a slow computer — how colibrì streams a 744B model from disk

  colibrì is a ~1,300-line pure-C inference engine that runs GLM-5.2, a 744B-parameter MoE model, on a consumer PC with 25GB of RAM. The trick is MoE sparsity plus disk streaming: only ~9.9GB of dense layers stay resident,

  2026-07-11 · 5 min read #ai#llm#local-inference#moe#quantization

• The State of LLM Quantization — From GPTQ and AWQ to FP8, MXFP4, and KV-Cache Quantization ♪ Listenable

  Quantization represents a model's numbers in fewer bits to cut memory and cost. Why quality survives fewer bits (outliers and scaling), how GPTQ and AWQ differ in approach, llama.cpp GGUF k-quants, QLoRA's NF4, the 2026

  2026-07-08 · 7 min read #ai#llm#quantization#inference#optimization

• The KV Cache and PagedAttention — Everything About Inference Memory

  A deep dive into the KV cache, the single biggest consumer of memory in LLM inference. We cover what the KV cache is and why it eats memory, the memory arithmetic and fragmentation problem, PagedAttention block managemen

  2026-06-26 · 12 min read #kv-cache#paged-attention#inference#gpu-memory#quantization

• Making Inference Fast — Quantization, Sparsity, and Dataflow from a Hardware Lens

  We break down the cost structure of inference through the memory-wall lens, then connect quantization (INT8/FP8/FP4), structured sparsity (2:4), dataflow architectures, operator fusion, batching and KV caching into one p

  2026-06-16 · 18 min read #inference#quantization#sparsity#dataflow#gpu

• The Memory Wall and HBM — The Real Bottleneck That Divides AI Performance ♪ Listenable

  In an era where compute is cheap and data movement is expensive, the real bottleneck of AI performance is memory. From the memory-wall concept to HBM generations, the roofline model and arithmetic intensity, the KV cache

  2026-06-16 · 21 min read #memory-wall#hbm#bandwidth#roofline#inference

• Edge AI and the NPU — On-Device Inference Accelerators ♪ Listenable

  We lay out why edge AI runs inference on the device instead of the cloud (latency, privacy, cost) and the concept of the NPU that makes it possible. From Apple Neural Engine, Qualcomm, Edge TPU, and ARM Ethos to model co

  2026-06-16 · 22 min read #edge-ai#npu#on-device#inference#quantization

• Local LLM Inference Optimization — From Quantization to Breaking the VRAM Ceiling ♪ Listenable

  Privacy concerns, cost pressure, and big-tech fatigue are driving a local LLM revival. We map the entire landscape of local inference optimization: VRAM-first hardware thinking, GGUF and AWQ quantization, llama.cpp vs vL

  2026-06-12 · 16 min read #llm#inference#quantization#llama-cpp#vllm

• LLM Serving & Local Inference in 2026 — vLLM / llama.cpp / MLX / Ollama / LM Studio / SGLang / TGI Deep Dive ♪ Listenable

  A map of the 2026 LLM serving and inference landscape. Datacenter camp (vLLM, SGLang, TGI, Triton, TensorRT-LLM), local camp (llama.cpp, MLX, llamafile, Ollama, LM Studio, GPT4All), emerging camp (KTransformers, MLC LLM,

  2026-05-16 · 25 min read #llm#model-serving#inference#vllm#llama-cpp

• Local AI & On-Device LLMs 2026 — Ollama · LM Studio · Jan · Msty · Open WebUI · GPT4All · AnythingLLM · Faraday Deep Dive ♪ Listenable

  By May 2026, local AI is no longer a hobby. An M4 Max MacBook Pro runs Llama 4 Scout 109B MoE at 24 tokens per second. Desktop runtimes like Ollama, LM Studio, Jan, and Msty unify GUI and CLI, while Open WebUI, AnythingL

  2026-05-16 · 23 min read #local-ai#on-device-llm#ollama#lm-studio#jan

• Complete LLM Inference Optimization Guide 2025: vLLM, TensorRT-LLM, KV Cache, Speculative Decoding ♪ Listenable

  Everything about LLM inference optimization! vLLM (PagedAttention), TensorRT-LLM (FP8/INT4), KV Cache management, Speculative Decoding, Continuous Batching, FlashAttention, Quantization (GPTQ/AWQ/GGUF), Model Serving (Tr

  2026-04-14 · 25 min read #llm-inference#vllm#tensorrt-llm#kv-cache#speculative-decoding