Tag: #local-llm
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 5 posts
5 posts are tagged #local-llm; the most recent was published on 2026-08-14.
Often tagged together: #apple-silicon 2 #inference 2 #llm 2 #quantization 2 #ai-desktop 1
Most read in the last 90 days:
The Hybrid Attention in Qwen3.8-27B — Only 16 of 64 Layers Grow a KV Cache ♪ Listenable
The reason a 27B model can carry a 262K-token context on a laptop is not its parameter count but its layer layout. Qwen3.8-27B places 48 of its 64 layers as linear attention (Gated DeltaNet) and only 16 as ordinary atten
2026-08-14 · 8 min read #llm#inference#quantization#local-llm#attentionHow Much VRAM to Run an LLM Locally — Compute It From Formulas, Not Tables ♪ Listenable
The answer to "how many GB does an 8B model need?" is not a table but two formulas. Weights are parameter count times bpw divided by 8; the KV cache is 2 times layers times KV heads times headdim times bytes times tokens
2026-07-17 · 24 min read #llm#quantization#kv-cache#local-llm#gpuWhy the Mac mini Became an On-Device AI Machine — What Apple's Silicon Exec Said, and What He Left Out
Apple Silicon senior product manager Doug Brooks talked to The Deep View about demand for the Mac mini and Mac Studio and where on-device AI is heading. Why developers and small teams reach for this little desktop as a l
2026-07-11 · 6 min read #apple-silicon#on-device-ai#local-llm#mac-mini#inferenceMLX Deep Dive — Apple's ML Framework for Apple Silicon: Unified Memory, Lazy Graphs, and the Mac-Native Flow (2026 Hands-On) ♪ Listenable
MLX is the array framework that Apple's ML team — the same people who previously built PyTorch and JAX — rewrote from scratch for Apple Silicon. The core thesis is one sentence: the M-series GPU shares RAM with the CPU,
2026-05-14 · 19 min read #mlx#apple-silicon#ml-framework#unified-memory#metalAI Desktop Apps in 2026 — A Snapshot of Granola, Cleft, Lex, Highlight, Raycast AI, Ollama, and the Quiet Rise of the Ambient-AI Category ♪ Listenable
Spring 2026: the center of gravity for everyday AI use has quietly moved from chatbot tabs to desktop apps. Granola for meeting notes, Cleft / Superwhisper / MacWhisper for local dictation, Lex for AI-first writing, High
2026-05-14 · 36 min read #ai-desktop#granola#cleft#lex-app#highlight-ai