Tag: #deepspeed
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 6 posts
6 posts are tagged #deepspeed; the most recent was published on 2026-08-02.
Often tagged together: #distributed-training 4 #fsdp 3 #cuda 2 #english 2 #llm 2
Most read in the last 90 days:
- AI Inference Engines 2026 - vLLM · SGLang · llama.cpp · TGI · TensorRT-LLM · MLX · mistral.rs · DeepSpeed-MII · Aphrodite Deep Dive
- Distributed Training & GPU Infrastructure 2026 Deep-Dive — DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, Blackwell GB200, MI325X, TPU v5p
- LLM Fine-tuning Frameworks 2026 — A Deep Dive into Axolotl, Unsloth, LLaMA-Factory, TRL, PEFT, and TorchTune
The Four Kinds of Multi-GPU Parallelism — What You Split and What You Communicate
A numbers-first breakdown of what data parallelism, tensor parallelism, pipeline parallelism, and context parallelism each split, and what they pay in communication for it. It starts by building the per-parameter 16-byte
2026-08-02 · 13 min read #mlops#distributed-training#multi-gpu#fsdp#deepspeedLLM Fine-tuning Frameworks 2026 — A Deep Dive into Axolotl, Unsloth, LLaMA-Factory, TRL, PEFT, and TorchTune ♪ Listenable
A complete map of the 2026 LLM fine-tuning ecosystem. Open-source frameworks like Axolotl, Unsloth, LLaMA-Factory, TRL, PEFT, and TorchTune. LLM Foundry (MosaicML, acquired by Databricks). Cloud fine-tuning APIs from Mod
2026-05-16 · 28 min read #llm#finetuning#axolotl#unsloth#llama-factoryDistributed Training & GPU Infrastructure 2026 Deep-Dive — DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, Blackwell GB200, MI325X, TPU v5p
A comparison of DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, and Composer — plus NVIDIA Blackwell GB200 NVL72, AMD MI325X, Intel Gaudi 3, AWS Trainium 2, and Google TPU v5p/v6e Trillium. 3D parallelism, ZeR
2026-05-16 · 14 min read #distributed-training#deepspeed#fsdp#megatron-lm#rayAI Inference Engines 2026 - vLLM · SGLang · llama.cpp · TGI · TensorRT-LLM · MLX · mistral.rs · DeepSpeed-MII · Aphrodite Deep Dive
In 2026, LLM engineering is no longer about which model — it is about which inference engine. We dissect vLLM V1, SGLang 0.4, TensorRT-LLM, TGI 3.x, llama.cpp, MLX-LM, mistral.rs, DeepSpeed-MII, Aphrodite, CTranslate2, E
2026-05-16 · 20 min read #llm-inference#vllm#sglang#llama-cpp#tgiDeepSpeed Complete Guide: ZeRO Optimization and Large-Scale Model Training
A complete guide to mastering Microsoft DeepSpeed. Learn ZeRO-1/2/3 optimization, Offload, pipeline parallelism, mixed precision, MoE, and DeepSpeed Inference with practical configurations and code examples.
2026-03-17 · 15 min read #deepspeed#zero-optimization#distributed-training#llm#pytorchMastering Slurm: A Practical Guide to the HPC/AI Cluster Workload Manager
A comprehensive, hands-on guide to the Slurm workload manager. Covers architecture (slurmctld/slurmd/slurmdbd), core concepts (Partitions/QoS/Fairshare), essential commands (sbatch/srun/salloc), GPU scheduling (GRES/MIG/
2026-03-01 · 15 min read #slurm#hpc#gpu#distributed-training#cluster