Tag: #fsdp
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 5 posts
5 posts are tagged #fsdp; the most recent was published on 2026-08-02.
Often tagged together: #deepspeed 3 #distributed-training 3 #2026-03 2 #llm 2 #pytorch 2
Most read in the last 90 days:
- Distributed Training & GPU Infrastructure 2026 Deep-Dive — DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, Blackwell GB200, MI325X, TPU v5p
- LLM Fine-tuning Frameworks 2026 — A Deep Dive into Axolotl, Unsloth, LLaMA-Factory, TRL, PEFT, and TorchTune
- Torch-Titan Complete Guide: Everything About Large-Scale Distributed Training with PyTorch
The Four Kinds of Multi-GPU Parallelism — What You Split and What You Communicate
A numbers-first breakdown of what data parallelism, tensor parallelism, pipeline parallelism, and context parallelism each split, and what they pay in communication for it. It starts by building the per-parameter 16-byte
2026-08-02 · 13 min read #mlops#distributed-training#multi-gpu#fsdp#deepspeedLLM Fine-tuning Frameworks 2026 — A Deep Dive into Axolotl, Unsloth, LLaMA-Factory, TRL, PEFT, and TorchTune ♪ Listenable
A complete map of the 2026 LLM fine-tuning ecosystem. Open-source frameworks like Axolotl, Unsloth, LLaMA-Factory, TRL, PEFT, and TorchTune. LLM Foundry (MosaicML, acquired by Databricks). Cloud fine-tuning APIs from Mod
2026-05-16 · 28 min read #llm#finetuning#axolotl#unsloth#llama-factoryDistributed Training & GPU Infrastructure 2026 Deep-Dive — DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, Blackwell GB200, MI325X, TPU v5p
A comparison of DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, and Composer — plus NVIDIA Blackwell GB200 NVL72, AMD MI325X, Intel Gaudi 3, AWS Trainium 2, and Google TPU v5p/v6e Trillium. 3D parallelism, ZeR
2026-05-16 · 14 min read #distributed-training#deepspeed#fsdp#megatron-lm#rayTorch-Titan Complete Guide: Everything About Large-Scale Distributed Training with PyTorch
A complete guide to mastering large-scale LLM distributed training with PyTorch Titan (torchtitan). Covers FSDP2, pipeline parallelism, tensor parallelism, 4D parallelism, Flash Attention, and mixed precision with hands-
2026-03-17 · 19 min read #torch-titan#distributed-training#pytorch#fsdp#deep-learningPyTorch Internals & Advanced Optimization: autograd, torch.compile, FSDP, and Triton
A complete guide to PyTorch internals: autograd engine, torch.compile() with TorchInductor, FSDP distributed training, gradient checkpointing, and custom CUDA operations.
2026-03-17 · 10 min read #pytorch#torch-compile#fsdp#triton#mixedprecision