Tag: #distributed-training

← All posts

• The Four Kinds of Multi-GPU Parallelism — What You Split and What You Communicate

  A numbers-first breakdown of what data parallelism, tensor parallelism, pipeline parallelism, and context parallelism each split, and what they pay in communication for it. It starts by building the per-parameter 16-byte

  2026-08-02 · 13 min read #mlops#distributed-training#multi-gpu#fsdp#deepspeed

• Running a GPU Cluster with Slurm — Knowing Why a Job Will Not Run Matters More Than Submitting

  Everything you need to actually use Slurm on a GPU cluster in practice. Sets up the coordinate system of partition, QoS, and account first, then covers how to request GPUs, CPUs, and memory in an sbatch script and the bi

  2026-08-02 · 14 min read #mlops#slurm#hpc#gpu-cluster#distributed-training

• torchcomms in PyTorch 2.13 — The New Distributed Communication Backend Coming for c10d

  Among the highlights of PyTorch 2.13, released on July 8, 2026, the most structural change is torchcomms — PyTorch Distributed's new communication backend has begun landing in core's CI and DeviceMesh paths. torchcomms i

  2026-07-17 · 10 min read #pytorch#distributed-training#nccl#deep-learning

• The Workload/PodGroup API in Kubernetes v1.36 — Gang Scheduling Is Moving Into kube-scheduler ♪ Listenable

  Gang scheduling for AI training and batch workloads has so far been the job of external schedulers like Volcano or Kueue, but Kubernetes has started pulling this capability into its core. v1.35 shipped the Workload API a

  2026-07-16 · 19 min read #kubernetes#scheduling#gang-scheduling#distributed-training#dra

• Multi-GPU, Multi-Node Training Platforms: The Complete Map — from the Framework Ecosystem to Hands-On Slurm and Kubeflow Guides

  A one-page map of the full landscape of training models across multiple GPUs and multiple nodes. The AI library and framework ecosystem map (PyTorch, JAX, HuggingFace, DeepSpeed, Ray), when to pick which parallelization

  2026-07-09 · 8 min read #ai#ml#distributed-training#slurm#kubeflow

• AI Interconnect — NVLink, NVSwitch, UALink, and the Art of Scaling Up ♪ Listenable

  In large-scale AI training and inference, the real bottleneck is not compute but communication. This article walks through the principles and practice of AI interconnect — the scale-up domains built by NVLink and NVSwitc

  2026-06-16 · 29 min read #gpu-cuda#nvlink#nvswitch#ualink#interconnect

• Distributed Training & GPU Infrastructure 2026 Deep-Dive — DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, Blackwell GB200, MI325X, TPU v5p

  A comparison of DeepSpeed, FSDP2, Megatron-LM, Ray Train, JAX, TorchTitan, and Composer — plus NVIDIA Blackwell GB200 NVL72, AMD MI325X, Intel Gaudi 3, AWS Trainium 2, and Google TPU v5p/v6e Trillium. 3D parallelism, ZeR

  2026-05-16 · 14 min read #distributed-training#deepspeed#fsdp#megatron-lm#ray

• Torch-Titan Complete Guide: Everything About Large-Scale Distributed Training with PyTorch

  A complete guide to mastering large-scale LLM distributed training with PyTorch Titan (torchtitan). Covers FSDP2, pipeline parallelism, tensor parallelism, 4D parallelism, Flash Attention, and mixed precision with hands-

  2026-03-17 · 19 min read #torch-titan#distributed-training#pytorch#fsdp#deep-learning

• Federated Learning Complete Guide: Privacy-Preserving Distributed AI ♪ Listenable

  A complete guide to mastering Federated Learning. Learn how to train distributed models while preserving data privacy, covering FedAvg, FedProx, differential privacy, and the Flower framework with hands-on code.

  2026-03-17 · 23 min read #federated-learning#privacy#distributed-training#differential-privacy#ai

• DeepSpeed Complete Guide: ZeRO Optimization and Large-Scale Model Training

  A complete guide to mastering Microsoft DeepSpeed. Learn ZeRO-1/2/3 optimization, Offload, pipeline parallelism, mixed precision, MoE, and DeepSpeed Inference with practical configurations and code examples.

  2026-03-17 · 15 min read #deepspeed#zero-optimization#distributed-training#llm#pytorch

• Large-Scale Model Training Complete Guide: Strategies for Pre-training 100B+ Parameter LLMs ♪ Listenable

  A complete practical guide to training hundred-billion parameter LLMs. Covers scaling laws (Chinchilla), Megatron-LM, 3D parallelism, checkpointing strategies, training stability, and data mixing with hands-on examples.

  2026-03-17 · 21 min read #large-scale-training#llm#megatron-lm#distributed-training#scaling-laws

• Deep Learning Training Methods Complete Guide: From Optimization to Distributed Training ♪ Listenable

  A comprehensive guide covering all techniques for effectively training deep learning models. Learn gradient descent, optimizers, learning rate scheduling, regularization, batch normalization, transfer learning, fine-tuni

  2026-03-17 · 28 min read #deep-learning#training#optimization#regularization#distributed-training

• Ring Attention Paper Analysis: Implementing Infinite Context Window Training in Distributed Environments ♪ Listenable

  Analyzes the Ring Attention paper exploring methods to overcome context length limitations in distributed environments. Covers the connection with Blockwise Parallel Transformer, implementation details, performance bench

  2026-03-08 · 33 min read #ai-papers#ring-attention#distributed-training#long-context#transformer

• Complete Guide to Multi-GPU Distributed Training: DDP, FSDP, DeepSpeed ♪ Listenable

  Systematically analyze the core components of multi-GPU distributed training including DDP, FSDP, and DeepSpeed ZeRO based on PyTorch official documentation, with practical setup instructions.

  2026-03-01 · 21 min read #gpu#cuda#distributed-training#deep-learning#pytorch

• Mastering Slurm: A Practical Guide to the HPC/AI Cluster Workload Manager

  A comprehensive, hands-on guide to the Slurm workload manager. Covers architecture (slurmctld/slurmd/slurmdbd), core concepts (Partitions/QoS/Fairshare), essential commands (sbatch/srun/salloc), GPU scheduling (GRES/MIG/

  2026-03-01 · 15 min read #slurm#hpc#gpu#distributed-training#cluster

• Kubernetes AI Training Pipeline: Analyzing Volcano, Training Operator, and Kueue ♪ Listenable

  A deep dive into building distributed AI training pipelines on Kubernetes based on the official documentation of Volcano, Kubeflow Training Operator, and Kueue.

  2026-03-01 · 21 min read #kubernetes#ai#distributed-training#volcano#kubeflow