C29
Distributed Training
8 lessons4 hoursPlannedFree
Outcome
Train models that don’t fit on one GPU, choose a parallelism strategy, and measure scaling efficiency.
Prerequisites
- Required: C10 Training in Practice
- Recommended: C11 Transformers and Attention from Scratch
Lessons
- L01Memory math: parameters, gradients, optimizer states, activations · coming soonExercise: Estimate GPU memory to train a 7B model with Adam in mixed precision.
- L02Collective communication: all-reduce, all-gather, reduce-scatter · coming soonExercise: Simulate ring all-reduce and compute its communication cost.
- L03Data parallelism with PyTorch DDP · coming soonExercise: Train with DDP on two processes and verify the weights stay identical.
- L04Sharding with ZeRO and FSDP · coming soonExercise: Compare per-GPU memory for DDP vs. FSDP on the same model.
- L05Tensor and pipeline parallelism · coming soonExercise: Split an MLP across two devices column-wise and row-wise and check outputs match.
- L06Activation checkpointing and gradient accumulation · coming soonExercise: Trade compute for memory and measure both.
- L07Checkpointing, fault tolerance, and elastic training · coming soonExercise: Kill a worker mid-run and resume from the last checkpoint.
- L08Measuring scaling efficiency and MFU · coming soonExercise: Compute MFU and scaling efficiency from a multi-GPU run.