C30
Distributed Inference
7 lessons4 hoursPlannedFree
Outcome
Serve large models across GPUs and nodes, and route traffic across replicas efficiently.
Prerequisites
- Required: C21 Serving at Scale
- Recommended: C29 Distributed Training
Lessons
- L01Inference memory math: weights and KV cache · coming soonExercise: Compute the maximum concurrent sequences for a model on one GPU.
- L02PagedAttention and prefix caching · coming soonExercise: Measure the gain from prefix caching on requests with a shared system prompt.
- L03Tensor and pipeline parallel inference across GPUs · coming soonExercise: Serve a model with tensor parallelism and compare latency and throughput.
- L04Prefill/decode disaggregation · coming soonExercise: Model throughput and latency with and without disaggregation.
- L05Speculative decoding · coming soonExercise: Measure acceptance rate and speedup with a draft model.
- L06Multi-model and multi-LoRA serving · coming soonExercise: Serve several LoRA adapters on one base model and load test it.
- L07Routing across replicas: load-aware and cache-aware · coming soonExercise: Compare round-robin and cache-aware routing in a simulation.