C21
Serving at Scale
5 lessons3.5 hoursPlannedFree
Outcome
Serve models with batching and reason about cost.
Prerequisites
- Required: C20 Model Packaging and Inference
Lessons
- L01Request batching and throughput · coming soonExercise: Measure throughput vs. batch size.
- L02vLLM and continuous batching · coming soonExercise: Serve a small LLM and load test it.
- L03Latency vs. cost tradeoffs · coming soonExercise: Compute cost per 1,000 requests.
- L04Capacity planning · coming soonExercise: Size a GPU fleet for a target load.
- L05Autoscaling and cold starts · coming soonExercise: Simulate a traffic spike.