C15
RLHF Basics
5 lessons3 hoursPlannedFree
Outcome
Explain and implement the core pieces of RLHF.
Prerequisites
- Required: C14 Policy Gradients and PPO
- Required: C12 Fine-Tuning with LoRA and PEFT
Lessons
- L01Preference data and pairwise comparisons · coming soonExercise: Build a preference dataset.
- L02Reward modeling · coming soonExercise: Train a small reward model.
- L03KL penalty and reference models · coming soonExercise: Explain why it prevents drift.
- L04DPO as a simpler alternative · coming soonExercise: Compare the loss to RLHF.
- L05Evaluating aligned models · coming soonExercise: Run a side-by-side evaluation.