Welcome to our KDD’25 Tutorial, “Enhancing Reasoning in LLMs through Distillation and Reinforcement Learning”.

Website: Reasoning Model Tutorial

Tutorial Abstract

This hands-on tutorial provides comprehensive guidance on generating reasoning capabilities in large language models (LLMs) through two primary approaches: knowledge distillation from advanced reasoning models and post-training with reinforcement learning (RL). We demonstrate how to distill reasoning abilities from state-of-the-art models like DeepSeek R1 into smaller, more efficient open-source models such as Qwen and Llama. Additionally, we explore how RL techniques can be applied to further enhance reasoning capabilities. The tutorial includes step-by-step demonstrations using Jupyter notebooks that cover extracting reasoning data from advanced models via APIs, fine-tuning with distilled data, and implementing RL strategies. Participants will gain hands-on experience with implementing these techniques to enhance reasoning capabilities in LLMs, with applications across various domains requiring logical reasoning, mathematical problem-solving, and scientific analysis.

Target Audience and Prerequisites

Target audience: Data scientists, machine learning engineers, and researchers interested in enhancing reasoning capabilities in LLMs.

Prerequisites: Basic knowledge of machine learning concepts, familiarity with Python programming, and some experience with training or fine-tuning neural networks. Participants should have a working understanding of LLMs and their applications.

Tutorial Outline

1. Introduction to Reasoning in LLMs (10 minutes)
   • Brief overview of reasoning capabilities and their importance
   • Quick summary of knowledge distillation and reinforcement learning approaches
2. Core Concepts of Reasoning Enhancement (20 minutes)
   • Key architectures of reasoning models
   • Fundamentals of distillation and RL for reasoning
   • Evaluation metrics for reasoning capabilities
3. Hands-on: Setting Up the Environment (20 minutes)
   • Installing required libraries and frameworks
   • Preparing datasets and model access
   • Quick verification of setup
4. Hands-on: Extracting Reasoning Data (40 minutes)
   • Using APIs to access various reasoning models
   • Generating long-CoT data
   • Data preprocessing for distillation
5. Hands-on: Distillation Implementation (60 minutes)
   • Implementing distillation in Qwen and Llama
   • Monitoring training progress
   • Evaluating reasoning improvements
6. Hands-on: RL Post-training (60 minutes)
   • Setting up RL environments
   • Implementing reward modeling
   • Fine-tuning models with RL
   • Comparing results with distillation approach
7. Applications, Limitations, and Future Directions (15 minutes)
   • Real-world use cases across domains
   • Current limitations and ethical considerations
   • Future research directions

Key References Published by Lecturers

1. DeepSeek R1 paper: "DeepSeek R1: An Open-Source LLM for Reasoning"
2. NeMo 2.0 framework documentation: https://nvidia.github.io/NeMo/
3. Relevant research on knowledge distillation in LLMs: [List 3-5 key papers]
4. RL for LLM training: [List 3-5 key papers]

Resource Requirements

• Jupyter notebooks (generate_reasoning_data.ipynb, qwen2_distill_nemo.ipynb, and rl_post_training.ipynb)
• APIs for accessing various reasoning models (free tiers)
• NeMo 2.0 framework (open-source)
• Sample datasets for distillation and RL training
• Open-source models (Qwen, Llama)

Short Bio of Tutors