1 00:00:00,040 --> 00:00:02,880 Welcome back to the Deep Dive. We're glad to have you with us. 2 00:00:03,280 --> 00:00:06,280 Today we are tackling a subject that feels a bit like looking 3 00:00:06,280 --> 00:00:09,280 under the hood of a Ferrari, only to find out the engine is 4 00:00:09,280 --> 00:00:13,160 being driven by a very polite, very specific set of traffic 5 00:00:13,160 --> 00:00:15,440 laws. We're looking at the Practical 6 00:00:15,440 --> 00:00:18,880 AI Digest edition of our show, and the focus is squarely on the 7 00:00:18,880 --> 00:00:20,600 architecture that really changed everything. 8 00:00:20,840 --> 00:00:23,320 Instruct GPT. It's great to be here. 9 00:00:23,320 --> 00:00:25,560 And you know, that Ferrari analogy isn't too far off 10 00:00:25,560 --> 00:00:27,040 actually. We are talking about some 11 00:00:27,040 --> 00:00:28,520 serious high performance machinery. 12 00:00:28,520 --> 00:00:31,520 But the steering mechanism, the thing that makes it useful for 13 00:00:31,520 --> 00:00:34,440 an actual driver, that's what we're dissecting today, 14 00:00:34,440 --> 00:00:36,840 specifically reinforcement learning from human feedback. 15 00:00:36,840 --> 00:00:39,160 Or you know, the famous acronym RLHF. 16 00:00:39,160 --> 00:00:41,680 Right, RLHF. And before we get into the nuts 17 00:00:41,680 --> 00:00:44,360 and bolts, and there are a lot of them today, we really need to 18 00:00:44,360 --> 00:00:47,360 set the stage with the problem because before instruct GPT, we 19 00:00:47,360 --> 00:00:51,840 had GPT 3 and well, GPT 3. Was this just technological 20 00:00:51,840 --> 00:00:54,280 marvel? It had a bit of a personality 21 00:00:54,280 --> 00:00:56,000 problem. We call this the misaligned 22 00:00:56,000 --> 00:00:58,800 genius paradox. Oh yeah, it's a paradox that 23 00:00:58,800 --> 00:01:02,200 anyone who used those early API's remembers vividly. 24 00:01:02,600 --> 00:01:05,840 I mean, you have a model like GPT 3 that has read essentially 25 00:01:05,840 --> 00:01:08,320 the entire public Internet. It knows everything. 26 00:01:08,560 --> 00:01:11,600 But if you ask the original GPT 3 to write a recipe for 27 00:01:11,600 --> 00:01:15,160 chocolate cake, well, it might not actually give you a recipe. 28 00:01:15,160 --> 00:01:18,280 No, I remember this so clearly. You'd type that in and it might 29 00:01:18,280 --> 00:01:20,400 just complete the sentence that say something like write a 30 00:01:20,400 --> 00:01:22,800 recipe for chocolate cake and then eat it. 31 00:01:23,440 --> 00:01:25,600 Or it would just list ingredients with no 32 00:01:25,600 --> 00:01:28,200 instructions. Or worse, it might just start 33 00:01:28,200 --> 00:01:31,560 reciting a Wikipedia article about the history of cocoa 34 00:01:31,560 --> 00:01:33,480 beans. Which, to be fair, is 35 00:01:33,480 --> 00:01:37,240 technically correct English. It's grammatically perfect, but 36 00:01:37,240 --> 00:01:39,880 it is totally useless if you're standing in your kitchen holding 37 00:01:39,880 --> 00:01:41,720 a whisk. So why did it do that? 38 00:01:42,040 --> 00:01:44,040 I mean, if it's so smart, why couldn't it understand the 39 00:01:44,040 --> 00:01:46,320 simple command? And that brings us to the 40 00:01:46,320 --> 00:01:48,680 absolute core concept you have to understand. 41 00:01:48,680 --> 00:01:51,920 It's this objective mismatch. This is the root tension in 42 00:01:51,920 --> 00:01:55,040 these large language models. You have to remember what pre 43 00:01:55,040 --> 00:01:58,120 training actually is. That massive phase where the 44 00:01:58,120 --> 00:02:02,680 model reads the Internet, it optimizes for one and only one 45 00:02:02,680 --> 00:02:05,520 specific thing, predicting the next token. 46 00:02:05,720 --> 00:02:08,600 It's a completion engine. It is not by default an 47 00:02:08,600 --> 00:02:11,800 instruction following engine. OK, so when I type write a 48 00:02:11,800 --> 00:02:13,880 recipe, the model isn't thinking. 49 00:02:13,880 --> 00:02:17,080 Oh the user wants instructions. It's thinking based on all the 50 00:02:17,080 --> 00:02:20,000 text I've ever seen. What words usually come after 51 00:02:20,000 --> 00:02:23,160 this exact phrase? Precisely, and on the Internet, 52 00:02:23,160 --> 00:02:26,280 a phrase like write a recipe might be a header on a blog or a 53 00:02:26,280 --> 00:02:29,240 form title, so it just predicts the next likely text. 54 00:02:29,440 --> 00:02:32,040 The objective of predict the next word on a web page is 55 00:02:32,040 --> 00:02:34,440 fundamentally different from helpfully follow this user 56 00:02:34,440 --> 00:02:36,800 specific request. So the model wasn't broken, it 57 00:02:36,800 --> 00:02:40,320 was just misunderstood. Or rather, it misunderstood us. 58 00:02:40,440 --> 00:02:42,920 It was misaligned. And that is the whole mission of 59 00:02:42,920 --> 00:02:45,000 today's deep dive. We're going to break down 60 00:02:45,000 --> 00:02:48,360 exactly how open AI solve this with instruct GPT. 61 00:02:48,600 --> 00:02:50,720 And here's the kicker, The thing that really shocked me when I 62 00:02:50,720 --> 00:02:53,600 went back through the papers, we aren't talking about making the 63 00:02:53,600 --> 00:02:56,360 models bigger. In fact, spoiler alert, we're 64 00:02:56,360 --> 00:03:01,080 going to see that a 1.3 billion parameter instruct GPT model, 65 00:03:01,080 --> 00:03:04,200 which is, you know, relatively tiny in this world, can actually 66 00:03:04,200 --> 00:03:08,560 outperform the massive 175 billion parameter GPT. 67 00:03:08,840 --> 00:03:10,200 See. That is the part that blows my 68 00:03:10,200 --> 00:03:11,920 mind. It just disrupts that whole 69 00:03:12,080 --> 00:03:14,600 bigger is better narrative that we heard for years. 70 00:03:15,000 --> 00:03:17,640 It's not about size, it's about the training methodology. 71 00:03:17,800 --> 00:03:19,400 It is all about the objective function. 72 00:03:19,400 --> 00:03:23,000 We are going to dissect the methodology known as RLHF, and 73 00:03:23,000 --> 00:03:25,200 it breaks down into three very distinct steps. 74 00:03:25,200 --> 00:03:27,360 If you're an engineer building your own models, or you're just 75 00:03:27,360 --> 00:03:30,280 trying to understand how ChatGPT actually works, this is your 76 00:03:30,280 --> 00:03:32,480 road map. Step one is supervised fine 77 00:03:32,480 --> 00:03:35,840 tuning or SFT, Step 2 is training a reward model, and 78 00:03:35,840 --> 00:03:38,520 Step 3 is the actual reinforcement learning loop 79 00:03:38,520 --> 00:03:40,840 using PO. OK, let's unpack this 80 00:03:40,840 --> 00:03:43,800 systematically. Step one, supervised fine tuning 81 00:03:44,000 --> 00:03:45,920 the warm up. The warm up is a really good way 82 00:03:45,920 --> 00:03:48,840 to put it, because before you can use reinforcement learning, 83 00:03:48,840 --> 00:03:52,880 which is a pretty complex optimization process, you need a 84 00:03:52,880 --> 00:03:56,520 model that at least vaguely understands what an instruction 85 00:03:56,520 --> 00:03:58,800 even looks like. You can't just start from a raw 86 00:03:58,800 --> 00:04:01,800 completion engine. You're facing what's basically a 87 00:04:01,800 --> 00:04:04,280 cold start problem. Right, because the model has 88 00:04:04,280 --> 00:04:08,640 never seen a prompt like explain quantum physics to a 5 year old 89 00:04:08,960 --> 00:04:11,400 followed by an actual good explanation. 90 00:04:11,480 --> 00:04:16,279 It's only seen raw Internet text, so where do they get this 91 00:04:16,279 --> 00:04:18,880 initial training data? It didn't just exist naturally. 92 00:04:18,880 --> 00:04:22,200 They had to build it Open AI had to go out and hire about 40 93 00:04:22,200 --> 00:04:24,200 contractors. And I want to emphasize this 94 00:04:24,200 --> 00:04:26,640 wasn't just some random crowdsourcing on Mechanical 95 00:04:26,640 --> 00:04:28,120 Turk. These people were screened. 96 00:04:28,360 --> 00:04:30,200 They looked for high performance, but also 97 00:04:30,400 --> 00:04:33,640 sensitivity to demographic groups and an ability to 98 00:04:33,640 --> 00:04:36,400 identify harmful content. They needed really high quality 99 00:04:36,400 --> 00:04:39,360 labelers because you know, garbage in, garbage out and 100 00:04:39,360 --> 00:04:39,960 what? Exactly. 101 00:04:39,960 --> 00:04:41,680 Were these 40 people writing all day? 102 00:04:41,840 --> 00:04:43,880 They were creating the ground truth. 103 00:04:44,360 --> 00:04:47,280 They produced a data set of prompts and crucially, the 104 00:04:47,280 --> 00:04:50,000 desired outputs. They were working with three 105 00:04:50,000 --> 00:04:53,680 main types of data sources here. First, there were labeler 106 00:04:53,680 --> 00:04:55,880 written prompts. They just asked the contractors 107 00:04:55,880 --> 00:04:59,120 to come up with plain tasks. You know, open-ended questions, 108 00:04:59,120 --> 00:05:02,480 brainstorming lists. Tell me a story about a frog or 109 00:05:02,800 --> 00:05:05,920 list five ways to save money. OK, so just generating a bunch 110 00:05:05,920 --> 00:05:08,800 of basic tasks. Right then second they had few 111 00:05:08,800 --> 00:05:10,800 shot variations. This is where they give the 112 00:05:10,800 --> 00:05:13,760 model and instruction and a few examples of how to solve it, and 113 00:05:13,760 --> 00:05:17,080 then ask the labelers to write the final perfect response. 114 00:05:17,080 --> 00:05:19,680 And what was the third source? The third source is fascinating 115 00:05:19,680 --> 00:05:20,960 because it came from the real world. 116 00:05:21,400 --> 00:05:24,040 They took actual prompts submitted to the early Open 117 00:05:24,040 --> 00:05:27,000 AIAPI, filtered for any personally identifiable 118 00:05:27,000 --> 00:05:30,320 information of course, and had the labelers write the perfect 119 00:05:30,320 --> 00:05:32,480 response to those real world queries. 120 00:05:32,640 --> 00:05:36,040 So a human looks at a prompt like list 5 ideas for birthday 121 00:05:36,040 --> 00:05:39,000 party and they actually sit down and write out a good list 1 122 00:05:39,160 --> 00:05:42,600 bowling, 2 pizza and so on. Exactly. 123 00:05:42,600 --> 00:05:46,280 They're creating demonstrations. This is supervised learning in 124 00:05:46,280 --> 00:05:49,520 its purest form. You feed the prompt and the 125 00:05:49,520 --> 00:05:52,840 human written answer into the model and you say copy this 126 00:05:52,840 --> 00:05:55,440 behavior. You are literally showing the 127 00:05:55,440 --> 00:05:57,600 model. When you see an instruction, you 128 00:05:57,600 --> 00:06:00,000 don't just complete the sentence, you answer the 129 00:06:00,000 --> 00:06:02,120 question. OK, let's get technical for a 130 00:06:02,120 --> 00:06:04,600 second. For the engineers listening who 131 00:06:04,600 --> 00:06:07,440 might want to replicate this or just understand the mechanics, 132 00:06:07,680 --> 00:06:10,160 what does this training route actually look like? 133 00:06:10,400 --> 00:06:14,760 So they took a pre trained GPT 3 model and fine-tuned it on this 134 00:06:14,760 --> 00:06:18,200 new demonstration data. They trained it for 16 epochs. 135 00:06:18,600 --> 00:06:22,280 They used a cosine learning rate decay, which is pretty standard 136 00:06:22,280 --> 00:06:25,160 for stabilizing convergence as you get closer to the optimal 137 00:06:25,160 --> 00:06:28,720 weights, and they used a residual drop out of .2. 138 00:06:28,840 --> 00:06:32,280 Wait 16 epochs? That seems like a lot for fine 139 00:06:32,280 --> 00:06:34,320 tuning. Usually when you fine tune on a 140 00:06:34,320 --> 00:06:37,040 small data set like that, you really worry about overfitting 141 00:06:37,040 --> 00:06:39,040 right? Where the model just memorizes 142 00:06:39,040 --> 00:06:41,560 the training data and then can't generalize to new things. 143 00:06:41,720 --> 00:06:44,640 That is a fantastic point, and it leads to one of the most 144 00:06:44,640 --> 00:06:47,240 interesting nuances in the whole paper. 145 00:06:47,480 --> 00:06:49,880 They actually found that the model did overfit on the 146 00:06:49,880 --> 00:06:52,480 validation loss after just one epoch. 147 00:06:52,520 --> 00:06:55,400 Hold on, so the validation loss, the main metric that tells you 148 00:06:55,400 --> 00:06:57,400 if the model is learning general patterns. 149 00:06:57,760 --> 00:07:00,760 It started getting worse after only one epoch, yes. 150 00:07:01,120 --> 00:07:03,880 In a traditional ML workflow, you would stop training 151 00:07:03,880 --> 00:07:05,760 immediately. You'd say, OK, we're overfitting 152 00:07:05,760 --> 00:07:07,880 the model is memorizing the test answers, cut it. 153 00:07:08,360 --> 00:07:11,040 But here is where it gets really, really interesting. 154 00:07:11,360 --> 00:07:15,080 Even though the validation loss was spiking the human preference 155 00:07:15,080 --> 00:07:18,400 scores, the actual rating of how good the answers were from a 156 00:07:18,400 --> 00:07:21,640 person continued to improve. That is so counterintuitive. 157 00:07:21,640 --> 00:07:25,840 The math is screaming stop, but the product quality says keep 158 00:07:25,840 --> 00:07:27,000 going. Exactly. 159 00:07:27,480 --> 00:07:30,160 It suggests that the validation metrics we traditionally use for 160 00:07:30,160 --> 00:07:32,640 language modeling like perplexity, which measures how 161 00:07:32,640 --> 00:07:36,080 surprised the model is by text. They aren't perfectly correlated 162 00:07:36,080 --> 00:07:37,960 with what a human perceives as quality. 163 00:07:38,200 --> 00:07:41,080 So they just ignored the validation loss spike and kept 164 00:07:41,080 --> 00:07:44,720 training for all 16-8 box. Wow, living dangerously. 165 00:07:44,720 --> 00:07:47,080 I love it. So at the end of step one we 166 00:07:47,080 --> 00:07:50,840 have the SFT model. It's better than raw GPT 3 at 167 00:07:50,840 --> 00:07:53,880 following instructions, but it's still pretty limited by the fact 168 00:07:53,880 --> 00:07:56,280 that it's just mimicking a small group of humans. 169 00:07:56,480 --> 00:07:58,840 Correct, it's the baseline policy. 170 00:07:59,120 --> 00:08:02,920 But stealing this is impossible. You just cannot hire humans to 171 00:08:02,920 --> 00:08:06,880 write a perfect paragraph for every possible query in the 172 00:08:06,880 --> 00:08:08,800 universe. It's way too slow and way too 173 00:08:08,800 --> 00:08:11,040 expensive. You hit a bottleneck really 174 00:08:11,040 --> 00:08:13,240 fast. Which brings us perfectly to 175 00:08:13,240 --> 00:08:17,400 Step 2, the reward model, or, as I like to call it, building the 176 00:08:17,400 --> 00:08:19,240 Critic. This is the real pivot point. 177 00:08:19,240 --> 00:08:22,400 The realization is that it's much, much faster for a human to 178 00:08:22,400 --> 00:08:24,680 rank outputs than to write them from scratch. 179 00:08:25,240 --> 00:08:27,200 Look, if I ask you to write a sonnet about a toaster, that 180 00:08:27,200 --> 00:08:29,040 might take you 10 minutes of creative effort. 181 00:08:29,320 --> 00:08:32,080 If I show you 2 sonnets about toasters and just ask which one 182 00:08:32,080 --> 00:08:34,440 is better, you can probably decide in 10 seconds. 183 00:08:34,440 --> 00:08:36,480 So we switch from creation to curation. 184 00:08:36,480 --> 00:08:39,280 We move from writing to judging. How does the data collection 185 00:08:39,280 --> 00:08:41,960 work for this part? So the system presents a labeler 186 00:08:42,159 --> 00:08:45,360 with one single prompt and then K different model outputs. 187 00:08:45,840 --> 00:08:47,840 In the paper. K is usually somewhere between 188 00:08:47,840 --> 00:08:51,800 4:00 and 9:00, so the model generates A9 different versions 189 00:08:51,800 --> 00:08:54,360 of an answer. So one question, 9 different 190 00:08:54,360 --> 00:08:57,520 answers generated by the model and the labeler just picks the 191 00:08:57,520 --> 00:08:59,920 best one. Not just the best one, they rank 192 00:08:59,920 --> 00:09:02,880 them from best to worst one through 9. 193 00:09:04,120 --> 00:09:06,600 That seems straightforward enough, but how do you train a 194 00:09:06,600 --> 00:09:10,040 model on rankings? I mean, a neural network usually 195 00:09:10,040 --> 00:09:12,040 wants a target number like a score. 196 00:09:12,280 --> 00:09:14,520 It wants to know this is a 7 out of 10. 197 00:09:14,640 --> 00:09:17,240 This is where they get into a really clever efficiency hack. 198 00:09:17,600 --> 00:09:21,560 They train a reward model or RM to predict which output a human 199 00:09:21,560 --> 00:09:24,720 would prefer, but instead of running a forward pass for every 200 00:09:24,720 --> 00:09:27,720 single pair of answers which would be computationally heavy, 201 00:09:28,040 --> 00:09:31,240 they treat all the comparisons from a single prompt as one 202 00:09:31,240 --> 00:09:32,880 batch. OK, walk me through that math. 203 00:09:32,880 --> 00:09:35,360 If I have 9 outputs, how many comparisons is that? 204 00:09:35,880 --> 00:09:38,200 It's a combinatorics problem. It's K choose 2. 205 00:09:38,200 --> 00:09:41,000 So if you have 9 outputs, that's 36 unique pairs. 206 00:09:41,040 --> 00:09:43,040 You know, output one versus output 2, one versus 3, two 207 00:09:43,040 --> 00:09:46,000 versus 3, and so on. If you process those separately, 208 00:09:46,000 --> 00:09:47,880 you'd be running the model 36 times. 209 00:09:48,440 --> 00:09:51,520 By doing it as a batch. They only run the forward pass 210 00:09:51,680 --> 00:09:55,280 once per completion, and then they calculate the loss for all 211 00:09:55,280 --> 00:09:57,400 36 pairs at the same time. It's a. 212 00:09:57,400 --> 00:10:00,280 Huge compute saving. You're essentially getting 36 213 00:10:00,280 --> 00:10:03,360 pieces of training data for the price of nine generations. 214 00:10:03,640 --> 00:10:07,440 A massive efficiency game, and for the loss function itself 215 00:10:07,440 --> 00:10:10,800 they use a cross entropy loss that's based on the log odds. 216 00:10:11,320 --> 00:10:14,600 Basically, the reward model outputs A scaler score, just a 217 00:10:14,600 --> 00:10:16,320 single number for each piece of text. 218 00:10:16,800 --> 00:10:19,280 The loss function then tries to maximize the gap between the 219 00:10:19,280 --> 00:10:21,440 score of the winning text and the losing text. 220 00:10:21,520 --> 00:10:24,560 So the reward model is learning to assign a high number to the 221 00:10:24,560 --> 00:10:27,680 text the human liked, and a low number to the text the human 222 00:10:27,680 --> 00:10:28,960 hated. Exactly. 223 00:10:29,000 --> 00:10:31,240 It becomes a numerical proxy for human taste. 224 00:10:31,320 --> 00:10:34,080 It's an automated critic. Now I noticed a detail in the 225 00:10:34,080 --> 00:10:36,040 architecture choice here that I really want to flag. 226 00:10:36,600 --> 00:10:41,360 The main GPT 3 model is 175 billion parameters, but for the 227 00:10:41,360 --> 00:10:44,480 reward model they used the 6 billion parameter model. 228 00:10:44,600 --> 00:10:47,480 Why go so much smaller? Wouldn't a bigger brain make a 229 00:10:47,480 --> 00:10:49,480 better critic? You would think so, but they ran 230 00:10:49,480 --> 00:10:53,600 into some very practical issues. They found that training 175 B 231 00:10:53,600 --> 00:10:56,600 parameter reward models was just really unstable. 232 00:10:56,800 --> 00:10:58,880 It's hard to get them to converge reliably. 233 00:10:59,240 --> 00:11:01,760 Plus, you have to remember that in step three, we're going to be 234 00:11:01,760 --> 00:11:03,600 using this reward model inside a loop. 235 00:11:03,920 --> 00:11:07,600 It needs to be fast. A 6B model was the sweet spot 236 00:11:07,600 --> 00:11:11,640 for stability and compute cost. It's the Goldilocks zone, big 237 00:11:11,640 --> 00:11:14,680 enough to understand the nuance of the text, but small enough to 238 00:11:14,680 --> 00:11:16,280 run efficiently over and over again. 239 00:11:16,520 --> 00:11:19,080 And one final technical note on the reward model before we move 240 00:11:19,080 --> 00:11:20,720 on. They normalize the output. 241 00:11:21,040 --> 00:11:24,240 They set the bias in the final layer such that the mean score 242 00:11:24,240 --> 00:11:26,560 of the original labeler demonstrations is 0. 243 00:11:26,560 --> 00:11:28,920 Why does that matter? It just acts as a baseline. 244 00:11:29,200 --> 00:11:31,800 It means positive scores are better than an average human 245 00:11:31,800 --> 00:11:34,320 demonstration and negative scores are worse. 246 00:11:35,360 --> 00:11:37,840 It really helps with interpretability when you start 247 00:11:37,840 --> 00:11:40,840 the reinforcement learning. It gives you a kind of C level 248 00:11:40,840 --> 00:11:44,280 to measure against. OK, so to recap, we have our SFT 249 00:11:44,280 --> 00:11:48,040 model, the writer and our reward model, the critic. 250 00:11:48,760 --> 00:11:53,600 Now we can fight Step 3, reinforcement learning via PPO. 251 00:11:54,200 --> 00:11:56,160 Or collaborate depending on how you look at it. 252 00:11:56,360 --> 00:11:59,760 This is the optimization loop. We are taking that SFT model 253 00:11:59,760 --> 00:12:02,560 which we now call the policy and we want to fine tune its 254 00:12:02,560 --> 00:12:05,920 parameters to maximize the score it gets from the reward model. 255 00:12:05,920 --> 00:12:08,240 This is the environment set up for the RL problem. 256 00:12:08,240 --> 00:12:10,040 Exactly. It's a banded environment. 257 00:12:10,320 --> 00:12:13,920 The model sees a random customer prompt, it generates a response, 258 00:12:13,920 --> 00:12:16,080 and then bam, the reward model gives it a score. 259 00:12:16,480 --> 00:12:18,520 There is no human involved in this loop anymore. 260 00:12:18,520 --> 00:12:20,560 It's just machine teaching machine. 261 00:12:20,800 --> 00:12:23,400 And the algorithm they use is PPO, Proximal Policy 262 00:12:23,400 --> 00:12:25,480 Optimization. We don't need to get into all 263 00:12:25,480 --> 00:12:28,960 the math here, but what is the intuition behind using PPO? 264 00:12:28,960 --> 00:12:31,400 Specifically, why not just standard gradient descent? 265 00:12:31,520 --> 00:12:34,920 PPO is great because it's stable in reinforcement learning. 266 00:12:34,920 --> 00:12:37,720 If you update your policy too aggressively based on a single 267 00:12:37,720 --> 00:12:40,920 piece of feedback, you can actually collapse the model. 268 00:12:40,920 --> 00:12:42,960 It just starts doing bizarre things. 269 00:12:43,600 --> 00:12:47,160 PPO limits how much the policy can change in any single step. 270 00:12:47,640 --> 00:12:51,040 It forces the model to learn gradually, which prevents it 271 00:12:51,040 --> 00:12:54,240 from going off the rails because of one weird data point. 272 00:12:54,360 --> 00:12:56,240 This raises a really important question though. 273 00:12:56,600 --> 00:13:00,200 If the models only goal is to get a high score from the reward 274 00:13:00,200 --> 00:13:02,440 model, why doesn't it just cheat? 275 00:13:02,840 --> 00:13:04,760 That is the classic alignment problem. 276 00:13:04,760 --> 00:13:08,520 It's called reward hacking. If the reward model has some 277 00:13:08,520 --> 00:13:11,200 weird blind spots, say it really, really likes the word 278 00:13:11,200 --> 00:13:14,960 fantastic, the policy might just learn to output fantastic, 279 00:13:14,960 --> 00:13:17,680 fantastic, fantastic over and over to get a high score. 280 00:13:17,800 --> 00:13:20,200 Or it might just output gibberish that just happens to 281 00:13:20,200 --> 00:13:22,600 trigger the right neurons in the reward model. 282 00:13:22,640 --> 00:13:24,920 Right, it's because student realizing the teacher gives A's 283 00:13:24,920 --> 00:13:27,880 for long essays so they just write pages of nonsense to get 284 00:13:27,880 --> 00:13:29,960 the grade. How do they stop the model from 285 00:13:29,960 --> 00:13:32,040 gaming the system? They introduced something called 286 00:13:32,040 --> 00:13:34,880 AKL penalty. This is an absolutely critical 287 00:13:34,880 --> 00:13:38,600 design choice. KLD Virgin that measures how 288 00:13:38,840 --> 00:13:41,200 different 2 probability distributions are, right? 289 00:13:41,200 --> 00:13:43,640 Exactly. So while the RL model is 290 00:13:43,640 --> 00:13:47,360 training, they calculate the KL divergent between the new RL 291 00:13:47,360 --> 00:13:49,840 policy and the original SFT model. 292 00:13:50,200 --> 00:13:54,120 They then add this value as a penalty term to the reward. 293 00:13:54,800 --> 00:13:56,040 OK, let me see if I have this right. 294 00:13:56,560 --> 00:13:59,720 The model gets points for pleasing the reward model, but 295 00:13:59,720 --> 00:14:03,040 it loses points if it writes text that's too different from 296 00:14:03,040 --> 00:14:05,680 what the original SFT model. Would have written precisely. 297 00:14:05,680 --> 00:14:09,400 It's like a rubber band. The SFT model represents natural 298 00:14:09,400 --> 00:14:13,120 human like English. The KL penalty anchors the RL 299 00:14:13,120 --> 00:14:16,200 model to that baseline. It says, look, you can optimize 300 00:14:16,200 --> 00:14:18,960 for the reward, but don't drift so far away that you stop 301 00:14:18,960 --> 00:14:21,960 sounding like a human. That is such a smart constraint. 302 00:14:22,200 --> 00:14:25,200 It forces the optimization to happen within the realm of 303 00:14:25,200 --> 00:14:27,800 coherent language. It prevents the model from 304 00:14:27,800 --> 00:14:30,840 turning into a gibberish spewing robot just to maximize the 305 00:14:30,840 --> 00:14:31,760 score. Absolutely. 306 00:14:31,800 --> 00:14:34,320 Without that KL penalty, the model would almost certainly 307 00:14:34,320 --> 00:14:37,560 drift into adversarial examples texts that looks like noise to 308 00:14:37,560 --> 00:14:40,040 us, but looks like pure gold to the reward model. 309 00:14:40,120 --> 00:14:41,720 And for the engineers taking notes, what are the 310 00:14:41,720 --> 00:14:45,720 hyperparameters here? They set the PPO clip ratio to 311 00:14:45,960 --> 00:14:49,120 .2, which is fairly standard. They sampled rollouts at a 312 00:14:49,120 --> 00:14:51,720 temperature of 1, which helps ensure diversity during 313 00:14:51,720 --> 00:14:54,640 training, and they initialize the value function from the 314 00:14:54,640 --> 00:14:57,360 reward model itself. These are the specific dials 315 00:14:57,360 --> 00:14:59,160 they turn to make the whole system stable. 316 00:14:59,360 --> 00:15:03,040 OK, so we have this beautiful three-step loop, SFT to RM to 317 00:15:03,040 --> 00:15:05,760 PPO. Yeah, but it wasn't all smooth 318 00:15:05,760 --> 00:15:07,920 sailing. They hit a problem they called 319 00:15:07,920 --> 00:15:10,760 the alignment tax, and this is something that I think is so 320 00:15:10,760 --> 00:15:13,040 important for people to understand about modern AI. 321 00:15:13,480 --> 00:15:17,600 Yes, the alignment tax. This is a fascinating trade off. 322 00:15:17,600 --> 00:15:20,080 What they found was as they trained the model to be super 323 00:15:20,080 --> 00:15:22,240 helpful to customers. You know, answering questions, 324 00:15:22,240 --> 00:15:26,080 writing lists, being polite. The model actually got stupider 325 00:15:26,080 --> 00:15:28,640 at standard NLP benchmarks. Stupider in what way? 326 00:15:28,880 --> 00:15:31,840 It started failing at things like Squad which is a question 327 00:15:31,840 --> 00:15:35,800 answering data set or DRP, a reading comprehension task or 328 00:15:35,800 --> 00:15:39,240 even basic translation tasks. The model started to forget the 329 00:15:39,240 --> 00:15:42,000 fundamental facts and logic it learned during pre training 330 00:15:42,000 --> 00:15:45,480 because it was so hyper focused on this new chat style. 331 00:15:45,680 --> 00:15:48,840 It's like a student who crams so hard for the etiquette exam that 332 00:15:48,840 --> 00:15:52,160 they forget all their calculus. That is a perfect analogy. 333 00:15:52,320 --> 00:15:55,720 The alignment to the user instructions came at a tax to 334 00:15:55,760 --> 00:15:58,920 its general capabilities, and this is obviously bad. 335 00:15:58,960 --> 00:16:01,280 You don't want a helpful assistant that can't do math 336 00:16:01,280 --> 00:16:03,920 anymore or forgets how to translate French. 337 00:16:04,000 --> 00:16:06,680 So how do they fix it? This is where this PPO PTX thing 338 00:16:06,680 --> 00:16:08,560 comes in. Right, PPO PTX. 339 00:16:08,760 --> 00:16:11,600 It stands for PPO with a pre training mix. 340 00:16:11,960 --> 00:16:13,960 The solution was actually surprisingly elegant. 341 00:16:14,160 --> 00:16:17,000 During the RL update step they didn't just update based on the 342 00:16:17,000 --> 00:16:20,600 reward model, they also mixed in gradients from the original pre 343 00:16:20,600 --> 00:16:23,000 training data. So they are actively reminding 344 00:16:23,000 --> 00:16:25,680 it of the basics while teaching it all the new skills. 345 00:16:25,680 --> 00:16:29,560 Exactly the objective function became maximize the reward from 346 00:16:29,560 --> 00:16:33,800 the critic plus maximize the log likelihood of the original pre 347 00:16:33,800 --> 00:16:36,080 training text. So they're balancing two goals 348 00:16:36,080 --> 00:16:38,760 at the same time. Be helpful to this user right 349 00:16:38,760 --> 00:16:42,920 now, but also don't forget how to predict the the next token in 350 00:16:42,920 --> 00:16:46,800 a Wikipedia article. And they use a coefficient gamma 351 00:16:46,800 --> 00:16:50,760 to tune this balance. The result was that PPOPTX fixed 352 00:16:50,760 --> 00:16:53,760 the performance regression on benchmarks like Helleswag and 353 00:16:53,760 --> 00:16:56,960 Squad. And, crucially, it didn't hurt 354 00:16:56,960 --> 00:17:00,000 the human preference score. So they effectively evaded the 355 00:17:00,000 --> 00:17:02,240 tax. They got the alignment without 356 00:17:02,240 --> 00:17:04,280 losing the intelligence. Almost entirely yes. 357 00:17:04,440 --> 00:17:07,960 It's one of those rare have your cake and eat it 2 moments in 358 00:17:07,960 --> 00:17:11,000 machine learning engineering. It proves you can have a model 359 00:17:11,000 --> 00:17:14,520 that is both polite and smart if you are just careful about how 360 00:17:14,520 --> 00:17:16,839 you train it. All right, let's talk results. 361 00:17:17,200 --> 00:17:19,760 Does all this complexity, this three-step process, the 362 00:17:19,760 --> 00:17:23,240 labeling, the PPO, the KL penalty, I mean, does it 363 00:17:23,240 --> 00:17:25,440 actually pay off? The headline metric is just 364 00:17:25,440 --> 00:17:27,598 undeniable. In head to head comparisons, 365 00:17:27,598 --> 00:17:30,439 labelers significantly preferred the outputs from the Instruct 366 00:17:30,439 --> 00:17:33,399 GPT model over the original GPT 3. 367 00:17:33,400 --> 00:17:35,720 But here's the real kicker, the size difference. 368 00:17:35,720 --> 00:17:37,560 You mentioned this earlier, but I just want to double click on 369 00:17:37,560 --> 00:17:39,440 it. Right, so they compared a 1.3 370 00:17:39,440 --> 00:17:43,640 billion parameter instruct GPT model against the 175 billion 371 00:17:43,640 --> 00:17:47,080 parameter GPT 3, the one point 3B Model 1. 372 00:17:47,160 --> 00:17:50,160 That is just insane. A model more than 100 times 373 00:17:50,160 --> 00:17:52,320 smaller was perceived as better by humans. 374 00:17:52,760 --> 00:17:56,560 Proves that alignment is more important than raw capacity for 375 00:17:56,560 --> 00:17:59,920 user satisfaction. I mean, a massive genius that 376 00:17:59,920 --> 00:18:03,560 ignores you or speaks in riddles is way less useful than a 377 00:18:03,560 --> 00:18:06,360 reasonably smart assistant that listens intently and just 378 00:18:06,360 --> 00:18:09,320 answers your question directly. What about specific metrics? 379 00:18:09,320 --> 00:18:12,120 We talked about hallucinations earlier where the model just 380 00:18:12,120 --> 00:18:15,000 makes stuff up. On closed domain tasks, where 381 00:18:15,000 --> 00:18:17,960 the model is supposed to use only the provided text to answer 382 00:18:17,960 --> 00:18:21,600 a question, instruct GQC hallucinated significantly less. 383 00:18:21,840 --> 00:18:27,240 The rate went from 41% with GPT 3 down to 21% with Instruct GPT. 384 00:18:27,320 --> 00:18:29,600 That's a huge drop. That's cutting the lyre factor 385 00:18:29,600 --> 00:18:32,120 almost in half. It is so truthfulness improved, 386 00:18:32,360 --> 00:18:34,480 but there's a nuance with toxicity that I find really 387 00:18:34,480 --> 00:18:36,400 important, and it's a bit counterintuitive. 388 00:18:36,400 --> 00:18:38,160 This is the. Respectful versus obedient 389 00:18:38,160 --> 00:18:39,200 tension. Exactly. 390 00:18:39,280 --> 00:18:42,160 So when you instruct the model to be respectful, instruct GPT 391 00:18:42,160 --> 00:18:45,600 is much less toxic than GPT 3. It filters itself better, 392 00:18:45,840 --> 00:18:49,040 however, because it is so well trained to follow instructions 393 00:18:49,280 --> 00:18:51,640 if you explicitly order it to be toxic. 394 00:18:51,720 --> 00:18:53,840 Right and nasty consult about my neighbor. 395 00:18:54,040 --> 00:18:57,720 Instruct GPT is actually better at being toxic than GPT 3. 396 00:18:58,040 --> 00:19:01,400 It complies. GPT 3 might wander off topic or 397 00:19:01,400 --> 00:19:05,840 write some generic sentence but instruct GPT says yes Sir here 398 00:19:05,840 --> 00:19:09,120 is the toxicity you ordered. That is such a double edged 399 00:19:09,120 --> 00:19:11,600 sword. It follows instructions for 400 00:19:11,600 --> 00:19:14,200 better or for worse. It's not necessarily good in a 401 00:19:14,200 --> 00:19:17,520 moral sense, it's just obedient. Correct. 402 00:19:17,920 --> 00:19:20,240 It reflects the data it was trained on and learned that 403 00:19:20,240 --> 00:19:23,080 following instructions is the prime directive, even if the 404 00:19:23,080 --> 00:19:26,000 instruction is nasty. This is a key distinction 405 00:19:26,000 --> 00:19:30,160 between alignment and safety. It's aligned to your intent, 406 00:19:30,280 --> 00:19:33,480 even if your intent is bad. Now, one of the most surprising 407 00:19:33,480 --> 00:19:35,480 sections of the paper for me was about generalization. 408 00:19:35,480 --> 00:19:38,160 They call it a free lunch. And in ML, there's usually no 409 00:19:38,160 --> 00:19:40,440 such thing as a free lunch. Every game usually comes with a 410 00:19:40,440 --> 00:19:43,280 cost somewhere else, but here they found that the model 411 00:19:43,280 --> 00:19:46,120 improved on tasks it wasn't explicitly trained for during 412 00:19:46,120 --> 00:19:48,440 the RHF process. For example, non-english 413 00:19:48,440 --> 00:19:53,280 languages, the training data was 96% English, and yet Instruct 414 00:19:53,280 --> 00:19:56,720 GPT got better at following instructions in French, Spanish 415 00:19:56,720 --> 00:19:58,320 and Japanese. Main code right? 416 00:19:58,320 --> 00:19:59,560 I saw that too. Yes. 417 00:20:00,120 --> 00:20:02,800 Summarizing code and answering questions about code. 418 00:20:03,080 --> 00:20:06,520 They didn't have a massive code instruction data set in that 419 00:20:06,520 --> 00:20:09,360 fine tuning mix. So how on earth did it get 420 00:20:09,360 --> 00:20:12,880 better at French and code if it wasn't studying them during the 421 00:20:12,880 --> 00:20:16,320 fine tuning? The hypothesis is that the model 422 00:20:16,320 --> 00:20:19,440 already knew French and code from the pre training phase. 423 00:20:19,440 --> 00:20:22,840 From reading the Internet, the RLHF process didn't teach at 424 00:20:22,840 --> 00:20:25,280 French, it taught it the concept of instruction. 425 00:20:25,280 --> 00:20:28,160 Following, it learned a generalized behavior pattern. 426 00:20:28,640 --> 00:20:31,200 Identify the request, execute the request. 427 00:20:31,360 --> 00:20:34,200 It could then apply that pattern to its latent knowledge of 428 00:20:34,200 --> 00:20:35,880 French. That is just fascinating. 429 00:20:35,880 --> 00:20:38,440 So it learned the shape of being helpful and then applied it 430 00:20:38,520 --> 00:20:40,360 across its entire knowledge base. 431 00:20:40,720 --> 00:20:43,440 It suggests that alignment generalizes better than we might 432 00:20:43,440 --> 00:20:45,040 have thought. You don't necessarily need to 433 00:20:45,040 --> 00:20:47,080 align every single sub domain separately. 434 00:20:47,160 --> 00:20:49,560 You teach it the rules of the road once and it drives 435 00:20:49,560 --> 00:20:52,080 correctly on every highway. But we have to ground this. 436 00:20:52,360 --> 00:20:54,520 It's not perfect. The authors are very clear about 437 00:20:54,520 --> 00:20:56,840 the limitations. We shouldn't pretend this solved 438 00:20:56,840 --> 00:20:58,560 AGI or anything. Oh, absolutely. 439 00:20:58,640 --> 00:21:00,560 Really not. It is still a statistical model. 440 00:21:00,560 --> 00:21:05,120 It still makes very simple mistakes. 1 funny behavior, they 441 00:21:05,120 --> 00:21:08,280 noted is hedging. This is the model acting too 442 00:21:08,280 --> 00:21:10,880 humble, Yeah. If you ask a simple question, it 443 00:21:10,880 --> 00:21:13,680 might give you this long winding answer saying well there are 444 00:21:13,680 --> 00:21:16,480 many perspectives on this instead of just answering the 445 00:21:16,480 --> 00:21:17,880 question. Why would it do that? 446 00:21:17,880 --> 00:21:20,760 Is it just being annoying? It's likely because the labelers 447 00:21:20,760 --> 00:21:24,480 favored epistemic humility. They rated cautious answers 448 00:21:24,480 --> 00:21:26,720 higher than confident but wrong answers. 449 00:21:27,400 --> 00:21:29,880 So the model learned that playing it safe, sounding like a 450 00:21:29,880 --> 00:21:33,200 diplomatic bureaucrat is the best way to get a reward. 451 00:21:33,280 --> 00:21:36,320 It learned that if it sounds uncertain, it won't get punished 452 00:21:36,320 --> 00:21:39,200 for being wrong. That's pure game theory. 453 00:21:39,240 --> 00:21:42,680 In a way, yes, and it also struggles with false premises. 454 00:21:42,840 --> 00:21:45,800 If a user asks something like why is it important to eat socks 455 00:21:45,800 --> 00:21:49,120 after meditating, the model might try to invent a reason. 456 00:21:49,240 --> 00:21:51,400 Eating socks grounds your spiritual energy. 457 00:21:51,400 --> 00:21:54,200 Something like that. Exactly because it wants to be 458 00:21:54,200 --> 00:21:56,880 helpful, it assumes the user knows what they're talking 459 00:21:56,880 --> 00:21:58,440 about. It tries to rationalize the 460 00:21:58,440 --> 00:22:01,200 premise rather than correcting the user and saying hey, you 461 00:22:01,200 --> 00:22:03,800 shouldn't eat socks. That is a dangerous trait in a 462 00:22:03,800 --> 00:22:06,200 product. It reinforces user 463 00:22:06,200 --> 00:22:09,560 misconceptions because it's trying so hard to be polite. 464 00:22:09,680 --> 00:22:12,040 It is. And finally, we have to talk 465 00:22:12,040 --> 00:22:15,040 about cultural bias. Who are these 40 labelers? 466 00:22:15,280 --> 00:22:17,840 We keep mentioning them, but their identity really matters 467 00:22:17,840 --> 00:22:19,520 here. The paper says they were mostly 468 00:22:19,520 --> 00:22:22,000 English speaking US and Southeast Asia based 469 00:22:22,000 --> 00:22:24,720 contractors. Right, so the model is aligned 470 00:22:24,720 --> 00:22:27,840 to their preferences. It's not aligned to universal 471 00:22:27,840 --> 00:22:30,120 human values, because those don't really exist. 472 00:22:30,720 --> 00:22:33,960 It's aligned to the specific values, norms and biases of 473 00:22:33,960 --> 00:22:37,280 those 40 people and the open AI researchers who wrote the 474 00:22:37,280 --> 00:22:40,120 labeling guidelines for them. So if you're from a culture that 475 00:22:40,120 --> 00:22:44,040 values directness and the labelers valued politeness, the 476 00:22:44,040 --> 00:22:46,320 model might feel annoying or even fake to you. 477 00:22:46,480 --> 00:22:48,600 Exactly. Alignment is subjective. 478 00:22:48,800 --> 00:22:51,720 We are not aligning to truth in some cosmic sense. 479 00:22:52,080 --> 00:22:55,120 We are aligning to a specific data set of human feedback. 480 00:22:55,320 --> 00:22:58,400 The AI is a mirror of the people who graded its homework. 481 00:22:58,600 --> 00:23:01,640 So what does this all mean for the engineer or the builder 482 00:23:01,640 --> 00:23:03,920 who's listening right now? They want to implement this. 483 00:23:03,920 --> 00:23:06,880 What is the core lesson? The take away is crystal clear. 484 00:23:07,000 --> 00:23:10,400 If you want a model to act as a product, as an assistant, you 485 00:23:10,400 --> 00:23:13,560 simply cannot rely on next token prediction alone. 486 00:23:13,760 --> 00:23:16,880 It is not enough. You need a signal for human 487 00:23:16,880 --> 00:23:18,440 preference. And the recipe is now 488 00:23:18,440 --> 00:23:21,040 established. The recipe is demonstration 489 00:23:21,040 --> 00:23:25,000 which is SFT, then ranking which is the RM, then optimization 490 00:23:25,000 --> 00:23:27,960 which is PPO. That is the pipeline that turned 491 00:23:27,960 --> 00:23:31,320 GPT 3 into ChatGPT and completely changed the industry. 492 00:23:31,600 --> 00:23:34,520 It is a standard operating procedure for modern LLMS. 493 00:23:34,640 --> 00:23:37,920 SFTRMPPO read it on a post it note. 494 00:23:38,000 --> 00:23:41,160 It's the formula for modern AI. Before we sign off, I do want to 495 00:23:41,160 --> 00:23:43,280 leave you with a thought. We talked about the alignment 496 00:23:43,280 --> 00:23:45,800 tax and how they fixed it by mixing in old data. 497 00:23:46,120 --> 00:23:49,280 But it makes me wonder, does aiming for helpfulness 498 00:23:49,280 --> 00:23:51,880 inherently conflict with truthfulness in the long? 499 00:23:51,880 --> 00:23:53,520 Run. That is the big unanswered 500 00:23:53,520 --> 00:23:55,200 question facing the field right now. 501 00:23:55,520 --> 00:23:58,640 If users want comforting lies and labelers reward those 502 00:23:58,640 --> 00:24:02,080 comforting lies, the reward model will train the AI to lie. 503 00:24:02,240 --> 00:24:05,640 And as these models get smarter, you know, smarter than the 504 00:24:05,640 --> 00:24:08,520 labelers themselves, we'll will humans even be able to rank the 505 00:24:08,520 --> 00:24:11,840 outputs correctly? If the AI writes a proof for a 506 00:24:11,840 --> 00:24:14,840 physics theorem that the labeler doesn't understand, does it just 507 00:24:14,840 --> 00:24:16,720 get a low score because it went over their head? 508 00:24:16,800 --> 00:24:18,880 That's the scalable oversight problem. 509 00:24:18,960 --> 00:24:22,280 Can dumb humans supervise smart AI? 510 00:24:22,280 --> 00:24:24,800 We might need a whole other deep dive just for that one. 511 00:24:24,800 --> 00:24:27,800 Indeed, we might. That's all for today's Practical 512 00:24:27,840 --> 00:24:29,920 AI Digest. Thanks for diving deep with us. 513 00:24:30,000 --> 00:24:30,760 Until next time.