1 00:00:00,080 --> 00:00:05,520 ML projects tend to fail in this common failure modes. 1 is what 2 00:00:05,520 --> 00:00:09,360 I call POC health. Another failure mode is that 3 00:00:09,840 --> 00:00:14,360 it's easy to get maybe the first thing out of the door, but then 4 00:00:14,480 --> 00:00:19,040 it's kind of stuck together with duct tape and sticky gum and 5 00:00:19,040 --> 00:00:23,200 it's caught to test or change. In my experience of working with 6 00:00:23,760 --> 00:00:28,560 data intensive ML initiatives or other technology 7 00:00:28,560 --> 00:00:32,280 specializations, I'd found that the ability to build the thing 8 00:00:32,479 --> 00:00:34,960 wasn't always the major success factor. 9 00:00:34,960 --> 00:00:38,280 Often it was understanding what the right thing to build was, 10 00:00:38,280 --> 00:00:41,400 making sure that there was alignment between technical 11 00:00:41,400 --> 00:00:44,080 teams and and business teams and product teams on the right thing 12 00:00:44,080 --> 00:00:46,080 to build. You can't ML OPS your problems 13 00:00:46,080 --> 00:00:49,120 away just like how you can't dev OPS your problems away. 14 00:00:49,600 --> 00:00:53,040 ML OPS helps in a few ways, but ML OPS not going to run your 15 00:00:53,040 --> 00:00:55,440 tests for you. They're not going to talk to 16 00:00:55,440 --> 00:00:58,000 users and make sure you're writing the same right features 17 00:00:58,000 --> 00:00:59,400 or implementing the right features. 18 00:00:59,760 --> 00:01:03,680 It's useful to think about the essential characteristics of of 19 00:01:03,680 --> 00:01:07,400 ML solutions that we have to do things at speed and scale that 20 00:01:07,400 --> 00:01:09,880 no human could. But they'll make mistakes. 21 00:01:10,200 --> 00:01:12,320 We wouldn't consider an ML solution if we knew the right 22 00:01:12,320 --> 00:01:14,800 answer every time. We'd write some rules instead. 23 00:01:15,040 --> 00:01:18,120 One of the quotes opening quotes in our book was from Edward 24 00:01:18,120 --> 00:01:21,360 Stemming saying a bad system would be a good person every 25 00:01:21,360 --> 00:01:23,800 time. The whole thesis of Outlook is 26 00:01:23,800 --> 00:01:28,040 how do we create those systems to help teams build the right 27 00:01:28,040 --> 00:01:31,520 thing, build the thing right, and in a way that's right for 28 00:01:31,520 --> 00:01:46,360 people. Hello everyone, welcome back to 29 00:01:46,360 --> 00:01:48,680 another new episode of the Technician podcast. 30 00:01:48,680 --> 00:01:51,400 Today I have with me two David's very excited. 31 00:01:51,400 --> 00:01:55,280 One is David Tan, he is actually my ex colleague in thought books 32 00:01:55,280 --> 00:01:56,960 and the other one is David Coles. 33 00:01:57,240 --> 00:02:00,800 So they are the co-authors together with another one Ada 34 00:02:00,800 --> 00:02:04,200 who couldn't join us today. They wrote a book titled 35 00:02:04,200 --> 00:02:07,640 Effective Machine Learning Teams, even though it's a 36 00:02:07,640 --> 00:02:09,280 machine learning as part of the book. 37 00:02:09,280 --> 00:02:11,920 But I think we are not going to cover solely just machine 38 00:02:11,920 --> 00:02:15,320 learning because I'm also not an ML expert, But we'll discuss 39 00:02:15,320 --> 00:02:17,840 things like how we can build effective machine learning 40 00:02:17,840 --> 00:02:21,200 teams, what kind of practices that we should be aware of and 41 00:02:21,200 --> 00:02:23,280 things like that. So welcome to the show. 42 00:02:23,280 --> 00:02:28,240 I'll maybe mention Dave as David calls right, and David for David 43 00:02:28,240 --> 00:02:29,600 Tan. So welcome to the show guys. 44 00:02:30,200 --> 00:02:31,600 Yeah. Thanks for having us, Henry. 45 00:02:31,800 --> 00:02:34,280 Happy to be here. Thanks Henry, nice to be here. 46 00:02:35,080 --> 00:02:36,880 Right. I always love to ask my guests 47 00:02:36,880 --> 00:02:40,280 maybe first to share a little bit of themselves by sharing 48 00:02:40,280 --> 00:02:42,880 your career turning points that you think we all can learn from 49 00:02:42,880 --> 00:02:44,960 that so any one of you can start first. 50 00:02:45,760 --> 00:02:50,160 Cool so I'm currently engineering manager in the Air 51 00:02:50,200 --> 00:02:55,320 Products group in zero and my career into tech hasn't always 52 00:02:55,320 --> 00:02:57,960 been straight. Like I was actually 53 00:02:57,960 --> 00:03:00,480 non-technical when I graduated from university. 54 00:03:00,840 --> 00:03:05,800 So I was working in government for maybe 2-3 years and I got by 55 00:03:05,800 --> 00:03:08,440 accident handful and mouth disease and I had to like be 56 00:03:08,440 --> 00:03:11,480 quarantined for eight days. And from there I started like, 57 00:03:11,480 --> 00:03:15,400 you know, watching some YouTube videos about data analytics or 58 00:03:15,400 --> 00:03:18,560 coding and then, you know, try some tutorials and, you know, 59 00:03:18,560 --> 00:03:20,960 it's really cool. So then decided to, you know, 60 00:03:20,960 --> 00:03:24,560 join the programming boot camp, quit my job, did that. 61 00:03:24,680 --> 00:03:28,160 And then giant ThoughtWorks. And yeah, learnt a lot about 62 00:03:28,160 --> 00:03:32,360 agile, about test driven development, refactoring, CICD, 63 00:03:33,160 --> 00:03:37,640 got into Mr. Engineering and so, you know, finally got into this 64 00:03:37,640 --> 00:03:41,000 exciting space of Mr. Engineering and OPS building Mr. 65 00:03:41,000 --> 00:03:43,280 products. So yeah, I have to thank this 66 00:03:43,360 --> 00:03:46,480 eight day quarantine back in the day for my career. 67 00:03:46,480 --> 00:03:48,520 Turning point? Yeah, maybe. 68 00:03:48,520 --> 00:03:51,800 How about you? To to pick on an external theme 69 00:03:51,800 --> 00:03:55,520 like that, I'd probably choose deciding to play ultimate 70 00:03:55,520 --> 00:03:59,640 Frisbee early in my career where I met a number of people 71 00:03:59,640 --> 00:04:03,440 involved in the software scene, but in particular a thought 72 00:04:03,440 --> 00:04:05,960 worker who encouraged me to interview at ThoughtWorks. 73 00:04:06,400 --> 00:04:09,640 And when I think about turning points, I guess that kind of 74 00:04:09,640 --> 00:04:13,320 leads me into the fact that I started as a developer in 75 00:04:13,440 --> 00:04:17,279 technical data intensive roles. I then, you know, looked for 76 00:04:17,320 --> 00:04:19,959 ways to make more impact in the leadership space. 77 00:04:20,320 --> 00:04:22,840 And when I joined ThoughtWorks, although I'd actually passed a 78 00:04:22,840 --> 00:04:25,520 coding interview, I joined as a project manager. 79 00:04:25,800 --> 00:04:29,080 Then I've worked for a number of years in agile transformations 80 00:04:29,080 --> 00:04:32,320 and organizational transformation before then 81 00:04:32,480 --> 00:04:36,480 picking up the data practice. So I guess my career has turned 82 00:04:36,800 --> 00:04:40,840 from deeply technical to leadership positions and 83 00:04:40,840 --> 00:04:44,640 organizational design a number of times over that period. 84 00:04:45,480 --> 00:04:46,960 Thank you so much for sharing your story. 85 00:04:46,960 --> 00:04:49,920 Very interesting, right? So the kind of serendipity that 86 00:04:49,920 --> 00:04:51,640 could happen through our career, right? 87 00:04:51,640 --> 00:04:54,400 And what led us to where we are at the moment. 88 00:04:54,880 --> 00:04:57,840 So today we're going to discuss topics from your book, Effective 89 00:04:57,840 --> 00:05:00,160 Machine Learning Teams. Maybe in the beginning, let's 90 00:05:00,320 --> 00:05:02,720 share what made you wrote this book. 91 00:05:03,520 --> 00:05:06,680 Yeah, this book started as a blog post actually. 92 00:05:06,680 --> 00:05:11,880 So back in 2019 we had wrapped up a project where it involved a 93 00:05:11,880 --> 00:05:15,160 lot of refactoring of data science code bases. 94 00:05:15,600 --> 00:05:18,920 So we had data science piece for wrote code in the Jupiter 95 00:05:18,920 --> 00:05:21,840 notebooks. No tests, a lot of lines. 96 00:05:21,840 --> 00:05:24,480 You know, it's good like it was solving real business problem 97 00:05:25,120 --> 00:05:27,120 and then we had to kind of productionize that. 98 00:05:27,720 --> 00:05:30,560 And so we started writing about how we did that. 99 00:05:30,840 --> 00:05:34,080 How do we encapsulate code into functions? 100 00:05:34,080 --> 00:05:36,840 How do we have automated tests? So we are, you know, clear about 101 00:05:36,840 --> 00:05:40,880 that, about the quality. How do we do CICD for those 102 00:05:40,880 --> 00:05:43,160 changes? And then we wrote a blog post 103 00:05:43,160 --> 00:05:46,960 and it kind of like exploded in back then Twitter, someone from 104 00:05:46,960 --> 00:05:50,000 the our community shared it and a lot of likes and it's like, 105 00:05:50,480 --> 00:05:52,000 OK, I think we're on to something here. 106 00:05:52,000 --> 00:05:56,000 There's this desire, I think in the data sense community or ML 107 00:05:56,000 --> 00:05:59,520 community to have more kind of engineering and robust 108 00:05:59,520 --> 00:06:02,360 practices. So, yeah, well then we started 109 00:06:02,960 --> 00:06:07,440 writing more and more about from the subsequent projects or what 110 00:06:07,440 --> 00:06:13,160 we did to kind of large scale ML training pipelines, large scale 111 00:06:13,160 --> 00:06:16,640 high volume ML products. And you know, the practices that 112 00:06:16,640 --> 00:06:21,240 helped us build that reliably build ML products that are safe, 113 00:06:21,800 --> 00:06:23,600 you get fast feedback on your changes. 114 00:06:23,840 --> 00:06:25,920 And some of those things worked really well. 115 00:06:25,920 --> 00:06:29,000 So we wanted to share it more broadly with our readers. 116 00:06:29,680 --> 00:06:31,360 And that's just the engineering space. 117 00:06:31,360 --> 00:06:35,000 And I think Dave also has some other kind of yeah, aspects that 118 00:06:35,040 --> 00:06:37,680 we know. We started with this thread only 119 00:06:37,680 --> 00:06:41,200 engineering practices for ML and then we discovered this whole 120 00:06:41,200 --> 00:06:44,440 kind of world of other practices that also help ML teams. 121 00:06:44,880 --> 00:06:46,920 Yeah. So yeah, in my experience of 122 00:06:47,080 --> 00:06:52,920 working with, I guess, data intensive ML initiatives or 123 00:06:53,280 --> 00:06:56,080 other technology specializations, I'd found that 124 00:06:56,440 --> 00:07:00,600 the ability to build the thing wasn't always the major success 125 00:07:00,600 --> 00:07:02,960 factor. Often it was understanding what 126 00:07:02,960 --> 00:07:05,480 the right thing to build was, making sure that there was 127 00:07:05,480 --> 00:07:09,040 alignment between technical teams and and business teams and 128 00:07:09,040 --> 00:07:11,960 product teams on the right thing to build, which could be complex 129 00:07:11,960 --> 00:07:14,200 when people don't speak the same language. 130 00:07:14,520 --> 00:07:17,840 So, you know, what sort of processes and techniques can you 131 00:07:17,840 --> 00:07:21,200 use to build that alignment, especially when there's a lot of 132 00:07:21,200 --> 00:07:23,880 uncertainty both about the problem and the solution. 133 00:07:24,400 --> 00:07:28,080 And then also because of the specialized nature of this work, 134 00:07:28,440 --> 00:07:32,440 it's often hard to build a team that can execute end to end. 135 00:07:32,840 --> 00:07:37,600 So then the factors around either building in a single 136 00:07:37,600 --> 00:07:40,040 team, how do you bring all of those different perspectives 137 00:07:40,040 --> 00:07:42,480 together? Or when you have to rely on 138 00:07:42,480 --> 00:07:45,800 multiple teams to get a piece of work out the door, what's the 139 00:07:45,800 --> 00:07:49,960 best way to achieve fast flow and high quality in a way that's 140 00:07:49,960 --> 00:07:52,440 sustainable for the people working in that environment? 141 00:07:52,800 --> 00:07:55,760 So for me, those were the perspectives I wanted to bring 142 00:07:56,120 --> 00:07:59,360 to machine learning product development, which is an 143 00:07:59,680 --> 00:08:01,840 essentially multidisciplinary activity. 144 00:08:02,040 --> 00:08:03,800 You know, how do we do that better as teams? 145 00:08:04,720 --> 00:08:08,280 Right, when I write your book, I must admit that I was thinking 146 00:08:08,280 --> 00:08:11,440 that I was going to learn a lot of the technical things about 147 00:08:11,440 --> 00:08:15,200 specific ML, you know, projects and ML products and things like 148 00:08:15,200 --> 00:08:16,440 that. But actually you cover kind of 149 00:08:16,440 --> 00:08:19,000 like a holistic approach. It's not just the engineering, 150 00:08:19,280 --> 00:08:21,680 but also like product delivery and things like that. 151 00:08:22,000 --> 00:08:24,120 And actually there are so many things that any software 152 00:08:24,120 --> 00:08:26,600 engineering team I believe could learn just by reading the book 153 00:08:26,600 --> 00:08:28,520 as well. I mean, putting aside some parts 154 00:08:28,520 --> 00:08:32,640 of the ML, but let's start maybe in the 1st place because I'm 155 00:08:32,640 --> 00:08:35,960 sure many listeners here not coming from the ML background, 156 00:08:36,240 --> 00:08:39,159 what are the differences between, you know, ML 157 00:08:39,159 --> 00:08:41,880 engineering and the typical software engineering these days 158 00:08:41,880 --> 00:08:44,400 where people build websites, APIs and things like that? 159 00:08:45,560 --> 00:08:46,960 Yeah. But that's a great question. 160 00:08:47,040 --> 00:08:51,000 And I think ML systems fundamentally it's orders of 161 00:08:51,000 --> 00:08:56,000 magnitude, I think harder and more complex than software 162 00:08:56,000 --> 00:08:58,360 engineering in some ways that of course software engineering is 163 00:08:58,360 --> 00:09:02,560 harder than we've got micro services, you've got distributed 164 00:09:02,560 --> 00:09:06,120 computing, you've got a lot of things like in itself it's an 165 00:09:06,120 --> 00:09:12,040 art and a whole discipline. ML itself, the difficulties 166 00:09:12,600 --> 00:09:16,080 manifest themselves in some common ways, like, you know, 167 00:09:16,640 --> 00:09:19,600 it's not just one service or one component. 168 00:09:19,800 --> 00:09:23,480 Usually it's like a data pipeline dependencies upstream 169 00:09:23,920 --> 00:09:25,560 that you know is complex in itself. 170 00:09:25,840 --> 00:09:29,800 You've got your skill or compute requirements for your own ML 171 00:09:29,800 --> 00:09:34,120 workload, which is also can be done in many ways. 172 00:09:34,120 --> 00:09:37,240 You know, you can have a really large instance with really large 173 00:09:37,240 --> 00:09:40,080 memory, do everything there, or you know, at some point that 174 00:09:40,080 --> 00:09:43,480 breaks down. So how do you architect or make 175 00:09:43,480 --> 00:09:45,600 it simple to scale up large workloads? 176 00:09:46,320 --> 00:09:47,760 And then there's also the monitoring. 177 00:09:47,760 --> 00:09:50,560 Like when you think about software products, typically 178 00:09:50,560 --> 00:09:54,160 your monitoring is kind of your 4 golden signals, right? 179 00:09:54,640 --> 00:09:57,080 Success rate, latency, things like that. 180 00:09:57,520 --> 00:10:00,960 You can do that for ML and you can be all green and good. 181 00:10:01,440 --> 00:10:05,200 But that says nothing about like the quality of the predictions 182 00:10:05,200 --> 00:10:09,400 that models are producing and then did the correctness of 183 00:10:09,400 --> 00:10:11,280 those. So it adds another layer of 184 00:10:11,800 --> 00:10:14,560 challenges that I think the ML OPS community have come to solve 185 00:10:14,720 --> 00:10:17,800 in recent years. And then further, even further 186 00:10:17,800 --> 00:10:21,240 down, like, yes, we've built this product is giving accurate 187 00:10:21,240 --> 00:10:25,040 predictions to an extent. What is the kind of business 188 00:10:25,040 --> 00:10:29,280 impact or what levers on our Dios is it moving? 189 00:10:29,840 --> 00:10:33,000 So that's like kind of essentially the fundamental 190 00:10:33,000 --> 00:10:36,240 difference between the ML product and software engineering 191 00:10:36,240 --> 00:10:38,800 products. There's a small business diagram 192 00:10:38,800 --> 00:10:41,960 from Google where like ML is just like a little box and 193 00:10:41,960 --> 00:10:45,160 there's like infra, there's monitoring, there's data 194 00:10:45,160 --> 00:10:48,720 quality, data skew. So a lot of moving parts and 195 00:10:48,960 --> 00:10:51,880 that's why we wanted to bring a lot of practices that kind of 196 00:10:51,880 --> 00:10:54,560 help you obtain the complexity in these different 197 00:10:54,560 --> 00:10:58,680 subcomponents. And when you look at how to 198 00:10:58,680 --> 00:11:01,960 manage the work as well, then I guess one of the characteristics 199 00:11:01,960 --> 00:11:05,600 of building an ML feature is there's a lot less certainty 200 00:11:05,880 --> 00:11:08,000 about how it's going to perform upfront. 201 00:11:08,520 --> 00:11:12,200 And so often you have to integrate this exploratory data 202 00:11:12,200 --> 00:11:16,360 and analysis or building prototypes or architecture 203 00:11:16,360 --> 00:11:20,080 search as it might be described into the product delivery 204 00:11:20,200 --> 00:11:23,920 process as well. And then as you converge on a 205 00:11:23,920 --> 00:11:26,920 potential solution, then you're also dealing with another vector 206 00:11:26,920 --> 00:11:30,800 of change, which is the data and the world changing as well. 207 00:11:30,800 --> 00:11:34,200 As David said around monitoring, often that changes slowly, but 208 00:11:34,200 --> 00:11:36,320 sometimes we can see step changes. 209 00:11:36,800 --> 00:11:42,000 COVID was a classic example with recommender systems moving from 210 00:11:42,400 --> 00:11:46,160 lowest price to highest to best availability and you know, ML 211 00:11:46,160 --> 00:11:48,000 models needing to catch up with that. 212 00:11:48,360 --> 00:11:50,800 So yeah, you both have less certainty and you have another 213 00:11:50,800 --> 00:11:53,920 vector of change to deal with when it comes to managing that 214 00:11:53,920 --> 00:11:55,440 work. Yeah. 215 00:11:55,440 --> 00:11:58,080 So thanks for sharing some of these complexities, right. 216 00:11:58,080 --> 00:12:01,480 So definitely for people who may not be able to relate, I think 217 00:12:01,600 --> 00:12:04,320 typically from what I can see, right, there are so many data 218 00:12:04,320 --> 00:12:07,560 pipelines when you build ML, you probably need more than just a 219 00:12:07,560 --> 00:12:09,880 small data, right? Something like a medium data of 220 00:12:10,120 --> 00:12:12,840 big data and a kind of a compute that you need. 221 00:12:12,840 --> 00:12:15,400 Sometimes it's more like a distributed computing with large 222 00:12:15,400 --> 00:12:18,600 scale kind of a pipelines. And I think typically what I can 223 00:12:18,600 --> 00:12:21,760 see as well, the feedback loop might be long in an ML project 224 00:12:21,760 --> 00:12:24,280 simply because you have this training kind of thing in the 225 00:12:24,280 --> 00:12:25,800 1st place. Maybe let's clarify. 226 00:12:25,920 --> 00:12:29,040 When you say ML, I think we are not just associating it with 227 00:12:29,040 --> 00:12:31,080 LLM, right? Like these days people are crazy 228 00:12:31,080 --> 00:12:34,320 about LLM and maybe they associate ML now as LLM. 229 00:12:34,480 --> 00:12:36,960 So maybe a little bit of insight here, like what do you mean by 230 00:12:36,960 --> 00:12:39,280 ML projects? Yeah, I guess we, yeah, Yeah, 231 00:12:39,520 --> 00:12:42,560 great observation, Henry. You know, was thinking, you 232 00:12:42,560 --> 00:12:45,120 know, as you talked about feedback cycles and big data 233 00:12:45,120 --> 00:12:48,240 and, and all of those elements, yes, those are the things we 234 00:12:48,240 --> 00:12:51,480 traditionally associate with supervised machine learning, 235 00:12:51,560 --> 00:12:55,240 which I guess is our sort of primary focus in the book, which 236 00:12:55,240 --> 00:12:58,440 might be anything from a binary classification problem. 237 00:12:58,440 --> 00:13:01,280 Like is this transaction a fraudulent transaction? 238 00:13:01,600 --> 00:13:04,600 And, you know, to solve that with the supervised paradigm, we 239 00:13:04,600 --> 00:13:07,480 get a lot of historical data about transactions, their 240 00:13:07,480 --> 00:13:10,920 amount, the the age of the account, you know, maybe some 241 00:13:10,920 --> 00:13:13,560 information about the network around that account. 242 00:13:13,560 --> 00:13:16,200 And then we label those transactions whether they were 243 00:13:16,200 --> 00:13:18,680 fraudulent or not. That's actually a nice example 244 00:13:18,680 --> 00:13:21,240 of where the labels might be easy to get because we might get 245 00:13:21,240 --> 00:13:23,080 them from customer reports of fraud. 246 00:13:23,480 --> 00:13:25,880 But in general, that effort of labelling the data set can be 247 00:13:25,880 --> 00:13:28,800 really intensive as well. So that's the sort of main 248 00:13:28,800 --> 00:13:31,040 paradigm we're looking at. But we think that, you know, 249 00:13:31,040 --> 00:13:33,920 looking through this lens of working with uncertainty, 250 00:13:33,920 --> 00:13:37,480 working with data, working with specializations, but there's a 251 00:13:37,480 --> 00:13:41,680 whole lot of other paradigms of MLAI that that might fit into 252 00:13:41,680 --> 00:13:44,760 that. So obviously working with a 253 00:13:45,280 --> 00:13:48,360 large language model in inference mode, that's still a 254 00:13:48,360 --> 00:13:51,520 challenging environment. The supervised paradigm might 255 00:13:51,520 --> 00:13:55,360 apply to fine tuning or even training from scratch your own 256 00:13:55,480 --> 00:13:59,320 large learning model as well. But then beyond that sort of 257 00:13:59,320 --> 00:14:03,120 paradigm, we're also looking at initiatives that might be might 258 00:14:03,160 --> 00:14:05,160 use reinforcement learning, for instance, might. 259 00:14:05,240 --> 00:14:07,720 It's another approach to recommender systems. 260 00:14:08,040 --> 00:14:10,680 And this is where you might not need big data to start with. 261 00:14:11,080 --> 00:14:13,920 You might not have historical data to start with, but you 262 00:14:13,920 --> 00:14:17,320 might actually build or tune your data set as you go. 263 00:14:17,800 --> 00:14:20,400 We might also look further afield to techniques like 264 00:14:20,400 --> 00:14:23,640 simulation or operations research for optimization. 265 00:14:23,880 --> 00:14:25,640 They share a lot of the same characteristics. 266 00:14:25,640 --> 00:14:28,480 Simulation is actually very similar to machine learning when 267 00:14:28,480 --> 00:14:31,440 it comes to productionizing it. It's just that instead of a 268 00:14:31,440 --> 00:14:34,120 learned model of the world, you're working with an explicit 269 00:14:34,120 --> 00:14:36,480 model of the world. But you still need to go through 270 00:14:36,480 --> 00:14:39,560 feature engineering ability to monitor and assess business 271 00:14:39,560 --> 00:14:43,320 impact and align to stakeholder expectations too. 272 00:14:43,720 --> 00:14:47,800 So while we're focused on supervised machine learning as a 273 00:14:47,800 --> 00:14:49,680 paradigm, yes, there's a big world out there. 274 00:14:50,000 --> 00:14:53,520 Anything that's decision making driven off data or models of the 275 00:14:53,520 --> 00:14:56,560 world, or requires specialization might fit into 276 00:14:56,640 --> 00:14:58,600 this view that we've put forward in the book as well. 277 00:14:59,680 --> 00:15:02,360 Thanks for the clarification. So maybe let's start going 278 00:15:02,360 --> 00:15:04,880 deeper into the book, right? I think one thing that picked my 279 00:15:04,880 --> 00:15:07,560 interest when I read the book in the first few chapters of the 280 00:15:07,560 --> 00:15:09,120 book, right? I think you cover some 281 00:15:09,120 --> 00:15:12,160 statistics here. The first one is actually many 282 00:15:12,160 --> 00:15:15,400 ML projects doesn't make it to production, right? 283 00:15:15,800 --> 00:15:18,400 So and the second one, even though they reach production, 284 00:15:18,600 --> 00:15:20,920 they actually don't solve the real business problem or they 285 00:15:20,920 --> 00:15:23,520 don't bring any value, right? So maybe let's share a little 286 00:15:23,520 --> 00:15:26,680 bit why this is the case. What do you see out there as ML 287 00:15:26,680 --> 00:15:29,480 practitioners, right? So why is it so hard to actually 288 00:15:29,480 --> 00:15:31,600 bringing ML projects to production? 289 00:15:33,000 --> 00:15:36,040 Yeah, thanks, Henry. Like that's the crux of the book 290 00:15:36,040 --> 00:15:41,280 in that ML projects tend to fail in this, the common failure 291 00:15:41,280 --> 00:15:45,840 modes that we describe, and I can go through them and it's 292 00:15:46,240 --> 00:15:49,240 kind of preventable types of failure. 293 00:15:49,600 --> 00:15:52,840 Like if we can learn from experience, from history, and 294 00:15:53,200 --> 00:15:56,200 then we would bring in the right tactics to make sure that, as 295 00:15:56,200 --> 00:15:59,240 you mentioned, let's say a project that, you know, let's 296 00:15:59,240 --> 00:16:03,440 say we invest half a year, nine months into it and ship it. 297 00:16:03,440 --> 00:16:06,360 And we found that actually it's not solving the right problem or 298 00:16:06,520 --> 00:16:08,200 users didn't really care about it. 299 00:16:08,680 --> 00:16:11,320 It's like then what techniques can we bring in there with 300 00:16:11,440 --> 00:16:13,800 prototype testing or customer research? 301 00:16:14,640 --> 00:16:19,840 So that's kind of the whole kind of the main thesis of the book 302 00:16:19,840 --> 00:16:23,760 is like how have we failed and what have we learnt and how can 303 00:16:23,760 --> 00:16:28,160 we do better. So 1 common failure mode in ML 304 00:16:28,160 --> 00:16:30,600 projects is what I call POC health. 305 00:16:31,120 --> 00:16:33,280 It's like a combination of factors. 306 00:16:33,280 --> 00:16:37,920 Maybe business don't have enough of the appetite to release 307 00:16:37,920 --> 00:16:42,440 something to users. So we might take the first easy 308 00:16:42,440 --> 00:16:46,600 step to build a prototype to test it, maybe internal demo of 309 00:16:46,600 --> 00:16:53,080 it, but the resources or risk to productionized it and put it in 310 00:16:53,080 --> 00:16:54,960 the hands of users is kind of too great. 311 00:16:55,080 --> 00:16:59,760 So what the lack of commitment there means that teams on the 312 00:16:59,760 --> 00:17:02,960 ground just kind of build Pocs of the Pocs of the Pocs. 313 00:17:03,600 --> 00:17:07,040 So that's detrimental to, you know, the business in some ways 314 00:17:07,040 --> 00:17:09,920 that you're missing out on opportunities, so detrimental to 315 00:17:09,920 --> 00:17:12,800 the morale of the team. You do a few and then, you know, 316 00:17:12,800 --> 00:17:14,400 after a while you say, what's the point? 317 00:17:14,400 --> 00:17:17,040 So, you know, you move on and then you've got churn and you've 318 00:17:17,079 --> 00:17:19,760 got onboarding and you're losing talent, right? 319 00:17:20,480 --> 00:17:24,720 Another common, I think a failure mode is that it's easy 320 00:17:24,720 --> 00:17:27,599 to get maybe the first thing out of the door. 321 00:17:27,800 --> 00:17:31,920 You hustle, you get the data, you get the compute, you train 322 00:17:31,920 --> 00:17:35,560 the model, you deploy the thing. But then it's kind of stuck 323 00:17:35,560 --> 00:17:40,440 together with duct tape and sticky gum and it's caught to 324 00:17:40,480 --> 00:17:45,720 test or change after Evolve based on, you know, new customer 325 00:17:45,720 --> 00:17:48,200 feedback, things like that. So then that comes to kind of 326 00:17:48,200 --> 00:17:52,880 continuous delivery CICD. How confident can we be to test 327 00:17:52,880 --> 00:17:55,960 and deploy a change? And so this set of problems is 328 00:17:55,960 --> 00:18:00,040 solved by ML OPS, by continuous delivery for machine learning. 329 00:18:00,520 --> 00:18:03,880 There's probably a couple more. I don't want to hop the mic. 330 00:18:04,200 --> 00:18:06,480 Dave, was there any other failure modes that came to mind, 331 00:18:06,480 --> 00:18:09,360 all these challenges? Yeah, those are two big ones. 332 00:18:09,360 --> 00:18:13,160 It's, you know, is there any value in doing this at all? 333 00:18:13,280 --> 00:18:16,240 And that can be really challenging when their 334 00:18:16,240 --> 00:18:19,200 responsibility is divided across different teams who might even 335 00:18:19,200 --> 00:18:20,920 be in different parts of the organization. 336 00:18:21,480 --> 00:18:27,800 There might be AI guess the release of ChatGPT sparked a lot 337 00:18:27,800 --> 00:18:31,560 of curiosity in how do we best use LLMS in products. 338 00:18:31,560 --> 00:18:35,520 So, but we might have responsibility diffused across 339 00:18:35,520 --> 00:18:37,720 the organization. We might have an the ML team 340 00:18:37,720 --> 00:18:41,000 that's interested in looking at the use of large language models 341 00:18:41,360 --> 00:18:44,200 and we might have a product team that has their own road map 342 00:18:44,560 --> 00:18:47,560 where they don't have these features anywhere on that road 343 00:18:47,560 --> 00:18:49,720 map. And then we might rely on data, 344 00:18:50,160 --> 00:18:54,040 especially in the case of Gen. AI, that's unstructured data 345 00:18:54,040 --> 00:18:57,400 that's not well governed. It's hard to make available in a 346 00:18:57,720 --> 00:18:59,800 responsible way to these systems. 347 00:19:00,200 --> 00:19:04,640 And so while each group can do what's within their control to 348 00:19:05,000 --> 00:19:08,040 move a little bit towards a future state, it's not aligned 349 00:19:08,160 --> 00:19:12,240 or stitched up in a in a cohesive way that allows us to 350 00:19:12,640 --> 00:19:15,600 establish whether there is some value quickly with cheap and and 351 00:19:15,600 --> 00:19:18,360 low risk tests. And then, as David said, once 352 00:19:18,360 --> 00:19:21,880 you've established this some value, you know, if you have 353 00:19:22,600 --> 00:19:25,600 productionize something that's not supported by a lot of these 354 00:19:25,600 --> 00:19:28,920 good practices around ML OPS and and continuous delivery, which 355 00:19:28,920 --> 00:19:32,400 is often often the case to, to, you know, understand if there is 356 00:19:32,400 --> 00:19:35,560 value there, then you start to understand how much value you're 357 00:19:35,560 --> 00:19:38,080 leaving on the table. And then this can be an 358 00:19:38,080 --> 00:19:42,280 opportunity to invest in those practices to be able to maximize 359 00:19:42,280 --> 00:19:44,480 the lifetime value of an ML product. 360 00:19:44,960 --> 00:19:47,320 But then that requires a certain, again, a certain degree 361 00:19:47,320 --> 00:19:50,800 of maturity in and a certain ability to work across the 362 00:19:50,800 --> 00:19:53,240 existing organizational boundaries or reshape them. 363 00:19:54,240 --> 00:19:57,240 Yeah. And that's also kind of the flip 364 00:19:57,240 --> 00:20:00,200 side of the failure modes is the success, right? 365 00:20:00,200 --> 00:20:02,960 So we've also worked with and seen teams, ML teams 366 00:20:02,960 --> 00:20:06,320 successfully deliver, you know, really great demo products and 367 00:20:06,920 --> 00:20:09,880 what tactics do they use to succeed like we could see. 368 00:20:10,360 --> 00:20:12,960 Things like customer testing and user testing. 369 00:20:12,960 --> 00:20:17,200 So before they build out a prototype or MVP, which is a 370 00:20:17,200 --> 00:20:20,400 really expensive way to test something, even just talking to 371 00:20:20,400 --> 00:20:24,120 users, understanding what is the pinpoint, what are their jobs to 372 00:20:24,120 --> 00:20:28,440 be done, where could ML help? And then once they bought in, 373 00:20:28,440 --> 00:20:31,720 they say, OK, this is the right ML product that will solve those 374 00:20:31,720 --> 00:20:35,600 problems. Then combating that failure mode 375 00:20:35,600 --> 00:20:38,960 of what they've mentioned, like multiple teams kind of throwing 376 00:20:38,960 --> 00:20:42,480 across each other. If you remember the DevOps comic 377 00:20:42,480 --> 00:20:44,480 where you have devs on one side, you have OPS. 378 00:20:44,840 --> 00:20:49,560 So instead of throwing code between scientists and you know, 379 00:20:49,880 --> 00:20:53,200 ML OPS engineers, you know, we do that inverse Conway manoeuvre 380 00:20:53,200 --> 00:20:57,120 where we have a cross functional team shipping end to end for 381 00:20:57,120 --> 00:21:00,320 this initiative. And when we talk to the team 382 00:21:00,320 --> 00:21:04,160 members who worked on that project, they enjoyed that end 383 00:21:04,160 --> 00:21:06,280 to end ownership, the satisfaction of putting 384 00:21:06,280 --> 00:21:08,320 something releasing to customers. 385 00:21:08,720 --> 00:21:11,720 You get to touch different parts of the stack, learn about data 386 00:21:11,720 --> 00:21:14,560 science, you learn about OPS, you know, so they help them. 387 00:21:14,560 --> 00:21:16,000 You know, you can get fast feedback. 388 00:21:16,000 --> 00:21:19,480 You have the same standard. You ship things iteratively. 389 00:21:19,480 --> 00:21:22,120 You showcase it every, you know, 2 springs, 3 springs. 390 00:21:22,400 --> 00:21:25,520 So that feeling of flow and speed was like really amazing. 391 00:21:26,240 --> 00:21:28,280 And then, you know, when something is released, you know, 392 00:21:28,280 --> 00:21:31,760 you've got your continuous delivery, production monitoring 393 00:21:32,000 --> 00:21:35,760 when things are not going well. Yeah, and you'll have safety for 394 00:21:35,760 --> 00:21:38,120 changes. If I need to make a refactoring 395 00:21:38,120 --> 00:21:43,200 upgrade library, If your CI checks are all green, your model 396 00:21:43,200 --> 00:21:46,920 monitoring dashboard is saying performance is good, then you 397 00:21:46,920 --> 00:21:50,040 know, you merge that PR and you don't feel nervous or stressful 398 00:21:50,040 --> 00:21:52,560 about any releases. So yeah, there there. 399 00:21:52,560 --> 00:21:55,760 There are also bright spots in in how teams deliver ML 400 00:21:55,760 --> 00:21:57,560 products. Right. 401 00:21:57,720 --> 00:21:59,640 So I think when I heard you mentioned, you know in the 402 00:21:59,640 --> 00:22:02,520 beginning about POC, hell, you know, so many duct tapes. 403 00:22:02,520 --> 00:22:04,520 So when you bring it to production, I could relate to 404 00:22:04,520 --> 00:22:06,920 some other ML projects that I knew of before. 405 00:22:07,200 --> 00:22:10,560 So I think many, many ML teams probably in this mode, right? 406 00:22:10,560 --> 00:22:14,240 They keep building POC because maybe the expectation from the 407 00:22:14,240 --> 00:22:17,040 stakeholders or from the business also kind of like with 408 00:22:17,040 --> 00:22:19,360 a lot of hype, right? Because these days people think 409 00:22:19,360 --> 00:22:22,960 AI can do a lot of magic, right? And especially with all the 410 00:22:22,960 --> 00:22:26,320 success of GGPT and all that, they even think that it is easy 411 00:22:26,320 --> 00:22:28,640 to do. But I think the reality may not 412 00:22:28,640 --> 00:22:31,200 be the case, right? And the other thing is about the 413 00:22:31,200 --> 00:22:34,520 practices, right? So many ML teams first, either 414 00:22:34,520 --> 00:22:37,680 they don't have many good software engineers, so they are 415 00:22:37,680 --> 00:22:41,760 like data scientists or you know, ML engineers who just have 416 00:22:41,760 --> 00:22:43,560 a lot of expertise in building the models. 417 00:22:43,560 --> 00:22:46,080 But actually they don't have all these other skills like 418 00:22:46,080 --> 00:22:48,560 refactoring, continuous delivery that you mentioned, right, 419 00:22:48,560 --> 00:22:51,400 testing as well. Or the other flip side, which is 420 00:22:51,600 --> 00:22:54,680 a lot of software engineers being turned into ML engineers. 421 00:22:55,080 --> 00:22:58,120 So they are not necessarily an ML engineer, but they just come 422 00:22:58,120 --> 00:23:00,840 from, you know, like web application development and turn 423 00:23:00,840 --> 00:23:03,360 into ML engineer. So I think some of these things 424 00:23:03,360 --> 00:23:06,120 definitely I can relate. And just to add to that, there's 425 00:23:06,120 --> 00:23:10,560 also the flip side of engineers or ML engineers being excellent 426 00:23:10,720 --> 00:23:14,280 engineers, but not treating it as a data science problem Where 427 00:23:14,640 --> 00:23:19,480 there is uncertainties is the need for experimentation to get 428 00:23:19,480 --> 00:23:22,320 fast feedback. So yeah, I think the kind of 429 00:23:23,120 --> 00:23:26,360 emphasizes the need I think for cost functional collaboration 430 00:23:26,920 --> 00:23:31,360 from both like data science from I know OPS from SRE, things like 431 00:23:31,360 --> 00:23:33,720 that. So maybe let's go there, right? 432 00:23:33,720 --> 00:23:36,600 So the composition of the team, you mentioned a few times about 433 00:23:36,600 --> 00:23:39,640 cross functional teams and you know, these silos between, for 434 00:23:39,640 --> 00:23:43,560 example, data science or maybe the ML OPS or whoever operates 435 00:23:43,560 --> 00:23:46,320 the system in the end, maybe there are also other functions 436 00:23:46,320 --> 00:23:49,040 like for example, other teams, because ML typically also 437 00:23:49,040 --> 00:23:52,240 realized a lot of dependencies like data or maybe it's some 438 00:23:52,240 --> 00:23:53,680 kind of model or things like that. 439 00:23:54,000 --> 00:23:56,840 So maybe what are the best composition here in your 440 00:23:56,840 --> 00:23:58,480 experience? Maybe you can advise us? 441 00:23:59,360 --> 00:24:03,440 I guess here there's another question that is and it's what 442 00:24:03,600 --> 00:24:05,480 we tackle towards the end of the book. 443 00:24:05,920 --> 00:24:10,680 We look at sort of multiple levels of team effectiveness and 444 00:24:10,960 --> 00:24:13,800 one of the levels we look at is an individual in a team, the 445 00:24:13,800 --> 00:24:16,960 types of practices you can use, the expertise you can build as 446 00:24:16,960 --> 00:24:19,640 an individual. The next level we look at is 447 00:24:19,640 --> 00:24:22,120 within teams. Again, how those practices 448 00:24:22,120 --> 00:24:26,160 around technology delivering product augment teams but also 449 00:24:26,440 --> 00:24:29,320 broader team dynamics. But then the third level we look 450 00:24:29,320 --> 00:24:33,440 at is between teams. So how can teams be effective 451 00:24:33,440 --> 00:24:36,000 between teams? And so the so the best make up 452 00:24:36,000 --> 00:24:39,960 of a team depends a bit on the shape of the team and its 453 00:24:39,960 --> 00:24:43,160 interactions with other teams. And so this is where we use the 454 00:24:43,160 --> 00:24:47,040 team topologies model to identify the different types of 455 00:24:47,040 --> 00:24:51,120 ML teams that you might sit in. And so to run through it quickly 456 00:24:51,120 --> 00:24:53,960 and we can come back and dive into details, you know, you, you 457 00:24:53,960 --> 00:24:57,800 might have a streamline team and this we'd say this is the basic 458 00:24:57,800 --> 00:25:01,480 unit to think of first ML project, a streamline team that 459 00:25:01,480 --> 00:25:03,720 can deliver end to end. You're not going to be 460 00:25:04,120 --> 00:25:06,200 conducting ground breaking ML research. 461 00:25:06,200 --> 00:25:09,760 You're going to be using tried and true techniques in this 462 00:25:09,760 --> 00:25:12,080 case. Or at the other end, where you 463 00:25:12,080 --> 00:25:15,680 have an established ML ecosystem internally, you can add another 464 00:25:15,680 --> 00:25:18,560 streamline team that draws on those existing services 465 00:25:18,560 --> 00:25:23,720 internally quite easily. Then as you scale beyond one 466 00:25:23,720 --> 00:25:26,120 team, there might be two routes that you take. 467 00:25:26,560 --> 00:25:29,880 And so you might go down the route of where you have a sort 468 00:25:29,880 --> 00:25:33,880 of low level common concerns across different initiatives. 469 00:25:33,920 --> 00:25:36,520 That might be the opportunity for a technology platform to 470 00:25:36,520 --> 00:25:41,120 support ML at at at a lower level like compute and, and, and 471 00:25:41,120 --> 00:25:42,720 data and and feature engineering. 472 00:25:43,520 --> 00:25:46,400 And so that would be a platform team in the the team topologies 473 00:25:46,400 --> 00:25:51,200 team shape and that would aim to provide as a service and that 474 00:25:51,200 --> 00:25:53,680 would be composed differently from a stream aligned team. 475 00:25:54,080 --> 00:25:56,600 The other route you might take to scale is you might identify 476 00:25:56,600 --> 00:25:59,880 business clusters of ML needs. So there might be something 477 00:25:59,880 --> 00:26:03,040 around audience engagement or there might be something around 478 00:26:03,240 --> 00:26:06,240 asset valuation or there might be something around content 479 00:26:06,240 --> 00:26:08,160 moderation. And so those are sort of 480 00:26:08,160 --> 00:26:11,040 specific set of business needs that also come with an 481 00:26:11,040 --> 00:26:14,600 associated set of ML paradigms as well and that don't 482 00:26:14,600 --> 00:26:17,960 necessarily have a common support in a technology platform 483 00:26:18,160 --> 00:26:20,400 at that right business level of abstraction. 484 00:26:20,920 --> 00:26:23,760 So then you might have a what's called a complicated subsystem 485 00:26:23,760 --> 00:26:25,760 team. And again, the makeup of that 486 00:26:25,760 --> 00:26:28,920 might be a little bit different. And then as you, you know, as 487 00:26:28,920 --> 00:26:32,040 you have a bigger ecosystem, then there'll be a range of 488 00:26:32,040 --> 00:26:36,080 problems that come up of a similar nature, but like have a 489 00:26:36,080 --> 00:26:39,400 unique presentation each time, which might be around, say 490 00:26:39,400 --> 00:26:44,840 privacy or ethical use of data or optimizing particular 491 00:26:44,840 --> 00:26:46,960 techniques. And this is where you might have 492 00:26:46,960 --> 00:26:51,080 an enabling ML team as well that sort of acts as a consultancy to 493 00:26:51,080 --> 00:26:53,160 other ML teams to make themselves redundant. 494 00:26:53,600 --> 00:26:57,720 And so that was a long way of saying it depends on the ideal 495 00:26:57,720 --> 00:27:01,720 make up of a team, but you might assume it's a stream aligned 496 00:27:01,720 --> 00:27:04,360 team and talk about the roles that sit in that. 497 00:27:04,360 --> 00:27:07,880 Maybe I'll throw to David. I think Team Topologies is a 498 00:27:07,880 --> 00:27:11,440 really useful set of constructs, the ones that they've went 499 00:27:11,440 --> 00:27:16,840 through because it helps teams scale through the team's API. 500 00:27:17,480 --> 00:27:22,120 So as Dave mentioned, if we treat everything, let's say as a 501 00:27:22,120 --> 00:27:24,480 stream align team, like in software engineering, we're very 502 00:27:24,480 --> 00:27:26,240 familiar with cross functional teams. 503 00:27:27,120 --> 00:27:30,720 But then the failure mode there is that teams like Conway's Law, 504 00:27:30,720 --> 00:27:32,000 right? We've got three teams. 505 00:27:32,000 --> 00:27:34,480 We're going to build three sets of, you know, basically some 506 00:27:34,640 --> 00:27:37,360 architecture, rebuild certain tools that we need. 507 00:27:37,920 --> 00:27:40,360 Then that's where like maybe platform team comes in to 508 00:27:40,360 --> 00:27:44,200 abstract all of that. So then with that, what we've 509 00:27:44,200 --> 00:27:47,800 seen worked really well in one particular case was this 510 00:27:47,800 --> 00:27:50,280 complicated subsystem team. Like as you mentioned, ML is 511 00:27:50,280 --> 00:27:54,320 hard, a lot of moving parts they took on the effort to build this 512 00:27:54,360 --> 00:27:58,640 ML product. Let's imagine it's, I can't say 513 00:27:58,640 --> 00:28:01,200 this specific example, but let's say it's a car valuations, 514 00:28:01,240 --> 00:28:03,440 right? That's kind of ML product data 515 00:28:03,440 --> 00:28:05,680 product. The API to this team or this 516 00:28:05,680 --> 00:28:11,480 product is I will give you the planning of cars and now that is 517 00:28:11,480 --> 00:28:15,160 self serviceable by other teams, they could embed it on the 518 00:28:15,160 --> 00:28:17,600 mobile. Mobile team could integrate with 519 00:28:17,600 --> 00:28:20,680 this API and expose that ML capability. 520 00:28:21,120 --> 00:28:24,320 A web team can do the same or e-mail marketing team can do the 521 00:28:24,320 --> 00:28:28,720 same and then send personalized emails to say you know about car 522 00:28:28,720 --> 00:28:31,360 evaluations. So then that was how that team 523 00:28:31,360 --> 00:28:34,080 scaled rather than having to integrate with each particular 524 00:28:34,080 --> 00:28:38,520 thing, encapsulating that complicated subsystem as kind of 525 00:28:38,520 --> 00:28:41,400 formal set of APIs either through batch or through real 526 00:28:41,400 --> 00:28:44,040 time that help the team like achieve more and get more 527 00:28:44,040 --> 00:28:46,360 mileage out of the ML product they built. 528 00:28:46,880 --> 00:28:48,560 Yes, yes, that's a great example. 529 00:28:48,560 --> 00:28:51,080 The complicated subsystem team is probably going to have a 530 00:28:51,240 --> 00:28:53,960 bunch of specialists. It's going to look maybe most 531 00:28:53,960 --> 00:28:56,720 like what people imagine an ML team looks like. 532 00:28:57,160 --> 00:28:59,480 But as David said, to scale their impact in the 533 00:28:59,480 --> 00:29:03,240 organization, they do need that business domain expertise. 534 00:29:03,320 --> 00:29:07,000 They do need to deliver as a service instead of constantly 535 00:29:07,000 --> 00:29:10,520 collaborating with other teams. You know, maybe some product 536 00:29:10,520 --> 00:29:13,000 thinking helps with that service definition as well. 537 00:29:13,520 --> 00:29:15,920 So, you know, that might be one team composition. 538 00:29:16,760 --> 00:29:20,440 Yeah, that's right. And if you replace that example 539 00:29:20,440 --> 00:29:23,240 there from correlations, which is a bit like I think not 540 00:29:23,240 --> 00:29:25,600 everybody can be with that, let's say it's like travel 541 00:29:25,600 --> 00:29:29,120 recommendations or product recommendations, then that one 542 00:29:29,120 --> 00:29:31,400 team that spent all the effort in building product 543 00:29:31,400 --> 00:29:35,280 recommendations now can impact multiple parts of the business 544 00:29:35,560 --> 00:29:39,040 by having the right team topology to have that fracture 545 00:29:39,040 --> 00:29:41,280 playing around. OK, my team is doing product 546 00:29:41,280 --> 00:29:45,080 recommendations and yeah, applying the kind of data 547 00:29:45,080 --> 00:29:49,040 product disciplines, exposing this as an API, encapsulating 548 00:29:49,040 --> 00:29:52,240 the details. Very exciting to hear about team 549 00:29:52,240 --> 00:29:55,280 topologies mentioned for ML product ML teams, right. 550 00:29:55,280 --> 00:29:56,960 So I think it's kind of like back then, right? 551 00:29:56,960 --> 00:30:00,480 It was a revolutionary approach to how we kind of like create 552 00:30:00,480 --> 00:30:02,480 different teams. And I, I'm glad that you brought 553 00:30:02,480 --> 00:30:06,040 it up because still I believe in many companies, they think, OK, 554 00:30:06,040 --> 00:30:08,800 we want to build ML product and ML project. 555 00:30:09,080 --> 00:30:11,920 They just hired a few data scientists or, you know, these 556 00:30:11,920 --> 00:30:15,400 ML experts and they just asked them to kind of like build the 557 00:30:15,400 --> 00:30:19,040 model first without actually involving the other aspects of 558 00:30:19,040 --> 00:30:21,520 software engineers. Could be the streamline team up 559 00:30:21,520 --> 00:30:24,480 from the product side, or it could be, you know, the data or 560 00:30:24,480 --> 00:30:27,160 it could be anything, right? But I think that tends to kind 561 00:30:27,160 --> 00:30:30,320 of like have its challenges. So I think bringing the concepts 562 00:30:30,320 --> 00:30:33,120 such as Steam Topologies to actually think holistically how 563 00:30:33,120 --> 00:30:36,040 we're going to the different ML projects is something right, 564 00:30:36,040 --> 00:30:37,560 really, really important. Yeah, yeah. 565 00:30:37,720 --> 00:30:40,400 So that's a really. Great point and I think one of 566 00:30:40,400 --> 00:30:43,680 the quotes opening quotes in our book was from Edwards Emming 567 00:30:43,880 --> 00:30:46,680 saying a bad system would be a good person every time. 568 00:30:47,120 --> 00:30:51,040 So you can hire the smartest data scientists, put them in an 569 00:30:51,040 --> 00:30:56,000 environment where it's not the right system, then, you know, we 570 00:30:56,000 --> 00:30:59,360 get what you described there. So the whole thesis of Outlook 571 00:30:59,360 --> 00:31:03,480 is how do we create those systems to help teams build the 572 00:31:03,480 --> 00:31:06,600 right thing, you know, solve the right problem, build the thing 573 00:31:06,600 --> 00:31:09,720 right, You know, engineering and my engineering data science. 574 00:31:10,120 --> 00:31:13,240 And then in a way that's right for people, I think what they've 575 00:31:13,240 --> 00:31:17,280 puts in a really good way, like it's not just shipping and 576 00:31:17,280 --> 00:31:20,680 shipping, but in a way that has the right team shape, right 577 00:31:20,680 --> 00:31:23,760 collaboration, more right trust, psychological safety and also 578 00:31:23,760 --> 00:31:26,840 the right processes to deliver ship early and often. 579 00:31:27,360 --> 00:31:29,880 So, yeah, I really was intrigued when you mentioned, you know, 580 00:31:29,880 --> 00:31:32,600 you can't just put a group of data scientists together or 581 00:31:32,720 --> 00:31:35,000 engineers together. You really got to create that 582 00:31:35,000 --> 00:31:38,440 system where they can, you know, ship to the right and solve the 583 00:31:38,440 --> 00:31:42,080 right problems. Yeah, thanks for adding that To 584 00:31:42,080 --> 00:31:44,480 come back to the theme of, you know this product discipline, 585 00:31:44,480 --> 00:31:46,000 right. So I think we know that a lot of 586 00:31:46,000 --> 00:31:49,240 ML projects doesn't make into production, right or they solve 587 00:31:49,240 --> 00:31:51,800 the wrong problem or they don't bring value, right. 588 00:31:52,120 --> 00:31:54,600 So I think they're, like they've mentioned in the beginning, 589 00:31:54,640 --> 00:31:58,560 there are a lot of uncertainties when you actually build ML model 590 00:31:58,560 --> 00:32:01,640 ML product, right? Because I mean, the way it works 591 00:32:01,640 --> 00:32:03,000 also, it's kind of like prediction. 592 00:32:03,000 --> 00:32:06,400 It's kind of like there's some kind of ambiguities inside LLM, 593 00:32:06,400 --> 00:32:09,600 there's a hallucination, right? So how can you actually come up 594 00:32:09,600 --> 00:32:12,720 with an approach such that, you know, when you first building 595 00:32:12,720 --> 00:32:16,000 the ML product, you can actually build something that is kind of 596 00:32:16,000 --> 00:32:19,640 like bringing business value either to the users or to the 597 00:32:19,640 --> 00:32:21,600 organization. So I think this is probably 1 598 00:32:21,960 --> 00:32:24,720 hard aspect as well. So typically how would you run 599 00:32:24,720 --> 00:32:27,080 this? It is hard and it's it goes 600 00:32:27,080 --> 00:32:30,320 beyond the technical, although the technical informs it. 601 00:32:30,320 --> 00:32:33,920 I, I find it's useful to think about the essential 602 00:32:33,920 --> 00:32:36,280 characteristics of of ML solutions. 603 00:32:37,000 --> 00:32:40,840 We're exploring them because we think they're going to be 604 00:32:40,840 --> 00:32:44,160 superhuman in some aspects. They're probably not going to 605 00:32:44,160 --> 00:32:45,960 beat the best experts in a field. 606 00:32:46,360 --> 00:32:49,960 That's maybe one myth that we should tackle straight up, but 607 00:32:50,320 --> 00:32:52,760 you know, they'll be able to do things at speed and scale that 608 00:32:52,760 --> 00:32:56,040 no human could. But then we need to also 609 00:32:56,040 --> 00:32:59,120 consider that they'll make mistakes by the very nature. 610 00:32:59,120 --> 00:33:01,480 Again, we wouldn't consider an ML solution if we knew the right 611 00:33:01,480 --> 00:33:04,040 answer every time. We'd write some rules instead. 612 00:33:04,360 --> 00:33:07,920 So there'll be some percentage of mistakes, however small, in, 613 00:33:07,920 --> 00:33:12,080 in any ML solution by design, as well as the unforeseen mistakes. 614 00:33:12,600 --> 00:33:15,800 And then when it comes to those mistakes by design, we really 615 00:33:15,800 --> 00:33:19,760 need to understand the the cost sensitivity of what's, you know, 616 00:33:19,840 --> 00:33:22,920 what value do we get out of a bunch of right answers and what 617 00:33:22,920 --> 00:33:25,720 is the impact of maybe a very small number of wrong answers, 618 00:33:25,720 --> 00:33:28,720 but they could have a very huge impact across all sorts of 619 00:33:28,720 --> 00:33:33,640 dimensions to financial as well as security bias and fair 620 00:33:33,640 --> 00:33:36,240 treatment of all our stakeholders. 621 00:33:37,280 --> 00:33:40,720 So yeah, we'd need to consider that fallible nature of them as 622 00:33:40,720 --> 00:33:42,800 well. And so, you know, we need to 623 00:33:43,320 --> 00:33:45,880 start with products that are designed to handle that failure. 624 00:33:45,880 --> 00:33:48,840 You know, they have some upside from when ML gets it right, but 625 00:33:48,840 --> 00:33:52,440 they're robust to the times when ML gets it wrong because it 626 00:33:52,440 --> 00:33:53,720 will. So starting from that 627 00:33:53,720 --> 00:33:56,760 perspective, you know, we can then identify some experiments 628 00:33:56,760 --> 00:33:58,880 about how well does this need to work. 629 00:33:59,080 --> 00:34:01,360 We can start with very simple baselines. 630 00:34:01,720 --> 00:34:05,320 Sometimes, you know, even just predicting the majority class or 631 00:34:05,320 --> 00:34:08,480 or random guessing and you know, seeing how that works as as a 632 00:34:08,480 --> 00:34:12,520 product, but you know, ideally yet to resolve this uncertainty, 633 00:34:12,520 --> 00:34:15,440 we're getting into some real data and understanding the 634 00:34:15,440 --> 00:34:17,159 predictive potential of the real data. 635 00:34:17,880 --> 00:34:20,280 And so this is again where it's like it's a challenging 636 00:34:20,280 --> 00:34:23,199 multidisciplinary exercise where we're trying to proceed on 637 00:34:23,199 --> 00:34:25,560 multiple fronts. So we building the right thing 638 00:34:25,880 --> 00:34:29,320 can out, yet can our solution support or a proposed solution 639 00:34:29,320 --> 00:34:32,239 support the performance that we expect and so on. 640 00:34:33,400 --> 00:34:35,199 Yeah. Well, just to add to that, I 641 00:34:35,199 --> 00:34:37,920 think that emphasizes the importance of that cross 642 00:34:37,920 --> 00:34:40,239 functional, the nature of the work as well. 643 00:34:40,239 --> 00:34:43,960 Like if we frame this as a data science or ML problem, then we 644 00:34:43,960 --> 00:34:49,679 can try our level best to go from, let's say 55% accuracy to 645 00:34:50,080 --> 00:34:52,800 99. You will never get to 100 and we 646 00:34:52,800 --> 00:34:56,639 will spend many, many weeks and months trying to get there. 647 00:34:57,320 --> 00:35:01,160 So it's not just the ML problem where we try to improve the 648 00:35:01,160 --> 00:35:05,640 model's accuracy or recall precision, but also how can we 649 00:35:05,640 --> 00:35:08,960 design for these failure modes of the ML model? 650 00:35:09,560 --> 00:35:13,640 So like for example, you know, hacks, displaying, designing a 651 00:35:13,640 --> 00:35:17,160 product in a way, right, to show users that, OK, this is not a 652 00:35:17,160 --> 00:35:20,880 competent prediction or this is a prediction, but would you 653 00:35:20,880 --> 00:35:24,040 correct that? And also maybe even giving users 654 00:35:24,040 --> 00:35:27,480 options like these are the top three, like the models, top one 655 00:35:28,200 --> 00:35:31,520 prediction may be, you know, not where we want it to be, but the 656 00:35:31,720 --> 00:35:34,720 top three, OK, is much higher. So designing the product in a 657 00:35:34,720 --> 00:35:38,480 way that mitigates these failure modes of the ML model. 658 00:35:39,240 --> 00:35:43,960 And yeah, I it's easy for teams if they don't have the right 659 00:35:43,960 --> 00:35:47,560 capabilities or right skill sets to try to solve it. 660 00:35:47,680 --> 00:35:50,840 Like if you are a hammer, everything is a nail and it's 661 00:35:50,840 --> 00:35:53,240 very costly to try to level up the accuracy. 662 00:35:54,080 --> 00:35:56,400 We may never get there. But yeah, having that cross 663 00:35:56,400 --> 00:35:58,720 functional approach, like how do we design it in a different way? 664 00:35:58,720 --> 00:36:01,200 How do we talk to users? Like the users find this OK, 665 00:36:01,560 --> 00:36:03,960 that's a kind of more holistic way to solve the problem. 666 00:36:04,200 --> 00:36:07,240 Yeah, that, that hammer and nail is, is really important as you 667 00:36:07,720 --> 00:36:13,080 start to shift to OK, yeah, how do we make this viable or how do 668 00:36:13,080 --> 00:36:15,600 we, yeah, make it viable from the perspective of solving the 669 00:36:15,600 --> 00:36:18,800 problem effectively, but also from being economically 670 00:36:18,800 --> 00:36:22,080 sustainable to to maintain it. And I guess there's another 671 00:36:22,080 --> 00:36:25,320 essential characteristic around it, the fact that ML solutions 672 00:36:25,320 --> 00:36:28,520 are narrow. The very training process is to 673 00:36:28,520 --> 00:36:30,920 optimize a loss function, which is a narrow definition of 674 00:36:30,920 --> 00:36:34,760 success, but they're composable. And you know, this is, I think 675 00:36:34,760 --> 00:36:37,360 where Gen. AI can be really interesting. 676 00:36:37,360 --> 00:36:40,120 And I'm not the only one to take this perspective, but I've 677 00:36:40,120 --> 00:36:43,400 described like Gen. AI is a stone soup for 678 00:36:43,400 --> 00:36:46,400 innovation. So the story of Stone Soup if if 679 00:36:46,400 --> 00:36:49,760 you haven't heard it, is that a weary traveller arrives at a 680 00:36:49,760 --> 00:36:52,840 village late at night and all they have in their knapsack is a 681 00:36:52,840 --> 00:36:55,040 stone. So they go to the first house in 682 00:36:55,040 --> 00:36:57,800 the village and they ask the villager there, could I have an 683 00:36:57,800 --> 00:37:01,040 onion to make stone soup? You know, I've got the stone, 684 00:37:01,040 --> 00:37:03,720 all I need from you is an onion. That villager says, oh great, 685 00:37:03,760 --> 00:37:06,440 yeah, I'll provide an onion. They go to the next house and 686 00:37:06,840 --> 00:37:09,480 repeat and ask for carrots and, you know, proceed around the 687 00:37:09,480 --> 00:37:11,840 village. By the end of that process, they 688 00:37:11,840 --> 00:37:15,000 cook up an amazing soup that feeds the whole village and it 689 00:37:15,000 --> 00:37:18,680 was all cooked from a stone. Considering that, you know, you 690 00:37:18,680 --> 00:37:21,000 might have a spark of an idea, you might be able to prototype 691 00:37:21,000 --> 00:37:24,080 it easily, but it might actually be made-up of many different 692 00:37:24,080 --> 00:37:26,840 components composed together to produce something that looks 693 00:37:26,840 --> 00:37:30,960 like that behaves intelligently. It needs to be factored into 694 00:37:30,960 --> 00:37:33,840 that process as well. And you know, one of those big 695 00:37:33,840 --> 00:37:36,560 components will be the differentiated data that you 696 00:37:36,560 --> 00:37:41,720 bring and often like the step from prototyping something in a 697 00:37:42,280 --> 00:37:45,560 experimental environment to actually plumbing those data 698 00:37:45,560 --> 00:37:48,840 pipelines in a way that's sustainable for production use. 699 00:37:49,120 --> 00:37:50,800 You know, that can be a major step as well. 700 00:37:50,800 --> 00:37:53,160 That needs to be considered upfront. 701 00:37:53,160 --> 00:37:55,720 And so, you know, when we've talked about, when we've 702 00:37:55,720 --> 00:37:58,720 explored AI and ML initiatives, you know, we put a heavy 703 00:37:58,720 --> 00:38:01,880 weighting on that factor of where will the data come from 704 00:38:01,880 --> 00:38:04,840 and yeah, how will you deliver it to the product or 705 00:38:04,840 --> 00:38:06,920 application. You know, that's again, you 706 00:38:06,920 --> 00:38:09,840 know, it's one of those laws where it always takes longer 707 00:38:09,840 --> 00:38:12,760 than you expect, even when you plan for it to take longer than 708 00:38:12,760 --> 00:38:16,360 you expected. Like any software engineering 709 00:38:16,360 --> 00:38:18,760 project out there, right? So I think the most important 710 00:38:18,760 --> 00:38:21,360 thing is ML product ML project, right? 711 00:38:21,360 --> 00:38:24,440 So you need to still have the product thinking concept in the 712 00:38:24,440 --> 00:38:26,440 very beginning, right? So you, you've mentioned a 713 00:38:26,440 --> 00:38:29,520 little bit about, you know, is user interviews experiment, you 714 00:38:29,520 --> 00:38:32,920 know, building prototypes, making sure the data that you 715 00:38:32,920 --> 00:38:36,120 feed into the OR the ML training and all that is also 716 00:38:36,120 --> 00:38:39,160 appropriate, right? And I, I like the mentioning 717 00:38:39,160 --> 00:38:42,560 about failure modes, right? Because unlike other products 718 00:38:42,560 --> 00:38:45,000 out there, we kind of like know the input and output that we 719 00:38:45,000 --> 00:38:49,120 want the features to be, right? So ML product typically could 720 00:38:49,120 --> 00:38:51,880 fail in AI don't know unpredictable way, right? 721 00:38:51,880 --> 00:38:54,280 So we have so many things mentioned in the news, like for 722 00:38:54,280 --> 00:38:57,640 example, Google, you know, image classification, you know, the 723 00:38:57,640 --> 00:39:02,440 ChatGPT and you know, Gemini or bot giving wrong hallucinating 724 00:39:02,440 --> 00:39:05,200 answers. So like, how do you tackle that? 725 00:39:05,200 --> 00:39:08,640 Plus, I think the whole aspect of, you know, data security 726 00:39:08,920 --> 00:39:11,880 bias, right? And also others associated 727 00:39:11,880 --> 00:39:15,120 aspects of Fair data that you use in the training, right? 728 00:39:15,440 --> 00:39:18,480 I think it's also another thing that you should put your product 729 00:39:18,480 --> 00:39:21,320 thinking concept holistically so that you can come up with a very 730 00:39:21,320 --> 00:39:25,040 useful and valuable product. So let's go to the other 731 00:39:25,040 --> 00:39:27,480 discipline, which is the engineering side, right? 732 00:39:27,720 --> 00:39:31,280 So I think what I could see in the past as well, like a lot of 733 00:39:31,280 --> 00:39:35,560 ML code is kind of like highly unstructured Jose, right? 734 00:39:35,560 --> 00:39:38,600 So it's like procedural. There's no proper modeling. 735 00:39:38,600 --> 00:39:41,880 It's just function calls over function calls, very complex to 736 00:39:41,880 --> 00:39:44,640 trace. So maybe in your view, and you 737 00:39:44,640 --> 00:39:47,040 mentioned in the beginning as well, you took a project to 738 00:39:47,040 --> 00:39:51,240 refactor ML project, right? So what are the disciplines that 739 00:39:51,240 --> 00:39:54,680 typically are lacking in the engineering aspect of ML product 740 00:39:55,000 --> 00:39:59,840 and how we could do better? Yeah, I personally can resonate 741 00:39:59,840 --> 00:40:03,400 and relate with that experience. I have to get glasses recently, 742 00:40:04,000 --> 00:40:07,480 maybe because of oh, but I think mainly because looking at too 743 00:40:07,480 --> 00:40:09,800 much code all the time and sometimes in the late nights 744 00:40:10,120 --> 00:40:11,880 because of strengths of you know, which is. 745 00:40:12,440 --> 00:40:16,680 Other things which we as a healthy team, you know, if we do 746 00:40:16,680 --> 00:40:20,600 those right practices with automated testing, with 747 00:40:20,680 --> 00:40:23,320 continuous delivery, automated deployment, then nobody should 748 00:40:23,320 --> 00:40:26,440 need to work late night. So I think a couple of things 749 00:40:26,440 --> 00:40:30,040 that you mentioned there. Number one, I think test 750 00:40:30,040 --> 00:40:33,800 automation is a big part. Like every ML engineer, data 751 00:40:33,800 --> 00:40:37,760 scientist we've worked with, they enjoy the automated tests 752 00:40:37,840 --> 00:40:41,840 that we introduced and added. And so it's, I think here at 753 00:40:41,840 --> 00:40:44,720 this point is a problem of information asymmetry. 754 00:40:45,120 --> 00:40:48,360 Like we've got pockets of teams of people who know how to do 755 00:40:48,440 --> 00:40:50,160 automated testing for ML systems. 756 00:40:50,600 --> 00:40:53,960 And every team that I've joined and worked with, they say, oh, I 757 00:40:53,960 --> 00:40:55,720 didn't know this, you could do that. 758 00:40:56,200 --> 00:40:59,600 So I think that's the desire and demand for, you know, more 759 00:40:59,600 --> 00:41:05,000 automated testing ML systems. 1 encapsulating story was we had a 760 00:41:05,000 --> 00:41:08,280 project, it has kind of close to 0 test coverage. 761 00:41:08,280 --> 00:41:12,480 It was an LLM system. So over time we added the test 762 00:41:12,480 --> 00:41:16,280 pyramid like the simple things like unit tests, integration 763 00:41:16,280 --> 00:41:19,840 tests to touch our whole LLM application, check that it's OK, 764 00:41:20,440 --> 00:41:22,320 those are still point based tests, right? 765 00:41:22,360 --> 00:41:27,000 And we also have that kind of more like deeper like model eval 766 00:41:27,000 --> 00:41:31,720 tests to suite off however many examples we run it 5 minutes 767 00:41:31,720 --> 00:41:35,840 later, we know OK model accuracy is 75% or whatever number that 768 00:41:35,840 --> 00:41:38,680 is. So then we had this automated 769 00:41:38,720 --> 00:41:44,800 dependency manager like on upgrade called SNC or renovate 770 00:41:44,800 --> 00:41:48,880 or dependent board. So SNC open up APR saying you 771 00:41:48,880 --> 00:41:51,400 need to upgrade this it automatically. 772 00:41:51,400 --> 00:41:54,360 The PR have all these screen takes tests for passing. 773 00:41:54,760 --> 00:41:56,640 We automatically trigger model eval. 774 00:41:56,640 --> 00:41:58,680 We know, OK, performance is just as good. 775 00:41:59,240 --> 00:42:02,760 So in 15 minutes we could merge the PR, no stress, no effort. 776 00:42:03,000 --> 00:42:07,000 So that's how we grow capacity of a team just by having this 777 00:42:07,400 --> 00:42:10,800 automated testing model eval. You touched a little bit on 778 00:42:10,840 --> 00:42:13,600 software design or code design as well. 779 00:42:14,240 --> 00:42:18,720 And so yeah, that's, I think any code base, not just ML is 780 00:42:18,720 --> 00:42:23,040 susceptible to that. And so I think taking that next 781 00:42:23,040 --> 00:42:28,880 level of discipline to say, can I attract function for this, Can 782 00:42:28,880 --> 00:42:33,920 I have a readable variable name, not just DF or X, all of those 783 00:42:34,200 --> 00:42:37,760 software hedging practices which we can link some in the show 784 00:42:37,760 --> 00:42:42,280 notes, single responsibility principle, my favorite one is 785 00:42:42,280 --> 00:42:45,840 open close principle like can you design something that is 786 00:42:45,840 --> 00:42:49,400 open to extension, but you don't have to modify it every time? 787 00:42:50,080 --> 00:42:54,680 Probably I'm going to too much detail there, but I think the 788 00:42:54,680 --> 00:42:58,640 design of it or lack of design sometimes stems from the lack of 789 00:42:58,640 --> 00:43:00,640 tests. As you know, if there's no test, 790 00:43:00,640 --> 00:43:03,880 then nobody can refactor. Refactoring so scary, so risky, 791 00:43:03,880 --> 00:43:06,440 like nobody does that. We pick the path of the least 792 00:43:06,440 --> 00:43:08,400 resistance. So yeah, I think the teams that 793 00:43:08,400 --> 00:43:12,480 we've worked with, the moment we added that safety harness, when 794 00:43:12,520 --> 00:43:15,600 you have that test on the path to production, then a lot of 795 00:43:15,600 --> 00:43:19,080 things can happen. One time we did massive 796 00:43:19,080 --> 00:43:23,480 refactoring of like this variable that was LinkedIn all 797 00:43:23,480 --> 00:43:26,320 different places. You know, it was, you use ID 798 00:43:26,320 --> 00:43:29,320 shortcut, replaced it in like, I don't know, 50 places. 799 00:43:29,920 --> 00:43:33,000 And then, you know, test pass commit done, right. 800 00:43:33,000 --> 00:43:35,720 It was like either effort reduced from maybe days of 801 00:43:35,720 --> 00:43:39,640 testing to again minutes and then in 20 minutes it was 802 00:43:39,640 --> 00:43:42,800 running in production. So yeah, it's like a lot of 803 00:43:42,800 --> 00:43:45,080 these practices that have been emerging. 804 00:43:45,080 --> 00:43:47,240 I think it's just about spreading it more and it's why 805 00:43:47,240 --> 00:43:51,360 we wrote the book so that our teams don't do late nights 806 00:43:51,360 --> 00:43:53,600 writing code like I did in one project. 807 00:43:54,080 --> 00:43:59,880 And yeah, just enjoy the flow and work life balance and yeah, 808 00:44:00,640 --> 00:44:03,440 zero stress and production deployments because, you know, 809 00:44:03,440 --> 00:44:05,440 tester passing, you've got production monitoring. 810 00:44:05,640 --> 00:44:08,800 So a lot of these engineering practices will, you know, really 811 00:44:08,800 --> 00:44:12,440 help team feel the joy and flow of with ML products. 812 00:44:13,320 --> 00:44:17,200 Yeah, I think the call out around testing is really crucial 813 00:44:17,200 --> 00:44:20,640 and actually allocating your effort effectively. 814 00:44:20,640 --> 00:44:24,400 So in a regular software product we might use the test pyramid to 815 00:44:24,760 --> 00:44:28,680 direct effort so that we have you know large number of cheap 816 00:44:28,680 --> 00:44:31,360 high level tests. You know, we have a medium 817 00:44:31,360 --> 00:44:35,040 number of integration level tests and then, you know, we 818 00:44:35,040 --> 00:44:37,520 won't have a small number of end to end tests. 819 00:44:38,000 --> 00:44:40,960 We can actually in when we're looking at ML applications and 820 00:44:40,960 --> 00:44:43,320 other data intensive applications, we can add a 821 00:44:43,320 --> 00:44:46,120 second dimension to that. So it's agreed instead of a 822 00:44:46,240 --> 00:44:48,720 instead of a period. And that dimension is the data 823 00:44:48,720 --> 00:44:51,320 dimension. So you know, we might have, you 824 00:44:51,360 --> 00:44:55,080 know, a lot of small cheap tests around individual data points. 825 00:44:55,480 --> 00:44:58,720 So Canaries, I guess you might, you might also describe those as 826 00:44:59,200 --> 00:45:03,120 we might also then have samples tests at a sample level. 827 00:45:03,120 --> 00:45:07,200 So these are going to give us a little bit more insight than a 828 00:45:07,200 --> 00:45:09,800 point data test. They're going to have some 829 00:45:09,800 --> 00:45:11,200 variability. So there's going to be some 830 00:45:11,200 --> 00:45:16,200 tuning there, but they offer us much faster feedback than the 831 00:45:16,200 --> 00:45:19,160 final level, which might be a kind of global, let's test on 832 00:45:19,160 --> 00:45:22,360 all the data that we have. And often people don't think 833 00:45:22,360 --> 00:45:25,760 hard enough about balancing the tests across that spectrum of 834 00:45:25,760 --> 00:45:27,760 data. So, you know, you might be able 835 00:45:27,760 --> 00:45:31,680 to do a very quick training run on a very small subset of data. 836 00:45:32,120 --> 00:45:34,160 If anything's misconfigured in the training run and the 837 00:45:34,160 --> 00:45:36,520 training doesn't work properly, for instance, you know, that 838 00:45:36,520 --> 00:45:38,840 will break and you'll get that feedback really quickly rather 839 00:45:38,840 --> 00:45:41,600 than waiting hours for it. But you might also, you know, 840 00:45:41,640 --> 00:45:44,640 the training might pass, you might have a, a lower benchmark 841 00:45:44,640 --> 00:45:47,640 or threshold for acceptable performance on that low run. 842 00:45:47,840 --> 00:45:50,360 But at least you've tested end to end and got some feedback 843 00:45:50,760 --> 00:45:55,160 with a, with a sample data set that it's, it's likely to work 844 00:45:55,440 --> 00:45:57,320 at the large scale. So it's all about, you know, 845 00:45:57,320 --> 00:46:01,920 bringing that feedback back. And I think when when we come to 846 00:46:01,920 --> 00:46:05,760 that uncertainty in the front end as well, also being 847 00:46:05,760 --> 00:46:09,200 thoughtful about how you test under those conditions of 848 00:46:09,200 --> 00:46:12,160 uncertainty, when you don't even know what it is you're looking 849 00:46:12,160 --> 00:46:14,600 for in exploratory data analysis. 850 00:46:15,240 --> 00:46:19,800 And so there it's kind of moving from the unknown unknowns to 851 00:46:19,800 --> 00:46:22,680 known unknowns to known knowns through testing. 852 00:46:23,040 --> 00:46:27,080 You know, visualisation is really key to be able to look at 853 00:46:27,080 --> 00:46:29,520 the data and understand what it's telling you or use 854 00:46:29,520 --> 00:46:32,200 automated tools to find relationships in the data. 855 00:46:32,760 --> 00:46:34,560 And then when you sort of understand what the data's 856 00:46:34,560 --> 00:46:37,240 telling you so that visualization doesn't look 857 00:46:37,240 --> 00:46:40,040 right, does it look like? I expect that can actually be a 858 00:46:40,040 --> 00:46:43,200 form of testing called a visualization driven development 859 00:46:43,240 --> 00:46:46,800 at times as well. But then once you understand 860 00:46:47,200 --> 00:46:50,040 qualitatively what you're looking for, then you know, 861 00:46:50,160 --> 00:46:52,720 there's a whole range of data science techniques that you can 862 00:46:52,720 --> 00:46:56,280 use to turn that into a binary expectation that can pass or 863 00:46:56,280 --> 00:46:58,800 fail. That, you know, might be useful 864 00:46:59,040 --> 00:47:01,560 in an exploratory environment, but then might also be something 865 00:47:01,560 --> 00:47:04,640 that you promote into a production integration pipeline 866 00:47:04,640 --> 00:47:07,480 as well as David was describing. So really, you know, thinking 867 00:47:07,480 --> 00:47:10,920 hard about how you use testing to get fast feedback all the way 868 00:47:10,920 --> 00:47:12,720 through the life cycle is pretty crucial. 869 00:47:13,520 --> 00:47:17,360 And just to add to that, LLMS, as you mentioned is all the rage 870 00:47:17,360 --> 00:47:20,600 for the past two years. And so one of the teams we 871 00:47:20,600 --> 00:47:23,920 worked with, we had to innovate and think about how to shift 872 00:47:23,920 --> 00:47:27,320 that. So as Dave mentioned, full 873 00:47:27,440 --> 00:47:31,240 evaluation can be costly, it takes time, it can cost money, 874 00:47:31,240 --> 00:47:34,680 especially for LRMS. So what we ended up doing was to 875 00:47:34,680 --> 00:47:38,360 shift again that left. So before we kick start with big 876 00:47:38,360 --> 00:47:43,720 eval or any further deployments, we had a integration test and 877 00:47:44,120 --> 00:47:46,920 one of the challenges that the team said was like how can you 878 00:47:46,920 --> 00:47:48,600 test something that's non deterministic? 879 00:47:48,600 --> 00:47:50,200 You know, the answer is different every time. 880 00:47:50,600 --> 00:47:54,920 So we had to, you know, we wrote a assertion function that 881 00:47:54,920 --> 00:47:57,760 asserts on intent rather than vocabulary. 882 00:47:58,040 --> 00:48:01,840 So we can still evaluate that this response given these 883 00:48:01,840 --> 00:48:04,440 conditions. Yes, it's the intent of what we 884 00:48:04,520 --> 00:48:10,080 we expect what we had expected. We were using pie ham Crest for 885 00:48:10,080 --> 00:48:14,080 that kind of amateur style extensions or you could use, you 886 00:48:14,080 --> 00:48:17,400 know, other tools as well. But yeah, I think sometimes at 887 00:48:17,400 --> 00:48:20,360 the forefront of this new capability, we have to be a bit 888 00:48:20,360 --> 00:48:23,680 creative on how can we shoot that left to get the feedback 889 00:48:23,680 --> 00:48:26,960 that they've mentioned. Right, thanks for mentioning 890 00:48:26,960 --> 00:48:29,640 some of these techniques. I'm always intrigued like you 891 00:48:29,640 --> 00:48:31,600 mentioned, right? So I'm always intrigued, how can 892 00:48:31,600 --> 00:48:34,680 you test something that is non deterministic and you know, so 893 00:48:34,680 --> 00:48:37,120 many variables that could come in into play, right? 894 00:48:37,400 --> 00:48:40,040 So I think thanks for bringing also the importance of automatic 895 00:48:40,040 --> 00:48:41,560 testing. I think I like what you 896 00:48:41,560 --> 00:48:43,080 mentioned when you explained that, right? 897 00:48:43,080 --> 00:48:45,320 There's a little bit of information asymmetry. 898 00:48:45,320 --> 00:48:48,160 Maybe there are some MLA engineers who are never exposed 899 00:48:48,160 --> 00:48:51,040 to some of these techniques and when they know it, actually they 900 00:48:51,040 --> 00:48:54,000 could actually follow the discipline and make sure that 901 00:48:54,000 --> 00:48:55,720 the products are getting better and better. 902 00:48:56,120 --> 00:48:59,280 And I like also the approach of, you know, slicing the data for 903 00:48:59,280 --> 00:49:02,800 different stages of tests. I think that's also key in 904 00:49:02,800 --> 00:49:05,920 making sure that the ML projects also kind of like still behaves 905 00:49:05,920 --> 00:49:08,720 as what we expect because like, for example, you can tweak the 906 00:49:08,720 --> 00:49:11,480 model a little bit, you know, the output can change so much, 907 00:49:11,480 --> 00:49:13,400 right? So we don't want that happen in 908 00:49:13,400 --> 00:49:16,080 the production. The other aspect of ML that 909 00:49:16,080 --> 00:49:19,160 people always talk about lately is about ML OPS, you know, 910 00:49:19,160 --> 00:49:23,520 building platforms, you know how to actually deploy a model and 911 00:49:23,520 --> 00:49:25,520 operate it. So maybe a little bit here. 912 00:49:25,560 --> 00:49:28,600 What do you think about ML OPS? Is it something that we all need 913 00:49:28,760 --> 00:49:31,920 for building ML product and what problem does it solve? 914 00:49:32,640 --> 00:49:35,880 Yeah, I think it's yet another tool in our toolkit, which I 915 00:49:35,880 --> 00:49:37,840 very much welcome. You know, back in the day we 916 00:49:37,840 --> 00:49:42,400 have to wrangle and think how to solve large scale distributed 917 00:49:42,400 --> 00:49:44,480 processing. But now there are these ML OPS 918 00:49:44,480 --> 00:49:48,240 tools that let you abstract away that concern. 919 00:49:48,440 --> 00:49:52,520 So in one case, one team we have built at ML platform where now 920 00:49:52,520 --> 00:49:55,800 the data center is anyone who doesn't have know anything about 921 00:49:55,800 --> 00:50:00,040 infrastructure or AWS or Kubernetes, they won't just 922 00:50:00,040 --> 00:50:03,880 write plain Python And say, I want to have this, you know, 923 00:50:03,880 --> 00:50:07,200 large vertical scaling this way, I want to fan out and all of 924 00:50:07,200 --> 00:50:11,200 that starting from Pythons Compute is one part of the ML op 925 00:50:11,200 --> 00:50:12,640 stack. You know, there's experiment 926 00:50:12,640 --> 00:50:14,960 tracking, which has really helped us as well. 927 00:50:15,400 --> 00:50:19,280 Every pull request runs an experiment that reports some 928 00:50:19,280 --> 00:50:23,000 results that we can kind of check over time if David creates 929 00:50:23,000 --> 00:50:26,400 a new PR that this is better than our champion model. 930 00:50:26,720 --> 00:50:29,680 So that ML OPS practice was really, really welcome as well. 931 00:50:30,320 --> 00:50:34,920 The challenge here is like too many tools and it's hard to 932 00:50:34,920 --> 00:50:37,680 navigate. Powerworks had this article 933 00:50:37,680 --> 00:50:41,160 called A Guide to Evaluating ML Platforms, which we could link 934 00:50:41,160 --> 00:50:43,000 in the show notes. That really helped. 935 00:50:43,000 --> 00:50:46,480 Like thinking about the capability, like some platforms 936 00:50:46,480 --> 00:50:48,640 try to do everything, some are narrow. 937 00:50:48,800 --> 00:50:51,800 So how do you pick what's right? And how do you avoid like 938 00:50:51,800 --> 00:50:54,760 shotgun surgery, like vendor coupling, things like that. 939 00:50:55,440 --> 00:50:57,840 But yeah, end of the day, I think ML OPS is about 940 00:50:58,680 --> 00:51:02,120 abstracting away complexity so that you can focus on solving 941 00:51:02,320 --> 00:51:06,040 the right problem, not having to deal with undifferentiated labor 942 00:51:06,480 --> 00:51:09,640 in your day-to-day work. And I think, yeah, coming back 943 00:51:09,640 --> 00:51:12,800 to the testing perspective as well, I think one of the things 944 00:51:12,800 --> 00:51:15,560 we highlight in the book to get the most out of ML OPS 945 00:51:15,560 --> 00:51:19,520 automation and abstraction, you also need to ensure that you're 946 00:51:19,520 --> 00:51:22,240 doing the right testing to give you confidence that when you're 947 00:51:22,240 --> 00:51:24,400 moving fast, you're doing so safely. 948 00:51:25,480 --> 00:51:29,200 Yep, and to add to that as well, like that's a key point of 949 00:51:29,200 --> 00:51:32,880 making the book in that ML OPS you can't ML OPS your problems 950 00:51:32,880 --> 00:51:35,920 away just like how you can't dev OPS your problems away. 951 00:51:36,360 --> 00:51:41,160 Last week you had on the show on last episode DX with Laura Tyco 952 00:51:41,840 --> 00:51:44,760 and talking about Dev X. So I really like this diagram of 953 00:51:44,760 --> 00:51:48,440 the DFX triangle. How do you get faster feedback 954 00:51:48,440 --> 00:51:50,000 loops? How do you manage cognitive 955 00:51:50,000 --> 00:51:51,360 load? How do you get in the flow 956 00:51:51,360 --> 00:51:54,720 state? ML OPS helps in a few ways, but 957 00:51:54,720 --> 00:51:56,760 ML is not going to write your tests for you. 958 00:51:57,360 --> 00:51:59,640 They're not going to make talk to users and make sure you're 959 00:51:59,640 --> 00:52:01,920 writing the same right features or implementing the right 960 00:52:01,920 --> 00:52:04,440 features. They're not going to make sure 961 00:52:04,440 --> 00:52:08,920 your code is nicely factored and readable so that you can stay in 962 00:52:08,920 --> 00:52:12,200 the flow. So, yeah, I think it's another 963 00:52:12,200 --> 00:52:14,480 tool button is to be coupled with these other disciplines 964 00:52:14,480 --> 00:52:17,800 that Dave and I mentioned in this podcast and in the bulk, 965 00:52:17,800 --> 00:52:20,480 Yeah. Yeah, thanks for the plug for 966 00:52:20,480 --> 00:52:22,120 the developer experience as well, right. 967 00:52:22,120 --> 00:52:25,080 So don't forget any kind of ML product essentially it's also 968 00:52:25,080 --> 00:52:27,120 like a software engineering problem, right? 969 00:52:27,120 --> 00:52:30,120 So it's a social technical. So don't forget also the aspect 970 00:52:30,120 --> 00:52:32,720 of this feedback loops, ecological safety you also 971 00:52:32,720 --> 00:52:33,920 mentioned at the beginning, right. 972 00:52:34,160 --> 00:52:36,360 So all this, like I mentioned at the very beginning, right, it's 973 00:52:36,360 --> 00:52:39,560 not just ML technicals that you need to understand, but it's 974 00:52:39,560 --> 00:52:42,040 actually at the end, it's a software engineering thing that 975 00:52:42,040 --> 00:52:44,080 you have to handle really, really well. 976 00:52:44,440 --> 00:52:46,760 So we have talked a lot about the other things as we move 977 00:52:46,760 --> 00:52:48,640 towards the end. Is there anything that we 978 00:52:48,640 --> 00:52:51,600 haven't covered that you think should be mentioned as well? 979 00:52:52,400 --> 00:52:56,680 I think one key take away is how do we make good easy as 980 00:52:56,680 --> 00:53:00,880 engineering leaders, as ML practitioners, We've talked a 981 00:53:00,880 --> 00:53:03,040 lot about a lot of different practices. 982 00:53:03,880 --> 00:53:07,920 If we can make good easy, then teams can kind of just by 983 00:53:08,000 --> 00:53:12,520 following the team practices, following exemplar repos, then 984 00:53:12,920 --> 00:53:17,680 you get that for free in your CICD setup, your test strategy, 985 00:53:18,160 --> 00:53:20,560 even maybe hygiene checks of talking to users. 986 00:53:20,560 --> 00:53:23,680 Have you put a business case together before you start asking 987 00:53:23,840 --> 00:53:25,480 people to work on this for six months? 988 00:53:25,960 --> 00:53:29,280 So yeah, it can get out of hand easily with so many moving 989 00:53:29,280 --> 00:53:31,000 parts. So I think as an engineering 990 00:53:31,000 --> 00:53:34,360 leader, how do we make good, easy, make teams on the ground 991 00:53:34,360 --> 00:53:37,880 when they get the mission to do a certain piece of work? 992 00:53:37,880 --> 00:53:40,120 Like it's kind of built into the way of working. 993 00:53:41,240 --> 00:53:42,760 Yeah. So I like it make good easy, 994 00:53:42,760 --> 00:53:44,840 right. So sometimes we all get excited 995 00:53:44,840 --> 00:53:47,080 about the technology, so many moving parts, so many 996 00:53:47,080 --> 00:53:48,880 technologies that we can play with, right. 997 00:53:48,880 --> 00:53:52,440 But we forgot to expect to make it easy for people to adopt, 998 00:53:52,480 --> 00:53:54,440 make it easy to get the buy in as well. 999 00:53:54,760 --> 00:53:56,120 So I think thanks for mentioning that. 1000 00:53:56,720 --> 00:53:58,640 So it's been an exciting conversation. 1001 00:53:58,640 --> 00:54:01,800 I learned a lot about what it takes to actually build an ML 1002 00:54:01,800 --> 00:54:03,760 projects, which I find it really complicated. 1003 00:54:04,320 --> 00:54:07,080 But as we reach the end of our conversation, I have one last 1004 00:54:07,080 --> 00:54:08,360 question that I'd like to ask you. 1005 00:54:08,520 --> 00:54:10,800 I call this the three technical leadership question. 1006 00:54:11,240 --> 00:54:14,240 Just treat a bit like an advice that you want to give to us as a 1007 00:54:14,240 --> 00:54:16,440 listeners. So what will be the three 1008 00:54:16,440 --> 00:54:18,680 technical leadership wisdom that you can share with us? 1009 00:54:19,480 --> 00:54:24,040 Shall I go first? I think again, in line with some 1010 00:54:24,040 --> 00:54:26,920 of the philosophy of the book, being able to take different 1011 00:54:26,920 --> 00:54:30,360 perspectives on technical problems is really key for your 1012 00:54:30,360 --> 00:54:34,320 leadership growth. And so looking for opportunities 1013 00:54:34,320 --> 00:54:38,000 to play different roles in projects, even for a short time, 1014 00:54:38,200 --> 00:54:41,040 it gives you that understanding of what other stakeholders 1015 00:54:41,040 --> 00:54:45,480 require and how to make them successful as you aim to be 1016 00:54:45,480 --> 00:54:49,240 successful yourself. Yep, I thought of 2. 1017 00:54:49,240 --> 00:54:53,720 So 1 is I call it focus, function and fire. 1018 00:54:54,360 --> 00:54:58,120 So this was an idea I got from Todd Henry in his book I think 1019 00:54:58,760 --> 00:55:02,920 Taming Tigers. So we are building things every 1020 00:55:02,920 --> 00:55:05,520 day. Teams can get distracted with 1021 00:55:05,520 --> 00:55:08,480 many things. So how can we to set our team up 1022 00:55:08,480 --> 00:55:10,840 for success? How can we give them that focus, 1023 00:55:11,040 --> 00:55:15,040 that clear mission, the milestone, the why we're doing 1024 00:55:15,040 --> 00:55:17,840 it for the customers, the benefit for the business, the 1025 00:55:17,840 --> 00:55:20,560 business case, Like how will this benefit, you know, the 1026 00:55:20,560 --> 00:55:23,240 metrics that we care about function, you know, the way of 1027 00:55:23,240 --> 00:55:27,440 working instead of throwing work or communications over two 1028 00:55:27,440 --> 00:55:29,800 teams. If have we got the right way of 1029 00:55:29,800 --> 00:55:33,840 working and fire, you know, meaning the kind of implicit 1030 00:55:33,840 --> 00:55:35,840 motivation like why are we doing this? 1031 00:55:35,840 --> 00:55:38,760 How is this helping people? How is this helping business? 1032 00:55:39,720 --> 00:55:44,040 So that that for me is one principle I take to my teams. 1033 00:55:44,520 --> 00:55:46,560 The second one was really interesting. 1034 00:55:46,560 --> 00:55:48,920 We covered this in the book as well about trust and 1035 00:55:48,920 --> 00:55:53,560 psychological safety. So when it's absent from the 1036 00:55:53,560 --> 00:55:58,200 room, then a simple conversation becomes a process of 1037 00:55:58,200 --> 00:56:00,160 bureaucracy. OK, I got to write up this 1038 00:56:00,160 --> 00:56:02,920 documentation. We've got to pre read it and 1039 00:56:02,920 --> 00:56:05,320 let's have a meeting in two weeks to discuss. 1040 00:56:05,960 --> 00:56:09,960 And when it's also not there, then team members are afraid to 1041 00:56:09,960 --> 00:56:12,160 voice concerns of how things might fail. 1042 00:56:12,160 --> 00:56:14,840 So then we continue to track down the wrong direction. 1043 00:56:15,240 --> 00:56:18,640 So, yeah, as kind of tech leaders, how do we make sure we 1044 00:56:18,640 --> 00:56:22,040 have, you know, embody and encourage and ensure we have the 1045 00:56:22,200 --> 00:56:24,960 psychological safety in the team so that we can all do our best? 1046 00:56:24,960 --> 00:56:28,920 Work on David's final point. This is what I would say that 1047 00:56:28,920 --> 00:56:32,520 idea of trust is, is also really important in a multidisciplinary 1048 00:56:32,520 --> 00:56:37,240 team and innovative initiatives. Under certain under conditions 1049 00:56:37,240 --> 00:56:41,200 of uncertainty, we want anyone to be able to speak out with 1050 00:56:41,680 --> 00:56:45,040 good ideas or concerns that things might be broken. 1051 00:56:45,400 --> 00:56:48,400 And so to be able to create those serendipitous moments as 1052 00:56:48,400 --> 00:56:51,200 well as the well understood moments that trust facilitates 1053 00:56:51,520 --> 00:56:53,120 is a pretty key focus for leaders. 1054 00:56:54,240 --> 00:56:56,520 Yeah, I like the last aspect that you mentioned about 1055 00:56:56,520 --> 00:56:59,320 psychological safety. So you also mentioned that if 1056 00:56:59,320 --> 00:57:01,920 it's not there, right, things can tend to become like a 1057 00:57:01,920 --> 00:57:03,240 bureaucratic kind of thing, right? 1058 00:57:03,240 --> 00:57:04,800 So I think that's really a good plot. 1059 00:57:05,360 --> 00:57:09,040 So if people want to, you know, learn more about these exciting 1060 00:57:09,040 --> 00:57:11,520 things that you mentioned in the book, or if they want to discuss 1061 00:57:11,520 --> 00:57:14,760 with you, maybe is there a place where they can find any of you 1062 00:57:14,760 --> 00:57:18,320 or both of you online? Yeah, you can find me on 1063 00:57:18,320 --> 00:57:22,720 LinkedIn, David Coles. Yep, and I'm David Feidt and I 1064 00:57:22,720 --> 00:57:25,280 can share a link in the show notes or we can share a link in 1065 00:57:25,280 --> 00:57:27,920 the show notes. And our book Effective Machine 1066 00:57:27,920 --> 00:57:30,280 Learning Teams is also the 1st chapter. 1067 00:57:30,280 --> 00:57:33,360 And the preface, actually, sorry, the preface is available 1068 00:57:33,360 --> 00:57:35,960 for free on the link that we can share in the show notes. 1069 00:57:35,960 --> 00:57:39,320 So that gives you an overview of everything we've talked about in 1070 00:57:39,320 --> 00:57:42,920 this podcast in seven pages. And also the book itself. 1071 00:57:42,920 --> 00:57:45,760 We can look in the show note on where you can yeah, read or 1072 00:57:45,760 --> 00:57:47,840 listen to it. Thank you so much. 1073 00:57:47,840 --> 00:57:51,800 So it's been a pleasure to have both of you David's in the show. 1074 00:57:51,800 --> 00:57:54,440 So I hope people learn a lot about the aspects of machine 1075 00:57:54,440 --> 00:57:56,880 learning model and software engineering good practices 1076 00:57:56,880 --> 00:57:59,720 anyway at the end. Yeah, Thanks so much, Henry. 1077 00:57:59,720 --> 00:58:01,880 This is a great chat. Thanks for having us, Henry. 1078 00:58:01,920 --> 00:58:02,480 That was great.