1 00:00:00,080 --> 00:00:02,880 That's what dev X or develop experience is all about, finding 2 00:00:02,880 --> 00:00:05,800 those friction points in the complete software development 3 00:00:05,800 --> 00:00:07,560 life cycle. So that goes all the way from 4 00:00:07,560 --> 00:00:11,280 onboarding to debugging systems in production and everything in 5 00:00:11,280 --> 00:00:12,960 between. People are not so much tied to a 6 00:00:12,960 --> 00:00:15,360 hypothesis, but they're usually very tied to their opinions. 7 00:00:15,400 --> 00:00:18,440 Instead of saying our CICD pipeline is slow, we need to fix 8 00:00:18,440 --> 00:00:21,400 it like this, You could say, I think if we change this, our 9 00:00:21,400 --> 00:00:24,400 pipeline will become 30% faster. And then you have an experiment. 10 00:00:24,720 --> 00:00:27,120 Writing code is never been the biggest bottleneck, right? 11 00:00:27,360 --> 00:00:29,960 You are still responsible for what you put in the PR and 12 00:00:30,240 --> 00:00:32,240 you're responsible for understanding the solution that 13 00:00:32,360 --> 00:00:34,240 the LLM came up with. Because in the end, you know 14 00:00:34,240 --> 00:00:37,040 that the LLM is not going to get paid at 3:00 AM at night, but 15 00:00:37,040 --> 00:00:39,440 you are. If you're curious about 16 00:00:39,440 --> 00:00:42,200 developer experience then this episode is for you. 17 00:00:42,360 --> 00:00:45,440 Especially nowadays with the genetic tooling in the mix. 18 00:00:45,880 --> 00:00:49,400 I'm joined by Buster Rd., he's tech lead software engineer and 19 00:00:49,400 --> 00:00:52,920 has always been in between hands on roles, developer experience 20 00:00:52,920 --> 00:00:55,920 and platform engineering, which makes him the right person to 21 00:00:55,920 --> 00:00:57,160 talk to. So enjoy. 22 00:01:00,960 --> 00:01:03,800 I had this conversation on the podcast and it was quite 23 00:01:03,800 --> 00:01:05,920 controversial, especially in the comment section where people 24 00:01:05,920 --> 00:01:08,560 say, well, if we don't accommodate for the future to a 25 00:01:08,560 --> 00:01:11,040 certain degree, and in this case we can say hexagonal 26 00:01:11,040 --> 00:01:13,960 architecture, then moving towards that, either it's never 27 00:01:13,960 --> 00:01:16,360 going to get prioritized or we're going to have too much 28 00:01:16,360 --> 00:01:19,320 sunken cost in a certain way that it's very hard to steer. 29 00:01:20,080 --> 00:01:22,160 But I'm hearing you say there's a there's a different way. 30 00:01:23,200 --> 00:01:25,440 Yeah, I think this is the the discussion you always have with 31 00:01:25,440 --> 00:01:27,480 your team, right? And something you have to fix 32 00:01:27,480 --> 00:01:30,640 within your team, I think. And then to choose a part that 33 00:01:30,640 --> 00:01:32,320 works for for everyone, I would say. 34 00:01:33,000 --> 00:01:36,640 But I've seen over the course of the past 10 years so many times 35 00:01:36,640 --> 00:01:39,760 that you accommodated for future change that never came. 36 00:01:40,400 --> 00:01:42,840 But you do get all the all the downsides. 37 00:01:43,800 --> 00:01:46,760 Like just the other day we had, we had to build a feature to 38 00:01:46,760 --> 00:01:49,240 import either ACSV file or an XML file. 39 00:01:49,640 --> 00:01:54,080 Don't get me started on that. But what's the requirement and 40 00:01:55,240 --> 00:01:56,440 why? Also, so you made this nice, 41 00:01:56,440 --> 00:01:58,720 nice interface so you accommodate for both of the file 42 00:01:58,720 --> 00:02:02,680 types and while you were finishing up the CSV part, the 43 00:02:02,680 --> 00:02:05,800 XML part got scrapped. OK, so now we have this 44 00:02:05,800 --> 00:02:08,160 interface that accommodates for both, but we only use one, 45 00:02:08,160 --> 00:02:11,360 right. And that that's, that was even 46 00:02:11,360 --> 00:02:12,760 something that was planned. Yeah. 47 00:02:12,760 --> 00:02:14,440 And even then you don't need it in the end, so. 48 00:02:14,680 --> 00:02:17,080 What did you do then? Did you scrap the XML part even 49 00:02:17,080 --> 00:02:19,480 though you built it or was the team like, well, we, we 50 00:02:19,480 --> 00:02:22,720 obviously didn't scrap it. No, see, I would have scrapped 51 00:02:22,720 --> 00:02:23,640 it. It's like we're never going to 52 00:02:23,640 --> 00:02:26,440 use this. Yeah, this is a funny, but it 53 00:02:26,440 --> 00:02:29,600 like people are so opinionated. I feel like it's very inherent. 54 00:02:29,600 --> 00:02:32,000 And I also like that. Let's have a solid discussion. 55 00:02:32,480 --> 00:02:34,640 The hard part is people don't budget right. 56 00:02:34,760 --> 00:02:36,440 In the end, someone needs to make a decision. 57 00:02:36,440 --> 00:02:40,040 Do we go for simplicity or do we indeed develop the feature with 58 00:02:40,040 --> 00:02:42,800 future in mind? And then there's a fine line. 59 00:02:42,800 --> 00:02:44,880 It's not that black and white that you could just be like, we 60 00:02:44,880 --> 00:02:47,280 do this or we do that. There's a lot of grey in the 61 00:02:47,280 --> 00:02:50,080 middle and in the end you have to compromise somewhere. 62 00:02:50,560 --> 00:02:53,080 But it's very hard to bump heads and like find a resolution 63 00:02:53,080 --> 00:02:55,360 within a team. For sure, as I think it's very 64 00:02:55,360 --> 00:02:58,640 interesting point because you see this so often, right, that 65 00:02:58,640 --> 00:03:02,120 you have this discretion. And I, what I really like is 66 00:03:02,120 --> 00:03:04,440 have fair discussions where indeed, if you have good 67 00:03:04,440 --> 00:03:07,840 arguments, people are able to change their mind and go another 68 00:03:07,840 --> 00:03:10,280 direction, even if it wasn't, even if that was not the 69 00:03:10,280 --> 00:03:12,840 direction they originally thought of. 70 00:03:13,960 --> 00:03:17,320 And I think what really helps to achieve that is to form 71 00:03:17,320 --> 00:03:23,040 hypothesis over opinions. So instead of saying or CICD 72 00:03:23,040 --> 00:03:25,040 pipeline is slow, we need to fix it like this. 73 00:03:25,520 --> 00:03:28,360 You could say, I think if we change this, our pipeline will 74 00:03:28,360 --> 00:03:31,440 become 30% faster. And then you have an experiment. 75 00:03:32,040 --> 00:03:34,840 And if it doesn't work out, you know, people are not so much 76 00:03:34,840 --> 00:03:37,160 tied to a hypothesis, but they're usually very tied to 77 00:03:37,160 --> 00:03:39,080 their opinions. I like that a lot. 78 00:03:39,800 --> 00:03:42,600 I always think that the people I want to work with, I want them 79 00:03:42,600 --> 00:03:46,560 to be open minded and I also want them to be opinionated, 80 00:03:46,560 --> 00:03:49,560 right? But if you combine the 2, I feel 81 00:03:49,560 --> 00:03:52,600 like I as a person, I'm very stubborn, but I have no trouble 82 00:03:52,600 --> 00:03:55,240 saying, well, I believe this and then I get proven otherwise. 83 00:03:55,240 --> 00:03:56,520 I'm like, well, I've changed my mind. 84 00:03:56,680 --> 00:03:59,400 I agree with this. It's I have no issues with that. 85 00:03:59,760 --> 00:04:02,000 But now to be honest, like that. And I want the people that have 86 00:04:02,000 --> 00:04:04,880 the flexibility that I can also say, OK, I was wrong, right? 87 00:04:04,880 --> 00:04:08,200 And with new insights, with proven results, I now have a 88 00:04:08,200 --> 00:04:11,120 different mindset because we've shown results and that fits in 89 00:04:11,120 --> 00:04:12,920 this context. And maybe in another context 90 00:04:12,920 --> 00:04:15,040 it's different. But I want people to think like 91 00:04:15,040 --> 00:04:18,120 that and also introspect and not be afraid to say, well I was 92 00:04:18,120 --> 00:04:20,320 wrong in this case and it doesn't matter in the end. 93 00:04:20,600 --> 00:04:22,520 Yeah, exactly. You don't want to get married to 94 00:04:22,520 --> 00:04:24,240 your solutions, right? In the end, you want to be able 95 00:04:24,240 --> 00:04:28,040 to experiment and change things if the reality is a bit 96 00:04:28,040 --> 00:04:29,280 different than what you thought it was. 97 00:04:29,320 --> 00:04:32,080 Yeah, yeah. I think developer experience is 98 00:04:32,120 --> 00:04:35,120 is very interesting nowadays. You already mentioned that if 99 00:04:35,120 --> 00:04:37,840 you buy into a certain complexity for the future. 100 00:04:38,160 --> 00:04:41,680 I've been in that situation and then developing new features 101 00:04:41,880 --> 00:04:44,200 with my reference. I feel like I could, I can go 102 00:04:44,200 --> 00:04:47,520 really fast, but then through the architecture choices or 103 00:04:47,520 --> 00:04:50,200 through the complexity we already have inherently at a 104 00:04:50,200 --> 00:04:53,320 very early stage projects. I've had examples where I'm just 105 00:04:53,320 --> 00:04:56,520 slowed down or I have to do many things to get one thing in order 106 00:04:56,880 --> 00:04:58,720 on different places. And I'm like, well, this should 107 00:04:58,720 --> 00:05:00,800 not be the case at this stage yet. 108 00:05:00,960 --> 00:05:04,040 We should be able to fly because we're in Greenfield, but certain 109 00:05:04,040 --> 00:05:06,400 architectural decisions prevent us from doing so. 110 00:05:07,240 --> 00:05:09,040 And I, I cannot do that for long. 111 00:05:09,040 --> 00:05:11,760 I've noticed like I will fight and be like, we need to simplify 112 00:05:11,760 --> 00:05:13,520 because this cannot stand for a long time. 113 00:05:13,960 --> 00:05:15,760 Yeah. What has been your experience so 114 00:05:15,760 --> 00:05:18,440 far with battling complexity and developer experience? 115 00:05:18,920 --> 00:05:21,400 I think it's much broader than just architecture. 116 00:05:21,960 --> 00:05:25,840 So if you if you take a step back and see how we got here, 117 00:05:25,840 --> 00:05:27,800 right, we had the whole DevOps movement. 118 00:05:27,800 --> 00:05:30,640 I think what was it 1015 years ago that it started a little bit 119 00:05:30,800 --> 00:05:35,400 like that and DevOps brings like so many advantages. 120 00:05:36,000 --> 00:05:37,400 Yeah, but we're shifting everything left. 121 00:05:37,760 --> 00:05:41,240 So there's coming that that much more burden on the on the 122 00:05:41,240 --> 00:05:44,080 software engineers because in the past you just write your 123 00:05:44,080 --> 00:05:46,760 code and throw it over the fence and some teams can deploy it for 124 00:05:46,760 --> 00:05:48,600 you. And now you need to know about 125 00:05:48,800 --> 00:05:51,600 of course your code, but also Kubernetes, the cloud stuff, 126 00:05:52,200 --> 00:05:54,440 database, Estella form, etcetera, etcetera. 127 00:05:54,560 --> 00:05:56,200 It's a lot. It's it's really a lot. 128 00:05:56,200 --> 00:05:59,560 Yeah. I had a colleague, she used to 129 00:05:59,560 --> 00:06:02,320 say a shift left until there's only shift left. 130 00:06:02,360 --> 00:06:06,040 OK. And it's it's it's kind of true, 131 00:06:06,040 --> 00:06:09,720 I think because there's so much burden on these developers and 132 00:06:09,840 --> 00:06:12,240 that's what Dev X or developer experience is all about. 133 00:06:12,400 --> 00:06:15,640 It's about finding those friction points in the complete 134 00:06:15,640 --> 00:06:18,560 software development life cycle. So that goes all the way from on 135 00:06:18,560 --> 00:06:21,880 boarding, like your first day at a new client or a new project 136 00:06:22,480 --> 00:06:25,800 all the way to debugging systems in production and everything in 137 00:06:25,800 --> 00:06:28,760 between. So the very technical stuff like 138 00:06:28,760 --> 00:06:32,880 your architectural decisions, but also more the social stuff, 139 00:06:33,120 --> 00:06:36,160 it's also very important. So it's not just about tooling 140 00:06:36,160 --> 00:06:41,520 or CLIS or technologies, but also about, hey, do I actually 141 00:06:41,520 --> 00:06:45,240 get solid requirements? Yeah, how how many times do I 142 00:06:45,240 --> 00:06:48,040 deliver a feature and it needs to rework because the the 143 00:06:48,040 --> 00:06:50,560 requirements weren't crystal clear at the beginning, for 144 00:06:50,560 --> 00:06:54,440 example. Yeah. 145 00:06:54,440 --> 00:06:55,880 So that that's what Dev X is all about for me. 146 00:06:55,880 --> 00:06:58,080 It's about removing those fiction points from the entire 147 00:06:58,080 --> 00:07:00,880 software development life cycle, and not just on the technical 148 00:07:00,880 --> 00:07:03,400 side. Yeah, I had a conversation, this 149 00:07:03,400 --> 00:07:05,440 was yesterday actually. And when you're listening to 150 00:07:05,440 --> 00:07:08,400 this aired last week, but it was a, it was about platform 151 00:07:08,400 --> 00:07:10,240 engineering. And then we were discussing 152 00:07:10,240 --> 00:07:14,240 indeed very similar to developer experience, friction somewhere 153 00:07:14,240 --> 00:07:17,280 in the development life cycle going from idea to production, 154 00:07:17,280 --> 00:07:20,160 but then very specifically for tooling, I was like, why is this 155 00:07:20,160 --> 00:07:22,320 not called developer experience in the 1st place? 156 00:07:22,320 --> 00:07:25,040 Why is it platform engineering? And they mentioned that what we 157 00:07:25,040 --> 00:07:27,800 do focus a lot on tooling. There's a big emphasis on 158 00:07:27,800 --> 00:07:29,880 tooling. And when the problem is then 159 00:07:29,880 --> 00:07:33,800 process, when people only can spend 10 percent, 20% of their 160 00:07:33,800 --> 00:07:37,440 time hands on coding and 80% they just get dragged into 161 00:07:37,440 --> 00:07:40,240 meetings because of middle management or because of bad 162 00:07:40,240 --> 00:07:44,280 processes, then that's also a bigger part than just platform 163 00:07:44,280 --> 00:07:45,840 engineering. That's all of developer 164 00:07:45,840 --> 00:07:48,680 experience. I feel like in bigger 165 00:07:48,680 --> 00:07:51,640 organizations, the more you focus on developer experience. 166 00:07:51,640 --> 00:07:54,280 If you have a lot of engineers and you can make all of them a 167 00:07:54,280 --> 00:07:57,040 little bit more efficient or a little bit more effective, if 168 00:07:57,040 --> 00:07:59,440 we're talking about small percentages, then that's going 169 00:07:59,440 --> 00:08:01,560 to pay dividends. But I'm curious, from your 170 00:08:01,560 --> 00:08:03,920 perspective, even in the smallest organizations with only 171 00:08:03,920 --> 00:08:07,320 a few engineers, does developer experience matter and who should 172 00:08:07,320 --> 00:08:09,200 focus on Dev X in the first place? 173 00:08:10,000 --> 00:08:14,760 I think there is like a flipping point at around 50 engineers. 174 00:08:14,760 --> 00:08:19,200 I would say I only did Dev X at companies of yeah, with with 200 175 00:08:19,200 --> 00:08:22,120 or more engineers, SO2234500 engineers. 176 00:08:22,680 --> 00:08:26,200 And there you can get in a very big benefits if you make small 177 00:08:26,400 --> 00:08:31,600 gains because they're compound. But I think at around 50 178 00:08:31,600 --> 00:08:33,760 engineers, it's a bit like a guest guest for me. 179 00:08:33,760 --> 00:08:36,720 I'm not entirely sure because I never tried it, but I can 180 00:08:36,880 --> 00:08:38,840 imagine the flipping point is around somewhere that that 181 00:08:38,840 --> 00:08:43,200 stage, you know, that stage where you can't just where 182 00:08:43,200 --> 00:08:45,960 you're big enough so that you just can't just get everything 183 00:08:45,960 --> 00:08:49,080 done on the fly, right. If you're responsible for 184 00:08:49,080 --> 00:08:52,320 provisioning databases, and I'm in the development team, and if 185 00:08:52,320 --> 00:08:54,480 you're a small company, I can just send you a select message, 186 00:08:54,480 --> 00:08:55,960 say, hey man, can we know the best from you? 187 00:08:56,000 --> 00:08:59,640 Yeah, we know each other. And there's only like maybe two 188 00:08:59,680 --> 00:09:01,200 or three other development teams, right? 189 00:09:01,200 --> 00:09:03,800 So if they all ask you for a database, you're not kind of 190 00:09:03,800 --> 00:09:06,160 swamped in work. But if there's 10 or 20 191 00:09:06,160 --> 00:09:08,680 development teams and you have to provision databases for all 192 00:09:08,680 --> 00:09:10,880 of those, you're not going to be happy. 193 00:09:11,400 --> 00:09:15,000 That's our database profiter. The people that focus on 194 00:09:15,200 --> 00:09:18,520 developer experience, their knowledge also needs to be quite 195 00:09:18,520 --> 00:09:20,760 broad. Do they typically come from a 196 00:09:20,760 --> 00:09:23,040 hands on role? Do they come from a product or 197 00:09:23,040 --> 00:09:25,920 process role? Or who is typically responsible 198 00:09:25,920 --> 00:09:29,360 for developer experience? I think it really helps to have 199 00:09:29,360 --> 00:09:33,520 a deep technical understanding of how code is being written and 200 00:09:33,520 --> 00:09:35,240 how the software development life cycle works. 201 00:09:36,680 --> 00:09:39,440 Yeah, that's, that's for sure, for sure for a valuable because 202 00:09:39,440 --> 00:09:42,240 in the end what you want to do is or what we usually do is we 203 00:09:42,240 --> 00:09:45,200 start measuring the state of the effects within a company. 204 00:09:45,960 --> 00:09:50,120 So by usually by doing surveys and just getting sentimental 205 00:09:50,120 --> 00:09:52,600 data on a lot of possible friction points. 206 00:09:53,480 --> 00:09:55,840 So for example, you ask like how confident are you in shipping 207 00:09:55,840 --> 00:09:59,440 changes that nothing breaks and a whole bunch of other 208 00:09:59,440 --> 00:10:03,720 questions. And what you get then is a set 209 00:10:03,720 --> 00:10:08,520 of sentiments basically. And we allow the respondents 210 00:10:08,520 --> 00:10:13,040 also to rank the items in what I think is most important. 211 00:10:13,760 --> 00:10:15,920 So in the end, you have this list of sentiments and ranked 212 00:10:15,920 --> 00:10:18,320 based on importance. So you are looking for things 213 00:10:18,320 --> 00:10:21,560 that have low sentiment for whatever reason and have a 214 00:10:21,560 --> 00:10:27,040 high-ranking on priority. And once you have that list, 215 00:10:27,800 --> 00:10:29,560 yeah, you know, there's a friction point somewhere, but 216 00:10:29,560 --> 00:10:31,400 you don't know what is causing it. 217 00:10:31,760 --> 00:10:34,960 So the next step is to talk to engineers and of course the 218 00:10:34,960 --> 00:10:37,440 survey part you can do pretty well if you have the questions 219 00:10:37,600 --> 00:10:40,520 without any technical knowledge. But then the part where you talk 220 00:10:40,520 --> 00:10:43,240 to engineers and to interviews, yeah, you will need a lot of 221 00:10:43,240 --> 00:10:47,320 technical knowledge if the if the friction point is technical, 222 00:10:47,320 --> 00:10:51,840 of course, to determine what is actually causing it and how to 223 00:10:51,840 --> 00:10:54,200 possibly fix it. Those surveys, are they 224 00:10:54,480 --> 00:10:57,440 templates that you have online or do you make them custom for 225 00:10:57,440 --> 00:10:59,320 an organization that you've been involved with? 226 00:10:59,720 --> 00:11:01,840 How do the questions actually make sense in the end? 227 00:11:02,200 --> 00:11:04,480 Yeah, there's, there's different approaches you can take. 228 00:11:04,840 --> 00:11:08,360 So there's a couple of open source, how do you say it 229 00:11:08,560 --> 00:11:11,000 frameworks you can use. So there's the hard framework. 230 00:11:11,040 --> 00:11:12,320 I think there's a space framework. 231 00:11:12,320 --> 00:11:15,040 I also always forget what the abbreviation stands for. 232 00:11:15,040 --> 00:11:18,520 But there's, there's a pretty well documented and they explain 233 00:11:18,520 --> 00:11:20,680 you what, what kind of questions you can ask and how you can 234 00:11:20,680 --> 00:11:23,440 build this survey. And it's also commercial tooling 235 00:11:23,560 --> 00:11:29,840 like DX, which was accompanied by of acquired by Atlassian not 236 00:11:29,880 --> 00:11:32,720 a long ago. And I think that's, I think 237 00:11:32,720 --> 00:11:35,640 that's a great tool to do those surface because they spend a lot 238 00:11:35,640 --> 00:11:39,840 of time on the questions and yeah, the tooling itself, which 239 00:11:39,840 --> 00:11:42,040 is really nice to use. And what, what I think is really 240 00:11:42,040 --> 00:11:45,720 cool about DX also is that they gather like a lot of data from 241 00:11:45,720 --> 00:11:48,480 different companies. And of course I anonymize it, 242 00:11:48,920 --> 00:11:52,440 but you're able to compare yourself to other companies. 243 00:11:53,440 --> 00:11:57,120 So for example, you can you have this topic of requirements 244 00:11:57,120 --> 00:12:00,800 quality and you can compare your scores to the scores of other 245 00:12:00,800 --> 00:12:02,400 companies. And of course, you don't see the 246 00:12:02,400 --> 00:12:05,440 company names, but you can see in what percentile, yeah, you 247 00:12:05,440 --> 00:12:07,240 are scoring. Which is pretty cool. 248 00:12:07,680 --> 00:12:10,880 So if what we have is normal or is really just skewed? 249 00:12:10,880 --> 00:12:12,680 In a yeah, or maybe really good. Right. 250 00:12:12,680 --> 00:12:15,880 Yeah, that's interesting. I'm on Reddit a lot and I just 251 00:12:15,880 --> 00:12:17,720 get spammed with DX advertisement. 252 00:12:17,760 --> 00:12:18,720 Oh, really? Yeah, yeah, yeah. 253 00:12:19,040 --> 00:12:20,600 So when you said that, I was like, I know that whole yeah, 254 00:12:21,440 --> 00:12:24,160 that's really funny. In those conversations that you 255 00:12:24,160 --> 00:12:26,880 have, let's say we have a problem and you go to teams to 256 00:12:26,880 --> 00:12:29,680 kind of figure out what their workflow looks like and what 257 00:12:29,680 --> 00:12:32,720 that bottleneck actually is. How do you approach this 258 00:12:33,080 --> 00:12:35,360 interview or discovery conversation with them? 259 00:12:36,320 --> 00:12:38,760 Usually really open, right, because you have this list of 260 00:12:38,800 --> 00:12:43,040 where you have to survey results and usually they are are 261 00:12:43,040 --> 00:12:46,200 anonymised. So yeah, maybe send out a Slack 262 00:12:46,200 --> 00:12:48,680 message to an entire company saying, hey, we, we saw this in 263 00:12:48,680 --> 00:12:50,960 the server results. If you think this is 264 00:12:50,960 --> 00:12:53,640 interesting, you want to talk about it, you know, let's let's 265 00:12:53,640 --> 00:12:56,720 have a chat and that's usually the way we we get started. 266 00:12:56,720 --> 00:12:59,080 And then your interview like maybe, yeah, depending on the 267 00:12:59,080 --> 00:13:01,880 company size, maybe 5-6, seven different people and doing start 268 00:13:01,880 --> 00:13:04,480 hearing the same things over and over again and they usually know 269 00:13:04,480 --> 00:13:08,920 you have enough information. And then of course, it's really 270 00:13:08,920 --> 00:13:11,240 dependent on what the issue is. Now you're going to fix it. 271 00:13:11,440 --> 00:13:14,320 Yeah, interesting. I had a person on that was 272 00:13:14,320 --> 00:13:16,440 responsible for product at VID VID. 273 00:13:16,440 --> 00:13:19,000 Is this in the browser video editing tool, which is the 274 00:13:19,000 --> 00:13:22,440 largest one out there right now. And he said for us in a company, 275 00:13:22,440 --> 00:13:25,360 it's really important to talk to customers and they do discovery 276 00:13:25,360 --> 00:13:28,480 those interviews and they record the transcript of those 277 00:13:28,480 --> 00:13:31,040 conversations and they put it in a location somewhere. 278 00:13:31,040 --> 00:13:34,480 They do, if a lot of people do them weekly, you get a lot of 279 00:13:34,480 --> 00:13:37,160 transcripts of a lot of conversations, and then they 280 00:13:37,160 --> 00:13:39,240 built an AI tool that goes through all of the transcripts 281 00:13:39,240 --> 00:13:42,000 and finds like commonalities of user problems and stuff like 282 00:13:42,000 --> 00:13:44,800 that. When the organization gets that 283 00:13:44,800 --> 00:13:48,720 big in conversations with engineers, kind of scale to fix 284 00:13:48,720 --> 00:13:51,000 developer issues. I think that's a very 285 00:13:51,000 --> 00:13:53,040 interesting approach, but I haven't seen that applied to 286 00:13:53,040 --> 00:13:55,800 that level yet. I do feel like developers are 287 00:13:55,800 --> 00:13:58,880 the people that typically have a solution in their mind, but for 288 00:13:58,880 --> 00:14:01,400 some reason they're not enabled or they don't get the time to 289 00:14:01,400 --> 00:14:04,560 actually apply that. But then those solutions might 290 00:14:04,560 --> 00:14:07,120 also be incorrect sometimes and they don't really know what the 291 00:14:07,120 --> 00:14:10,360 problem actually is, which is the same as any other user to be 292 00:14:10,360 --> 00:14:12,320 honest. Is that also what you've seen? 293 00:14:12,800 --> 00:14:15,000 I really like the idea of, you know, recording those 294 00:14:15,000 --> 00:14:17,840 conversations and then putting it in an LLMI think that's, you 295 00:14:17,840 --> 00:14:19,480 know, I'm going to take that with me, I guess. 296 00:14:20,200 --> 00:14:21,200 So thanks for. Sure. 297 00:14:21,200 --> 00:14:26,360 Yeah, yeah, great. Sorry, what was the question 298 00:14:26,360 --> 00:14:28,520 again? The developers, I think they 299 00:14:28,520 --> 00:14:31,320 typically have a solution in mind, but they could either be 300 00:14:31,440 --> 00:14:34,520 on the spot and they're just not enabled to act upon that, or 301 00:14:34,520 --> 00:14:37,400 they don't have time, or they're not in a department to fix that, 302 00:14:37,400 --> 00:14:39,760 or they're actually incorrect and the problem is elsewhere. 303 00:14:40,000 --> 00:14:41,520 I'm wondering what your experience has been. 304 00:14:41,760 --> 00:14:44,240 Yeah, that's also why it's so valuable to start measuring 305 00:14:44,240 --> 00:14:48,560 those things because in those measurements without those 306 00:14:48,560 --> 00:14:51,400 measurements like dev access, abstract concepts of which we 307 00:14:51,400 --> 00:14:55,640 all think it should be improved, but you nobody knows quite how 308 00:14:55,640 --> 00:14:58,120 you know, you don't know if you start turning some knobs, what's 309 00:14:58,120 --> 00:15:00,240 going to happen and if it has the effect that you want. 310 00:15:00,880 --> 00:15:03,120 So by measuring it like continuously, like every 311 00:15:03,120 --> 00:15:07,520 quarter, every every six months, you get this very nice baseline 312 00:15:07,520 --> 00:15:11,760 in, in this tracking system to see if your improvements also 313 00:15:11,760 --> 00:15:12,880 make sense. Right. 314 00:15:14,240 --> 00:15:17,960 And I think one of the examples we had at the company is that we 315 00:15:18,880 --> 00:15:23,040 there was, there was like this already this idea that the gate 316 00:15:23,040 --> 00:15:25,320 lab pipelines were not so optimal because there were no 317 00:15:25,320 --> 00:15:29,000 spot instances and sometimes build builds would get killed 318 00:15:29,160 --> 00:15:30,840 halfway during the builds. You have to research builds 319 00:15:30,840 --> 00:15:32,160 quite annoying, takes a lot of time. 320 00:15:32,160 --> 00:15:34,280 Feedback loops are slow. It's not what you want. 321 00:15:34,960 --> 00:15:37,640 And it was not, it was a known problem, right? 322 00:15:38,120 --> 00:15:39,960 Or at least it there was a problem known to a lot of 323 00:15:39,960 --> 00:15:43,440 engineers. And then we did this survey and 324 00:15:43,440 --> 00:15:45,640 because we already knew there might be a friction point there, 325 00:15:45,640 --> 00:15:47,080 we ask a question in that direction. 326 00:15:47,200 --> 00:15:49,440 And we indeed saw like a huge percentage of the respondees 327 00:15:49,440 --> 00:15:51,000 saying, Hey, this is need a problem. 328 00:15:51,920 --> 00:15:54,240 And now we had like this, this data bit that we could take to 329 00:15:54,240 --> 00:15:56,240 the team responsible for the GitLab runners and say, hey, 330 00:15:56,480 --> 00:15:58,320 maybe we should have a look at this. 331 00:15:59,120 --> 00:16:01,480 And they made some fixes and they allowed GitLab runners to 332 00:16:01,480 --> 00:16:05,640 run on non spot instances. And in the next survey, we asked 333 00:16:05,640 --> 00:16:08,640 the question again and we see a huge improvement on that, on 334 00:16:08,640 --> 00:16:11,600 that specific topic. And now of course, this is a 335 00:16:11,600 --> 00:16:16,160 very, this is a thing you might not be able to change if you 336 00:16:16,160 --> 00:16:18,440 have much more local scope, like if you are an engineer in a 337 00:16:18,440 --> 00:16:21,120 team, you might not be able to make these kinds of changes. 338 00:16:23,000 --> 00:16:26,280 But I think a lot of effective defects initiatives, they start 339 00:16:26,440 --> 00:16:29,080 very much from bottom up. So they start really small. 340 00:16:30,160 --> 00:16:33,720 And I think a great way to make improvements for you as a team 341 00:16:33,720 --> 00:16:37,360 is just to take your refinement and ask like what slowed you 342 00:16:37,360 --> 00:16:40,200 down in the past quarter or in the past month or something. 343 00:16:40,920 --> 00:16:44,040 I see if if there are friction points that are small enough for 344 00:16:44,040 --> 00:16:47,680 your for your own team to solve. You know, because you might be 345 00:16:47,680 --> 00:16:51,200 able to write rewrite your whole CICD setup, but you might be 346 00:16:51,200 --> 00:16:55,440 able to make huge improvements to your test suite speeds for 347 00:16:55,440 --> 00:16:57,120 example. Yeah, I like that a lot. 348 00:16:57,560 --> 00:17:01,400 Is this survey that you mentioned you do quarterly or 349 00:17:01,400 --> 00:17:05,160 maybe a little less frequent, is that the most important way for 350 00:17:05,160 --> 00:17:07,400 you that gathers data that you've seen? 351 00:17:07,520 --> 00:17:10,200 Because I've also heard quantitative data durometrics, 352 00:17:10,520 --> 00:17:12,960 and I'm not sure how much people manipulate those to actually 353 00:17:12,960 --> 00:17:16,160 skew to the right way, but the survey seems very interesting 354 00:17:16,400 --> 00:17:18,240 actually. Yeah, the the the good thing 355 00:17:18,240 --> 00:17:19,720 about the surface, it's low cost. 356 00:17:19,880 --> 00:17:21,319 It's really easy to get started with you. 357 00:17:21,599 --> 00:17:24,359 You usually don't need a lot of permissions to send out the 358 00:17:24,359 --> 00:17:29,240 survey within your company. Dora metrics are also great, but 359 00:17:29,240 --> 00:17:32,920 I think they're way more valuable to measure how far 360 00:17:32,920 --> 00:17:36,280 along a company is into watch moving to a DevOps way of 361 00:17:36,280 --> 00:17:38,280 working. Because once you've hit that 362 00:17:38,280 --> 00:17:40,120 DevOps way of working, you're deploying multiple times to 363 00:17:40,120 --> 00:17:42,120 production a day. You're not going to see huge 364 00:17:42,120 --> 00:17:44,400 shifts in your Dora metrics, hopefully. 365 00:17:46,080 --> 00:17:48,520 So then your friction points will become a bit harder to 366 00:17:48,520 --> 00:17:51,200 detect because in Dora, you only have, I think, 4 metrics that 367 00:17:51,200 --> 00:17:53,320 you can use. And with the survey, you can 368 00:17:53,320 --> 00:17:54,680 have much more touch points of course. 369 00:17:54,880 --> 00:17:57,960 But the survey is really important and easy to get 370 00:17:57,960 --> 00:18:01,560 started with low cost. I think if you move along a 371 00:18:01,560 --> 00:18:04,360 little further in your defect journey, you also want to start 372 00:18:04,360 --> 00:18:06,960 looking at system data. And that sounds maybe more 373 00:18:06,960 --> 00:18:08,960 complicated than it is because you can get started pretty 374 00:18:08,960 --> 00:18:11,320 easily, right? If you are using Gitap or 375 00:18:11,320 --> 00:18:16,680 GitLab, there's already metrics there that for example track how 376 00:18:16,680 --> 00:18:19,360 many PRS people are doing. Not to measure individual 377 00:18:19,360 --> 00:18:25,240 productivity, but measuring time to 10 PR from people onboarding 378 00:18:25,240 --> 00:18:27,400 is a really good indicator of how good your onboarding process 379 00:18:27,400 --> 00:18:29,720 is. So it's not measuring like how 380 00:18:29,720 --> 00:18:35,440 fast can this guy do 10 PRS, but it's about how well are we able 381 00:18:35,440 --> 00:18:38,840 as your company to onboard new people because by the time they 382 00:18:38,840 --> 00:18:42,160 made that 10 PR, you can be pretty sure they're are working 383 00:18:42,160 --> 00:18:44,720 on their own. Exactly, I'm wondering what you 384 00:18:44,720 --> 00:18:48,760 think of people nowadays have an opinion on developer 385 00:18:48,760 --> 00:18:50,120 productivity. Right? 386 00:18:50,120 --> 00:18:53,160 There's a lot of agentic tooling and people can vibe code that 387 00:18:53,160 --> 00:18:56,480 are non-technical and they can show a feature and they'll be 388 00:18:56,480 --> 00:18:58,960 like, well why doesn't this happen within the team or why is 389 00:18:58,960 --> 00:19:01,960 the team estimation 2 weeks and I can do it in two days, for 390 00:19:01,960 --> 00:19:03,280 example. I've seen a lot of those 391 00:19:03,280 --> 00:19:05,680 conversations and they're quite funny to be honest. 392 00:19:06,360 --> 00:19:09,400 But yeah, it does say something about the organization, a big 393 00:19:09,440 --> 00:19:13,240 oil tanker versus an individual that just goes Greenfield and 394 00:19:13,240 --> 00:19:17,080 shows results in that way. But it forms opinions and all of 395 00:19:17,080 --> 00:19:20,520 a sudden people are more. So I think rather than bottom up 396 00:19:20,520 --> 00:19:24,200 being enabled, also top down, there's this expectation of we 397 00:19:24,200 --> 00:19:26,960 need to be more productive as an organization because if I talk 398 00:19:26,960 --> 00:19:28,800 to other people, they're doing the same thing in other 399 00:19:28,800 --> 00:19:31,040 organisations, they're trying to find the gold as well. 400 00:19:31,480 --> 00:19:32,880 I'm curious what you've seen so far. 401 00:19:34,120 --> 00:19:36,520 I I think like writing code is never been the biggest 402 00:19:36,520 --> 00:19:39,280 bottleneck, right? There's, like I said in the 403 00:19:39,280 --> 00:19:41,080 beginning, defects is about more than just tooling or 404 00:19:41,080 --> 00:19:44,280 technologies or coding. A lot of it is also processes, 405 00:19:44,280 --> 00:19:47,280 for example. Yeah, so let's say you come in 406 00:19:47,280 --> 00:19:49,440 on Monday morning and you're tasked with building a new 407 00:19:49,440 --> 00:19:51,680 Greenfield project. Very nice, right? 408 00:19:51,680 --> 00:19:53,800 Cool project. So you start building, you're 409 00:19:53,800 --> 00:19:56,680 happy, and at some point you need a database because every 410 00:19:56,680 --> 00:19:58,560 good product needs a needs a database at some point. 411 00:20:00,280 --> 00:20:04,560 And you start looking at documentation, trying to see how 412 00:20:04,560 --> 00:20:06,200 you need to request this database. 413 00:20:06,880 --> 00:20:10,280 And in the end you find some doc that's outdated and who knows 414 00:20:10,280 --> 00:20:12,320 what. And in the end, you find someone 415 00:20:12,320 --> 00:20:14,040 that can actually explain to you, hey, you need to create a 416 00:20:14,040 --> 00:20:16,560 ticket in the ticketing system to request a database. 417 00:20:17,480 --> 00:20:19,440 Of course, the ticketing system is already the place where 418 00:20:19,440 --> 00:20:24,920 innovation dies usually. But fine, you know, you put the 419 00:20:24,920 --> 00:20:28,400 ticket in, you wait for a couple of days, nothing happens because 420 00:20:28,400 --> 00:20:32,440 it turns out you also need to submit an e-mail to some other 421 00:20:32,560 --> 00:20:34,360 department for your ticket to be picked up. 422 00:20:35,240 --> 00:20:37,920 And then in the end, it turns out there's this sacred cloud 423 00:20:37,920 --> 00:20:39,400 committee that's meet once a month. 424 00:20:39,600 --> 00:20:41,600 It's a group of non-technical stakeholders and they're going 425 00:20:41,760 --> 00:20:44,280 to decide if your database is worthy of existence. 426 00:20:44,760 --> 00:20:46,480 And of course, you're in a last meeting last yesterday. 427 00:20:46,480 --> 00:20:48,840 So you have to wait a full month for the next meeting. 428 00:20:50,880 --> 00:20:53,880 So you you could have written all the code for the application 429 00:20:53,960 --> 00:20:58,000 in maybe a couple of days. But because of this very yeah, 430 00:20:58,000 --> 00:21:01,120 bad process, it's going to take a full month for you to deploy 431 00:21:01,120 --> 00:21:03,520 your service. Yeah, when when you were 432 00:21:03,520 --> 00:21:05,040 describing that, I was just laughing. 433 00:21:05,160 --> 00:21:09,280 I I feel like I would really experience a lot of pain going 434 00:21:09,280 --> 00:21:11,760 through this process and a lot of frustration to be honest. 435 00:21:12,240 --> 00:21:15,880 Now of course it's an execuration, but for waiting 436 00:21:15,880 --> 00:21:17,520 full month of course of the Sacred Cloud committee. 437 00:21:17,520 --> 00:21:22,200 But yeah, there's, there's a lot of ticket OPS still going on. 438 00:21:22,200 --> 00:21:25,360 A lot of companies you have to supply a ticket, wait for a bit, 439 00:21:25,360 --> 00:21:29,440 and then yeah, you get your database or you don't get it and 440 00:21:29,440 --> 00:21:31,120 it it wrecks team autonomy. Really. 441 00:21:31,240 --> 00:21:32,880 Yeah. Have you seen people? 442 00:21:33,240 --> 00:21:37,360 Because I was wondering if I I'm in a consultancy, I go from 443 00:21:37,360 --> 00:21:40,480 project to project, so I kind of get a look behind the scenes in 444 00:21:40,480 --> 00:21:43,400 a lot of organisations. So then I get a preference 445 00:21:43,400 --> 00:21:46,520 because I want to work in an organization and I'm going to 446 00:21:46,520 --> 00:21:49,240 compare to the best experience I've had developer experience 447 00:21:49,240 --> 00:21:50,680 wise. So I'm going to bring that to an 448 00:21:50,680 --> 00:21:53,480 organization and be like, well why don't we have this or what 449 00:21:53,480 --> 00:21:56,480 can we do to improve? But if I've been in the same 450 00:21:56,480 --> 00:21:59,440 organization for a long, long time, ticket OPS, this is the 451 00:21:59,440 --> 00:22:02,920 way it is, I don't think I'll know what a better way of 452 00:22:02,920 --> 00:22:06,080 working would be or what kind of other organisations are doing 453 00:22:06,080 --> 00:22:08,200 out there or what I could improve locally. 454 00:22:09,280 --> 00:22:12,880 Then I'm just going to kind of do the status quo, to be honest, 455 00:22:12,880 --> 00:22:15,400 because this is it and maybe it doesn't get better than this. 456 00:22:16,200 --> 00:22:18,960 I think it's interesting that if you have the ability to switch 457 00:22:19,360 --> 00:22:21,680 between companies, then that is going to give you a lot of 458 00:22:21,680 --> 00:22:24,680 insights on that aspect. But yeah, there's also benefits 459 00:22:24,680 --> 00:22:27,480 in staying with the same org, just maybe not with regards to 460 00:22:27,480 --> 00:22:28,800 the way of working or innovating. 461 00:22:29,200 --> 00:22:30,800 Is that also how you've experienced it? 462 00:22:31,160 --> 00:22:33,360 Yeah, I think that's what you'll see also see in the interviews I 463 00:22:33,360 --> 00:22:36,720 have with with engineers like the people that that not only 464 00:22:36,720 --> 00:22:38,640 the consultants, but also the people that change maybe 465 00:22:38,760 --> 00:22:40,520 employers every five or six years. 466 00:22:40,680 --> 00:22:44,200 They get to see more internals from different companies and 467 00:22:44,200 --> 00:22:47,640 they usually have more ideas on how things could be or can be. 468 00:22:49,040 --> 00:22:50,280 So yeah, you definitely see a difference there. 469 00:22:50,280 --> 00:22:51,320 Yeah, for sure. Yeah. 470 00:22:51,800 --> 00:22:54,720 It's interesting that you can you can leverage seeing 471 00:22:54,720 --> 00:22:57,560 different organisations and then applying that in this org. 472 00:22:58,200 --> 00:23:01,760 The team that is responsible for Dev X, do they only focus on Dev 473 00:23:01,760 --> 00:23:04,400 X or is that like a side thing to also do in product? 474 00:23:04,400 --> 00:23:07,680 Or how's typically the balance with regards to what they do? 475 00:23:08,080 --> 00:23:10,280 Oh, it's really different for for each company. 476 00:23:10,280 --> 00:23:12,760 Of course, sometimes you have like a dedicated Dev X team, 477 00:23:13,920 --> 00:23:17,400 sometimes they're part of platform teams because yeah, 478 00:23:17,440 --> 00:23:20,080 like you said before, Devx and platform, they're they're quite 479 00:23:20,080 --> 00:23:23,080 closely related. Yeah, for both your customers 480 00:23:23,080 --> 00:23:24,760 are the engineers within your company. 481 00:23:24,760 --> 00:23:32,640 Yeah, yeah, I've, I've worked in both Ryan's and I think having a 482 00:23:32,640 --> 00:23:35,360 dedicated Devx team was really was great because you can really 483 00:23:35,360 --> 00:23:38,560 focus on on the essence of of the whole thing. 484 00:23:38,560 --> 00:23:43,440 But then you still have a lot of collaboration with, for example 485 00:23:43,440 --> 00:23:45,160 platform and with engineers. Yeah. 486 00:23:46,160 --> 00:23:50,000 How does a Dev X team in the end when we're talking about from 487 00:23:50,000 --> 00:23:54,240 the standpoint of investments, if I am owner of a business or 488 00:23:54,240 --> 00:23:56,080 the owner of a certain department And I do think we 489 00:23:56,080 --> 00:23:58,160 need to focus on Dev X, but we don't have any metrics yet. 490 00:23:58,160 --> 00:24:01,000 So we don't have a baseline. But I do think we need to focus 491 00:24:01,000 --> 00:24:03,840 on a few things because there's a lot of signals and a lot of 492 00:24:04,560 --> 00:24:06,520 symptoms that I think we can improve. 493 00:24:07,280 --> 00:24:11,840 I will start with either a small Dev X responsibility person or a 494 00:24:11,840 --> 00:24:15,360 team, but in the end there will need to be certain measurements 495 00:24:15,360 --> 00:24:17,680 and then people are going to move towards certain outcomes. 496 00:24:18,000 --> 00:24:20,560 But I also have to have trust that this in the end is going to 497 00:24:20,560 --> 00:24:23,320 be a better outcome. So metrics help, but how do you 498 00:24:23,320 --> 00:24:26,640 typically see people get buy in or people get investments 499 00:24:26,640 --> 00:24:29,880 towards certain initiatives that actually move organizations? 500 00:24:31,160 --> 00:24:33,800 Yeah, can you can start very small bottom up. 501 00:24:34,200 --> 00:24:37,120 So even if you don't have like a very good baseline of the whole 502 00:24:37,120 --> 00:24:40,040 company and how they're performing, if you would in your 503 00:24:40,040 --> 00:24:44,720 team can make some local changes and you can measure improvements 504 00:24:44,720 --> 00:24:48,640 there because you probably have the data and things that weren't 505 00:24:48,640 --> 00:24:50,680 going so well before because that's why you're going to 506 00:24:50,680 --> 00:24:53,680 change something. You can make the changes there. 507 00:24:54,560 --> 00:24:56,760 And then you have, you might have a business case, right? 508 00:24:56,960 --> 00:25:01,240 If you can speed up your CCICD pipelines by X percent, or you 509 00:25:01,240 --> 00:25:03,680 can lower your memory consumption by X percent or 510 00:25:03,680 --> 00:25:05,480 whatever, you might have a business case. 511 00:25:05,760 --> 00:25:09,520 Interesting. I'm interested in more about you 512 00:25:09,520 --> 00:25:12,560 personally because you've been in Dev X teams, you said you've 513 00:25:12,560 --> 00:25:14,440 seen different organisations a lot as well. 514 00:25:14,920 --> 00:25:18,240 Where do you get your energy from or what still gives you 515 00:25:18,240 --> 00:25:20,640 energy when you're part of that Dev X teams specifically? 516 00:25:22,160 --> 00:25:25,480 I yeah, I just like coding a lot still. 517 00:25:25,520 --> 00:25:28,400 Yeah. And like you said, like 518 00:25:28,400 --> 00:25:31,480 yourself, I worked as a consultant for my entire career. 519 00:25:32,560 --> 00:25:35,640 And you're constantly comparing, Indeed your experience to the 520 00:25:35,640 --> 00:25:40,160 best companies you've worked at. And I know how nice it was to 521 00:25:40,160 --> 00:25:43,320 work with those companies where I could be super effective as a 522 00:25:43,320 --> 00:25:46,080 software engineer. So I think what gives me most 523 00:25:46,080 --> 00:25:50,080 energy is trying to achieve that for other engineers or different 524 00:25:50,080 --> 00:25:53,240 companies as well. And how do you then get the 525 00:25:53,240 --> 00:25:55,400 feeling of fulfilment? Is that when you see people 526 00:25:55,400 --> 00:25:57,520 really fly based on the changes that you've made? 527 00:25:58,320 --> 00:26:00,880 Yeah. So it's when you, when you do 528 00:26:00,880 --> 00:26:03,400 defx work, the feedback loops are completely different than 529 00:26:03,400 --> 00:26:04,760 what you have as a software engineer, right? 530 00:26:04,760 --> 00:26:07,760 Because you run your tests and they go green or red. 531 00:26:07,840 --> 00:26:11,560 Green is good and green is good usually or you're, you would do 532 00:26:11,560 --> 00:26:14,280 your and it takes maybe a couple of seconds while you do your 533 00:26:14,280 --> 00:26:17,320 deployment to acceptance or pro op might take 10 minutes, 20 534 00:26:17,320 --> 00:26:22,360 minutes if you're a bit unlucky and you have your feedback for 535 00:26:22,360 --> 00:26:24,040 defx work because it's much different. 536 00:26:24,040 --> 00:26:26,000 I had to get used to that a little bit in the beginning 537 00:26:26,320 --> 00:26:29,960 because you'll make some changes and then usually if you don't 538 00:26:29,960 --> 00:26:32,280 hear anything for for a while, then it's good. 539 00:26:32,560 --> 00:26:34,800 Oh, it's good. Because if people are not happy, 540 00:26:34,800 --> 00:26:35,840 they're you're going to hear them. 541 00:26:36,120 --> 00:26:38,040 They're vocal, yeah, they're vocal about it. 542 00:26:38,640 --> 00:26:42,120 And yeah, it doesn't happen often, but sometimes people are 543 00:26:42,120 --> 00:26:43,960 so happy with the changes they they will come to you say, hey 544 00:26:43,960 --> 00:26:46,120 man, I'm really happy with this or with this bit of 545 00:26:46,120 --> 00:26:49,760 documentation and great job. But that doesn't happen so much. 546 00:26:49,800 --> 00:26:51,920 That's right. Usually silence is, is good. 547 00:26:52,000 --> 00:26:54,200 And then hopefully you see the results in the surface or in the 548 00:26:54,200 --> 00:26:57,040 metrics that you are tracking. Yeah, so that's what you get 549 00:26:57,040 --> 00:26:59,640 your fulfilment then out of. If it's silence, you're like, 550 00:26:59,640 --> 00:27:02,440 it's quite as good. Yeah, it sounds a bit 551 00:27:02,440 --> 00:27:05,840 disappointing, but. That was really funny. 552 00:27:05,840 --> 00:27:09,120 Like I, I started my career in operations and to be honest, 553 00:27:09,520 --> 00:27:13,080 with a really good operations team, the dream is you don't 554 00:27:13,080 --> 00:27:15,960 really need a lot of operations or people being standby, right? 555 00:27:15,960 --> 00:27:17,800 Because if you have great systems, then you don't actually 556 00:27:17,800 --> 00:27:19,360 need to do a lot when you're standby. 557 00:27:19,360 --> 00:27:23,240 That's the dream. And if no one finds out, then 558 00:27:23,240 --> 00:27:25,240 just keep quiet basically. Exactly. 559 00:27:25,240 --> 00:27:29,760 I feel like developer experience is, is interesting. 560 00:27:30,040 --> 00:27:32,760 I am wondering though, because I'm getting really enthusiastic 561 00:27:32,760 --> 00:27:37,000 about Dev X in the first place, if I specialized in this, if I 562 00:27:37,000 --> 00:27:39,760 become a person that can orchestrate surveys, that can 563 00:27:39,760 --> 00:27:42,760 show metrics, that can communicate and get by in, will 564 00:27:42,760 --> 00:27:46,760 I tie myself to be most effective in big organisations? 565 00:27:46,760 --> 00:27:49,600 Will that prevent me from going to smaller organisations or how 566 00:27:49,600 --> 00:27:55,280 do you view that? I guess it depends how you what 567 00:27:55,280 --> 00:27:58,680 you find a small organization. Like I said, 50 people instead 568 00:27:58,680 --> 00:28:00,880 of the 100 people instead 10. Yeah. 569 00:28:02,080 --> 00:28:04,080 Any of that or even start-ups? Yeah. 570 00:28:05,080 --> 00:28:09,240 Yeah, I think for start-ups there's a little less need for a 571 00:28:09,240 --> 00:28:11,840 dedicated defects team because as you already manage it like 572 00:28:11,960 --> 00:28:14,240 you're going to get the biggest results if your company's a bit 573 00:28:14,240 --> 00:28:16,960 bigger and where you can make small changes and they compound, 574 00:28:17,520 --> 00:28:19,080 you know, your time or money or whatever. 575 00:28:21,040 --> 00:28:25,960 But still, by working on these defects issues or these defects 576 00:28:25,960 --> 00:28:29,480 friction points, you experience so many different issues people 577 00:28:29,480 --> 00:28:31,920 have in their sofa development life cycle. 578 00:28:32,320 --> 00:28:35,440 And you learn a lot just by looking at those things. 579 00:28:35,440 --> 00:28:38,640 Because when I look at myself, the times I've learnt the most 580 00:28:38,920 --> 00:28:41,040 is when you're solving nasty issues. 581 00:28:41,280 --> 00:28:43,200 And that's basically what you're doing all the time. 582 00:28:43,320 --> 00:28:46,920 Yeah, if you're if you're doing Devx in a more dedicated manner. 583 00:28:47,360 --> 00:28:51,400 Interesting, you also mentioned that you really get joy out of 584 00:28:51,400 --> 00:28:54,080 coding specifically, but how much do you actually do hands on 585 00:28:54,080 --> 00:28:55,600 stuff when you're responsible for devx? 586 00:28:56,200 --> 00:28:57,760 Do you still dive deep when it matters? 587 00:28:57,760 --> 00:28:59,600 But you're not really creating I think as much. 588 00:29:00,440 --> 00:29:02,480 Yeah, that's true. So you dive deep when it matters 589 00:29:03,600 --> 00:29:06,760 and there's of course a social side, also the technical side. 590 00:29:06,760 --> 00:29:09,080 And for the technical side, you might have to do some coding, 591 00:29:09,080 --> 00:29:12,440 right. So I'm thinking of an example. 592 00:29:12,440 --> 00:29:16,760 I had a company where they were using, I think Datadog, and the 593 00:29:16,760 --> 00:29:18,240 company's running a lot of Scala code. 594 00:29:18,880 --> 00:29:23,120 For some reason, the, the tracing didn't work quite all 595 00:29:23,120 --> 00:29:25,920 right in Datadog for, for Scala applications with a certain 596 00:29:25,920 --> 00:29:28,200 framework. And then we spent quite some 597 00:29:28,200 --> 00:29:30,840 time coding to fix that issue actually. 598 00:29:30,960 --> 00:29:31,800 Gotcha. Yeah. 599 00:29:32,040 --> 00:29:36,240 Is that then because there's no one is responsible for that bit 600 00:29:36,320 --> 00:29:39,080 or they lack the expertise? Because I'm wondering why that 601 00:29:39,080 --> 00:29:43,040 goes to the Dev X team and not there's not a team or ownership 602 00:29:43,160 --> 00:29:46,480 there without it. Yeah, that's a good question. 603 00:29:46,720 --> 00:29:48,600 I think it's, it's a bit of both, right. 604 00:29:48,600 --> 00:29:51,520 So it was a quite a complicated problem. 605 00:29:51,880 --> 00:29:54,520 So in the end, what we did, we started a working group with 606 00:29:54,520 --> 00:29:56,720 some very experienced scholar developers as well to work 607 00:29:56,720 --> 00:30:00,120 together to just fix this issue. But by starting this this 608 00:30:00,120 --> 00:30:04,720 working group, we also got some mandates and some budget to 609 00:30:04,720 --> 00:30:06,760 actually work on this fix in a dedicated manner. 610 00:30:07,560 --> 00:30:09,280 Because otherwise you have all these scatter teams that all off 611 00:30:09,280 --> 00:30:11,440 the issue and they're gonna you know, maybe stare at each other 612 00:30:11,440 --> 00:30:13,320 and say maybe they're gonna fix it. 613 00:30:13,520 --> 00:30:15,040 Maybe they are gonna fix it. And then just. 614 00:30:15,040 --> 00:30:17,280 They're gonna wait. No one fixes it, no ownership. 615 00:30:18,000 --> 00:30:21,720 I like this concept of task force and I, I'm now at Iholt 616 00:30:21,720 --> 00:30:23,640 and Iholt. They also have that for specific 617 00:30:23,640 --> 00:30:26,560 initiatives, people come together from a dispersed group 618 00:30:26,560 --> 00:30:30,360 of teams, start a task force, solve the issue, go back and 619 00:30:30,360 --> 00:30:32,760 it's the first time I've actually seen a really good like 620 00:30:32,760 --> 00:30:36,520 orchestrated process around solving root cause problems. 621 00:30:37,240 --> 00:30:40,520 I feel like if I have a lot of symptoms and that kind of in 622 00:30:40,520 --> 00:30:43,600 inhibits my way of working, it's very hard if I don't feel 623 00:30:43,600 --> 00:30:46,920 empowered to fix it. I don't think I could last like 624 00:30:46,920 --> 00:30:49,480 years and years on end like I see other people do it. 625 00:30:49,960 --> 00:30:52,720 I've always given the example of I started out in operations and 626 00:30:52,720 --> 00:30:55,760 then before going manually into orders and changing things 627 00:30:55,760 --> 00:30:57,680 because they were ending up in an error directory. 628 00:30:57,960 --> 00:31:00,920 I was like, I already week one. I was like, I cannot see myself 629 00:31:00,920 --> 00:31:03,360 doing this. Like I'm a big fan of automating 630 00:31:03,360 --> 00:31:06,600 and solving problems and developer experience or teams 631 00:31:06,600 --> 00:31:09,440 that are dedicated for that, also orchestrating task forces, 632 00:31:09,840 --> 00:31:12,080 fixing technical problems, but also non-technical. 633 00:31:12,440 --> 00:31:14,960 I think it's just very cool that people are spending time and 634 00:31:14,960 --> 00:31:17,600 attention towards that because yeah, if you have 1000 635 00:31:17,600 --> 00:31:20,320 engineers, it's just a lot of people to make more effective 636 00:31:20,320 --> 00:31:22,840 and small changes can have big benefits. 637 00:31:23,080 --> 00:31:25,760 Yeah, for sure, for sure. I think that that sums it up 638 00:31:25,760 --> 00:31:28,200 quite nicely. Yeah, yeah, yeah, definitely. 639 00:31:28,480 --> 00:31:30,960 The part with AI tooling for me is just very interesting. 640 00:31:31,120 --> 00:31:34,920 And I'm wondering for you is that if people already have a 641 00:31:34,920 --> 00:31:37,400 really good way of looking at developer experience and then 642 00:31:37,400 --> 00:31:40,880 they add AI tooling in the mix, I'm assuming it's nothing new 643 00:31:40,920 --> 00:31:43,600 than just any other tool. But for companies that don't 644 00:31:43,600 --> 00:31:46,520 have that, then adding AI on top of it might just create more 645 00:31:46,520 --> 00:31:48,360 chaos. How do you see that? 646 00:31:48,800 --> 00:31:52,800 Yeah, The funny thing is that most friction points that hurt 647 00:31:53,040 --> 00:31:58,480 humans also hurt agents, right? If you have very unclear 648 00:31:58,480 --> 00:32:02,760 processes or slow feedback loops or flaky tests killing for 649 00:32:02,760 --> 00:32:04,240 agents, of course, flaky testing. 650 00:32:06,040 --> 00:32:08,520 What else do we have? Port documentation? 651 00:32:08,600 --> 00:32:12,040 Out of date documentation that also, yeah, it's also going to 652 00:32:12,040 --> 00:32:13,880 hurt agents, It's going to hurt humans, also going to hurt 653 00:32:13,880 --> 00:32:16,360 agents. And then I think it's even more 654 00:32:16,360 --> 00:32:20,160 important to have good baseline measurements. 655 00:32:20,320 --> 00:32:24,800 So if you start introducing AI within your company and you 656 00:32:24,800 --> 00:32:28,080 don't have these baseline measurement measurements, how 657 00:32:28,080 --> 00:32:31,560 are you going to determine if the cost, because those tokens 658 00:32:31,560 --> 00:32:34,320 are not free, right, that you sent to your LLM, How are you 659 00:32:34,320 --> 00:32:36,240 going to determine that they actually bring value? 660 00:32:37,160 --> 00:32:39,880 Because indeed, maybe you might see features get completed 661 00:32:39,880 --> 00:32:43,800 faster, but maybe there's a big increase in production 662 00:32:43,800 --> 00:32:46,160 incidents. Yeah, I was talking to a person 663 00:32:46,160 --> 00:32:49,120 recently and they were talking about a governance framework 664 00:32:49,560 --> 00:32:52,440 because people are being a enabled with a gentic tooling 665 00:32:52,840 --> 00:32:55,480 and things are accelerating. But their governance framework 666 00:32:55,480 --> 00:33:00,320 was indeed for I principle with regards to pull requests, not 667 00:33:00,320 --> 00:33:04,200 being able to push the main directly. 668 00:33:04,200 --> 00:33:05,920 And I was like, well, nothing of this is new. 669 00:33:06,200 --> 00:33:08,840 There was also something additionally that was a list of 670 00:33:08,840 --> 00:33:11,800 check boxes that developer would be like, yeah, I've reviewed 671 00:33:11,800 --> 00:33:13,720 this even though it was AI code. And I'm like, man, then you're 672 00:33:13,720 --> 00:33:16,640 just trying to finger point to the problem that people actually 673 00:33:16,640 --> 00:33:19,080 constantly do something. I'm like, I don't know if this 674 00:33:19,080 --> 00:33:21,480 is the way. Like I don't know what the way 675 00:33:21,480 --> 00:33:22,960 is. I don't think there needs to be 676 00:33:22,960 --> 00:33:24,840 anything extra. Like I haven't seen anything 677 00:33:24,840 --> 00:33:27,640 extra that needs to be there from a process perspective with 678 00:33:27,640 --> 00:33:31,680 agents in the mix other than if you are really have autonomous 679 00:33:31,680 --> 00:33:34,000 agents specifically. But then your your code review 680 00:33:34,000 --> 00:33:37,040 process should be able to catch that I would think. 681 00:33:37,280 --> 00:33:39,400 So I haven't seen anything new that needs to be added. 682 00:33:40,000 --> 00:33:42,480 But then, are the code reviews done by humans or agents? 683 00:33:42,680 --> 00:33:46,680 Yeah, that's an interesting one. I would say right now they're 684 00:33:46,680 --> 00:33:49,440 done by humans, but I think they can done be done by both. 685 00:33:49,760 --> 00:33:53,000 If I have an agent that reviews and then another agent that 686 00:33:53,000 --> 00:33:55,720 picks up those reviews at some point will come into a state 687 00:33:55,720 --> 00:33:58,720 where a human can actually look. Doesn't have to be first pass, 688 00:33:58,720 --> 00:34:01,040 but it can be a LastPass if we need to. 689 00:34:01,560 --> 00:34:04,720 And then if people have so much confidence in due to setting up 690 00:34:04,720 --> 00:34:08,199 agents with regards to specific context, specific knowledge and 691 00:34:08,199 --> 00:34:11,760 conventions, strong conventions, that they have high confidence 692 00:34:11,760 --> 00:34:15,199 that if an agent says it's good, then it's good, then I think it 693 00:34:15,199 --> 00:34:17,360 should be good. And if it's not, then we will 694 00:34:17,360 --> 00:34:19,480 experience, but we do need to own those problems. 695 00:34:20,120 --> 00:34:23,840 Yeah, for sure for for us, the team I'm working right now, 696 00:34:23,920 --> 00:34:27,000 we're not there yet to have that much confidence in the agents. 697 00:34:27,600 --> 00:34:30,360 So what we did, what we saw a few weeks back is that a lot 698 00:34:30,360 --> 00:34:33,639 more codes being produced by Junie, the LLM from Jetbrains. 699 00:34:34,800 --> 00:34:38,560 And what we noticed in interviews happening a bit more 700 00:34:38,560 --> 00:34:41,719 is that you would would flag something in a in a code review 701 00:34:42,280 --> 00:34:43,960 saying, hey, why is this like this and this? 702 00:34:44,760 --> 00:34:46,159 And then the person would play. Yeah, Junie. 703 00:34:46,159 --> 00:34:47,440 Junie made that. OK. 704 00:34:47,800 --> 00:34:50,639 E so that yeah, you know, and then we, we had to talk about it 705 00:34:50,639 --> 00:34:52,920 with the team and update their golden guidelines a little bit. 706 00:34:52,920 --> 00:34:58,040 And we said, OK, using LLM to to generate code is perfectly fine, 707 00:34:58,040 --> 00:34:59,960 right? I think think it's a good idea. 708 00:35:00,480 --> 00:35:03,920 But in the end, you are the expert in the lead and you are 709 00:35:03,920 --> 00:35:07,960 still responsible for what you put in the PR and you're 710 00:35:07,960 --> 00:35:11,200 responsible for understanding the solution that the LLM came 711 00:35:11,200 --> 00:35:12,400 up with. Yeah, because in the end, you 712 00:35:12,400 --> 00:35:15,200 know, the LLM is not going to get paid at 3:00 AM at night, 713 00:35:15,600 --> 00:35:18,400 but you are. Yeah, Hey, I had a conversation 714 00:35:18,400 --> 00:35:21,320 with Gregor Hope I, I don't think I shared that podcast with 715 00:35:21,320 --> 00:35:22,800 you, but I definitely would recommend it. 716 00:35:22,920 --> 00:35:25,200 It's really funny how he put it. He said either the tool's really 717 00:35:25,200 --> 00:35:28,920 good, which means we don't need you, so if you don't do 718 00:35:28,920 --> 00:35:30,480 anything, then we just use the tool. 719 00:35:31,040 --> 00:35:33,840 Or he says the tool's not that good and you don't do anything 720 00:35:34,240 --> 00:35:36,600 and you're still responsible. So that's that's your problem. 721 00:35:36,920 --> 00:35:38,320 Basically, it's one of those two. 722 00:35:38,600 --> 00:35:41,080 You have to add value on top. For sure, yeah. 723 00:35:41,240 --> 00:35:44,120 Yeah, that's also how I view it. And I think if we put it in that 724 00:35:44,120 --> 00:35:46,000 way, people will, it really resonates with people. 725 00:35:46,720 --> 00:35:49,120 You mentioned that you like still the hands on part, the 726 00:35:49,120 --> 00:35:52,080 coding part specifically, but I feel like that's going away. 727 00:35:52,400 --> 00:35:56,280 Are you on board with regards to agentic workflows or how do you 728 00:35:56,280 --> 00:35:58,360 view that and how do you get fulfilment out of that part? 729 00:35:59,440 --> 00:36:02,920 I still do get a little fulfillment out of just thinking 730 00:36:02,920 --> 00:36:04,320 about how a solution should be, right? 731 00:36:04,360 --> 00:36:07,720 Yeah, maybe not about just typing the code letter by 732 00:36:07,720 --> 00:36:10,080 letter. And I'm not not quite sure if it 733 00:36:10,080 --> 00:36:12,840 ever was like the the fulfilling part. 734 00:36:12,840 --> 00:36:16,760 Yeah, now that I think about it. But I think a lot of the fun for 735 00:36:16,760 --> 00:36:19,040 me also to think about how, you know, what's going to go wrong 736 00:36:19,040 --> 00:36:21,040 with the solution. You know, what are their unhappy 737 00:36:21,040 --> 00:36:22,920 parts? How does this tie into our 738 00:36:22,920 --> 00:36:26,440 domain? What kind of other things that 739 00:36:26,440 --> 00:36:28,160 we need to consider to make a good solution? 740 00:36:29,320 --> 00:36:32,160 And I still get that with agentic coding, at least for 741 00:36:32,160 --> 00:36:33,840 now. Yeah, yeah, same here. 742 00:36:34,080 --> 00:36:38,320 I did like always being a good hands on on the keyboard, but 743 00:36:38,320 --> 00:36:39,880 that's just to accelerate the process. 744 00:36:40,040 --> 00:36:43,120 And now I have a process that accelerates it even further. 745 00:36:43,600 --> 00:36:46,560 So then I really get enjoyment out of solving problems and 746 00:36:46,560 --> 00:36:50,040 working towards outcomes to the point where now I've downloaded 747 00:36:50,040 --> 00:36:51,960 this tool. I'll show it to you after it 748 00:36:51,960 --> 00:36:54,640 downloads a local speech to text model. 749 00:36:54,960 --> 00:36:58,040 And I just talk because a lot of prompting I'm typing and I'm 750 00:36:58,040 --> 00:36:59,520 like, I can talk way faster than type. 751 00:36:59,760 --> 00:37:02,120 So that's my latest optimization. 752 00:37:02,600 --> 00:37:05,440 And I'm also using it in Teams where people, I'm interacting 753 00:37:05,440 --> 00:37:07,680 with a lot of people and I'm just speech to text. 754 00:37:07,800 --> 00:37:10,280 Let's go speech to text, let's go. 755 00:37:10,320 --> 00:37:12,600 Great. It's actually quite fun. 756 00:37:13,200 --> 00:37:16,200 And because everything is local, I'm like, yeah, I get the local 757 00:37:16,200 --> 00:37:18,160 transcripts. I can even read the way I talk. 758 00:37:18,160 --> 00:37:20,960 If I see a lot of the Fiddler words, I try and improve my 759 00:37:20,960 --> 00:37:23,880 speech, which then comes back to the podcast. 760 00:37:23,880 --> 00:37:25,880 Yeah, I'm having fun, to be honest. 761 00:37:26,480 --> 00:37:28,200 Yeah. How happy are people would you 762 00:37:28,200 --> 00:37:29,960 when you work in an open office and you were. 763 00:37:30,000 --> 00:37:32,760 Oh, no, I, I actually, I, this is the part I don't like. 764 00:37:32,760 --> 00:37:35,760 I don't like going to the Openoffice anymore because maybe 765 00:37:35,760 --> 00:37:38,400 it's, I'm a bit awkward. I don't like it when people call 766 00:37:38,400 --> 00:37:39,920 in an open office or something like that. 767 00:37:39,920 --> 00:37:42,200 So I was trying to find a quiet spot and then I talked to my 768 00:37:42,200 --> 00:37:45,640 laptop. I, I saw 2A while back and I 769 00:37:45,640 --> 00:37:47,680 think it's called whisper AI or something. 770 00:37:47,680 --> 00:37:51,560 And you can like in the in the prompt by voice. 771 00:37:51,560 --> 00:37:55,160 Yeah, I don't know, whispering tone, which my I I didn't try 772 00:37:55,160 --> 00:37:56,640 it. It felt too uncomfortable, but. 773 00:37:56,960 --> 00:38:00,160 Yeah, that feels, I, I know whisper flow, but then I have to 774 00:38:00,160 --> 00:38:02,600 pay really quickly and because it's a SAS, I'm like, I don't 775 00:38:02,600 --> 00:38:04,760 want to pay monthly. So I found this tool, it's a one 776 00:38:04,760 --> 00:38:07,720 time payment and then I have it and I'm using it. 777 00:38:08,920 --> 00:38:10,240 Cool. Yeah, you, you have to show me 778 00:38:10,240 --> 00:38:11,240 afterwards. Sam, I will. 779 00:38:12,400 --> 00:38:14,440 Do you have any tools that you've really fallen in love 780 00:38:14,440 --> 00:38:18,920 with lately? Lately I just start using open 781 00:38:18,920 --> 00:38:20,680 code, which I think is really nice. 782 00:38:20,680 --> 00:38:23,200 I've heard it's good, but I still have to set up the 783 00:38:23,200 --> 00:38:25,360 workflow with all the skills and everything because that's, 784 00:38:25,360 --> 00:38:28,720 that's something I'm seeing also in the community right now. 785 00:38:28,880 --> 00:38:32,080 There's so much tooling, so much new tooling, but the golden part 786 00:38:32,080 --> 00:38:34,560 on how to use the tool, those tools are not, are not yet 787 00:38:34,560 --> 00:38:36,840 really crystallized yet. So you have to figure it out 788 00:38:36,840 --> 00:38:38,320 yourself a little bit, try what works for you. 789 00:38:38,320 --> 00:38:42,000 And I'm, yeah, also liking that experimental process. 790 00:38:43,000 --> 00:38:47,400 But yeah, it's it's three or an hour, a little bit for now. 791 00:38:47,480 --> 00:38:49,800 Yeah, definitely. I'm wondering if we ever get 792 00:38:50,240 --> 00:38:52,960 kind of a platform engineering for AI tooling, because DevOps 793 00:38:52,960 --> 00:38:55,560 was kind of introducing a lot of tools in that aspects and 794 00:38:55,560 --> 00:38:57,080 platform engineering is trying to fix that. 795 00:38:57,640 --> 00:39:00,000 I feel like AI tools are just popping up like mushrooms, but I 796 00:39:00,000 --> 00:39:03,800 still want to see in the end what sticks and there are just 797 00:39:03,800 --> 00:39:06,920 more and more. So wondering if there needs to 798 00:39:06,920 --> 00:39:10,600 be, Yeah, kind of a golden path for that as well in the end, 799 00:39:10,680 --> 00:39:13,160 maybe a few years down the line, which could be interesting. 800 00:39:13,480 --> 00:39:14,400 Yeah. And I think also you have 801 00:39:14,400 --> 00:39:16,920 different levels, right. So if you have the the open code 802 00:39:16,920 --> 00:39:19,440 or cloud code skills, you of course can share them with your 803 00:39:19,440 --> 00:39:22,400 team or maybe with your company or there's a lot of lot of 804 00:39:22,760 --> 00:39:24,920 different levels of abstraction you have there as well. 805 00:39:25,600 --> 00:39:27,640 Yeah, definitely. We've shared a lot about 806 00:39:27,640 --> 00:39:30,120 developer experience and I feel like we've covered a lot of 807 00:39:30,120 --> 00:39:31,760 bases. Is there anything we didn't 808 00:39:31,760 --> 00:39:34,760 discuss that you still want to add before we round off? 809 00:39:35,560 --> 00:39:37,160 Nothing that comes to mind right now. 810 00:39:37,440 --> 00:39:40,000 Probably like after after we sign off, immediately after. 811 00:39:40,000 --> 00:39:42,280 Recording. That's always how this goes. 812 00:39:43,040 --> 00:39:44,960 And thank you so much, Buzz. This is real fun. 813 00:39:45,320 --> 00:39:46,360 Thank you. Cool. 814 00:39:46,360 --> 00:39:48,440 We'll round it off here. If you're still with us, let me 815 00:39:48,440 --> 00:39:50,600 know in the comments section what you thought of this episode 816 00:39:50,600 --> 00:39:51,680 and we'll see you in the next one.