1 00:00:00,080 --> 00:00:02,680 Hi everyone, My name is Patrick Akil and if you're interested in 2 00:00:02,680 --> 00:00:06,320 how they build AI features over at Microsoft Copilot, this 3 00:00:06,320 --> 00:00:09,000 episode is for you because I'm joined by Stephanie Fisher, 4 00:00:09,280 --> 00:00:11,640 Senior Product Manager over at Microsoft Copilot. 5 00:00:11,960 --> 00:00:15,800 And a lot has changed the expectations for engineers and 6 00:00:15,800 --> 00:00:20,600 data scientists, the focus on delivery as fast as possible and 7 00:00:20,600 --> 00:00:22,800 doing that through many, many experiments. 8 00:00:23,120 --> 00:00:24,720 I really enjoyed this conversation. 9 00:00:24,720 --> 00:00:30,320 So enjoy. I was wondering because you you 10 00:00:30,320 --> 00:00:33,840 mentioned that you were PM before anything copilot related. 11 00:00:33,880 --> 00:00:35,760 Yes. What has changed now that 12 00:00:35,760 --> 00:00:37,880 Copilot is there with regards to PM? 13 00:00:37,960 --> 00:00:40,680 Everything, literally everything is start. 14 00:00:40,960 --> 00:00:47,880 Yeah, such a good start. I think like what has changed is 15 00:00:47,880 --> 00:00:50,480 what makes a good product, therefore what has changed it, 16 00:00:50,480 --> 00:00:52,520 what makes a good product manager, right? 17 00:00:52,520 --> 00:00:55,200 Because you have to adapt your skills to whatever you want your 18 00:00:55,200 --> 00:00:57,520 product to be. And then also just, I think the 19 00:00:57,520 --> 00:01:00,720 variety of things that you're now doing as APM is so different 20 00:01:00,720 --> 00:01:03,200 than it was before. Yeah. 21 00:01:03,200 --> 00:01:06,680 So I don't know which aspect. Why? 22 00:01:06,760 --> 00:01:09,920 Why did the product change that much from your perspective? 23 00:01:10,760 --> 00:01:15,840 So, so I was in, I was building office products before, right? 24 00:01:15,840 --> 00:01:18,560 I'm a product manager for worst copilot right now. 25 00:01:18,560 --> 00:01:21,360 Before that I was a product manager for like other office 26 00:01:21,360 --> 00:01:25,120 applications in general. And I feel that back then 27 00:01:25,520 --> 00:01:28,280 everything was about like a great product was a great UX, 28 00:01:28,280 --> 00:01:29,560 right? Really good user flow. 29 00:01:29,560 --> 00:01:33,480 For example, one of the products that I worked on just before 30 00:01:33,480 --> 00:01:36,760 Copilot's game was, for example, multilingual proofing, which 31 00:01:36,760 --> 00:01:41,000 meant if I'm in Outlook and I'm a multilingual speaker, like I 32 00:01:41,000 --> 00:01:43,640 am like, that means that I'm sometimes writing in English. 33 00:01:43,640 --> 00:01:45,720 That means that I'm sometimes writing in Dutch, right? 34 00:01:45,960 --> 00:01:48,640 And it's really annoying if my spelling suggestions, if I'm 35 00:01:48,640 --> 00:01:51,640 writing a, a e-mail that's mixed English in Dutch. 36 00:01:51,640 --> 00:01:54,160 So that's squiggles of the Dutch, for example, you get red 37 00:01:54,160 --> 00:01:56,360 lines everywhere. So I was building a product 38 00:01:56,360 --> 00:01:58,800 related to that multilingual proofing. 39 00:01:59,600 --> 00:02:02,200 And then like the thing that I thought through as a product 40 00:02:02,200 --> 00:02:06,040 manager was just simply OK, what's you know, how will a user 41 00:02:06,040 --> 00:02:07,560 use it? What is a good user flow? 42 00:02:07,560 --> 00:02:09,639 What is the best UX? You know, what does the system 43 00:02:09,639 --> 00:02:11,840 have to do? We're talking pre copilot and 44 00:02:11,840 --> 00:02:14,520 pre AI about deterministic systems as well, right? 45 00:02:16,160 --> 00:02:20,680 The system just had to work. And as a product manager, my job 46 00:02:20,680 --> 00:02:25,080 was to create an excellent spec. And in that spec I had to take 47 00:02:25,080 --> 00:02:27,200 all the edge cases into account. And that's what I did. 48 00:02:27,200 --> 00:02:29,600 And then a good product was a great user flow, but also a 49 00:02:29,600 --> 00:02:32,680 product that followed the spec to AT, right? 50 00:02:33,480 --> 00:02:39,000 Right now, I think in AI we're moving towards outputs like it's 51 00:02:39,000 --> 00:02:41,320 all about output quality as opposed to UX. 52 00:02:41,800 --> 00:02:45,760 So we're moving kind of from pixels to tokens, right? 53 00:02:45,760 --> 00:02:50,400 And no good UI right now in AI roles can make up for the fact 54 00:02:50,400 --> 00:02:51,880 that your output quality isn't good. 55 00:02:52,480 --> 00:02:55,880 So a great product now is not a great user flow, but a great 56 00:02:55,880 --> 00:02:58,640 product is, is your output accurate? 57 00:02:58,720 --> 00:03:01,600 Is your output relevant customer satisfaction? 58 00:03:02,040 --> 00:03:05,280 And that also means that as APM, your job becomes really 59 00:03:05,280 --> 00:03:10,480 different because I'm going to have to constantly test the 60 00:03:10,480 --> 00:03:13,520 prompts that my scientists are using, are building on top of 61 00:03:13,520 --> 00:03:16,160 the models and see, oh, how's the output quality on these 62 00:03:16,160 --> 00:03:18,240 types of documents? How's the output quality on 63 00:03:18,640 --> 00:03:21,600 other types of documents? Like, And then I communicate 64 00:03:21,600 --> 00:03:25,320 back to my scientists and say, hey, listen, guys, like this is, 65 00:03:25,480 --> 00:03:27,520 this is kind of the golden standard for looking for this 66 00:03:27,520 --> 00:03:29,880 type of outputs or this is really bad, right? 67 00:03:30,280 --> 00:03:33,400 But then again, like the systems right now we're building are non 68 00:03:33,400 --> 00:03:38,400 deterministic like like we can't plan for the outputs, right. 69 00:03:38,400 --> 00:03:40,560 We can build great problems if we don't know what the output 70 00:03:40,560 --> 00:03:43,760 will be ever. So just kind of that whole 71 00:03:43,760 --> 00:03:48,120 testing, experimenting mindset is something that's we never did 72 00:03:48,240 --> 00:03:51,680 before AI world. Interesting. 73 00:03:51,880 --> 00:03:54,880 With deterministic stuff you could aim for like let's say 74 00:03:54,880 --> 00:03:58,280 your idea of perfection and then it works over and over and over 75 00:03:58,280 --> 00:04:00,000 again. Do you think you can still do 76 00:04:00,000 --> 00:04:02,680 that in this non deterministic way or will there always be edge 77 00:04:02,680 --> 00:04:04,160 cases that you just can't accommodate? 78 00:04:04,160 --> 00:04:06,560 For I think there will always be edge cases you can't accommodate 79 00:04:06,560 --> 00:04:09,600 for like you can make a great problems, obviously. 80 00:04:09,600 --> 00:04:12,560 And we have all these evaluations from the science 81 00:04:12,560 --> 00:04:15,560 side as well where we can measure, oh, how well is this 82 00:04:15,560 --> 00:04:18,880 model or this problem doing in terms of accuracy, relevancy, 83 00:04:18,880 --> 00:04:21,680 all these types of things. But then again, like you never 84 00:04:21,680 --> 00:04:23,520 know indeed, because it's non deterministic. 85 00:04:23,520 --> 00:04:27,960 So yeah, so, so, so you can't, can't plan for everything. 86 00:04:28,160 --> 00:04:30,560 How do, how do you decide then if you're experimenting with, 87 00:04:30,560 --> 00:04:33,640 let's say a new AI feature, when to go live? 88 00:04:33,640 --> 00:04:37,400 Because the only experience I had was we're working towards an 89 00:04:37,400 --> 00:04:40,000 outcome and the outcome was making people more productive. 90 00:04:40,440 --> 00:04:44,640 And even if our AI solution was only X amount of percent 91 00:04:44,640 --> 00:04:46,680 accurate, I was like, let's do user testing. 92 00:04:46,680 --> 00:04:49,560 Let's measure if we actually contribute towards that outcome. 93 00:04:50,000 --> 00:04:52,800 And we also have to test kind of the engagement and adoption and 94 00:04:52,800 --> 00:04:55,800 stuff like that. But even if we're not 95% 95 00:04:55,800 --> 00:04:58,400 accurate, but we're hitting 80% and people are already 96 00:04:58,400 --> 00:05:00,920 significantly more productive, we can go live and like release 97 00:05:00,920 --> 00:05:02,880 that. What is your thought on 98 00:05:03,120 --> 00:05:06,440 releasing new AI features? I think it's a good point. 99 00:05:06,440 --> 00:05:09,640 Like as a like I do have together with my scientists, 100 00:05:09,640 --> 00:05:13,040 just certain baselines that we want to hit in terms of metrics, 101 00:05:13,040 --> 00:05:15,640 right? Like we do want the outputs to 102 00:05:15,640 --> 00:05:21,160 be above certain accuracy. However, like the point that 103 00:05:21,160 --> 00:05:25,400 you're touching on is really interesting because what you 104 00:05:25,400 --> 00:05:28,880 also don't know now in using AI products is how people are 105 00:05:28,880 --> 00:05:33,120 actually going to use it, right? And the only way that we can 106 00:05:33,120 --> 00:05:36,720 know that is by bringing it out to users as fast as possible. 107 00:05:37,080 --> 00:05:40,360 So I think as BMS, we're really and like as feature crews in 108 00:05:40,360 --> 00:05:43,200 general, we're really going to have to switch towards a more OK 109 00:05:43,200 --> 00:05:47,560 experimental mindset. Like my entire work right now is 110 00:05:47,560 --> 00:05:51,040 building hypotheses super quickly, trying to validate 111 00:05:51,040 --> 00:05:54,320 those hypotheses and like throwing them away and failing 112 00:05:54,320 --> 00:05:56,920 fast and like modifying those hypotheses. 113 00:05:57,520 --> 00:06:04,000 So for me the process or or or the time towards shipping at 114 00:06:04,000 --> 00:06:08,480 least to internal users has become much smaller just because 115 00:06:08,480 --> 00:06:11,440 I want to see, OK, but like we can test everything within our 116 00:06:11,440 --> 00:06:13,840 own environment, but then we we can see all the accuracy, 117 00:06:13,840 --> 00:06:15,440 relevancy, all those kind of metrics. 118 00:06:15,600 --> 00:06:18,560 However, I've no idea how this will behave in a real world 119 00:06:18,800 --> 00:06:21,040 indeed because of the non deterministic aspect of it. 120 00:06:21,320 --> 00:06:28,040 So, so like my whole mindset of me mindset of me and my crew is 121 00:06:28,040 --> 00:06:31,640 to ship as soon as possible and then see real world usage 122 00:06:31,640 --> 00:06:35,880 because only then also you're going to encounter those those 123 00:06:35,880 --> 00:06:38,520 edge cases and moments where it doesn't work, right. 124 00:06:38,920 --> 00:06:41,040 Yeah. Do you always ship kind of in 125 00:06:41,040 --> 00:06:44,680 this phased approach to internal users first before releasing to 126 00:06:44,680 --> 00:06:46,560 the wider public? Yes, yeah, yeah. 127 00:06:46,560 --> 00:06:48,360 That's how we've always done it at Microsoft. 128 00:06:48,360 --> 00:06:52,200 So we have, yeah, First we ship to internal dog food, dog food 129 00:06:52,200 --> 00:06:54,520 users. So those are users who signed up 130 00:06:54,520 --> 00:06:55,920 for. I want the earliest earliest 131 00:06:55,920 --> 00:06:57,200 features. We call them dog food users. 132 00:06:57,200 --> 00:07:03,080 We call them dog food users, then we shift to the interactive 133 00:07:03,080 --> 00:07:05,720 Microsoft and then we start shipping to worlds towards 134 00:07:05,720 --> 00:07:09,040 external users. But what is really just like 135 00:07:09,040 --> 00:07:12,360 anecdotally, what's really interesting as well as before 136 00:07:12,520 --> 00:07:25,120 AI, we yeah, yeah, yeah, yeah. Before our AI world, whenever we 137 00:07:25,120 --> 00:07:28,400 were building a product like before we would even release it 138 00:07:28,480 --> 00:07:32,280 to those dog food users, we would have a certain confidence 139 00:07:32,280 --> 00:07:35,520 in a product. It was almost ready for external 140 00:07:35,520 --> 00:07:38,120 users as well. So the only reason why we would 141 00:07:38,120 --> 00:07:42,240 then ship to internal users is just to, you know, catch some 142 00:07:42,240 --> 00:07:44,920 bugs, etcetera, you know, see if there are certain parts of the 143 00:07:44,920 --> 00:07:47,680 products that aren't working. But at least back then, we had 144 00:07:47,680 --> 00:07:51,120 almost like a 9095% confidence, OK, the product is good enough 145 00:07:51,120 --> 00:07:55,200 to ship to external users. However, like talking again 146 00:07:55,200 --> 00:07:59,280 about that experimental mindset, that is not something that I 147 00:07:59,280 --> 00:08:01,600 want to do anymore. I don't want to kind of siloed 148 00:08:01,600 --> 00:08:04,720 together with my scientist, you know, try to optimize before 149 00:08:04,720 --> 00:08:06,480 we're even going out to actual users. 150 00:08:06,480 --> 00:08:10,280 Instead, I think we have to go to actual users as soon as 151 00:08:10,280 --> 00:08:13,520 possible. However, like imagine the shift 152 00:08:13,520 --> 00:08:15,880 within our organization that needed to happen. 153 00:08:15,880 --> 00:08:19,240 Like, I think a year ago I was working on a pretty 154 00:08:19,240 --> 00:08:23,840 controversial AI product, and my idea was, OK, you know, let's 155 00:08:23,840 --> 00:08:28,280 just ship it as soon as possible just in order to validate our 156 00:08:28,280 --> 00:08:31,040 riskiest hypothesis as soon as possible. 157 00:08:32,520 --> 00:08:35,679 But imagine you're a dog food user and, you know, you're used 158 00:08:35,679 --> 00:08:38,200 to seeing stable products, and you're used to seeing products 159 00:08:38,240 --> 00:08:42,320 as well that are meant to, you know, shift to actual customers. 160 00:08:42,320 --> 00:08:44,840 And then suddenly you're getting something that is like, to be 161 00:08:44,840 --> 00:08:48,720 frank, pretty unfinished. I was discussing with my 162 00:08:48,720 --> 00:08:50,920 software engineers, okay, you know, what is the bar? 163 00:08:51,160 --> 00:08:54,440 What is the bar in this new AI role to ship to dog food users? 164 00:08:54,800 --> 00:08:56,920 And we called it the non embarrassment bar. 165 00:08:56,920 --> 00:09:00,760 So the moment that there are not like it doesn't have to be 166 00:09:00,760 --> 00:09:02,440 great. It absolutely doesn't have to be 167 00:09:02,440 --> 00:09:04,960 great, but if we're not embarrassed, let's just ship it 168 00:09:04,960 --> 00:09:07,240 to users, internal users and see what's happening. 169 00:09:07,240 --> 00:09:09,200 Okay, how was the feedback? In the. 170 00:09:09,920 --> 00:09:12,640 Beginning. Yeah, yeah, I don't see was 171 00:09:12,840 --> 00:09:14,120 yeah, yeah, yeah. Because you mentioned you're a 172 00:09:14,120 --> 00:09:16,440 user and you think, oh, gosh, these products are actually 173 00:09:16,440 --> 00:09:19,560 going out to users, you know, to external users because that's 174 00:09:19,960 --> 00:09:22,240 what you think is happening. Interesting. 175 00:09:22,240 --> 00:09:24,200 Yeah. So I got so much negative 176 00:09:24,200 --> 00:09:28,080 feedback of people being really concerned. 177 00:09:28,080 --> 00:09:29,600 You know, you can't shift this to to user. 178 00:09:29,600 --> 00:09:31,760 And I'm like, yeah, yeah, yeah, I know guys, you know, but this 179 00:09:31,760 --> 00:09:34,040 is just part of the new experimentation products. 180 00:09:34,040 --> 00:09:35,200 Yeah. The process. 181 00:09:35,440 --> 00:09:39,960 I, I had Sam Bacon and he's the CPO of Veed, which is like a 182 00:09:39,960 --> 00:09:42,800 really big, I think it's the biggest online video editing 183 00:09:42,800 --> 00:09:46,080 tool out in the market. And he says what distinguishes 184 00:09:46,080 --> 00:09:49,280 us, and we used Google as an example, is that I can talk to 185 00:09:49,280 --> 00:09:52,280 customers like this and we try and do that every week. 186 00:09:52,720 --> 00:09:55,760 How easy is it for you to not just talk to dog food users but 187 00:09:55,760 --> 00:10:03,240 actually customers in the end? Real customers I, I think the 188 00:10:03,240 --> 00:10:07,240 great thing about working at such a large corporations that 189 00:10:07,240 --> 00:10:09,760 we have a ton of structures in place where we can actually talk 190 00:10:09,760 --> 00:10:12,200 to enterprise users like real life enterprise users. 191 00:10:12,400 --> 00:10:16,840 So we have programs in place where a ton of enterprise users 192 00:10:16,840 --> 00:10:19,640 from enterprise companies sign up and they say, hey, we want to 193 00:10:19,640 --> 00:10:22,040 develop these AI products together with Microsoft. 194 00:10:22,600 --> 00:10:25,600 So you know we're delivering features to them really early so 195 00:10:25,600 --> 00:10:27,400 it can they can give early feedback. 196 00:10:28,560 --> 00:10:30,640 So in that sense it is pretty easy. 197 00:10:32,280 --> 00:10:37,640 At the same time, like the hardest, like like again, the 198 00:10:37,640 --> 00:10:43,040 hardest thing is when I was building products before AI and 199 00:10:43,040 --> 00:10:47,040 again, it was very much based on user flow and UX, etcetera. 200 00:10:47,040 --> 00:10:49,200 I could go to platforms like user testing. 201 00:10:49,360 --> 00:10:51,360 There are platforms that are user testing, right? 202 00:10:51,840 --> 00:10:54,240 And just people all around the world can sign up. 203 00:10:54,920 --> 00:10:58,000 It was very easy because you could just like show a flow and 204 00:10:58,000 --> 00:11:00,040 think, man, people would comment on it, right? 205 00:11:00,400 --> 00:11:03,480 And then you would have at least a sense on how it would resonate 206 00:11:03,480 --> 00:11:05,840 with users. However, right now, in a world 207 00:11:05,840 --> 00:11:08,280 where it's so much about output quality, you can't do that 208 00:11:08,280 --> 00:11:10,040 anymore, right? Just because people can't. 209 00:11:10,120 --> 00:11:12,400 Yeah, yeah, yeah. And people can't really test it 210 00:11:12,400 --> 00:11:14,640 out anymore. So. 211 00:11:15,360 --> 00:11:18,600 So it so it is hard, but there are avenues simply because we 212 00:11:18,600 --> 00:11:20,640 have the benefit of being a large corporate and with our 213 00:11:20,640 --> 00:11:22,560 name, Yeah. I. 214 00:11:23,880 --> 00:11:26,600 Think it makes sense. Earlier you, you talked about 215 00:11:26,600 --> 00:11:29,160 kind of this change in behaviour, right? 216 00:11:29,680 --> 00:11:32,720 First delivering very high quality and also going to your 217 00:11:32,720 --> 00:11:35,240 dog food users with the assumption that this is 218 00:11:35,280 --> 00:11:37,360 basically a formality and it's already there. 219 00:11:37,360 --> 00:11:38,800 We're just trying to iron things out. 220 00:11:38,800 --> 00:11:40,160 Yeah, in case we miss things. Yeah. 221 00:11:40,600 --> 00:11:43,640 But also from your perspective as APM it it sounds like you 222 00:11:43,640 --> 00:11:46,360 made this shift like no problem at all and you're having a 223 00:11:46,400 --> 00:11:49,160 blast. Which is amazing. 224 00:11:49,240 --> 00:11:53,720 I grew up in, I, I don't have much PM experience and I did all 225 00:11:53,720 --> 00:11:57,080 of a sudden become responsible for an application and that also 226 00:11:57,080 --> 00:11:58,320 had a Gen. AI component. 227 00:11:58,800 --> 00:12:02,280 And because I think I have this engineering background, I always 228 00:12:02,280 --> 00:12:05,720 like just shipping and then seeing, OK, do we meet our use 229 00:12:05,720 --> 00:12:07,600 case? Do we meet whatever we aim to 230 00:12:07,600 --> 00:12:09,560 achieve? And if that's with 80% or if 231 00:12:09,560 --> 00:12:11,880 that's with 100%, I don't mind to be honest. 232 00:12:12,160 --> 00:12:14,360 And we can move on and deliver value in other aspects. 233 00:12:14,880 --> 00:12:17,560 So I've always kind of had that. But how is that shift then for 234 00:12:17,560 --> 00:12:20,000 you? I think I also maybe had the 235 00:12:20,000 --> 00:12:23,480 benefit that I was a software engineer before being APM and I 236 00:12:23,480 --> 00:12:28,360 was only APMI think for a year, 1, 1/2 year before the whole AI 237 00:12:28,360 --> 00:12:30,920 stuff hits. So I didn't have like a ton of 238 00:12:31,760 --> 00:12:33,920 habits or something like that, right. 239 00:12:33,920 --> 00:12:38,960 And what I think is kind of ingrained in engineers, which I 240 00:12:38,960 --> 00:12:41,320 noticed with myself and also with my team, like, yeah, 241 00:12:41,320 --> 00:12:44,120 shipping thinks it's fun, but experimenting is also like so 242 00:12:44,120 --> 00:12:46,960 much fun, right? So I could very like, I could 243 00:12:46,960 --> 00:12:50,320 very much like, easily tap into that by myself, but also like 244 00:12:50,880 --> 00:12:54,160 getting my engineers excited about just running experiments, 245 00:12:54,160 --> 00:12:57,080 looking at scorecards. And that data came pretty 246 00:12:57,080 --> 00:13:01,320 naturally, to be honest, just because, like, it's, it daps 247 00:13:01,320 --> 00:13:04,600 into some inherent curiosity, I think, which worked really well. 248 00:13:04,600 --> 00:13:05,440 That's amazing. Yeah. 249 00:13:05,440 --> 00:13:08,600 Yeah, yeah. Has your expectation of software 250 00:13:08,600 --> 00:13:10,760 engineers, let's start with software engineers first changed 251 00:13:10,760 --> 00:13:12,440 throughout kind of this experience? 252 00:13:12,760 --> 00:13:24,360 Yes, yeah, very much I think. I think as a like a, a good, 253 00:13:24,600 --> 00:13:27,000 this has always been my vision about product management. 254 00:13:27,080 --> 00:13:31,000 I think very traditionally, like really back in the days, it used 255 00:13:31,120 --> 00:13:34,000 to be pretty hierarchical in terms of how you build products. 256 00:13:34,000 --> 00:13:37,760 So the product manager sits in like an how do you say the ivory 257 00:13:37,760 --> 00:13:41,280 tower and you know, defines what needs to happen and writes A 258 00:13:41,280 --> 00:13:44,040 spec and then throws that over to design and design makes some 259 00:13:44,040 --> 00:13:46,920 sort of engineer design makes some sort of design, and then we 260 00:13:46,920 --> 00:13:48,840 throw that to engineering and engineering just needs to 261 00:13:48,840 --> 00:13:52,280 execute, in my opinion, from like PM. 262 00:13:52,280 --> 00:13:54,560 This is like not how you shoot PM anymore. 263 00:13:54,560 --> 00:13:58,280 I really see myself as APM that's facilitating all the 264 00:13:58,280 --> 00:14:00,520 other people in my team to be brilliant at what they do. 265 00:14:00,960 --> 00:14:03,680 And I really think product building is like an equal job 266 00:14:03,680 --> 00:14:06,000 between design, engineering and products. 267 00:14:07,320 --> 00:14:11,360 And now with the whole shift towards AI, where there's indeed 268 00:14:11,360 --> 00:14:13,480 like tons more experimentation, etcetera. 269 00:14:13,480 --> 00:14:16,600 I always also expect from my engineers that they step up 270 00:14:16,600 --> 00:14:18,720 their game in terms of experimentation, but also step 271 00:14:18,720 --> 00:14:21,960 up their game into what are those experiments that we can 272 00:14:21,960 --> 00:14:26,480 run, AKA what would resonate for a user in your opinion as a 273 00:14:26,480 --> 00:14:29,480 software engineer, right? So I'm really trying to 274 00:14:29,600 --> 00:14:32,720 stimulate them to also think about the user and the product, 275 00:14:32,720 --> 00:14:35,120 etcetera, because I also don't think that should be my role 276 00:14:35,120 --> 00:14:38,120 anymore. I have a pretty large team, 277 00:14:38,120 --> 00:14:40,240 right? I think I have like 30 software 278 00:14:40,240 --> 00:14:42,880 engineers, scientists, designers. 279 00:14:43,200 --> 00:14:44,400 It's really big. Yeah. 280 00:14:44,720 --> 00:14:47,280 OK. And that means I, I can't make 281 00:14:47,280 --> 00:14:50,360 all the decisions myself. You know, I really, yeah, really 282 00:14:50,360 --> 00:14:50,960 scary. Yes. 283 00:14:50,960 --> 00:14:55,040 And I also don't want to do that because, like, it's not the 284 00:14:55,040 --> 00:14:56,280 product. It's not going to be the best 285 00:14:56,280 --> 00:15:00,040 product if I'm by myself going to think through what they, what 286 00:15:00,040 --> 00:15:03,840 the product should look like. So I'm really trying to 287 00:15:03,840 --> 00:15:06,720 stimulate my engineers and my designers into product thinking, 288 00:15:06,720 --> 00:15:09,480 into users thinking. Thankfully they are really great 289 00:15:09,640 --> 00:15:11,920 product thinkers. Like all my engineers have a 290 00:15:11,920 --> 00:15:14,360 really good understanding of customers, etcetera. 291 00:15:15,440 --> 00:15:16,640 But there's also some work to do. 292 00:15:16,640 --> 00:15:20,240 So I invite them over to user interviews and those kind of 293 00:15:20,240 --> 00:15:22,520 things just so that they can get a better product sense. 294 00:15:23,720 --> 00:15:26,280 But because, yeah, I think they're going to need to step up 295 00:15:26,360 --> 00:15:29,440 in that regard as well. And they're for for everyone, by 296 00:15:29,440 --> 00:15:31,480 the way. I think PMS are going to need to 297 00:15:31,480 --> 00:15:33,920 learn how to code better and design better. 298 00:15:33,920 --> 00:15:35,840 Designers are going to need to learn how to code. 299 00:15:35,840 --> 00:15:37,920 You know, engineers are going to need to learn how to PM. 300 00:15:37,920 --> 00:15:40,440 Like, it's becoming so much more. 301 00:15:41,120 --> 00:15:45,280 Don't worry. So much more intertwined 302 00:15:45,320 --> 00:15:46,240 everything. Yeah. 303 00:15:46,360 --> 00:15:49,520 Yeah, what, when is the right time to involve engineering? 304 00:15:49,520 --> 00:15:51,840 Because I, I only have experimented with this. 305 00:15:52,600 --> 00:15:56,000 I did a kick off new milestone, engineering stakeholders all in 306 00:15:56,000 --> 00:15:57,600 one room. Let's discuss kind of the pain 307 00:15:57,600 --> 00:15:58,760 points and the problems. Yeah. 308 00:15:59,000 --> 00:16:01,640 And the feedback I got from some people, really enthusiastic, 309 00:16:01,640 --> 00:16:03,400 really excited from the beginning. 310 00:16:03,400 --> 00:16:06,080 Other people like this is too vague for me, like there's no 311 00:16:06,080 --> 00:16:07,520 structure yet. We don't know what we're doing. 312 00:16:07,520 --> 00:16:09,520 Yeah. I feel like it's a waste of kind 313 00:16:09,520 --> 00:16:11,520 of my energies and effort. Yeah, interesting. 314 00:16:11,520 --> 00:16:14,320 So then I'm like, OK, next milestone, I, I just do a raise 315 00:16:14,320 --> 00:16:16,200 of hands who wants to be there from the beginning? 316 00:16:16,200 --> 00:16:19,320 But then I think it's valuable, but I don't want to put this on 317 00:16:19,320 --> 00:16:21,200 people like it's always been this balance for me. 318 00:16:21,200 --> 00:16:24,520 Yeah, I think it is a balance like, but when I was a software 319 00:16:24,520 --> 00:16:28,960 engineer, one of the things that I used my PM for most is really 320 00:16:28,960 --> 00:16:32,840 understanding, oh, why am I taking up this user story again? 321 00:16:32,840 --> 00:16:34,960 Because like, if you're an engineer, you're involved so 322 00:16:34,960 --> 00:16:37,200 much in the reason you're kind of forgetting the big picture, 323 00:16:37,280 --> 00:16:40,440 right? And I think that is the main, 324 00:16:41,000 --> 00:16:43,240 the main difficulty that my engineers have. 325 00:16:43,240 --> 00:16:45,560 And I constantly have to remind myself like, oh, I need to 326 00:16:45,560 --> 00:16:47,880 communicate to my engineers like, why we're actually doing 327 00:16:47,880 --> 00:16:49,520 this? Like what are we contributing to 328 00:16:49,520 --> 00:16:53,080 and what are they directly contributing to, not only in 329 00:16:53,080 --> 00:16:58,480 terms of product, but in terms of success, You know, so we 330 00:16:58,560 --> 00:17:01,280 obviously want to increase engagement, activation, like all 331 00:17:01,280 --> 00:17:03,880 those sorts of metrics, like, and how does that trickle down 332 00:17:03,880 --> 00:17:06,079 to them? Because it gives them kind of a 333 00:17:06,079 --> 00:17:08,440 sense of, oh, This is why I'm working so hard on this. 334 00:17:08,440 --> 00:17:09,119 Yeah. Purpose. 335 00:17:10,839 --> 00:17:14,599 My sense is I'd love to involve engineers from the beginning 336 00:17:14,599 --> 00:17:18,920 when I have an idea of, hey, this is the user problem, right? 337 00:17:18,920 --> 00:17:22,319 So I've talked to a bunch of users and kind of consolidating 338 00:17:22,319 --> 00:17:24,960 all of that information. Then I'm presenting that out to 339 00:17:24,960 --> 00:17:26,760 my engineers. I'm like, OK, this is the 340 00:17:26,760 --> 00:17:29,120 direction that I'm thinking in, but I'm also open towards 341 00:17:29,120 --> 00:17:31,880 thinking like what you guys think. 342 00:17:32,120 --> 00:17:34,400 And the same thing with designers and scientists as 343 00:17:34,400 --> 00:17:37,080 well, by the way. I think we didn't work with 344 00:17:37,080 --> 00:17:39,520 scientists so much before, but they're so important right now 345 00:17:39,520 --> 00:17:41,000 in our work. So it's an entirely different 346 00:17:41,000 --> 00:17:43,000 type of colic that we're going to have to involve from the 347 00:17:43,000 --> 00:17:43,800 start. Yeah. 348 00:17:44,000 --> 00:17:46,560 Yeah. Are they ingrained and kind of 349 00:17:46,560 --> 00:17:50,600 part of the engineering team? Yes, it is a separate team with 350 00:17:50,600 --> 00:17:52,520 like a separate manager and things like that. 351 00:17:52,520 --> 00:17:57,280 But I usually work in we call ADRI Trios. 352 00:17:58,200 --> 00:18:03,360 DRI stands for designated responsible individual, 353 00:18:03,360 --> 00:18:07,480 something like that, whatever, another three word acronym. 354 00:18:10,920 --> 00:18:13,640 Anyway, it means there is someone responsible for every 355 00:18:13,640 --> 00:18:16,760 discipline. So I'm obvious, I'm whatever 356 00:18:16,760 --> 00:18:19,400 product I'm building, I'm always putting together APM. 357 00:18:19,920 --> 00:18:24,320 That's me, a scientist, scientist and engineer and a 358 00:18:24,320 --> 00:18:27,160 designer. And with the four of us, we're 359 00:18:27,160 --> 00:18:29,920 responsible for the products. Like doesn't matter. 360 00:18:30,160 --> 00:18:32,680 Like I don't care if you're an engineer, like if something 361 00:18:32,680 --> 00:18:35,360 falls into your place and I'm not there or I'm busy or 362 00:18:35,360 --> 00:18:38,120 whatsoever, you should solve it. Like we're really like with the 363 00:18:38,120 --> 00:18:40,280 four of us responsible for the products. 364 00:18:40,480 --> 00:18:43,320 And all those the same people kind of over and over again. 365 00:18:43,320 --> 00:18:46,080 Or do you switch with regards to the people in those little 366 00:18:46,080 --> 00:18:46,600 teams? Yeah. 367 00:18:46,600 --> 00:18:49,600 So it's a, it's a good question. So as I mentioned, I have a team 368 00:18:49,600 --> 00:18:54,320 of 3035 engineers approximately and within that we're building 369 00:18:54,320 --> 00:18:56,280 separate like all different products, right. 370 00:18:56,280 --> 00:19:00,800 So it's usually like well 35 / 4. 371 00:19:00,800 --> 00:19:03,080 So that's about like 8-8 different products are being 372 00:19:03,080 --> 00:19:06,400 built or something like that. So it's like constantly like 373 00:19:06,400 --> 00:19:09,840 different people and working on. Different products and you're 374 00:19:09,840 --> 00:19:11,800 the only product person that's there, yeah. 375 00:19:12,200 --> 00:19:16,400 Wow, I wish I had to help but I don't. 376 00:19:16,560 --> 00:19:17,880 Yes, that's super. Interesting. 377 00:19:18,040 --> 00:19:21,360 Like I've been in organizations where there's teams to the 378 00:19:21,360 --> 00:19:24,760 smallest level, 3-4 engineers, then there's also PM there. 379 00:19:24,760 --> 00:19:27,360 And then there's this hierarchy also of PMS, and they have one 380 00:19:27,360 --> 00:19:30,200 person that's product manager, and they also use terminologies 381 00:19:30,200 --> 00:19:32,440 of like product manager and product owner, kind of, Right? 382 00:19:32,880 --> 00:19:34,600 Yeah. And then in the end, there's one 383 00:19:34,600 --> 00:19:37,240 product manager without a team that just has a team of product 384 00:19:37,240 --> 00:19:40,040 people and they have smaller teams and like, how does this 385 00:19:40,040 --> 00:19:42,160 work? Yeah, Yeah. 386 00:19:42,160 --> 00:19:44,120 That is interesting. Yeah, we don't have that. 387 00:19:44,120 --> 00:19:46,280 And again, like I'm, but I'm outsourcing a lot of my 388 00:19:46,280 --> 00:19:50,400 engineers as well. I think like the PM role is very 389 00:19:50,560 --> 00:19:52,800 confluent as well. It means so many different 390 00:19:52,800 --> 00:19:56,240 things in many different organizations even need like 391 00:19:56,240 --> 00:19:59,160 product owner, product manager, project manager, etcetera. 392 00:20:00,040 --> 00:20:05,000 For example, as APMI personally don't run ADO boards or 393 00:20:05,000 --> 00:20:07,120 something like that or sprints or whatsoever. 394 00:20:07,120 --> 00:20:09,560 Like my engineers are responsible for that. 395 00:20:09,920 --> 00:20:11,880 They can do probably much better than I can. 396 00:20:11,880 --> 00:20:15,360 I don't add any value to that personally. 397 00:20:15,360 --> 00:20:17,080 It's not like within my skill sets. 398 00:20:17,800 --> 00:20:20,360 So I'm very consciously trying to think like, oh, what can my 399 00:20:20,360 --> 00:20:22,040 engineers do? What can my designers do and 400 00:20:22,040 --> 00:20:23,880 what do I do? And that's always kind of 401 00:20:23,880 --> 00:20:28,440 through the lens of. Where do I add unique value now? 402 00:20:28,640 --> 00:20:30,680 What are the decisions that you do take part of? 403 00:20:30,680 --> 00:20:34,920 Do you have cycles of retrospectives and plannings and 404 00:20:34,920 --> 00:20:37,680 forecasting or like OK ours, stuff like that? 405 00:20:37,960 --> 00:20:42,080 OK, ours, yes, that's a big one. That is a big one. 406 00:20:42,240 --> 00:20:45,760 Yeah, yeah, obviously we do make OK ours I think every half year 407 00:20:45,760 --> 00:20:48,640 or something just to know as an organization or as like a a part 408 00:20:48,640 --> 00:20:51,280 of the product, you know, what are the metrics that I want to 409 00:20:51,280 --> 00:20:56,680 move? And then obviously I have a like 410 00:20:56,920 --> 00:20:59,880 within Word copilot, I'm responsible for making people 411 00:20:59,880 --> 00:21:04,120 understand documents better. Like there are different jobs to 412 00:21:04,120 --> 00:21:06,640 be done whenever a user is opening a Word document, right? 413 00:21:06,640 --> 00:21:09,080 Like it is understanding a document or writing a document, 414 00:21:09,080 --> 00:21:11,440 one of these things, usually I'm responsible for that 415 00:21:11,440 --> 00:21:18,080 understanding part of it. So, so like I'm responsible for 416 00:21:18,080 --> 00:21:21,320 the road map within that part. So part of my job is just 417 00:21:21,320 --> 00:21:25,040 thinking about, OK, you know, what should AI help users with 418 00:21:25,280 --> 00:21:27,280 with this regards like upcoming year? 419 00:21:27,280 --> 00:21:31,560 So I've kind of a a backlog or things that I want to jobs to be 420 00:21:31,840 --> 00:21:33,520 to be done that I want to improve. 421 00:21:36,080 --> 00:21:38,520 And what were you asking again? What? 422 00:21:38,520 --> 00:21:40,960 What kind of cycles do you take part in with the team 423 00:21:40,960 --> 00:21:41,880 interaction? Yeah. 424 00:21:42,960 --> 00:21:45,480 So I make OK Rs for that. And I have like a road map in my 425 00:21:45,480 --> 00:21:48,400 head. And usually within Microsoft, we 426 00:21:48,400 --> 00:21:52,840 had planning cycles every three months or something. 427 00:21:52,840 --> 00:21:54,800 So every three months we would decide, oh, this is what we're 428 00:21:54,800 --> 00:21:57,440 going to work on. After months, that doesn't work 429 00:21:57,440 --> 00:21:59,880 anymore. And I really like literally like 430 00:21:59,880 --> 00:22:01,560 it's too long. It is way too long. 431 00:22:01,600 --> 00:22:03,680 It's way too long like. Three months already is too 432 00:22:03,680 --> 00:22:05,440 long. Yes, yeah, three months already 433 00:22:05,560 --> 00:22:08,560 luck, you know, like I mean you can make all the plans in the 434 00:22:08,560 --> 00:22:10,680 world that you want and then next week there's a new model 435 00:22:10,680 --> 00:22:12,560 being released or and and then you're done. 436 00:22:12,840 --> 00:22:15,400 I mean, I love this because always when people are like, 437 00:22:15,400 --> 00:22:16,480 where do you want to be in five years? 438 00:22:16,480 --> 00:22:18,480 I have no clue. Yeah, this is no clue. 439 00:22:18,520 --> 00:22:19,600 Insane. It is insane. 440 00:22:19,600 --> 00:22:22,520 Like, yeah, like big companies, you know, they're just like 441 00:22:22,880 --> 00:22:25,920 created out of the ground and then they're like next week 442 00:22:25,920 --> 00:22:27,880 they're acquired, next week there's a new model. 443 00:22:27,880 --> 00:22:31,120 It's like insane to keep track of, like what is happening in 444 00:22:31,120 --> 00:22:32,920 the industry. So you can't plan for three 445 00:22:32,920 --> 00:22:35,120 months. So we're trying to plan with my 446 00:22:35,120 --> 00:22:36,880 future crew every month. Yeah. 447 00:22:36,880 --> 00:22:38,960 So every month I'm kind of looking through whatever 448 00:22:38,960 --> 00:22:40,800 priorities in my opinion to work on. 449 00:22:40,800 --> 00:22:42,680 Then we're just like prioritizing them. 450 00:22:42,680 --> 00:22:46,640 And, and again, it might happen that during that month we're 451 00:22:46,640 --> 00:22:48,680 switching gears just because something else became more 452 00:22:48,680 --> 00:22:49,960 important. Yeah. 453 00:22:50,760 --> 00:22:53,560 I was wondering how you deal with bias because if I'm 454 00:22:53,560 --> 00:22:57,360 responsible for kind of understanding a document, I do 455 00:22:57,360 --> 00:22:59,600 that on a day-to-day. I open Word, this document, I 456 00:22:59,600 --> 00:23:01,960 have to understand it. I've done this so many times, 457 00:23:01,960 --> 00:23:06,400 so, so many biases I feel like. And then also when you have 40 458 00:23:06,400 --> 00:23:09,080 other people that contribute towards making this a better 459 00:23:09,400 --> 00:23:12,960 product or creating products and like improvements as part of 460 00:23:12,960 --> 00:23:16,240 that, how do you deal with bias and kind of remove that from 461 00:23:16,240 --> 00:23:18,560 people's way of working thinking? 462 00:23:18,560 --> 00:23:21,200 Well, I think we should do this. And that really anchors people. 463 00:23:22,360 --> 00:23:26,000 Yeah, because that's such an interesting question because 464 00:23:26,000 --> 00:23:31,240 what you don't want as a culture within a company is that you 465 00:23:31,240 --> 00:23:33,400 always have like these visionaries and innovators, 466 00:23:33,400 --> 00:23:35,120 etcetera, and they can talk really well and they've really 467 00:23:35,120 --> 00:23:37,760 great ideas, right. So it's really easy for them to 468 00:23:37,760 --> 00:23:40,400 influence the product just based on their gut feeling or like 469 00:23:40,400 --> 00:23:43,000 their visionary thinking. But that's not how we should 470 00:23:43,000 --> 00:23:46,880 build a product. Obviously in every product 471 00:23:46,880 --> 00:23:49,920 there's a bit of a bit of gut feeling, but that's not the way 472 00:23:49,920 --> 00:23:53,440 that we need to make decisions. So coming back to that 473 00:23:54,480 --> 00:23:59,200 controversial product that I was talking about, a ton of people 474 00:23:59,200 --> 00:24:00,480 didn't like that I was building it. 475 00:24:00,480 --> 00:24:02,760 Like absolutely not all the way to our leadership team. 476 00:24:03,160 --> 00:24:04,760 A ton of people didn't believe in it. 477 00:24:05,040 --> 00:24:08,240 The only way I could prove that it might have been a good idea 478 00:24:08,240 --> 00:24:11,760 was to experiment. So like purely data-driven, 479 00:24:11,760 --> 00:24:13,800 looking at the metrics that we care about and only 480 00:24:13,800 --> 00:24:18,360 communicating from that angle. So, so that's the best way to 481 00:24:18,360 --> 00:24:20,840 remove bias in my opinion. Like everyone can have opinions 482 00:24:20,840 --> 00:24:24,320 and that's great. I also have opinions and biases 483 00:24:24,320 --> 00:24:27,640 and those have been disproven so many times through running 484 00:24:27,640 --> 00:24:30,600 experiments where you feel like, oh, this is for sure going to 485 00:24:30,600 --> 00:24:32,000 move the metric this or this way. 486 00:24:32,200 --> 00:24:34,840 And it doesn't at all. It does the opposite thing, 487 00:24:34,840 --> 00:24:36,280 right? And that is super exciting. 488 00:24:36,280 --> 00:24:40,080 So we're doing that and we're actively communicating within 489 00:24:40,080 --> 00:24:43,240 our organization as well, whatever our findings are, how 490 00:24:43,240 --> 00:24:45,920 we're experimenting, etcetera. I like that a lot. 491 00:24:45,920 --> 00:24:48,080 I thought you were going to say because people that are 492 00:24:48,080 --> 00:24:49,880 stubborn, they like their own ideas, right? 493 00:24:49,880 --> 00:24:51,200 And they fall in love with their own ideas. 494 00:24:51,200 --> 00:24:54,240 So then when metrics and data comes and disproves it, it's 495 00:24:54,240 --> 00:24:56,920 like sad and disheartening. And the next one would be even 496 00:24:56,920 --> 00:24:58,960 harder. But I like you're saying it's 497 00:24:59,000 --> 00:25:00,280 super. Exciting, yeah. 498 00:25:00,440 --> 00:25:02,040 That means we can do many more things. 499 00:25:02,200 --> 00:25:03,160 Exactly. Go ahead. 500 00:25:03,200 --> 00:25:05,520 Yeah. Like I have so many like dailies 501 00:25:05,520 --> 00:25:07,440 with my engineers. We're looking at scorecards and 502 00:25:07,440 --> 00:25:11,120 we're like, oh, wow, this is like completely opposite to what 503 00:25:11,120 --> 00:25:13,080 we were thinking. And then we're discussing like, 504 00:25:13,080 --> 00:25:15,240 how come that we had that gut feeling, right? 505 00:25:15,240 --> 00:25:17,760 And what, what might these numbers actually say? 506 00:25:17,920 --> 00:25:19,840 So what do we have to rewire in our brain? 507 00:25:20,000 --> 00:25:22,520 So it's a really cool culture to build, kind of this experiment 508 00:25:22,520 --> 00:25:24,520 called experimentation culture. Yeah, Yeah. 509 00:25:24,520 --> 00:25:27,880 I can also see that it's very challenging because quantitative 510 00:25:27,880 --> 00:25:28,800 data is one thing. Yeah. 511 00:25:28,800 --> 00:25:30,280 And I think it's great to rely on that. 512 00:25:30,960 --> 00:25:33,360 And there's also then the qualitative part of it, like if 513 00:25:33,360 --> 00:25:36,080 you're doing things with AI, you're super trailblazing. 514 00:25:36,160 --> 00:25:39,440 So even if the data might say things yes, like talking to 515 00:25:39,440 --> 00:25:42,400 people might say otherwise, it's very hard to balance both 516 00:25:42,400 --> 00:25:44,480 feedback points. You're yeah, you're totally 517 00:25:44,480 --> 00:25:46,360 right. Also because like users don't 518 00:25:46,440 --> 00:25:49,920 like users usually don't really know what they want, right, 519 00:25:49,920 --> 00:25:51,960 Obviously. And especially in AI world, it's 520 00:25:51,960 --> 00:25:53,680 super hard. Like what do I have to build for 521 00:25:53,680 --> 00:25:54,960 you next week? That week? 522 00:25:57,280 --> 00:25:59,760 I wouldn't even imagine. I would have to see something 523 00:25:59,760 --> 00:26:02,000 and be like OK, just. Click exactly and you're like, 524 00:26:02,000 --> 00:26:05,040 ah, this works or this doesn't work, but it's so hard to 525 00:26:05,040 --> 00:26:09,120 rationalize. So the way that we now interact 526 00:26:09,120 --> 00:26:12,240 with like I'm finding those user problems, finding, finding the 527 00:26:12,240 --> 00:26:14,640 things that we need to work on really changed as well. 528 00:26:14,640 --> 00:26:18,760 Because like pre AI, again, I gave that example of 529 00:26:18,760 --> 00:26:23,320 multilingual proofing, right? Those were established. 530 00:26:23,320 --> 00:26:25,880 Like Outlook, for example, was an established product and we 531 00:26:25,880 --> 00:26:29,400 had like we had a backlog of users, like of of things users 532 00:26:29,400 --> 00:26:32,840 wanted us to work on, but that was all documented really 533 00:26:32,840 --> 00:26:34,280 nicely. You know, customers would reach 534 00:26:34,280 --> 00:26:35,960 out to us through different platforms. 535 00:26:35,960 --> 00:26:38,320 You know, we knew which features they wanted. 536 00:26:38,320 --> 00:26:40,960 It was just a matter of prioritization basically. 537 00:26:40,960 --> 00:26:42,960 It's not even that long ago, and it feels like so it. 538 00:26:42,960 --> 00:26:51,680 Feels so peaceful. Yes, yes, it feels like 539 00:26:51,680 --> 00:26:55,720 prehistoric time, you know, but now indeed, like you can't 540 00:26:55,720 --> 00:26:59,960 anymore. So so that also really changes 541 00:26:59,960 --> 00:27:04,480 how we're engaging with users. So what we now have to do is 542 00:27:04,480 --> 00:27:08,360 actually literally sit next to those users and asking, OK, just 543 00:27:08,360 --> 00:27:10,440 tell me about your workflows. Like what are you doing with 544 00:27:10,440 --> 00:27:13,040 words? And then just observing really 545 00:27:13,040 --> 00:27:16,800 well and then understanding like, oh, in these kind of 546 00:27:16,800 --> 00:27:21,600 moments in a workflow, we can inject AI in a very natural way. 547 00:27:21,600 --> 00:27:23,960 And that's the only way that we can kind of have that discovery 548 00:27:23,960 --> 00:27:25,840 process now. But it's super hard, so 549 00:27:25,840 --> 00:27:28,760 difficult. Yeah, discovery teams, I think 550 00:27:28,760 --> 00:27:30,680 those are going to be more and more at the forefront. 551 00:27:30,720 --> 00:27:33,760 I think so actually. Kind of pairing with users 552 00:27:33,760 --> 00:27:36,800 observing, users interviewing, investigating. 553 00:27:36,800 --> 00:27:38,200 And. Everything is still an 554 00:27:38,200 --> 00:27:39,800 assumption and then indeed testing. 555 00:27:39,800 --> 00:27:41,160 And then we're just experimenting. 556 00:27:41,160 --> 00:27:42,880 Yeah, exactly. And that's also, I think, a 557 00:27:42,880 --> 00:27:45,520 really big part of being APM. Like you can't get away with not 558 00:27:45,600 --> 00:27:47,880 deeply understanding a user anymore, right? 559 00:27:47,880 --> 00:27:50,160 You really have to deeply understand your customers at 560 00:27:50,160 --> 00:27:51,360 this point. Yeah, yeah. 561 00:27:51,680 --> 00:27:55,120 I get that. When everyone, or at least when 562 00:27:55,120 --> 00:27:57,400 you're in an organization where there's a lot of PMS and 563 00:27:57,400 --> 00:28:01,440 experimentation as a culture is kind of coming to the forefront 564 00:28:01,440 --> 00:28:04,440 because we have to, we have to deliver and then validate and do 565 00:28:04,440 --> 00:28:06,800 that over and over again until we have kind of a proof point of 566 00:28:06,800 --> 00:28:09,240 value. Does data become an issue? 567 00:28:09,600 --> 00:28:12,200 Do you have, for example, enough data or access to data to 568 00:28:12,200 --> 00:28:14,840 validate those assumptions? Or does it become a bottleneck 569 00:28:14,840 --> 00:28:16,800 because everyone needs it? Everyone needs it. 570 00:28:16,800 --> 00:28:21,520 Yeah, it is a bottleneck in a way that we just, I mean, we 571 00:28:21,520 --> 00:28:24,840 have so many users, but still it's not an infinite amount of 572 00:28:24,840 --> 00:28:27,480 users, right? So we can only run this very 573 00:28:27,560 --> 00:28:30,280 many experiments in parallel. And that's my main bottleneck 574 00:28:30,280 --> 00:28:32,640 right now. I've like 12 experiments that I 575 00:28:32,640 --> 00:28:35,440 want to run like at the same time, but my user base isn't 576 00:28:35,440 --> 00:28:37,920 that big. Like I count AB test 12 577 00:28:37,920 --> 00:28:40,960 different experiments, I still get statistically significant 578 00:28:40,960 --> 00:28:43,920 results right? Because we only care about when 579 00:28:43,920 --> 00:28:45,800 it's statistically really proven. 580 00:28:46,880 --> 00:28:50,360 So yeah, the bottleneck right now, which is a luxury position 581 00:28:50,360 --> 00:28:53,000 to have though like to want to run so many. 582 00:28:53,000 --> 00:28:56,680 Experiments I want to do 12 yes. Yeah, yeah, yeah. 583 00:28:56,960 --> 00:28:59,440 But right now, at this point, I think we're even that far in our 584 00:28:59,440 --> 00:29:03,480 experimentation culture within my team that if I didn't have 585 00:29:03,480 --> 00:29:05,040 this as a bottleneck, I would be worried. 586 00:29:05,520 --> 00:29:07,640 I'm like, oh, Stephanie, are you really experimenting? 587 00:29:07,680 --> 00:29:09,120 You know enough? Yeah, yeah, yeah. 588 00:29:09,120 --> 00:29:11,920 At least this tells me, oh, there are so many hypotheses 589 00:29:11,920 --> 00:29:14,560 that we're trying to investigate so that that that's cool. 590 00:29:14,600 --> 00:29:15,920 Yeah, that's so cool. Yeah. 591 00:29:16,040 --> 00:29:18,720 And you mentioned very strongly that you don't want to be the 592 00:29:18,720 --> 00:29:20,920 one person that comes up with all these experiments. 593 00:29:20,920 --> 00:29:23,480 So all these experiments are like a combination of all the 594 00:29:23,480 --> 00:29:25,680 disciplines coming together to try and solve this. 595 00:29:25,680 --> 00:29:27,360 Problem. Yeah, yeah, yeah, for sure. 596 00:29:27,360 --> 00:29:29,400 It's super cool. It's it's on every angle as 597 00:29:29,400 --> 00:29:32,280 well. It is designers wanting to try 598 00:29:32,280 --> 00:29:35,560 like a few things and see like how slightly different elements 599 00:29:35,560 --> 00:29:37,600 and design changes engagement, right? 600 00:29:37,600 --> 00:29:42,120 These engineers who are, for example, we're now very deep 601 00:29:42,120 --> 00:29:44,920 into chaos testing. So engineers who are trying to, 602 00:29:45,280 --> 00:29:47,840 I don't know, mess with our house booth and see how that 603 00:29:47,840 --> 00:29:50,600 influences engagement. This is me trying to experiment 604 00:29:50,600 --> 00:29:54,440 with different like features or or whatsoever and science when 605 00:29:54,440 --> 00:29:55,840 they can do things. So yeah, it's a really 606 00:29:55,840 --> 00:29:59,120 combination of things. It's still I think as ABM 607 00:29:59,160 --> 00:30:05,400 responsibility to kind of create order in that chaos of like of 608 00:30:05,440 --> 00:30:08,720 excitement and and think through, OK, this is a priority. 609 00:30:08,720 --> 00:30:10,760 This isn't priority. But yeah, it is sourced from 610 00:30:10,760 --> 00:30:13,320 everyone for sure. Have you ever had an experiment 611 00:30:13,320 --> 00:30:16,440 where there's a really good outcome, like things make sense, 612 00:30:16,440 --> 00:30:19,040 but from your product vision you're like, well, it's just not 613 00:30:19,040 --> 00:30:21,000 a good fit? Like it's not what we're trying 614 00:30:21,000 --> 00:30:26,320 to achieve with this. I've had that and therefore it 615 00:30:26,320 --> 00:30:29,440 is really important as well before you run an experiment 616 00:30:29,440 --> 00:30:32,760 like, OK, what am I going to do with this with the outputs? 617 00:30:32,760 --> 00:30:36,240 Because it's also very natural to be like, let's let's 618 00:30:36,240 --> 00:30:38,720 experiment of everything, you know? 619 00:30:39,080 --> 00:30:42,160 But if you're not going to take an actionable result based on 620 00:30:42,160 --> 00:30:45,280 that experiment just because indeed, like even if the outcome 621 00:30:45,280 --> 00:30:48,000 is this is great, it just goes against your product feeling 622 00:30:48,240 --> 00:30:50,720 that you just like, why are you running the experiments in 623 00:30:50,720 --> 00:30:52,800 general? It might be to gather 624 00:30:52,800 --> 00:30:54,480 information, and that could be valid, yes. 625 00:30:54,480 --> 00:30:56,440 But then it's something that we're discussing upfront with 626 00:30:56,440 --> 00:30:58,680 the engineers, like, hey, we can run this, but we're not going to 627 00:30:58,680 --> 00:31:01,040 make any decisions based on this. 628 00:31:01,040 --> 00:31:03,000 It's just to gather data and information. 629 00:31:04,040 --> 00:31:05,240 Yeah. So that happens. 630 00:31:05,400 --> 00:31:08,960 Gotcha. Yeah, I was wondering because 631 00:31:08,960 --> 00:31:10,440 this is mainly on the metrics level. 632 00:31:10,440 --> 00:31:12,720 I was responsible for this Gen. AI part and then it was like, 633 00:31:12,720 --> 00:31:16,600 OK, what are we aiming for with regards to accuracy with regards 634 00:31:16,640 --> 00:31:20,600 to like time reduced, I can have kind of a guesstimate metric, 635 00:31:20,600 --> 00:31:22,760 the dot on the horizon. So that's what I did. 636 00:31:22,960 --> 00:31:25,120 We're aiming for these things and then continuously we're 637 00:31:25,120 --> 00:31:27,120 evaluating if those metrics still make sense. 638 00:31:27,520 --> 00:31:29,360 How do you put your metrics kind of on the table? 639 00:31:29,360 --> 00:31:31,840 Because you mentioned with together with data science, we 640 00:31:31,840 --> 00:31:34,600 do have a few metrics that we're like striving for, but how do 641 00:31:34,600 --> 00:31:40,120 they originate? How do they originate So, so it 642 00:31:40,120 --> 00:31:42,200 kind of boils back maybe to like OK Rs. 643 00:31:42,200 --> 00:31:45,400 Like, I mean, I don't love OK Rs in terms of like writing them 644 00:31:45,520 --> 00:31:50,360 simply because it's hard, right? But I think it's also like a 645 00:31:50,360 --> 00:31:54,800 tendency to like, it's very easy to say I'm writing, OK, ours and 646 00:31:54,800 --> 00:31:57,840 let's just let's just increase engagement and that's it, right? 647 00:31:57,840 --> 00:32:00,920 But you really have like what you have to do is get back to, 648 00:32:00,960 --> 00:32:04,760 oh, like, what am I really trying to do for a user, right? 649 00:32:04,760 --> 00:32:07,280 And like a user's not necessarily helped with them 650 00:32:07,440 --> 00:32:09,400 clicking on something more, right? 651 00:32:09,600 --> 00:32:11,240 They're trying to achieve something else. 652 00:32:11,520 --> 00:32:16,680 So kind of writing that down kind of like and you said time 653 00:32:16,680 --> 00:32:18,960 to what, what were you trying to optimize time to? 654 00:32:18,960 --> 00:32:21,880 Making people more productive, so we're losing time spent. 655 00:32:21,880 --> 00:32:23,480 We're losing time spent. Yeah, exactly. 656 00:32:23,480 --> 00:32:26,280 But then so that is a really great, I think overarching goals 657 00:32:26,280 --> 00:32:30,120 which you shoot right down and then kind of distilling it back 658 00:32:30,120 --> 00:32:32,880 to, oh, that actually means that is obviously the hard part. 659 00:32:34,600 --> 00:32:37,960 I found that it really depends on the maturity as well of your 660 00:32:37,960 --> 00:32:43,840 team on how, how good you can or, or which metrics you can 661 00:32:43,840 --> 00:32:45,640 define. So for example, at the beginning 662 00:32:45,640 --> 00:32:49,600 of Copilot, we're really deep, like diving deep on those 663 00:32:49,600 --> 00:32:53,120 metrics for the first time. To be honest, in Copilot in the 664 00:32:53,120 --> 00:32:56,640 AI world, just because it was starting, we were only looking 665 00:32:56,640 --> 00:32:59,040 at, yeah, excavation and engagement, right. 666 00:32:59,600 --> 00:33:02,880 But now like 2 1/2 years in or two years in or something like 667 00:33:02,880 --> 00:33:05,000 that, we're starting to see more like, Oh yeah, OK. 668 00:33:05,000 --> 00:33:08,280 We're we can't go for like optimizing productivity to 669 00:33:08,480 --> 00:33:11,040 engagement. What is like, what is in between 670 00:33:11,040 --> 00:33:13,000 and how are we, how can we define that? 671 00:33:13,000 --> 00:33:15,400 And it's just a brainstorming exercise, to be honest, between 672 00:33:15,400 --> 00:33:19,120 ton of people, usually between PMS and managers, like 673 00:33:19,120 --> 00:33:20,800 engineering managers, but it's hard. 674 00:33:20,800 --> 00:33:23,200 It's such a hard thing to to create. 675 00:33:23,200 --> 00:33:26,000 Yeah, I feel like. Listening to your story and kind 676 00:33:26,000 --> 00:33:29,440 of the the scale of the operation you're in, you're at 677 00:33:29,440 --> 00:33:31,120 least engaging with a lot of people. 678 00:33:31,560 --> 00:33:34,600 How do you build kind of those relationships to have trust in 679 00:33:34,600 --> 00:33:36,920 the other people that they have kind of the same intentions or 680 00:33:37,280 --> 00:33:40,320 build the same level baseline of way of working and also 681 00:33:40,600 --> 00:33:43,120 communication cultures might be different completely. 682 00:33:43,520 --> 00:33:45,200 I feel like you need relationships to thrive. 683 00:33:45,560 --> 00:33:46,440 How do you build? Those. 684 00:33:46,480 --> 00:33:49,840 With so many people. It's, it's a really interesting 685 00:33:49,840 --> 00:33:51,960 question also because like you were chatting before the 686 00:33:51,960 --> 00:33:54,880 podcast, like all my engineers are in Serbia, right, Which is 687 00:33:55,160 --> 00:33:56,680 they're physically distant from me. 688 00:33:56,680 --> 00:33:58,400 It's like an entirely different culture. 689 00:33:58,800 --> 00:34:03,640 I've personally, and, and then like I'm also working with a ton 690 00:34:03,640 --> 00:34:06,440 of folks from the US, like, like none of my colleagues are in the 691 00:34:06,440 --> 00:34:07,720 Netherlands, right. So there's always. 692 00:34:07,720 --> 00:34:09,520 Like the only one. I'm literally the only one. 693 00:34:10,320 --> 00:34:14,080 Yeah, literally the only product organization in the Netherlands 694 00:34:14,080 --> 00:34:16,520 sits here. All of it, all of it. 695 00:34:19,960 --> 00:34:23,239 Like I personally always come from deeply understanding the 696 00:34:23,239 --> 00:34:25,480 person up so well. So like eventually your 697 00:34:25,480 --> 00:34:27,800 colleagues and like the the engineers you're working with 698 00:34:27,800 --> 00:34:29,880 are also kind of your customers in a way, right? 699 00:34:29,880 --> 00:34:32,520 You can see them like that. And just like you're trying to 700 00:34:32,520 --> 00:34:34,280 deeply understand your customers, you have to deeply 701 00:34:34,280 --> 00:34:35,639 understand your engineers as well. 702 00:34:37,400 --> 00:34:40,040 So I'm always trying to come from kind of the human and go 703 00:34:40,040 --> 00:34:42,280 things like, oh, why are you like, why are you here? 704 00:34:42,280 --> 00:34:43,880 Why are you working in Microsoft, right? 705 00:34:43,880 --> 00:34:47,000 Like what are you trying to achieve as an engineer? 706 00:34:47,000 --> 00:34:49,560 And how I also see responsibility for me as a PM, 707 00:34:49,560 --> 00:34:53,400 like how can I help you as an engineer reach your goals, 708 00:34:53,800 --> 00:34:55,600 right? And that means, so for example, 709 00:34:55,600 --> 00:34:58,400 I've had engineers that I knew had a passion in something that 710 00:34:58,400 --> 00:35:00,440 I couldn't execute it well. That's great. 711 00:35:00,440 --> 00:35:04,920 And it's like on me to make sure that if a project like that 712 00:35:04,920 --> 00:35:08,000 comes along, I can actually put them on that, right? 713 00:35:08,800 --> 00:35:11,800 Or Hyatt engineers reaching out and saying, hey, Stephanie, I 714 00:35:11,800 --> 00:35:15,120 really want to improve my product thinking. 715 00:35:15,120 --> 00:35:19,600 And I sometimes have ideas, but I feel afraid of speaking up in 716 00:35:19,600 --> 00:35:21,360 a large group. So can you help me with that? 717 00:35:21,360 --> 00:35:24,080 And then we have weekly calls where I'm just listening and 718 00:35:24,080 --> 00:35:25,760 they're pitching product ideas to me. 719 00:35:25,760 --> 00:35:28,160 And I'm saying, oh, yeah, you know, you did this work as well. 720 00:35:28,160 --> 00:35:30,240 This doesn't work well. Oh, maybe you can do this. 721 00:35:30,720 --> 00:35:34,480 So bottom line is really deeply trying to understand people 722 00:35:34,480 --> 00:35:38,560 helps building trust so much, I feel, because then you kind of 723 00:35:38,560 --> 00:35:40,760 mutually understand what they're coming from, where they're 724 00:35:40,760 --> 00:35:43,040 coming from. And especially also in 725 00:35:43,040 --> 00:35:46,000 organization of Microsoft. Like I, I work in words Copilot 726 00:35:46,200 --> 00:35:48,560 and then there are like 100 different products that 727 00:35:48,560 --> 00:35:51,760 Microsoft is building and I have to communicate with and, and 728 00:35:51,760 --> 00:35:55,960 align with those kind of product managers as well from different 729 00:35:55,960 --> 00:36:00,320 products. And sometimes obviously you see 730 00:36:00,320 --> 00:36:03,880 clashes, but those clashes always are coming from the fact 731 00:36:03,880 --> 00:36:06,640 that's what that person wants to achieve. 732 00:36:06,640 --> 00:36:08,920 It's just simply different from what you're trying to achieve. 733 00:36:08,920 --> 00:36:12,680 However, do I really understand what you're trying to achieve 734 00:36:12,680 --> 00:36:15,040 the other person and why that is? 735 00:36:15,480 --> 00:36:18,400 And if I understand that, then I can maybe find an angle or a 736 00:36:18,400 --> 00:36:21,120 way, you know, that's like that, that that's going to help both 737 00:36:21,120 --> 00:36:22,480 of us. But if I don't ask you, if I 738 00:36:22,480 --> 00:36:24,120 don't deeply understand you, I have no idea. 739 00:36:24,560 --> 00:36:26,520 And that's where you're getting these, like, surface level 740 00:36:26,520 --> 00:36:27,920 crashes, right? Yeah. 741 00:36:28,280 --> 00:36:30,880 I really recognize what you're saying and it's it's mainly what 742 00:36:30,880 --> 00:36:34,800 I see, what I'm just observing when I see two people trying to 743 00:36:34,800 --> 00:36:37,760 push their opinion on the other person and they're just going 744 00:36:37,840 --> 00:36:38,640 past. Each other. 745 00:36:38,680 --> 00:36:40,880 Yeah, and they're not noticing. No, no, exactly. 746 00:36:40,880 --> 00:36:41,960 And they're not listening either. 747 00:36:42,040 --> 00:36:44,520 No one's asking a question. Yeah, exactly. 748 00:36:44,840 --> 00:36:47,080 And there's probably like a base of understanding like they're 749 00:36:47,080 --> 00:36:49,240 shoot, there's somewhere a base where they can understand each 750 00:36:49,240 --> 00:36:51,640 other, right. But someone communicates it like 751 00:36:51,640 --> 00:36:53,720 this and it doesn't resonate with the other. 752 00:36:53,720 --> 00:36:55,240 So indeed, there's push back, right? 753 00:36:55,320 --> 00:36:58,240 Even though those people would step back a little bit, we would 754 00:36:58,560 --> 00:37:01,200 like find some common ground from where we can then, like 755 00:37:02,080 --> 00:37:03,480 collaborate together. Yeah. 756 00:37:04,000 --> 00:37:08,200 I feel like the people that are curious kind of innately also 757 00:37:08,200 --> 00:37:10,360 not just about products, but also about people where they're 758 00:37:10,360 --> 00:37:13,280 coming from, behaviour. You even mentioned kind of 759 00:37:13,280 --> 00:37:15,880 ambitions and goals. I think that's fantastic, 760 00:37:16,080 --> 00:37:19,320 especially as a person that's responsible for product and kind 761 00:37:19,320 --> 00:37:22,240 of work that gets done. You are equipped to put people 762 00:37:22,240 --> 00:37:25,720 in positions to thrive and if that, with their ambition and 763 00:37:25,720 --> 00:37:27,440 that's perfect. Yeah, yeah, exactly. 764 00:37:27,440 --> 00:37:29,560 And like eventually that creates a better product as well. 765 00:37:29,560 --> 00:37:31,520 If I have happy engineers working on the thing that 766 00:37:31,520 --> 00:37:34,120 they're so passionate about, that's only good for the 767 00:37:34,120 --> 00:37:36,440 product, right? So it is in my best interest as 768 00:37:36,440 --> 00:37:39,800 a product, as a person responsible for the product as 769 00:37:39,800 --> 00:37:43,040 well, that my engineers are as happy as possible and know what 770 00:37:43,040 --> 00:37:45,040 they're working on, they're excited what they're working on. 771 00:37:45,240 --> 00:37:47,800 It might be the most kind of undervalued statistics. 772 00:37:47,800 --> 00:37:49,600 Happy engineer. Yeah, Yeah, genuinely. 773 00:37:49,920 --> 00:37:55,840 Yeah, maybe it should be an OK, honestly, I think generally like 774 00:37:55,840 --> 00:37:57,920 people's skills in that way and deeply understanding people's a 775 00:37:57,920 --> 00:38:00,800 very unvalued skill. And like we always think about, 776 00:38:00,800 --> 00:38:03,400 oh, you know, like is it product manager or even an engineer or 777 00:38:03,400 --> 00:38:05,320 you're, you're as good as like the lines of code that you 778 00:38:05,320 --> 00:38:09,640 write, etcetera, but you're not. I was a when I got hired as a 779 00:38:09,640 --> 00:38:12,720 software engineer. I had absolutely no software 780 00:38:12,720 --> 00:38:14,800 engineering experience in real life. 781 00:38:15,520 --> 00:38:18,160 In in real life roles obviously like coding by myself. 782 00:38:18,560 --> 00:38:20,360 Otherwise I wouldn't. Get a job, otherwise I wouldn't 783 00:38:20,360 --> 00:38:22,760 get a job. But not in any real life 784 00:38:22,760 --> 00:38:25,720 systems. And, and so I got a software 785 00:38:25,720 --> 00:38:27,480 engineering job at Microsoft, right? 786 00:38:27,680 --> 00:38:30,080 And it wasn't because of like the brilliant code that I wrote, 787 00:38:30,080 --> 00:38:33,680 but it was because I, I was a customer facing software 788 00:38:33,680 --> 00:38:35,320 engineer. It was because I was good at 789 00:38:35,320 --> 00:38:38,280 understanding a customer and really trying like trying to see 790 00:38:38,280 --> 00:38:40,920 where they were coming from and what we needed to build and kind 791 00:38:40,920 --> 00:38:43,160 of, you know, build a bridge between the customer and our 792 00:38:43,160 --> 00:38:47,200 engineering team and not because I was like a the 10X coder, you 793 00:38:47,200 --> 00:38:48,160 know? Yeah, yeah. 794 00:38:48,320 --> 00:38:50,400 Yeah, I feel like that is such a life skill. 795 00:38:50,840 --> 00:38:54,640 Like you had that as a person, you got a job as a software 796 00:38:54,640 --> 00:38:56,920 engineer because of that. You're completely leveraging 797 00:38:56,920 --> 00:38:58,400 that and the role that you now. Have. 798 00:38:58,880 --> 00:39:01,440 If there's ever another role, it will still be valuable. 799 00:39:01,480 --> 00:39:02,880 Yeah, yeah, yeah, for sure. Yeah. 800 00:39:02,880 --> 00:39:05,000 It's like those transferable skills, right? 801 00:39:05,160 --> 00:39:08,960 Yeah, absolutely. As APN because you're also the 802 00:39:09,160 --> 00:39:11,680 only person in the Netherlands who challenges you like from a 803 00:39:11,680 --> 00:39:15,360 craft perspective. That's an excellent question. 804 00:39:15,800 --> 00:39:18,040 All my engineers, honestly. Yeah, yeah. 805 00:39:18,520 --> 00:39:21,680 I love it when I'm like kind of building a first version of a 806 00:39:21,680 --> 00:39:24,920 spec and I'm only building fee 1 specs at this point just because 807 00:39:24,920 --> 00:39:27,920 like I know that whenever you do, you get that feeling 808 00:39:27,920 --> 00:39:30,640 sometimes that you're building a spec and you've put so much love 809 00:39:30,640 --> 00:39:32,760 and effort into it and you're like, this is perfect and you're 810 00:39:32,760 --> 00:39:35,720 sharing a doubt and you get so much feedback that yeah, so many 811 00:39:35,720 --> 00:39:38,160 holes that it almost feels personal, right? 812 00:39:38,160 --> 00:39:38,880 I cannot do. That. 813 00:39:39,000 --> 00:39:42,560 Yeah, really bad at that. Exactly. 814 00:39:42,800 --> 00:39:46,640 So I'm like, I'm just like going to make like OK, fee 0.1 specs. 815 00:39:46,640 --> 00:39:48,960 I'm just going to send it out to all my engineers and my, and, 816 00:39:48,960 --> 00:39:51,840 and what I mentioned I'm very lucky to work with engineers who 817 00:39:51,840 --> 00:39:53,560 are so product and customer focused. 818 00:39:54,800 --> 00:39:59,000 They constantly challenge me. And then also kind of my my 819 00:39:59,000 --> 00:40:01,600 direct people that I work with are the engineering managers and 820 00:40:01,600 --> 00:40:05,840 the science managers. I have like often more than once 821 00:40:05,840 --> 00:40:07,280 a week I have meetings with them. 822 00:40:07,280 --> 00:40:10,840 We're just discussing products and ideas, etcetera. 823 00:40:11,720 --> 00:40:12,960 Yeah. So those are the people. 824 00:40:12,960 --> 00:40:14,600 Yeah, I heavily rely on them. Yeah, yeah. 825 00:40:15,120 --> 00:40:18,600 Some organizations that I've seen, I go in and out have like 826 00:40:18,600 --> 00:40:22,240 this, OK, you are part and responsible from your team, but 827 00:40:22,240 --> 00:40:24,400 then you have alignments with people like from the same craft. 828 00:40:24,400 --> 00:40:26,600 So I thought you would have a structure like that as well. 829 00:40:26,960 --> 00:40:30,000 But I really like that the challenge like on a day-to-day, 830 00:40:30,320 --> 00:40:31,640 I feel like it should come from a team. 831 00:40:31,800 --> 00:40:33,200 I think so, to raise the bar together. 832 00:40:33,200 --> 00:40:36,440 Yeah, exactly, exactly, exactly. And it doesn't become so siloed, 833 00:40:36,480 --> 00:40:38,520 right? You make like everyone 834 00:40:38,520 --> 00:40:40,320 accountable for what you're doing. 835 00:40:40,320 --> 00:40:45,560 And obviously I have other PMS, but I'm not, no, I'm not 836 00:40:45,560 --> 00:40:47,840 necessarily, they're not necessarily challenging me on a 837 00:40:47,840 --> 00:40:49,520 daily basis. It's not the same. 838 00:40:49,560 --> 00:40:52,920 Yeah, they also, they're also not so deep into like our aspect 839 00:40:52,920 --> 00:40:54,160 of the product that we're trying to build. 840 00:40:54,160 --> 00:40:56,120 And my engineers are right, yeah. 841 00:40:56,280 --> 00:40:57,680 That's super exciting. Yeah. 842 00:40:57,680 --> 00:41:00,040 You said you were lucky that you have people like that. 843 00:41:00,360 --> 00:41:04,000 But I feel like, I mean, if you were my PM, kind of the way you 844 00:41:04,000 --> 00:41:07,040 think about product and also the importance that you put on kind 845 00:41:07,040 --> 00:41:10,480 of feeling that other people need to think alike and think 846 00:41:10,480 --> 00:41:12,440 about the product. It's kind of also the culture 847 00:41:12,440 --> 00:41:13,680 that you've cultivated. Thank you. 848 00:41:13,800 --> 00:41:15,480 Yeah, yeah. At least that's my assumption. 849 00:41:15,480 --> 00:41:19,440 Yeah, that is it is. It is like I'm not trying to 850 00:41:19,480 --> 00:41:22,720 like my own horn, but I think it is something that I get back 851 00:41:22,720 --> 00:41:27,000 from my engineers and because I'm not like I've been product 852 00:41:27,000 --> 00:41:31,240 managing for not long, right? Like a few years, maybe 4 or 4 853 00:41:31,240 --> 00:41:34,920 1/2 years, something like that. So on paper, I'm for sure not 854 00:41:34,920 --> 00:41:37,680 the best PM that's out there because like I am, I work with 855 00:41:37,680 --> 00:41:40,400 people who've been PM ING for 15 years or longer, right? 856 00:41:40,400 --> 00:41:43,400 So they're probably better in kind of the, the technicalities 857 00:41:43,400 --> 00:41:45,960 of PM ING. At the same times, my engineers 858 00:41:45,960 --> 00:41:49,480 tell me that they really enjoy working with me indeed because 859 00:41:49,480 --> 00:41:53,360 of that reason, because I set a, I think someone even said at a 860 00:41:53,360 --> 00:41:55,920 certain point like, oh, you really put back the kind of the 861 00:41:55,920 --> 00:41:59,320 enjoyment in my work, right? And I think that's indeed what 862 00:41:59,320 --> 00:42:02,600 can really make you stand out as APM, just creating that culture. 863 00:42:02,640 --> 00:42:05,000 That's a beautiful compliment. It was such a good compliment. 864 00:42:06,760 --> 00:42:09,160 My job is done, you know, I can retire. 865 00:42:11,200 --> 00:42:13,240 Yeah. Are there because you went from 866 00:42:13,240 --> 00:42:17,040 software engineering, now you're PM and now you've dealing with 867 00:42:17,040 --> 00:42:20,240 kind of a lot of disciplines. Is PM like the thing for you, 868 00:42:20,240 --> 00:42:21,520 the thing that makes you most happy? 869 00:42:22,040 --> 00:42:24,920 Thank you so much. I love Yeming. 870 00:42:24,920 --> 00:42:27,440 Yeah, it's so good. It's a bit of everything. 871 00:42:27,480 --> 00:42:30,520 It is a bit of everything, yeah. And even before, like my road to 872 00:42:30,520 --> 00:42:32,200 here has been super weird, right? 873 00:42:32,200 --> 00:42:33,640 I studied hospitality management. 874 00:42:33,640 --> 00:42:36,040 So I come from a hospitality management background. 875 00:42:36,040 --> 00:42:38,040 I worked in hotels before moving into tech. 876 00:42:38,600 --> 00:42:41,600 I moved from hospitality management to tech by doing 877 00:42:41,600 --> 00:42:44,120 sales. So I was first doing sales and 878 00:42:44,120 --> 00:42:46,680 software engineering and PM ING. So I've seen like a whole bunch 879 00:42:46,680 --> 00:42:48,320 of weird stuff. And I think this is indeed like 880 00:42:48,320 --> 00:42:52,680 the perfect combination of like what I love about hospitality 881 00:42:52,680 --> 00:42:54,960 management is the whole customer focus, right? 882 00:42:54,960 --> 00:42:58,240 And I can totally use it here. What I love about engineering is 883 00:42:58,240 --> 00:42:59,440 that we're actually building products. 884 00:42:59,440 --> 00:43:01,200 I can totally do that here as AVM. 885 00:43:02,480 --> 00:43:04,160 I write marketing and stuff like that. 886 00:43:04,160 --> 00:43:06,040 So kind of the sales element isn't there as well. 887 00:43:06,040 --> 00:43:08,560 Like it is such a perfect blend of everything. 888 00:43:08,560 --> 00:43:12,320 And then it makes me so happy to work with people who are so good 889 00:43:12,320 --> 00:43:14,720 at what they're doing, right? There's engineer scientists and 890 00:43:14,720 --> 00:43:18,360 designers. And as a PM, just like, wow, 891 00:43:18,360 --> 00:43:21,120 this is amazing, you know, and we can build cool stuff 892 00:43:21,120 --> 00:43:23,200 together. So yeah, this is like, I'm 893 00:43:23,200 --> 00:43:26,160 staying here. Nobody's kicking me out of PMA. 894 00:43:27,520 --> 00:43:29,600 Yeah. I can truly relate to that 895 00:43:29,600 --> 00:43:32,200 experience. Like as PM, I feel like still 896 00:43:32,200 --> 00:43:34,240 you're kind of in a leadership position. 897 00:43:34,360 --> 00:43:37,280 And I truly believe that the way you behave, the way that every 898 00:43:37,280 --> 00:43:40,880 leader behaves, like it trickles down into whomever you engage 899 00:43:40,880 --> 00:43:43,440 with kind of in expectations behaviours. 900 00:43:43,440 --> 00:43:46,480 Do you own up to the problems that you've caused or like the 901 00:43:46,480 --> 00:43:48,640 the things that you've done, or do you shy away from them? 902 00:43:48,640 --> 00:43:50,920 Or do you like law or do you like, are you integral? 903 00:43:51,240 --> 00:43:53,480 All of that matters. In the people that you engage 904 00:43:53,480 --> 00:43:55,800 with, Yeah, yeah, you're right. Like you're responsible for the 905 00:43:55,800 --> 00:43:57,440 culture. Like not only for the product, 906 00:43:57,440 --> 00:44:00,040 but for the culture as well. And the culture trickles back 907 00:44:00,040 --> 00:44:02,720 into the product to which is really cool. 908 00:44:03,600 --> 00:44:04,960 Yeah. And like, the great thing about 909 00:44:04,960 --> 00:44:09,600 PM ING right now is like, it's so exciting to be responsible to 910 00:44:09,600 --> 00:44:12,000 think of. Oh, you know, what should AI be 911 00:44:12,000 --> 00:44:15,120 like in five years? Indeed, as you were saying, 912 00:44:15,120 --> 00:44:16,600 like, it's trailblazing. Nobody. 913 00:44:16,640 --> 00:44:19,360 Nobody knows the answer. And it's so cool to be 914 00:44:19,360 --> 00:44:21,920 responsible for thinking through that. 915 00:44:22,200 --> 00:44:24,080 Cool. Yeah, I have one last question. 916 00:44:24,160 --> 00:44:25,800 Yes, and that's more from a tooling perspective. 917 00:44:26,200 --> 00:44:29,200 Are there any tools that you've used AI related that make you 918 00:44:29,200 --> 00:44:32,680 more productive that kind of yes help you in your product? 919 00:44:32,680 --> 00:44:35,120 Yes, absolutely. Which ones do you love? 920 00:44:36,040 --> 00:44:39,120 Which ones do I love? I love Cursor, to be honest. 921 00:44:39,120 --> 00:44:44,920 Yeah, I don't do a ton of five coding just to like make designs 922 00:44:44,920 --> 00:44:47,400 and prototypes really quickly because I think that's such an 923 00:44:47,400 --> 00:44:49,880 easier way to communicate your idea. 924 00:44:50,120 --> 00:44:53,480 Like back in the days as well, when I had a feature idea, 925 00:44:53,480 --> 00:44:56,080 right, and I needed to convince people I would probably need to 926 00:44:56,080 --> 00:44:58,440 engage with designers. Designers wouldn't need to make 927 00:44:58,440 --> 00:45:01,200 time to kind of get my thinking on paper. 928 00:45:01,200 --> 00:45:03,040 Yeah, back. And forth is not right. 929 00:45:03,040 --> 00:45:05,680 Back and forth exactly. Maybe you would want to engage 930 00:45:05,680 --> 00:45:08,000 in engineers, make like a quick prototype or something like 931 00:45:08,000 --> 00:45:09,440 that. Well, how long would that take? 932 00:45:09,440 --> 00:45:11,960 Like maybe 4 weeks, six weeks or something like that. 933 00:45:12,280 --> 00:45:16,040 Now as APMI can just sit, sit behind my screen, five goes 934 00:45:16,040 --> 00:45:18,200 something in a day, shared up with people and people are like, 935 00:45:18,200 --> 00:45:19,680 oh, this is a good idea or not a good idea. 936 00:45:19,680 --> 00:45:22,400 Like the velocity is so much faster, which is really, really 937 00:45:22,400 --> 00:45:26,120 cool. So for short cursor, for five 938 00:45:26,120 --> 00:45:28,960 coding, I've tried obviously like lovable V-0, like all these 939 00:45:28,960 --> 00:45:32,320 kind of things. SOPM it's also your job to try 940 00:45:32,520 --> 00:45:34,520 out like ton of things play around. 941 00:45:34,520 --> 00:45:37,840 Yeah, obviously I'm loose using ChatGPT a lot as well. 942 00:45:38,520 --> 00:45:44,000 Also for consolidating user interviews, doing research, you 943 00:45:44,000 --> 00:45:47,800 know, all of those things. New entropic opus model is 944 00:45:47,800 --> 00:45:49,400 really cool. I'm not sure if you tried it. 945 00:45:49,400 --> 00:45:52,360 I I read about. It it's so good, yeah, it's so 946 00:45:52,360 --> 00:45:56,120 good as APM. You're also like, you have to be 947 00:45:56,120 --> 00:45:59,880 so model forward thinking, right? 948 00:45:59,880 --> 00:46:02,720 So you have to understand what each model can do and like how 949 00:46:02,720 --> 00:46:06,720 to grow prompts and how to get the most out of your models. 950 00:46:06,720 --> 00:46:10,000 So whatever works there, like can be, it can be claws, 951 00:46:10,000 --> 00:46:13,160 whatever. You're also like super pragmatic 952 00:46:13,160 --> 00:46:15,680 with regards to getting stuff. Done but. 953 00:46:15,720 --> 00:46:18,520 All these responsibilities and now also data science kind of 954 00:46:18,520 --> 00:46:21,400 coming on your plate. Is it too much like from a 955 00:46:21,400 --> 00:46:24,240 product sense? Do you think it's getting too 956 00:46:24,240 --> 00:46:26,920 wide with regards to things you need to know or kind of need to 957 00:46:26,920 --> 00:46:29,880 at least have an idea of? One interesting question I 958 00:46:29,880 --> 00:46:37,160 think, I think I had to, I had to shift a ton of my 959 00:46:37,160 --> 00:46:40,000 responsibilities to engineering and designers like making like 960 00:46:40,000 --> 00:46:42,760 small decisions, etcetera. I don't want to be responsible 961 00:46:42,760 --> 00:46:45,920 for them anymore. I'm just kind of observing and 962 00:46:45,920 --> 00:46:48,160 if it goes completely wrong direction, I'm kind of steering 963 00:46:48,160 --> 00:46:49,800 them. But like small decisions totally 964 00:46:49,800 --> 00:46:52,440 up to them. That makes that I have much more 965 00:46:52,440 --> 00:46:56,520 space for kind of experimenting and like vibe coding and 966 00:46:56,520 --> 00:47:00,600 understanding models etcetera. So it's indeed about being super 967 00:47:00,600 --> 00:47:03,520 deliberate with your time. And like what I mentioned 968 00:47:03,520 --> 00:47:06,920 before, thinking about, oh, where do I add unique value, 969 00:47:06,920 --> 00:47:09,800 what is super relevant of my role and that is understanding 970 00:47:09,800 --> 00:47:12,160 the industry, the mark, the models, et cetera. 971 00:47:12,760 --> 00:47:15,080 And then all of those product decisions, I just trust other 972 00:47:15,080 --> 00:47:17,200 people to do that because I've created that trust with the 973 00:47:17,200 --> 00:47:18,520 team. Yeah, that's amazing. 974 00:47:18,640 --> 00:47:20,360 Yeah, I've really enjoyed this conversation. 975 00:47:20,440 --> 00:47:23,600 I think you have super cool energy, super pragmatic, 976 00:47:23,680 --> 00:47:25,040 passionate about what you do. Thank. 977 00:47:25,040 --> 00:47:26,760 You so much, I really enjoyed it as well. 978 00:47:26,760 --> 00:47:28,480 Thank you for inviting me. Appreciate it. 979 00:47:28,480 --> 00:47:30,280 Of course, we're going to round it off here. 980 00:47:30,280 --> 00:47:31,880 Let us know in the comments section what you thought. 981 00:47:31,880 --> 00:47:34,520 Like the episode? If you did, it's free, so click 982 00:47:34,520 --> 00:47:36,320 the like button and we'll see you next time.