1 00:00:14,480 --> 00:00:18,160 Welcome to the APM podcast. APM is the childhood body for 2 00:00:18,160 --> 00:00:21,320 the project profession. My name is Emma David and I'm 3 00:00:21,320 --> 00:00:24,520 the editor of Project APM's quarterly journal and your host 4 00:00:24,520 --> 00:00:27,160 today. In this podcast, we're talking 5 00:00:27,160 --> 00:00:29,760 about how to measure anything in project management. 6 00:00:29,760 --> 00:00:32,119 Yes, you did hear me write anything. 7 00:00:32,759 --> 00:00:35,480 It's the title of a new book that is co-authored by Douglas 8 00:00:35,480 --> 00:00:38,080 Hubbard, Alexander Budzia and Andreas lead. 9 00:00:38,720 --> 00:00:41,440 I have the pleasure of being joined by Doug and Andreas Now. 10 00:00:42,320 --> 00:00:46,120 Doug is founder and president of Hubbard Decision Research, is a 11 00:00:46,120 --> 00:00:48,840 management consultant and the author of a series of books 12 00:00:48,840 --> 00:00:51,600 including the brilliantly titled How to Measure Anything. 13 00:00:52,480 --> 00:00:56,080 Andreas is head of Data Science at Oxford Global Projects and is 14 00:00:56,080 --> 00:00:58,960 a PhD Fellow at AAHSA University in Denmark. 15 00:00:59,840 --> 00:01:02,680 Be prepared for mind expanding conversation about project 16 00:01:02,680 --> 00:01:06,880 management and how any aspect of it can be measured, including 17 00:01:06,880 --> 00:01:09,600 those things you thought might be immeasurable and why we 18 00:01:09,600 --> 00:01:12,400 should all be engaged in the most important project of all, 19 00:01:12,720 --> 00:01:15,400 the meta project of understanding how to manage 20 00:01:15,400 --> 00:01:19,040 projects better. I'd like to begin by welcoming 21 00:01:19,040 --> 00:01:21,080 Andreas and Douglas to the APM Podcast. 22 00:01:21,080 --> 00:01:24,040 So thanks very much for finding the time to speak to us today. 23 00:01:25,040 --> 00:01:26,680 Thank you for having us. You bet. 24 00:01:27,880 --> 00:01:32,080 Let's begin by you telling us a little bit about your area of 25 00:01:32,080 --> 00:01:34,360 work and interest within project management. 26 00:01:34,360 --> 00:01:38,600 So Andreas, tell us a bit about what you're what you're working 27 00:01:38,600 --> 00:01:41,040 on right now. Yes, Sir, of course. 28 00:01:41,040 --> 00:01:44,760 So I'm, I'm Andreas lead and the head of Data Science at Oxford 29 00:01:44,760 --> 00:01:47,760 Global Projects. And Oxford Global projects is a 30 00:01:47,760 --> 00:01:51,960 small consultancy which is really about analysing vast 31 00:01:51,960 --> 00:01:58,000 amount of vast amounts of project data and and using it to 32 00:01:58,040 --> 00:02:00,400 to have a better decision making around projects. 33 00:02:01,120 --> 00:02:05,720 Another of what we do is about, you know, debiasing estimates 34 00:02:05,720 --> 00:02:09,600 and kind of assuring estimates, realistic, basing estimates on 35 00:02:09,800 --> 00:02:14,280 historical data rather than bottom up type approaches. 36 00:02:14,960 --> 00:02:17,000 But it's also kind of bigger than that. 37 00:02:17,000 --> 00:02:20,600 So as the head of data science, I did some of the more 38 00:02:20,600 --> 00:02:25,360 quantitative engagements, things like using AI for machine 39 00:02:25,360 --> 00:02:28,840 learning for forecasting project outcomes or doing kind of 40 00:02:28,840 --> 00:02:31,120 econometric cost driver analysis. 41 00:02:32,080 --> 00:02:35,760 And the big goal with all of this is, is to better projects 42 00:02:35,760 --> 00:02:39,720 because after having looked at all the project data we have 43 00:02:39,720 --> 00:02:43,440 access to, which is now about 20 data from 20,000 completed 44 00:02:43,440 --> 00:02:48,320 projects, the the big conclusion is that rojects aren't erforming 45 00:02:48,320 --> 00:02:50,520 that well. O we'd like to understand, you 46 00:02:50,520 --> 00:02:53,160 know, what leads to success and how can we imrove roject 47 00:02:53,160 --> 00:02:55,080 erformance? Thank you. 48 00:02:55,080 --> 00:02:58,080 So that's not necessarily a controversial finding, is it, 49 00:02:58,080 --> 00:03:00,840 unfortunately. But good to know. 50 00:03:01,080 --> 00:03:04,080 We're here today to talk about how we can make perhaps make 51 00:03:04,080 --> 00:03:07,120 projects more successful through the way they're measured or what 52 00:03:07,240 --> 00:03:10,240 what is in fact measured. So Douglas, tell us a bit about 53 00:03:10,240 --> 00:03:13,760 your area of interest and and actually tell us how the book 54 00:03:13,760 --> 00:03:16,040 came about as well, because that's always good to know. 55 00:03:17,600 --> 00:03:21,040 Yeah, sure. So I'm Doug Hubbard. 56 00:03:21,160 --> 00:03:24,600 I'm the president and founder of Hubbard Decision Research, and 57 00:03:24,600 --> 00:03:29,800 for the past 36 or 37 years or so I've been in some form of 58 00:03:29,800 --> 00:03:31,800 quantitative management consulting. 59 00:03:32,280 --> 00:03:37,280 So all of my work, including all my books, this was my fifth book 60 00:03:37,280 --> 00:03:42,560 actually, They've all been on some topic related to decisions 61 00:03:42,560 --> 00:03:45,560 under uncertainty and difficult measurements. 62 00:03:45,840 --> 00:03:49,680 So that includes risk analysis, making big bets, and lots of 63 00:03:49,680 --> 00:03:52,400 different areas of forecasting problems. 64 00:03:52,400 --> 00:03:56,960 And of course, that applies to projects in general. 65 00:03:57,080 --> 00:04:00,720 My previous four books were not about projects specifically, but 66 00:04:00,720 --> 00:04:04,160 certainly a lot of my work was about risk return analysis on 67 00:04:04,160 --> 00:04:08,640 major new investments, which were usually projects in regards 68 00:04:08,640 --> 00:04:11,720 to how we met. Actually, I'll let Andres 69 00:04:12,360 --> 00:04:17,079 describe this because initially 2 of you called me. 70 00:04:18,399 --> 00:04:20,320 Yeah, so. So I was writing an article 71 00:04:20,320 --> 00:04:24,680 together with a colleague about cost, benefit, cost benefit 72 00:04:24,680 --> 00:04:27,120 analysis. Now kind of alternatives to, to 73 00:04:27,280 --> 00:04:30,200 typical inclusions in cost benefit analysis. 74 00:04:30,640 --> 00:04:34,080 I, I also teach cost benefit analysis at the Department of 75 00:04:34,080 --> 00:04:36,360 economics here in all who's where I'm based. 76 00:04:37,800 --> 00:04:40,800 But we found that Doug's work was really interesting. 77 00:04:40,800 --> 00:04:45,160 So we, we asked him for an interview and kind of some ideas 78 00:04:45,160 --> 00:04:47,120 around what should we include in this article. 79 00:04:48,280 --> 00:04:50,160 And I guess it was, it was a good match. 80 00:04:50,600 --> 00:04:53,800 Doug was looking for his next book project and project 81 00:04:53,800 --> 00:04:56,880 management was one of the topics he was considering. 82 00:04:56,880 --> 00:04:59,160 And so because we had access to a lot of data and a lot of 83 00:04:59,160 --> 00:05:01,960 analysis that kind of fit in really well with the book. 84 00:05:02,640 --> 00:05:03,920 I think that's fair, isn't it, Doug? 85 00:05:04,640 --> 00:05:07,960 Oh yeah, absolutely. In fact, I I chose this book 86 00:05:07,960 --> 00:05:13,560 title based on analytics. I decided to go after How to 87 00:05:13,560 --> 00:05:16,960 Measure Anything in Project Management because it had 88 00:05:16,960 --> 00:05:20,600 previously tied as one of the top 2 with How to Measure 89 00:05:20,600 --> 00:05:23,920 Anything in Cybersecurity Risk, which I've already wrote now. 90 00:05:24,600 --> 00:05:27,720 So I wrote that one first. And what tipped the needle in 91 00:05:27,720 --> 00:05:32,120 the in favour of the first book of the the 4th, my 4th book. 92 00:05:32,120 --> 00:05:34,760 But the previous How to Measure Anything in Cybersecurity Risk 93 00:05:34,760 --> 00:05:38,680 book was that I had a a well qualified, enthusiastic 94 00:05:38,680 --> 00:05:42,440 co-author that said, Hey, I'll write this with you. 95 00:05:42,440 --> 00:05:44,480 And I said, OK, well, then let's get started on that. 96 00:05:44,880 --> 00:05:47,320 And so I was really looking for the right people. 97 00:05:47,320 --> 00:05:51,280 I actually talked to a bunch of other possible candidates for 98 00:05:51,280 --> 00:05:53,920 co-authors for how to measure anything in project management. 99 00:05:54,240 --> 00:05:57,600 And my strategy is always to find somebody who's a 100 00:05:57,600 --> 00:06:02,200 specialist, an expert in renowned in that field because 101 00:06:02,200 --> 00:06:04,840 I'm always the outsider, I'm always the outside quant 102 00:06:04,920 --> 00:06:07,480 analyst, applying it to different kinds of problems, 103 00:06:07,480 --> 00:06:09,600 etcetera. So what I wanted to do a spin 104 00:06:09,600 --> 00:06:12,120 off of my first book, my first book was just how to measure 105 00:06:12,120 --> 00:06:14,800 anything. So that and I meant it 106 00:06:14,800 --> 00:06:17,440 literally. So how to measure anything. 107 00:06:17,480 --> 00:06:20,280 And I decided to do a series of spin off books like that. 108 00:06:20,600 --> 00:06:25,800 And so I waited for the right co-authors and when I learned 109 00:06:25,800 --> 00:06:28,760 about Oxford Global projects, I thought, well, that's ideal. 110 00:06:28,760 --> 00:06:32,360 They've got all this data. They, they already have these 111 00:06:32,360 --> 00:06:35,480 guys that are, that have been studying projects and all the 112 00:06:35,480 --> 00:06:37,800 problems with projects for many years now. 113 00:06:37,800 --> 00:06:41,400 So it seemed like the ideal co-authors. 114 00:06:42,280 --> 00:06:47,000 That's really interesting. So Doug, can you measure 115 00:06:47,000 --> 00:06:49,040 anything? Sorry, I've got to ask you. 116 00:06:50,000 --> 00:06:52,400 Yeah, absolutely. Including the hardest most 117 00:06:52,520 --> 00:06:54,160 possible things you've ever heard of. 118 00:06:54,720 --> 00:06:58,440 Can you, I'll think of some ideas as I as as we progress 119 00:06:58,440 --> 00:07:01,280 about what might be really hard to measure, but it sounds as 120 00:07:01,280 --> 00:07:04,200 though the book then is a really nice mix of the quality of 121 00:07:04,200 --> 00:07:07,640 quantitative, someone who's very deep with people who are very 122 00:07:07,640 --> 00:07:10,600 deeply involved in the profession and then more of an 123 00:07:10,600 --> 00:07:13,400 outsider. And I think that always adds a 124 00:07:13,600 --> 00:07:16,600 really valuable perspective on the same topic. 125 00:07:16,920 --> 00:07:19,880 I'm going to dive in there and ask you a really hard question 126 00:07:20,880 --> 00:07:24,920 and probably very hard to answer in not very much time. 127 00:07:24,920 --> 00:07:28,160 But but what's new about what you're saying in project 128 00:07:28,160 --> 00:07:30,200 management and how it's measured? 129 00:07:30,200 --> 00:07:34,720 Because I've read in the forward something very ambitious that 130 00:07:34,720 --> 00:07:37,600 you wrote that was that you don't just want to challenge 131 00:07:37,600 --> 00:07:40,360 traditional project management practises, but replace them with 132 00:07:40,360 --> 00:07:43,760 something more of rigorous, transparent and effective. 133 00:07:44,480 --> 00:07:46,040 This sounds really interesting. So. 134 00:07:46,480 --> 00:07:49,480 So tell me more. I don't know who'd like to go 135 00:07:49,480 --> 00:07:52,360 press. Doug, would you like to share 136 00:07:52,360 --> 00:07:56,120 that with us? So I approach everything as a 137 00:07:56,120 --> 00:07:59,840 sceptic and in every field I've looked in, whether it was 138 00:07:59,840 --> 00:08:02,920 cybersecurity risk or operational risk management or 139 00:08:02,920 --> 00:08:06,080 portfolio prioritisation methods that businesses use or just 140 00:08:06,080 --> 00:08:10,880 decision making in general, I tend to find that there's a lot 141 00:08:10,880 --> 00:08:14,840 of, a lot of assumptions they make about what works and what 142 00:08:14,840 --> 00:08:16,880 does it. They don't really know, they 143 00:08:16,880 --> 00:08:19,040 just assumed it always has worked. 144 00:08:19,240 --> 00:08:22,880 They they never really measured it and they're often surprised 145 00:08:22,880 --> 00:08:25,440 when they do measure it. So it's the sort of thing where 146 00:08:25,440 --> 00:08:29,200 it's not really obvious though, like if you're, if you're 147 00:08:29,200 --> 00:08:32,240 golfing, it's going to be obvious whether or not you're 148 00:08:32,240 --> 00:08:34,600 good or bad at it, right? You get your score right at the 149 00:08:34,600 --> 00:08:38,320 end of things, right, other kinds of sports, etcetera. 150 00:08:38,600 --> 00:08:41,520 But in a lot of management decision making, you have this 151 00:08:41,520 --> 00:08:45,640 really inconsistent and highly delayed and ambiguous feedback 152 00:08:45,640 --> 00:08:49,200 cycle. So it's not really necessarily 153 00:08:49,200 --> 00:08:53,080 obvious that what you're using is really one of the better 154 00:08:53,080 --> 00:08:54,760 methods. There's probably many other 155 00:08:54,760 --> 00:08:56,440 methods. People tend to be very 156 00:08:56,440 --> 00:08:58,320 overconfident about their own judgement. 157 00:08:58,680 --> 00:09:01,440 We know this from a lot of other research that's been, you know, 158 00:09:01,800 --> 00:09:05,600 spearheaded by many others. We also know that they're highly 159 00:09:05,600 --> 00:09:08,840 inconsistent, that they're influenced by irrelevant random 160 00:09:08,840 --> 00:09:13,360 external factors, and they dismiss too many things as 161 00:09:13,360 --> 00:09:16,720 immeasurable when all the research shows that you're 162 00:09:16,720 --> 00:09:20,240 better off doing the math on a wide variety of problems. 163 00:09:20,240 --> 00:09:23,320 Even relatively simple statistical models seem to 164 00:09:23,320 --> 00:09:26,040 outperform human experts in a variety of different fields. 165 00:09:26,560 --> 00:09:30,320 You can't trust the humans. Well, you can't trust the humans 166 00:09:30,320 --> 00:09:32,960 for some things. It's really important you figure 167 00:09:32,960 --> 00:09:37,000 out how to use the humans, what to use them for, and what not to 168 00:09:37,000 --> 00:09:40,920 rely on them for. Quantitative methods in 169 00:09:40,920 --> 00:09:42,680 mathematics are really important. 170 00:09:42,680 --> 00:09:44,840 Human invention, slash discovery. 171 00:09:45,200 --> 00:09:48,280 All right. As important as language itself. 172 00:09:48,600 --> 00:09:51,960 OK. And it solves a lot of problems. 173 00:09:52,480 --> 00:09:54,600 It's a tool that we should use more often. 174 00:09:54,600 --> 00:09:59,280 We we revert too frequently to just rely on our own judgement. 175 00:09:59,640 --> 00:10:02,360 We're having too much trust in other human beings when in fact 176 00:10:02,360 --> 00:10:05,920 the data shows that simple statistical models are 177 00:10:05,920 --> 00:10:09,360 outperforming them. So does that pass the argument 178 00:10:09,360 --> 00:10:12,120 in the book? Oh yes, absolutely. 179 00:10:12,400 --> 00:10:15,840 We, we take a sceptical view of, of a lot of the methods that are 180 00:10:15,840 --> 00:10:18,280 used in project management and Andres can talk a little bit 181 00:10:18,280 --> 00:10:21,000 about that. But we did, in addition to the 182 00:10:21,000 --> 00:10:24,800 analysis of the 20,000 or so projects that they had in their 183 00:10:24,800 --> 00:10:28,440 database, we did our own original survey of little over 184 00:10:28,440 --> 00:10:32,760 200 project managers and projects and pretty extensive 185 00:10:32,760 --> 00:10:36,400 literature review, more than 100 different, you know, peer 186 00:10:36,400 --> 00:10:39,600 reviewed articles, standards, other sources and so forth. 187 00:10:40,000 --> 00:10:43,880 And we were looking for what's the evidence that some methods 188 00:10:43,880 --> 00:10:45,720 are measurably better than others. 189 00:10:46,880 --> 00:10:49,200 And that was one of the surprising things. 190 00:10:50,120 --> 00:10:51,680 I'll let Andreas talk about some of that. 191 00:10:51,680 --> 00:10:57,080 But really, what works doesn't is will be a surprise to many 192 00:10:57,120 --> 00:10:59,840 project managers I think. Well, what? 193 00:10:59,840 --> 00:11:02,240 What were these discoveries you made, Andreas? 194 00:11:03,320 --> 00:11:06,960 So I, I might just, I might just take a, a step back because one 195 00:11:06,960 --> 00:11:09,920 of the first things we did when we approached the book was, 196 00:11:10,360 --> 00:11:14,080 well, can can we say something about some of the, do some 197 00:11:14,080 --> 00:11:15,760 projects perform better than others? 198 00:11:15,760 --> 00:11:19,600 Do projects use specific types of methodologies or frameworks 199 00:11:20,040 --> 00:11:24,240 or different types of tools for estimating how do they perform 200 00:11:24,240 --> 00:11:27,080 compared to others? And so the the first thing that 201 00:11:27,080 --> 00:11:31,160 we actually did was to look at our large database of projects, 202 00:11:31,160 --> 00:11:34,960 which goes back to we have some historical projects from the 203 00:11:34,960 --> 00:11:38,200 1800s. And what we see there is kind of 204 00:11:38,200 --> 00:11:42,400 a very consistent trend in terms of projects not delivering to 205 00:11:42,400 --> 00:11:46,480 expectations in terms of cost and schedule and also benefits. 206 00:11:47,320 --> 00:11:51,120 And and that's despite the fact that since at least since the 207 00:11:51,120 --> 00:11:54,080 90s, we've gotten more and more kind of methodologies and 208 00:11:54,080 --> 00:11:56,840 frameworks and all these things that supposedly help us with 209 00:11:56,840 --> 00:12:00,840 projects. So we, if these frameworks and 210 00:12:00,840 --> 00:12:04,120 these tools we've developed works, we would expect to see, 211 00:12:04,120 --> 00:12:07,200 you know, budget overruns have gone down over time. 212 00:12:07,920 --> 00:12:11,040 We, we don't see as big delays. We are better at getting the 213 00:12:11,160 --> 00:12:14,320 benefits that we, we expect we will from projects. 214 00:12:14,840 --> 00:12:18,040 But when we look at kind of project performance over time, 215 00:12:18,040 --> 00:12:23,520 we don't see that a performance improvement since 1990s despite 216 00:12:23,520 --> 00:12:28,160 all this, all this work and improving projects in terms of 217 00:12:28,160 --> 00:12:30,240 methodologies and certifications and all this. 218 00:12:30,920 --> 00:12:32,760 So that was kind of the first clue. 219 00:12:34,480 --> 00:12:38,160 Afterwards, we, we launched this survey where we looked into, 220 00:12:38,160 --> 00:12:40,120 well, what, what do projects use? 221 00:12:40,120 --> 00:12:44,040 And can we see some kind of statistical differences between 222 00:12:44,040 --> 00:12:46,920 different types of approaches to delivering projects? 223 00:12:49,480 --> 00:12:54,080 And, and I think the, the evidence, what we found was was 224 00:12:54,080 --> 00:12:56,880 very contrasting because project management is a field where 225 00:12:56,880 --> 00:12:59,760 there's just a lot of claims, There's lots of methodologies 226 00:12:59,760 --> 00:13:03,760 that promise, you know, if you use this approach to deliver 227 00:13:03,760 --> 00:13:07,040 your project, you're going to get twice the benefits in half 228 00:13:07,040 --> 00:13:09,680 the time. But we didn't see that for any 229 00:13:09,840 --> 00:13:11,880 methods. We didn't see any methods really 230 00:13:11,880 --> 00:13:15,960 outperform others. We, we saw very few actually 231 00:13:15,960 --> 00:13:19,560 effects of applying any kind of methods at all to projects. 232 00:13:20,400 --> 00:13:23,800 I think that the one very consistent finding we had was 233 00:13:24,320 --> 00:13:27,440 that projects have spent more time in the front end 234 00:13:27,440 --> 00:13:30,720 development of projects. They they tend to perform better 235 00:13:31,360 --> 00:13:34,360 and and secondly projects that apply more quantitative and 236 00:13:34,360 --> 00:13:39,680 evidence based methods. They also tend to perform better 237 00:13:39,680 --> 00:13:42,640 in terms of adhering to their estimates compared to projects 238 00:13:42,640 --> 00:13:46,200 that use more qualitative and subjective estimating 239 00:13:46,200 --> 00:13:50,000 techniques. One of the things I thought of 240 00:13:50,240 --> 00:13:53,400 as I was going through the book was that I thought listeners 241 00:13:53,400 --> 00:13:56,000 might be interesting. Was was I, I? 242 00:13:57,120 --> 00:13:59,040 What can be measured in projects? 243 00:13:59,440 --> 00:14:04,040 Can everything be measured and what are the things you should 244 00:14:04,040 --> 00:14:06,080 be measuring? So the methodologies you're 245 00:14:06,080 --> 00:14:09,280 saying out there, many of them don't deliver any notable 246 00:14:09,280 --> 00:14:11,520 results. And is part of that because 247 00:14:11,520 --> 00:14:15,240 they're measuring the wrong things or what do you do with 248 00:14:15,240 --> 00:14:16,840 the measurements once you've got them? 249 00:14:17,040 --> 00:14:20,680 So big, a big couple of big questions there. 250 00:14:20,680 --> 00:14:25,280 So I guess if you want to take, can you measure everything in 251 00:14:25,280 --> 00:14:29,320 projects and then which are the ones you should be measuring? 252 00:14:29,320 --> 00:14:31,200 I think that would be really interesting. 253 00:14:32,320 --> 00:14:36,800 Well, the simple answer is yes. There's literally nothing, 254 00:14:37,400 --> 00:14:41,600 literally nothing that has any consequences at all for the real 255 00:14:41,600 --> 00:14:43,600 world. Every project that anybody has 256 00:14:43,600 --> 00:14:47,880 ever proposed had some expected consequences, right? 257 00:14:49,000 --> 00:14:53,440 And every consequence is necessarily observable. 258 00:14:53,560 --> 00:14:56,080 Otherwise, it wouldn't be something that everybody cared 259 00:14:56,080 --> 00:14:58,720 about, right? It would have to have observable 260 00:14:58,720 --> 00:15:00,600 consequences. No matter what you're thinking 261 00:15:00,640 --> 00:15:03,200 of, it's really hard to measure if you have a project that's 262 00:15:03,200 --> 00:15:06,400 supposed to improve collaboration or innovation or 263 00:15:06,400 --> 00:15:10,640 the environment or customer satisfaction, or some sort of 264 00:15:10,640 --> 00:15:13,920 social welfare or children's education, it doesn't really 265 00:15:13,920 --> 00:15:16,200 matter. In each of those cases, there 266 00:15:16,200 --> 00:15:18,080 are expected observable outcomes. 267 00:15:18,720 --> 00:15:20,800 Now, you might be highly uncertain about them. 268 00:15:21,000 --> 00:15:23,920 That's a different issue, but you can identify the 269 00:15:23,920 --> 00:15:29,000 observations first, then the rest is about observations and a 270 00:15:29,000 --> 00:15:31,400 little bit of math to reduce your uncertainty. 271 00:15:32,760 --> 00:15:36,800 Does that mean you need to know what you're observing before you 272 00:15:37,040 --> 00:15:38,960 start measuring? Do you know? 273 00:15:39,040 --> 00:15:41,240 Do you have to have an expectation of what there will 274 00:15:41,240 --> 00:15:44,680 be to observe before you can start thinking about measuring 275 00:15:44,680 --> 00:15:48,000 that thing? You don't have to, but you won't 276 00:15:48,000 --> 00:15:50,720 know the value that measurement until you know the reason for 277 00:15:50,720 --> 00:15:53,240 it. You can have purely exploratory 278 00:15:53,240 --> 00:15:56,560 discovery driven observations. A lot of science is this way. 279 00:15:56,840 --> 00:16:00,400 You you don't necessarily know what you're looking for, right? 280 00:16:01,520 --> 00:16:06,040 If I go to a, a, a, a new city and I'm walking around downtown, 281 00:16:06,040 --> 00:16:08,840 I don't know what I'm looking for necessarily, right? 282 00:16:08,840 --> 00:16:12,000 I might discover something. But if I'm going to spend a lot 283 00:16:12,000 --> 00:16:15,960 of time and effort measuring something in particular, I have 284 00:16:15,960 --> 00:16:21,040 to know what it's for, right? And all measurements are meant 285 00:16:21,040 --> 00:16:24,600 to reduce uncertainty, to make better bets. 286 00:16:24,960 --> 00:16:28,160 You're making better decisions. That's what all of this is 287 00:16:28,160 --> 00:16:31,080 about. Every project manager, what 288 00:16:31,080 --> 00:16:35,800 management means is all about making decisions before and 289 00:16:35,800 --> 00:16:38,760 during a project, sometimes even after a project. 290 00:16:39,320 --> 00:16:42,880 So you can think of projects, project decisions during the 291 00:16:42,880 --> 00:16:45,520 project as sort of intervention decisions. 292 00:16:46,240 --> 00:16:49,720 So what are they measuring if they're not going to do anything 293 00:16:49,720 --> 00:16:52,520 with the measurements during a project? 294 00:16:53,280 --> 00:16:55,400 That is, they can't think of anything they might do 295 00:16:55,400 --> 00:16:58,320 differently at all. If it was surprisingly high or 296 00:16:58,320 --> 00:17:03,080 surprisingly low, well then that has a information value of 0. 297 00:17:03,200 --> 00:17:05,560 This is a computable value, by the way, the value of 298 00:17:05,560 --> 00:17:08,680 information. So it's sort of like the cost of 299 00:17:08,680 --> 00:17:10,560 being wrong times the chance of being wrong. 300 00:17:11,160 --> 00:17:13,200 It can get a little bit more elaborate than that. 301 00:17:13,240 --> 00:17:16,480 But generally speaking, if there's, if you can't think of 302 00:17:16,560 --> 00:17:20,079 any consequence that would be different at all, or if there's 303 00:17:20,079 --> 00:17:22,760 no uncertainty about it already, well then there's no value to 304 00:17:22,760 --> 00:17:26,200 that that information. Do you think that project 305 00:17:26,200 --> 00:17:30,920 professionals. Oh, know that deep down that 306 00:17:30,920 --> 00:17:33,760 they do question why they're measuring something or is it 307 00:17:33,760 --> 00:17:36,400 just we're just measuring these things? 308 00:17:36,400 --> 00:17:39,640 So it's there's enough thought given to or consideration given 309 00:17:39,640 --> 00:17:43,800 to what's being measured and why, what the outcome you want 310 00:17:43,800 --> 00:17:46,040 from that measurement to be Andreas? 311 00:17:47,680 --> 00:17:50,640 So I think there's a lot of culture and project management. 312 00:17:50,640 --> 00:17:54,400 We typically measure the things we've measured before or that 313 00:17:54,600 --> 00:17:58,040 that other projects measure. And it's actually one of the 314 00:17:58,040 --> 00:18:01,080 things we go into that, you know, there's all these standard 315 00:18:01,080 --> 00:18:03,200 setting bodies, there's all these kind of influential 316 00:18:03,200 --> 00:18:05,840 organisations in, in the space of project management. 317 00:18:06,480 --> 00:18:08,000 And a lot of them just kind of repeat. 318 00:18:08,000 --> 00:18:10,640 This is how we've delivered projects before. 319 00:18:10,640 --> 00:18:13,360 This is how we've done project management and how we've thought 320 00:18:13,360 --> 00:18:16,440 about, you know, quantifying risk in some way. 321 00:18:16,800 --> 00:18:20,680 But they, they typically don't really distinguish between what 322 00:18:20,680 --> 00:18:25,200 are the methods that are proven to actually give you better 323 00:18:25,200 --> 00:18:30,840 results versus the ones that in, in the best case don't do 324 00:18:30,840 --> 00:18:33,680 anything, but at, at worst, they might add uncertainty. 325 00:18:34,160 --> 00:18:39,360 We use kind of a risk matrix as one of the examples of, so a way 326 00:18:39,360 --> 00:18:43,800 of we, we call it analysis placebo in the book, but it's, 327 00:18:43,800 --> 00:18:48,240 it's the idea that you, you feel you're kind of measuring a risk 328 00:18:48,320 --> 00:18:52,800 by using risk matrices to kind of quantify the impact versus 329 00:18:52,840 --> 00:18:56,320 the likelihood. But there's multiple problems 330 00:18:56,320 --> 00:19:00,680 with, with that type of more subjective scale approach where 331 00:19:01,600 --> 00:19:05,600 all the research from, from not just product management, but 332 00:19:05,600 --> 00:19:09,160 also different fields show that these types of quantifications, 333 00:19:09,160 --> 00:19:12,520 they, there's a lot of error with them and they don't 334 00:19:12,520 --> 00:19:14,720 actually contribute anything good. 335 00:19:15,120 --> 00:19:18,480 But in projects, we, we apply some of these tools and because 336 00:19:18,480 --> 00:19:21,120 we've been applying those tools, we feel that we've actually 337 00:19:21,440 --> 00:19:24,960 we've done some kind of assessment of a risk even though 338 00:19:25,360 --> 00:19:28,240 we haven't really, we call it analysis placebo that because 339 00:19:28,240 --> 00:19:32,440 we've spent time applying A structured framework, now we 340 00:19:32,440 --> 00:19:35,600 feel better about the situation even though we haven't reduced 341 00:19:35,600 --> 00:19:40,320 uncertainty at all. So, so I think that's a lot of 342 00:19:40,520 --> 00:19:44,720 that kind of culture. But another thing that's we 343 00:19:44,720 --> 00:19:47,760 found was pretty surprising in the book was that projects say 344 00:19:47,760 --> 00:19:50,800 spend a lot of time measuring things that have the lowest 345 00:19:50,800 --> 00:19:53,000 information values. If you want to stick to that 346 00:19:53,200 --> 00:19:56,640 framework of the value of information in terms of reducing 347 00:19:56,640 --> 00:20:00,280 uncertainty for our decision making projects, they spend a 348 00:20:00,280 --> 00:20:05,800 long time kind of estimating or measuring the CapEx cost, but 349 00:20:05,800 --> 00:20:09,480 they don't spend as much time measuring the OpEx cost of the 350 00:20:09,840 --> 00:20:13,280 kind of the maintenance of the the infrastructure, the project 351 00:20:13,280 --> 00:20:16,440 that they build afterwards. Where typically the 352 00:20:16,440 --> 00:20:21,360 uncertainties is typically in the latter and the Opecs the 353 00:20:21,360 --> 00:20:24,400 same with benefits. Often times big projects, 354 00:20:24,880 --> 00:20:28,720 especially big public projects have wider societal benefits and 355 00:20:28,720 --> 00:20:32,000 those are really uncertain. But projects they don't spend as 356 00:20:32,000 --> 00:20:35,400 much time trying to measure and reduce the uncertainty about the 357 00:20:35,400 --> 00:20:39,200 benefits side of the project as they do cost because they're the 358 00:20:39,200 --> 00:20:45,320 cost is the big typically it's, it's the big component that they 359 00:20:45,320 --> 00:20:47,760 could ask questions about. They have to deliver too. 360 00:20:48,640 --> 00:20:55,880 So should project professionals be focusing more on the benefits 361 00:20:56,200 --> 00:20:58,680 that will come from a project and measuring those? 362 00:20:59,520 --> 00:21:02,000 Yeah. Actually, when we compute the 363 00:21:02,000 --> 00:21:05,000 value of information, and we've done this many times on many 364 00:21:05,000 --> 00:21:07,280 different types of investments, including lots of different 365 00:21:07,280 --> 00:21:11,040 categories of projects, we tend to find the high information 366 00:21:11,040 --> 00:21:13,240 value variables. So I mentioned that's an 367 00:21:13,240 --> 00:21:16,960 algorithm you can actually run. These are more sensitive 368 00:21:16,960 --> 00:21:19,160 variables. They change the outcome of the 369 00:21:19,160 --> 00:21:22,920 decision more and they're more uncertain, right, than other 370 00:21:22,920 --> 00:21:25,000 variables. The high information value 371 00:21:25,000 --> 00:21:28,080 variables tend to be the things they haven't measured before, 372 00:21:28,080 --> 00:21:29,640 and that often includes benefits. 373 00:21:30,840 --> 00:21:33,520 They spend more time historically spending things 374 00:21:33,520 --> 00:21:37,680 that spending their time on, things that have lower 375 00:21:38,040 --> 00:21:41,440 information values, things that are statistically less likely to 376 00:21:41,440 --> 00:21:44,360 actually improve decisions. So we call this the measurement 377 00:21:44,360 --> 00:21:49,440 conversion in our books. I think this is a pervasive 378 00:21:49,440 --> 00:21:54,040 issue in every industry and projects and also outside of 379 00:21:54,040 --> 00:21:56,240 projects. I I don't know how it couldn't 380 00:21:56,240 --> 00:22:00,360 affect the GDP of countries. It would kind of have to. 381 00:22:00,840 --> 00:22:03,440 Why have we ended up this way? What? 382 00:22:03,440 --> 00:22:07,840 Why are we not measuring benefits better or measuring 383 00:22:07,840 --> 00:22:11,400 them at all? There's a general, there are a 384 00:22:11,400 --> 00:22:15,120 few different reasons I think and Andreas, you can comment on 385 00:22:15,120 --> 00:22:16,680 this. But First off, I think there's a 386 00:22:16,680 --> 00:22:20,360 habit of when you label something as intangible and 387 00:22:20,360 --> 00:22:22,760 immeasurable, it's never considered again. 388 00:22:24,040 --> 00:22:27,400 Once it's labelled intangible or immeasurable, no one ever goes 389 00:22:27,400 --> 00:22:30,040 back and ask, is it really immeasurable? 390 00:22:30,440 --> 00:22:34,280 Are there really no observable consequences at all that could 391 00:22:34,280 --> 00:22:35,880 reduce our uncertainty about this? 392 00:22:35,880 --> 00:22:38,480 Have we even figured out what it is that we're measuring right? 393 00:22:38,800 --> 00:22:41,400 Often things seem immeasurable because we have an ambiguous 394 00:22:41,400 --> 00:22:42,640 label. We're using terms like 395 00:22:42,640 --> 00:22:46,240 collaboration or innovation, but there's observable consequences 396 00:22:46,240 --> 00:22:48,400 that you can't identify out of each of those things. 397 00:22:48,960 --> 00:22:53,440 So there's that. Things get labelled too quickly 398 00:22:54,280 --> 00:22:57,320 or at all as immeasurable. That's one problem. 399 00:22:57,800 --> 00:23:02,520 They're all measurable. The other problem is that people 400 00:23:02,520 --> 00:23:05,280 are just as Andre has pointed out, it's just a matter of 401 00:23:05,280 --> 00:23:07,760 culture. They measure what they know how 402 00:23:07,760 --> 00:23:09,320 to measure. They don't really look at a 403 00:23:09,320 --> 00:23:13,280 problem and say, what should I measure given this problem and 404 00:23:13,280 --> 00:23:15,000 then learn how to measure those things. 405 00:23:15,000 --> 00:23:17,960 That's not the approach. The approach is I know how to 406 00:23:17,960 --> 00:23:19,640 measure these things. That's what I'm measuring. 407 00:23:21,440 --> 00:23:27,160 So what needs to change to make projects delivered more 408 00:23:27,160 --> 00:23:29,520 successfully or deliver better benefits? 409 00:23:29,520 --> 00:23:34,720 Is is part of that looking at benefits and realising that 410 00:23:34,720 --> 00:23:37,560 these can be measured in a useful way? 411 00:23:39,080 --> 00:23:44,160 Yes, that's part of it. The first big project management 412 00:23:44,160 --> 00:23:47,480 decision is approval and prioritisation. 413 00:23:47,520 --> 00:23:50,160 Should you even do the project in the first place? 414 00:23:51,440 --> 00:23:56,000 In those cases, even the decision of doing the project in 415 00:23:56,000 --> 00:23:58,800 the 1st place is. And that first big decision, 416 00:23:59,080 --> 00:24:02,040 modelling it quantitatively, is what's needed. 417 00:24:02,040 --> 00:24:06,240 Because when we start looking at that problem by itself, we find 418 00:24:06,240 --> 00:24:10,360 out that really most decision makers probably would not have 419 00:24:10,360 --> 00:24:14,080 accepted most of their projects. Seriously. 420 00:24:14,960 --> 00:24:17,880 Oh yes. So when you look at the data, 421 00:24:17,880 --> 00:24:20,200 there's a way to quantify risk aversion as well. 422 00:24:20,840 --> 00:24:23,160 And when and when you look at all the research on what 423 00:24:23,160 --> 00:24:26,440 executives say about how risk tolerant they are, how risk 424 00:24:26,440 --> 00:24:30,840 averse they are to, and then compare that to the actual 425 00:24:30,840 --> 00:24:34,840 uncertainties and risk of the big projects, they wouldn't have 426 00:24:34,840 --> 00:24:41,320 accepted almost any of them. So fewer projects, bigger wins. 427 00:24:41,520 --> 00:24:45,680 Is it one approach, right. And also spending more time on 428 00:24:45,680 --> 00:24:49,160 that early selection, you know, if somebody says, hey, this 429 00:24:49,160 --> 00:24:53,440 project is a good deal because we make more than our cost of 430 00:24:53,440 --> 00:24:56,480 money, that's not nearly good enough. 431 00:24:57,520 --> 00:25:00,880 If your return is just a little higher than your discount rates, 432 00:25:00,880 --> 00:25:02,800 you say, well, it's a good public project. 433 00:25:02,840 --> 00:25:06,680 You know, we're going to make the the cost of this big 434 00:25:06,680 --> 00:25:09,520 infrastructure project is justified or the cost of this 435 00:25:09,520 --> 00:25:13,480 big software project or Rd project is justified because we 436 00:25:13,480 --> 00:25:17,120 expect these returns. If those returns are modest, 437 00:25:17,360 --> 00:25:19,920 like modest improvements over your cost of money, your 438 00:25:19,920 --> 00:25:22,440 discount rates, that's not nearly good enough. 439 00:25:23,680 --> 00:25:27,160 If a project is expected, If we look at just all the projects in 440 00:25:27,160 --> 00:25:31,760 their database, if a project is expected to have a lot twice the 441 00:25:32,000 --> 00:25:35,440 benefits as costs, and this is present value discounted to 442 00:25:35,520 --> 00:25:39,200 current dollars, current units of monetary units. 443 00:25:39,640 --> 00:25:43,280 If you expect twice the benefits as you have cost, there's still 444 00:25:43,280 --> 00:25:45,640 so much uncertainty on benefits and costs. 445 00:25:45,640 --> 00:25:48,800 There's about a 14% chance of losing money still. 446 00:25:49,080 --> 00:25:51,720 In fact, there's even a chance of losing more than the money 447 00:25:51,720 --> 00:25:54,760 you put into it. And that's. 448 00:25:55,040 --> 00:25:59,280 Quite something, yeah. Yes, if you spend, if you spend, 449 00:25:59,280 --> 00:26:04,160 you know, £100 million on some big infrastructure project, is 450 00:26:04,160 --> 00:26:06,800 it possible you could lose more than 100 million? 451 00:26:08,000 --> 00:26:11,960 Yeah, actually there is. One of the cost that are not 452 00:26:11,960 --> 00:26:15,440 typically included in cost overruns is other process 453 00:26:15,440 --> 00:26:18,400 disruptions. So you've seen that's when 454 00:26:18,400 --> 00:26:20,760 there's a big infrastructure project, there's a bridge being 455 00:26:20,760 --> 00:26:23,680 built or a new road being repaid for something like this, and 456 00:26:23,680 --> 00:26:26,440 then there's a traffic congestion for a while because 457 00:26:26,440 --> 00:26:29,680 traffic has to be rerouted. Well, the project takes longer 458 00:26:29,680 --> 00:26:33,280 than expected. So that cost is the congestion 459 00:26:33,280 --> 00:26:36,640 cost is longer than expected, maybe even greater than 460 00:26:36,640 --> 00:26:40,200 expected. That's how you actually have 461 00:26:40,200 --> 00:26:44,640 cost that aren't in your budget. There's a other disruption cost 462 00:26:45,080 --> 00:26:49,000 in software cases. There's been cases of software 463 00:26:49,000 --> 00:26:51,880 where a lot of money has been spent to implement something. 464 00:26:52,240 --> 00:26:54,960 And then they figured out that this software that was supposed 465 00:26:54,960 --> 00:26:58,120 to improve productivity made things worse. 466 00:26:58,440 --> 00:27:01,640 So they had to unravel it. They had to undo it and take it 467 00:27:01,640 --> 00:27:06,120 out, go back to the old way. That's more that cost more than 468 00:27:06,120 --> 00:27:09,040 the cost of the project to do all of that. 469 00:27:09,520 --> 00:27:13,760 Now that doesn't happen every time, but it happens frequently 470 00:27:13,760 --> 00:27:18,440 enough that it's a risk we should consider and it's often 471 00:27:18,440 --> 00:27:21,400 not even acknowledges a possibility at the beginning of 472 00:27:21,400 --> 00:27:23,960 this approval process. So that's one big thing. 473 00:27:24,320 --> 00:27:29,360 Fewer projects, bigger wins, you know, a 20 or even 30 or 40% 474 00:27:29,360 --> 00:27:31,440 return. Not nearly good enough. 475 00:27:31,960 --> 00:27:36,640 You got to go bigger, spend more time in that upfront planning 476 00:27:36,880 --> 00:27:39,920 and look for better options. Keep exploring. 477 00:27:40,160 --> 00:27:42,320 There's one of the issues we talked about in the book is 478 00:27:42,320 --> 00:27:46,400 exploration versus exploitation. So this is kind of a classic 479 00:27:47,520 --> 00:27:49,880 measurement problem. It's a operations research 480 00:27:49,880 --> 00:27:52,400 problem. How long should I generate 481 00:27:52,400 --> 00:27:57,200 solutions before I pick the best one so far and and build on that 482 00:27:57,200 --> 00:27:59,240 one? Because there's a chance the 483 00:27:59,240 --> 00:28:01,920 next solution you generate is going to be better than any of 484 00:28:01,920 --> 00:28:04,160 the previous ones. On the other hand, you keep 485 00:28:04,200 --> 00:28:08,800 deferring the project, which has benefits itself presumably, 486 00:28:08,960 --> 00:28:13,000 right? So that's something that isn't 487 00:28:13,000 --> 00:28:15,760 typically looked at and it's something you could spend more 488 00:28:15,760 --> 00:28:19,000 time on in front end loading is you could spend more time 489 00:28:19,000 --> 00:28:24,040 generating solutions. It appears that people converge 490 00:28:24,040 --> 00:28:26,240 too quickly on a preferred solution. 491 00:28:27,080 --> 00:28:31,320 They should think of many more solutions and they should 492 00:28:31,320 --> 00:28:35,240 generate many more decision models and options to be used 493 00:28:35,240 --> 00:28:37,720 during the project. So it's not just the solutions, 494 00:28:37,720 --> 00:28:42,560 it's not the just the various designs of bridges, for example, 495 00:28:42,560 --> 00:28:45,560 that you're looking at. If you're also designing the 496 00:28:46,040 --> 00:28:50,000 measurement programme during the project, you're also designing 497 00:28:50,360 --> 00:28:53,960 the intervention options. These are decision models that 498 00:28:53,960 --> 00:28:57,000 run during the project. You're designing things. 499 00:28:57,080 --> 00:28:58,440 It's like what method should I use? 500 00:28:58,440 --> 00:29:01,240 Should this be an agile project or some modified version of 501 00:29:01,240 --> 00:29:03,400 that? That's all part of front end 502 00:29:03,400 --> 00:29:06,360 loading. And it's usually those things 503 00:29:06,360 --> 00:29:10,440 are they're not only not spend enough time on they're, they're 504 00:29:10,440 --> 00:29:13,880 just ignored as choices. They're just assumed that we're 505 00:29:13,880 --> 00:29:15,760 going to deal with this project with this way. 506 00:29:15,760 --> 00:29:18,520 We're going to follow this particular standard and we're 507 00:29:18,520 --> 00:29:21,120 going to measure these particular things and maybe 508 00:29:21,120 --> 00:29:23,320 after looking at a few solutions, we're going to pick 509 00:29:23,320 --> 00:29:26,080 that solution. That's a big problem. 510 00:29:27,240 --> 00:29:30,520 Just to add, when we talk about solutions, we mean solutions in 511 00:29:30,520 --> 00:29:34,240 a much broader kind of definition of that word. 512 00:29:34,320 --> 00:29:38,200 So when we talk about iterations of bridges, we really should 513 00:29:38,200 --> 00:29:41,440 spend more time thinking about alternatives to bridges as well 514 00:29:41,480 --> 00:29:44,440 that fulfil the same kind of requirement for the bridge. 515 00:29:44,880 --> 00:29:48,720 So thinking in fairies or tunnels or other types of 516 00:29:49,000 --> 00:29:52,960 solutions, not just iterations of bridges, because that's one 517 00:29:52,960 --> 00:29:56,600 thing that we saw too, is that we we fixate on a type of 518 00:29:56,600 --> 00:29:59,480 solution fairly quickly in projects and then we do 519 00:29:59,480 --> 00:30:01,400 iterations of that kind of design. 520 00:30:01,800 --> 00:30:04,440 How should that bridge look or how should that nuclear power 521 00:30:04,440 --> 00:30:07,000 plant be? Whereas we could consider much 522 00:30:07,640 --> 00:30:09,440 different types of solutions as well. 523 00:30:11,120 --> 00:30:13,520 I don't know where to start with everything I want to ask you, 524 00:30:13,520 --> 00:30:16,800 because that was quite macro. It's quite philosophical about 525 00:30:17,360 --> 00:30:20,680 what? What do we do and how do we do 526 00:30:20,680 --> 00:30:24,400 it in other alternatives, let alone the complication of 527 00:30:25,240 --> 00:30:29,320 political feeling or government. When you hit government mega 528 00:30:29,320 --> 00:30:34,160 projects for example, where the politics behind it can make it 529 00:30:34,880 --> 00:30:38,960 something happen without really being questioned why or is this 530 00:30:38,960 --> 00:30:44,160 the right thing to do. So for people listening, what is 531 00:30:44,160 --> 00:30:49,440 the how can they imagine someone just in a how I get a job 532 00:30:49,440 --> 00:30:55,480 working on a project? How can you begin to deliver, 533 00:30:55,480 --> 00:30:59,200 execute, think about your projects in a in a better way 534 00:30:59,200 --> 00:31:04,280 along the lines he suggests. And are you saying, let's look 535 00:31:04,280 --> 00:31:08,320 at the data, that let's make data more of a priority than 536 00:31:08,640 --> 00:31:12,400 these humans who might be too might be prone to optimism bias 537 00:31:12,400 --> 00:31:18,320 or and then with the rise of AI is, is this going to nudge 538 00:31:18,320 --> 00:31:22,640 projects more towards the way you're thinking that that if 539 00:31:22,640 --> 00:31:24,840 it's not measured, it should be measured, Everything could be 540 00:31:24,840 --> 00:31:27,760 measured and we should listen to the data and see the evidence of 541 00:31:27,760 --> 00:31:32,840 what actually works and be more objective about it. 542 00:31:33,800 --> 00:31:36,160 I think this is an interesting, interesting thing that comes up 543 00:31:36,160 --> 00:31:38,280 quite a lot. So many might say, but what 544 00:31:38,280 --> 00:31:40,440 about somebody who's working on a project right now? 545 00:31:42,000 --> 00:31:46,240 I think that's already too far down in the weeds because just 546 00:31:46,240 --> 00:31:49,720 organizationally that person doesn't really have the position 547 00:31:49,720 --> 00:31:52,000 or authority to make some of the changes we're talking about. 548 00:31:52,240 --> 00:31:55,520 We have to start much higher and much earlier in the 549 00:31:55,520 --> 00:31:57,560 organisation. We have to talk about 550 00:31:57,560 --> 00:32:00,480 stakeholders changing the way that they approach things, 551 00:32:00,480 --> 00:32:02,880 right? So it's stakeholders that are 552 00:32:02,880 --> 00:32:07,200 making ultimately making choices about where they should allocate 553 00:32:07,200 --> 00:32:10,680 funds in the first place. And something has to be 554 00:32:10,680 --> 00:32:15,400 communicated there that it's to their benefit to spend more time 555 00:32:15,400 --> 00:32:17,640 in this analysis before you make big bets. 556 00:32:18,120 --> 00:32:21,000 You know, if you're buying a house, you look at a few houses, 557 00:32:21,640 --> 00:32:23,880 right? If you're buying a car, you look 558 00:32:23,880 --> 00:32:25,440 at a few cars. You don't just pick the first 559 00:32:25,440 --> 00:32:30,640 one you see, right? And I think decision makers 560 00:32:30,720 --> 00:32:34,720 often are too quick to say we got to start coding or we got to 561 00:32:34,720 --> 00:32:40,040 start pouring concrete, right? I, that's a, that's a, this is a 562 00:32:40,040 --> 00:32:41,360 different way of thinking about it. 563 00:32:41,760 --> 00:32:45,280 We have to spend more time in that upfront analysis, analysing 564 00:32:45,280 --> 00:32:48,680 our choices. Again, fewer projects, bigger 565 00:32:48,680 --> 00:32:51,080 wins. That's the, the optimistic part 566 00:32:51,080 --> 00:32:52,960 here. I think it's the bigger wins 567 00:32:52,960 --> 00:32:56,280 part there. There's room for really 568 00:32:56,280 --> 00:32:59,680 impactful mega projects. That's where we should be 569 00:32:59,680 --> 00:33:05,640 thinking those really fundamentally life changing 570 00:33:05,640 --> 00:33:08,960 kinds of projects, socially changing kinds of projects. 571 00:33:09,520 --> 00:33:11,440 Like like what? Such as? 572 00:33:12,160 --> 00:33:15,200 Oh well, we have a couple of examples in the book, but at 573 00:33:15,200 --> 00:33:17,720 least in the states here, is it possible? 574 00:33:19,080 --> 00:33:21,520 Ideas have been proposed and it should be analysed. 575 00:33:21,520 --> 00:33:23,640 Is it possible to stop hurricanes? 576 00:33:25,920 --> 00:33:28,440 That's a hypothesis. And even if there's, if you work 577 00:33:28,440 --> 00:33:31,840 out the information value that if there were only a 5% chance 578 00:33:31,840 --> 00:33:35,080 of that being true, it'd be worth investigating just because 579 00:33:35,080 --> 00:33:40,160 it's so impactful. The cost of hurricanes is starts 580 00:33:40,160 --> 00:33:43,800 to compete with the cost of our entire aircraft carrier fleet, 581 00:33:44,320 --> 00:33:49,960 you know, in the US, right, or the entire Interstate system of 582 00:33:49,960 --> 00:33:54,680 expressways in the US. All of a sudden, when you add up 583 00:33:54,680 --> 00:33:57,320 a few big hurricanes like that, you say, well, yeah, actually, 584 00:33:57,520 --> 00:34:01,400 if it were even possible to even slightly reduce of the impact of 585 00:34:01,400 --> 00:34:04,560 hurricanes, what are the different ways we could do that? 586 00:34:04,560 --> 00:34:06,400 That's a big project. That could be a very big 587 00:34:06,400 --> 00:34:08,760 project. It might not even be a big 588 00:34:08,760 --> 00:34:11,199 project, Might be a bunch of little projects, by the way. 589 00:34:12,360 --> 00:34:14,679 But also, you think about AI itself. 590 00:34:15,239 --> 00:34:18,679 We talk about AI in the project and we tell people that you're 591 00:34:18,679 --> 00:34:21,600 single most important project, and this is the biggest project 592 00:34:21,600 --> 00:34:24,320 of all of them. By the way, the single biggest 593 00:34:24,320 --> 00:34:28,880 project the world is facing right now is how to do projects. 594 00:34:29,639 --> 00:34:32,120 That's the single biggest project. 595 00:34:32,120 --> 00:34:35,639 That's the most important mega project that any country or 596 00:34:35,639 --> 00:34:39,400 company is dealing with is how they should do projects. 597 00:34:39,400 --> 00:34:41,480 And this is a key message in the entire book. 598 00:34:42,400 --> 00:34:44,880 There's a lot of ways to run projects and a lot of ways to 599 00:34:44,880 --> 00:34:47,760 measure projects, and we're just not spending enough time 600 00:34:47,760 --> 00:34:49,560 figuring out which ways work better. 601 00:34:50,560 --> 00:34:53,480 Well, some are better than others, and we're still stuck on 602 00:34:53,480 --> 00:34:57,320 some, you know, habits that don't work as well as other 603 00:34:57,320 --> 00:34:59,680 things. Thanks Doug. 604 00:35:00,160 --> 00:35:02,920 Andreas, what? What are those better ways of 605 00:35:02,920 --> 00:35:07,160 running projects then what? What actually has does give some 606 00:35:07,560 --> 00:35:11,840 success. So we do have a lot of, I think 607 00:35:11,840 --> 00:35:14,840 one of the positive messages of the book is that there is a lot 608 00:35:14,840 --> 00:35:18,560 of potential, there are a lot of things you could be doing to 609 00:35:18,560 --> 00:35:20,840 improve the projects you're doing at the moment. 610 00:35:21,560 --> 00:35:25,720 So far we've talked a lot about, well, if we, we're able to 611 00:35:25,720 --> 00:35:28,600 reduce uncertainty about the bets that we're making, we would 612 00:35:28,680 --> 00:35:32,840 be making better bets, probably fewer bets, but the portfolio 613 00:35:32,840 --> 00:35:34,960 projects would be entirely different to what we're 614 00:35:34,960 --> 00:35:37,840 investing in now. And it would probably be better 615 00:35:37,840 --> 00:35:41,720 because a lot of what we, a lot of the projects we invest in now 616 00:35:41,720 --> 00:35:44,920 are the ones that look best on paper, but aren't necessarily 617 00:35:44,920 --> 00:35:48,680 the the best projects in in the book. 618 00:35:49,080 --> 00:35:50,600 Obviously the book is about measurement. 619 00:35:50,600 --> 00:35:53,720 So we talk about like what kind of measurement should you be 620 00:35:53,720 --> 00:35:55,640 making? Why are you doing? 621 00:35:55,640 --> 00:35:57,120 Why are you measuring in the 1st place? 622 00:35:57,120 --> 00:35:58,720 How should you be doing measurement? 623 00:36:00,240 --> 00:36:04,800 And one of the things when it comes to information value is, 624 00:36:04,800 --> 00:36:09,040 is often, if you don't know anything, almost anything will 625 00:36:09,040 --> 00:36:14,040 tell you something, right? So if the uncertainty is really 626 00:36:14,040 --> 00:36:16,720 high about something you haven't measured in your project, you 627 00:36:16,720 --> 00:36:20,480 don't need to collect a lot of data to reduce that uncertainty. 628 00:36:22,600 --> 00:36:25,240 And that's also part of the measurement inversion we talked 629 00:36:25,240 --> 00:36:27,400 about. We spend a lot of time fixating 630 00:36:27,400 --> 00:36:30,920 on the cost of the project. And sometimes to marginally 631 00:36:30,920 --> 00:36:33,480 reduce the uncertainty, we have to do a whole whole bunch of 632 00:36:33,480 --> 00:36:37,000 engineering in a project that might be really expensive and 633 00:36:37,000 --> 00:36:39,000 there's still a lot of uncertainty after we've done 634 00:36:39,000 --> 00:36:43,560 that detailed engineering. Whereas when it comes to some of 635 00:36:43,560 --> 00:36:46,360 the benefits coming out of projects, since we haven't 636 00:36:46,360 --> 00:36:50,120 observed it so far, we, we don't really need to make a lot of 637 00:36:50,120 --> 00:36:53,440 observation to be in a better place than we were before, to 638 00:36:53,440 --> 00:36:56,040 have reduced the uncertainty of what can we expect to get out of 639 00:36:56,040 --> 00:36:58,120 it and how can we manage that better. 640 00:36:58,800 --> 00:37:04,920 So we, we talk about, you know, you probably need less data than 641 00:37:04,920 --> 00:37:07,480 you think you need and you probably have more data than 642 00:37:07,520 --> 00:37:11,040 than you think you do. And that's generally going to be 643 00:37:11,040 --> 00:37:14,400 the case for organisations and for projects, especially on the 644 00:37:14,400 --> 00:37:16,280 things they aren't measuring so far. 645 00:37:17,680 --> 00:37:20,200 What what other things we talk about that you should be 646 00:37:20,200 --> 00:37:23,400 measuring as well as the the outside environment of projects. 647 00:37:23,680 --> 00:37:27,560 You mentioned politics as as an example of how it can add 648 00:37:27,560 --> 00:37:30,920 uncertainty to projects. And obviously that should be 649 00:37:30,920 --> 00:37:33,480 something if you're doing a big project that runs over multiple 650 00:37:33,480 --> 00:37:36,520 years, you should be measuring kind of the political context, 651 00:37:36,520 --> 00:37:39,560 political support of a project, because that's going to add 652 00:37:39,560 --> 00:37:42,640 uncertainty and it might mean that you're building in more 653 00:37:42,640 --> 00:37:45,920 decision options along the way of a project. 654 00:37:47,280 --> 00:37:49,560 But that's also one of the recommendations of the book 655 00:37:49,560 --> 00:37:53,120 that, you know, the projects, they aren't in their own bubble. 656 00:37:53,120 --> 00:37:56,840 They they do engage with and they are impacted by the 657 00:37:56,840 --> 00:37:59,320 external environment. So that's also some other things 658 00:37:59,320 --> 00:38:01,720 you should be measuring that you haven't been before. 659 00:38:03,120 --> 00:38:05,440 Thank you. Could you give an example of a 660 00:38:05,600 --> 00:38:09,920 project that perhaps does take on board the things you're 661 00:38:09,920 --> 00:38:12,760 telling me the recommend, the recommendations that you make in 662 00:38:12,760 --> 00:38:14,480 the book? Is there anything out there that 663 00:38:14,480 --> 00:38:16,560 people could find out more about? 664 00:38:17,520 --> 00:38:21,400 I've run into a couple of examples in in industries that 665 00:38:21,400 --> 00:38:23,360 do some of the things we're talking about. 666 00:38:23,800 --> 00:38:30,840 So for example, the option to cancel a project, that should be 667 00:38:30,840 --> 00:38:33,200 an option that's defined very well in advance. 668 00:38:33,200 --> 00:38:35,560 It's a part of a decision model that's run every day. 669 00:38:36,200 --> 00:38:39,600 Now nobody's doing quite that just yet, but in the 670 00:38:39,600 --> 00:38:42,920 Pharmaceutical industry and some related industries, they have a 671 00:38:42,920 --> 00:38:47,400 stage gate process. So only about 8% of all 672 00:38:47,400 --> 00:38:50,360 pharmaceutical products that are investigated ever make it to 673 00:38:50,360 --> 00:38:52,720 market. So how do they make any money? 674 00:38:53,240 --> 00:38:55,800 Well, when they quit, they quit early. 675 00:38:57,320 --> 00:38:59,760 And when you look at a lot of big projects that cancelled, and 676 00:38:59,760 --> 00:39:02,920 a lot of them do, they cancel with nothing to show for it 677 00:39:03,320 --> 00:39:05,920 after spending a lot of cost and having a lot of external 678 00:39:05,920 --> 00:39:09,440 consequences that are costly after. 679 00:39:09,440 --> 00:39:12,280 If you look at those kinds of projects, you often find out 680 00:39:12,280 --> 00:39:16,120 that they could have and should have cancelled it quite a long 681 00:39:16,120 --> 00:39:18,960 time ago. It they had all the data that it 682 00:39:18,960 --> 00:39:21,400 would would have justified cancellation quite a long time 683 00:39:21,400 --> 00:39:23,640 ago. It was just running on inertia. 684 00:39:24,720 --> 00:39:27,760 But also there's that feeling of, I think is it called sunk 685 00:39:27,760 --> 00:39:31,280 cost, sort of sunk cost of allergy where you've put all 686 00:39:31,280 --> 00:39:36,000 this money in already so it's too late to turn back. 687 00:39:36,960 --> 00:39:38,720 Right. It should always be on the 688 00:39:38,720 --> 00:39:41,880 forecast of the future. Here's what we here's our 689 00:39:41,880 --> 00:39:45,880 uncertainty about what there is remaining to be spent given what 690 00:39:45,880 --> 00:39:48,880 we've learned so far about the complexities of the project, 691 00:39:48,880 --> 00:39:52,200 because that might update our uncertainties about what's 692 00:39:52,440 --> 00:39:56,000 remaining to be spent. And we're comparing it to the 693 00:39:56,000 --> 00:39:57,440 outside world. It might not just be the 694 00:39:57,440 --> 00:39:59,960 project, it might be that something in the outside world 695 00:39:59,960 --> 00:40:02,680 has changed that changed the benefits of this thing. 696 00:40:04,360 --> 00:40:08,680 If I'm. 35% of the way into a project for some big 697 00:40:08,680 --> 00:40:13,720 infrastructure project and AI at the same time is improving 698 00:40:13,720 --> 00:40:17,840 because the and it will, because these are multi year projects, 699 00:40:17,960 --> 00:40:20,360 right? So there will be changes in not 700 00:40:20,360 --> 00:40:23,840 just AI, but materials and things like this and fabrication 701 00:40:23,840 --> 00:40:27,360 methods during the project. At what point during the project 702 00:40:27,360 --> 00:40:30,960 do we say there's enough that's changed about the outside world 703 00:40:31,040 --> 00:40:34,200 that I should now re engineer something about this project? 704 00:40:35,800 --> 00:40:39,240 That's a intervention decision. That's an intervention option 705 00:40:39,240 --> 00:40:40,960 that we should define in advance. 706 00:40:41,280 --> 00:40:46,120 So the pharmaceutical companies are one, but also aerospace does 707 00:40:46,120 --> 00:40:50,160 a little bit of something that we talked about where they do 708 00:40:50,160 --> 00:40:52,280 anticipate changes in technology. 709 00:40:52,280 --> 00:40:56,080 Because if they're developing a new crew re entry vehicle from 710 00:40:56,080 --> 00:41:00,160 the space station or something like this, they start designing 711 00:41:00,160 --> 00:41:04,560 it without necessarily assuming that the only materials they 712 00:41:04,560 --> 00:41:07,520 have are the current materials. Because it'll take a decade or 713 00:41:07,520 --> 00:41:11,400 more to design these things. And during that time there will 714 00:41:11,400 --> 00:41:14,080 be technological improvements. They're kind of counting on them 715 00:41:14,280 --> 00:41:17,360 in some ways. So we should take that into 716 00:41:17,360 --> 00:41:19,040 account. If we have a big infrastructure 717 00:41:19,040 --> 00:41:23,000 project, should we be saying asking questions like if AI 718 00:41:23,000 --> 00:41:27,680 could have changed this design if only I waited 2 1/2 years 719 00:41:27,680 --> 00:41:31,680 from now, what would I do differently? 720 00:41:31,680 --> 00:41:34,800 Is is there a point at which it might be worth waiting? 721 00:41:34,840 --> 00:41:37,720 There's something we introduced in the book called Technology 722 00:41:37,720 --> 00:41:41,760 Regret. Technology regret is includes 723 00:41:41,760 --> 00:41:44,960 things like you invest too soon in a technology that's rapidly 724 00:41:44,960 --> 00:41:49,600 changing, so you fixate on a bridge design or fixate on some 725 00:41:49,600 --> 00:41:54,600 software features at the same time while you're pouring 726 00:41:54,600 --> 00:41:57,880 concrete and writing code. AI is getting better. 727 00:41:59,160 --> 00:42:03,440 Not only AI, but materials and fabrication methods, 3D 728 00:42:03,440 --> 00:42:05,080 printing, hole bridges and so forth. 729 00:42:05,400 --> 00:42:07,000 That's getting better during this. 730 00:42:07,000 --> 00:42:10,240 Now AI is actually, I think improving faster than the other 731 00:42:10,240 --> 00:42:14,240 two items I mentioned. So we should at least be looking 732 00:42:14,240 --> 00:42:16,560 at there. There's not too many big 733 00:42:16,560 --> 00:42:19,560 projects that are going multiple years that shouldn't also be 734 00:42:19,560 --> 00:42:23,880 tracking what's going on with AI and asking questions about. 735 00:42:24,160 --> 00:42:27,400 Is there a point in time I should consider the possibility 736 00:42:27,400 --> 00:42:31,760 of right now and design an intervention option where I'm so 737 00:42:31,760 --> 00:42:34,920 far into the project and something the outside world has 738 00:42:34,920 --> 00:42:36,920 changed? And given how far I am into the 739 00:42:36,920 --> 00:42:40,160 project, and given the benefits of switching gears at that point 740 00:42:40,160 --> 00:42:43,720 in time, does it make sense to change something during the 741 00:42:43,720 --> 00:42:46,520 project? And you're saying that the 742 00:42:46,600 --> 00:42:49,920 answer could be yes, it could be, it could be beneficial to. 743 00:42:50,240 --> 00:42:56,840 So, so you're, you're planning in uncertainty, you're kind of 744 00:42:56,840 --> 00:43:00,600 planning that. That's quite probably quite a 745 00:43:00,600 --> 00:43:05,440 mindset shift for lots of people working on projects like this. 746 00:43:06,520 --> 00:43:10,480 Yeah, I think people are used to the concept of options, though 747 00:43:10,480 --> 00:43:12,880 some are. They're used to the concept of 748 00:43:12,960 --> 00:43:15,440 real options too. You know what's the what's the 749 00:43:15,440 --> 00:43:18,160 value of keeping an option open, right? 750 00:43:19,880 --> 00:43:23,240 Actually, even things like the option to cancel a project. 751 00:43:23,280 --> 00:43:26,320 This surprises people, but it de risk a project. 752 00:43:27,040 --> 00:43:30,760 Risk of a projects go down when you build in intervention 753 00:43:30,760 --> 00:43:35,360 options because in the case that you do need to cancel a project, 754 00:43:35,360 --> 00:43:37,320 you'll at least be able to cancel it sooner. 755 00:43:38,000 --> 00:43:41,920 So your expected losses would be lower if you do cancel a 756 00:43:41,920 --> 00:43:44,280 project. Now I've had lots of clients 757 00:43:44,280 --> 00:43:47,000 that would say, yeah, I've seen projects get cancelled, but 758 00:43:47,160 --> 00:43:51,240 prior to us developing decision models for them, they never saw 759 00:43:51,240 --> 00:43:55,160 a chance of cancellation on the business case for a project. 760 00:43:56,800 --> 00:44:01,280 How often do you see that right, if you're if you're presenting 761 00:44:01,280 --> 00:44:04,840 to a government agency or someone else in the corporation. 762 00:44:04,840 --> 00:44:06,560 I've got this idea for a project. 763 00:44:07,440 --> 00:44:10,000 Should I include the fact that there's based on historical 764 00:44:10,000 --> 00:44:12,920 data, there's about a 15% chance of cancellation with nothing to 765 00:44:12,920 --> 00:44:16,200 show for it. Some we've heard this, Andres 766 00:44:16,200 --> 00:44:18,840 and I were just talking to a group of the last time I was in 767 00:44:18,840 --> 00:44:21,040 the lending group of, you know, project managers and stuff like 768 00:44:21,040 --> 00:44:22,120 this. They'd say, well, you know, 769 00:44:23,280 --> 00:44:25,520 things wouldn't get approved if that were the case. 770 00:44:25,520 --> 00:44:27,640 If you're I said, well, if that's what you're looking at, 771 00:44:27,640 --> 00:44:29,240 then just leave out half the cost too. 772 00:44:30,040 --> 00:44:34,400 Why not? So the no, it's, it's about 773 00:44:34,400 --> 00:44:38,000 being transparent about these issues. 774 00:44:38,440 --> 00:44:42,080 Also, I think there's a fundamental conflict of interest 775 00:44:42,080 --> 00:44:46,600 here if the person opposing the project is the only one building 776 00:44:46,600 --> 00:44:50,800 the decision model for it, right, and the only one 777 00:44:50,800 --> 00:44:52,560 presenting it to decision makers. 778 00:44:53,280 --> 00:44:57,680 I do think there's room for some independent body of auditors. 779 00:44:57,680 --> 00:45:00,480 Actually, Oxford Global Projects does some of this right now 780 00:45:00,480 --> 00:45:03,840 because they audit people's projects, project plans and 781 00:45:03,840 --> 00:45:05,920 costs and schedules. That's part of the services that 782 00:45:05,920 --> 00:45:09,000 they do. And one of the values of having 783 00:45:09,000 --> 00:45:12,840 an outside consultant come come in, in general is that, hey, we 784 00:45:12,840 --> 00:45:14,600 don't have any skin in the game here. 785 00:45:14,720 --> 00:45:18,400 We're just going to do our analysis and give you an honest 786 00:45:19,080 --> 00:45:21,760 representation of the outcomes. Because usually we're working 787 00:45:21,760 --> 00:45:25,720 for the check writers, the people who were allocated funds, 788 00:45:25,720 --> 00:45:28,480 not necessarily the project manager or the person who wanted 789 00:45:28,480 --> 00:45:31,000 to promote the project. We're working for the people who 790 00:45:31,000 --> 00:45:34,520 have have to commit to the funds in the 1st place. 791 00:45:34,800 --> 00:45:39,600 So they want an honest answer and I think that's part of it. 792 00:45:39,600 --> 00:45:42,440 We have to think about some built in conflicts of interest 793 00:45:42,440 --> 00:45:46,320 in the way that we approach even proposing projects to begin 794 00:45:46,320 --> 00:45:49,920 with, but and then everything else that we just talked about. 795 00:45:51,040 --> 00:45:53,560 South more rigour what? Andreas, what would you like to 796 00:45:53,600 --> 00:45:56,720 add to that? I love this conversation has 797 00:45:56,720 --> 00:45:59,960 been about you know what, What type of projects should we be 798 00:45:59,960 --> 00:46:02,280 making? How can we plan projects better? 799 00:46:03,240 --> 00:46:06,240 But we do spend quite a long time in the book talking about 800 00:46:06,240 --> 00:46:09,520 different types of measurements for projects that are already in 801 00:46:09,520 --> 00:46:11,280 flights. What should you be start 802 00:46:11,280 --> 00:46:15,800 starting to measure and how can you make, how can you make kind 803 00:46:15,800 --> 00:46:20,320 of metrics more, more decision driven? 804 00:46:20,800 --> 00:46:24,240 So we, we talked a little about, we, we have specific chapters 805 00:46:24,240 --> 00:46:27,320 about, you know, cost and schedule and benefits progress, 806 00:46:27,320 --> 00:46:31,640 typical things that you need to measure on, on projects and that 807 00:46:31,640 --> 00:46:34,240 it's really useful to measure. But one of the things we 808 00:46:34,240 --> 00:46:38,400 observed was that a lot of the metrics that are currently being 809 00:46:38,440 --> 00:46:42,560 used and on all sorts of dashboards, they're very kind of 810 00:46:44,160 --> 00:46:46,720 exploratory, as Doug would put it. 811 00:46:46,840 --> 00:46:50,480 So they're there because decision makers are project 812 00:46:50,480 --> 00:46:53,640 managers, are risk managers. They feel that once they know 813 00:46:53,640 --> 00:46:56,480 that once they look at the dashboard full of metrics and 814 00:46:56,480 --> 00:47:01,000 they see like the right combination of different figures 815 00:47:01,000 --> 00:47:03,880 and charts and, and metrics, they'll have some kind of 816 00:47:04,040 --> 00:47:06,200 they'll know what to do in that kind of instance. 817 00:47:06,480 --> 00:47:08,920 They'll know how to react on the metrics on the dashboard. 818 00:47:09,680 --> 00:47:12,480 But a lot of the time what we discovered was that a lot of the 819 00:47:12,480 --> 00:47:16,120 metrics aren't geared for decision making really, and not 820 00:47:16,120 --> 00:47:19,120 about, you know, what intervention decision or how 821 00:47:19,120 --> 00:47:22,680 should we react if we see a specific metric. 822 00:47:23,000 --> 00:47:26,440 So the book spends a lot of time thinking about, you know, 823 00:47:27,400 --> 00:47:32,760 progress metrics like cost performance indicators or 824 00:47:32,760 --> 00:47:35,840 scheduled performance indicators and so forth, and thinking 825 00:47:35,840 --> 00:47:38,760 about, well, how could we make it reframe that into something 826 00:47:38,760 --> 00:47:43,160 that is easier to understand. And a lot of that is reframing 827 00:47:43,160 --> 00:47:46,680 it into, well, how much productivity increase would we 828 00:47:46,680 --> 00:47:50,640 need to have to actually catch up on our delay, for example? 829 00:47:51,960 --> 00:47:56,680 So we do give a lot of advice in terms of how should you be 830 00:47:56,680 --> 00:47:58,800 measuring, how should you be presenting metrics? 831 00:47:59,320 --> 00:48:02,400 And I think the other thing in the book that we do is we be 832 00:48:02,400 --> 00:48:07,160 sure that because this is a book about measurement, there is some 833 00:48:07,240 --> 00:48:10,440 maths in it. But we do show that, you know, 834 00:48:10,680 --> 00:48:12,600 all this maths isn't complicated. 835 00:48:12,680 --> 00:48:16,840 So alongside the book, we have a website where you uploaded a 836 00:48:16,840 --> 00:48:19,760 whole bunch of spreadsheets to show that all these things. 837 00:48:19,760 --> 00:48:23,800 We talk about the technology regret example, we have an 838 00:48:23,800 --> 00:48:26,360 example of using machine learning for predicting 839 00:48:26,680 --> 00:48:28,840 outcomes. We have different types of 840 00:48:28,840 --> 00:48:33,080 simulation exercises and we run it all in Excel, which you can 841 00:48:33,080 --> 00:48:35,560 download and apply it to your projects just to show that. 842 00:48:36,120 --> 00:48:39,760 It sounds pretty complex if you're used to using more 843 00:48:39,760 --> 00:48:43,600 subjective scores and scales and risk matrices, but it actually 844 00:48:43,600 --> 00:48:48,920 isn't that difficult and there's a lot of already built solutions 845 00:48:48,920 --> 00:48:50,520 out there that you can just apply. 846 00:48:52,000 --> 00:48:56,120 If there's one thing you'd like readers to take away from your 847 00:48:56,120 --> 00:49:00,080 book, what what might that be, Andreas? 848 00:49:02,800 --> 00:49:05,680 So if, if I can positively frame it, because you've talked about 849 00:49:05,800 --> 00:49:09,760 issues a lot on this on this podcast so far, is that there is 850 00:49:09,760 --> 00:49:13,440 actually a lot of evidence, even though there's in the industry, 851 00:49:13,440 --> 00:49:17,520 there's a lot of claims around or if you just apply our methods 852 00:49:17,520 --> 00:49:21,160 or our tools or our software, you're going to better projects. 853 00:49:21,680 --> 00:49:24,120 That's typically what we see. If we go out to conferences and 854 00:49:24,120 --> 00:49:26,960 we look at like vendor stands, it's all the same software 855 00:49:26,960 --> 00:49:30,520 companies with some kind of reframing of if you use our 856 00:49:30,520 --> 00:49:33,760 products, you're going to deliver to cost into schedule, 857 00:49:33,760 --> 00:49:35,640 right? That'll be the typical 858 00:49:35,640 --> 00:49:38,320 experience. But there's so many claims out 859 00:49:38,320 --> 00:49:41,400 there can be quite difficult to find out what works and what 860 00:49:41,400 --> 00:49:44,200 doesn't. And what we've tried to do is 861 00:49:44,200 --> 00:49:46,480 actually just to look at the data. 862 00:49:46,640 --> 00:49:48,800 The research has already been done. 863 00:49:49,480 --> 00:49:52,680 We've done some of our own research and we've identified 864 00:49:52,680 --> 00:49:56,040 that, you know, some things are better than others in terms of 865 00:49:56,440 --> 00:50:00,320 measurement or estimating or delivering projects. 866 00:50:01,120 --> 00:50:05,960 So that there's there's an opportunity to apply the things 867 00:50:05,960 --> 00:50:08,400 that work and kind of toss the things that don't work. 868 00:50:09,760 --> 00:50:12,440 Fantastic. That's very well put. 869 00:50:12,440 --> 00:50:16,600 Thank you, Doug. Any any one message you'd like 870 00:50:16,760 --> 00:50:22,400 to get across to to listeners? Yeah, we, we refer to the the 871 00:50:22,400 --> 00:50:26,680 most important project, the one that is about making projects 872 00:50:26,680 --> 00:50:29,120 better. We call that the meta project in 873 00:50:29,120 --> 00:50:32,160 the book. And everybody ought to start 874 00:50:32,160 --> 00:50:33,600 thinking about their meta project. 875 00:50:33,960 --> 00:50:39,080 You know, no matter how big your project portfolio is, no, no 876 00:50:39,080 --> 00:50:43,640 matter how big the individual projects are, some time is worth 877 00:50:43,640 --> 00:50:48,120 being spent on the meta project. Your most important decision is 878 00:50:48,120 --> 00:50:52,000 how to make decisions. By the way, that's in life and 879 00:50:52,000 --> 00:50:53,640 business and government and everything. 880 00:50:53,920 --> 00:50:56,240 Your most important decision is how to make decisions. 881 00:50:56,800 --> 00:51:01,200 So we should apply that here. And so the meta project is, is 882 00:51:01,200 --> 00:51:04,320 the project about making all the other projects better. 883 00:51:04,680 --> 00:51:08,320 What's a great way to spend 1% of a whole project portfolio 884 00:51:09,640 --> 00:51:11,520 trying to optimise the other 99%? 885 00:51:11,800 --> 00:51:14,120 That's the best way to spend 1%, right? 886 00:51:14,200 --> 00:51:18,600 And it might be about that much for a lot of project portfolios. 887 00:51:18,640 --> 00:51:21,240 If you look at some of the infrastructure portfolios, you 888 00:51:21,240 --> 00:51:25,600 know, even spending 1% of those portfolios is a significant meta 889 00:51:25,600 --> 00:51:29,480 project where a lot of analysis could be done. 890 00:51:29,840 --> 00:51:34,920 And so we recommend having a project called the Meta Project 891 00:51:34,920 --> 00:51:38,640 and Everybody's portfolio. Thank you, Doug. 892 00:51:38,800 --> 00:51:42,440 I feel as though we've just scrape the surface of what we 893 00:51:42,440 --> 00:51:45,280 could talk about in your book. I feel like I might have to do a 894 00:51:45,280 --> 00:51:49,400 follow up podcast sometime soon. But thank you very much for your 895 00:51:49,400 --> 00:51:50,800 time. Thank you for your thoughts. 896 00:51:51,400 --> 00:51:55,520 I wish you best of luck with the book, but it's been really 897 00:51:55,520 --> 00:51:59,080 fascinating talking to you and I'm sorry we couldn't cover much 898 00:51:59,080 --> 00:52:01,640 more of what you've what you've put in the book, but thanks very 899 00:52:01,640 --> 00:52:04,640 much for your time today. You best text. 900 00:52:05,040 --> 00:52:09,080 Thank you. Thanks again to Doug and Andreas 901 00:52:09,080 --> 00:52:12,040 for joining us and to you for listening to the APM podcast. 902 00:52:12,400 --> 00:52:16,000 I hope you've been left with much food for thought, so don't 903 00:52:16,000 --> 00:52:18,920 forget to look out for more episodes, alterate and reviews. 904 00:52:18,920 --> 00:52:22,120 Wherever you get your podcasts. We'd welcome you to get in touch 905 00:52:22,120 --> 00:52:25,400 with your comments, feedback and suggestions by emailing us at 906 00:52:25,440 --> 00:52:30,840 apmpodcast@thinkpublishing.co.uk. Spotify and YouTube users, 907 00:52:30,840 --> 00:52:32,920 please do also leave us your comments. 908 00:52:33,720 --> 00:52:37,080 This podcast has been brought to you by APM, the chartered body 909 00:52:37,080 --> 00:52:40,400 for the project profession. For more information on APM, 910 00:52:40,400 --> 00:52:42,320 visit apm.org.uk.