Welcome to the APM podcast. APM is the childhood body for
the project profession. My name is Emma David and I'm
the editor of Project APM's quarterly journal and your host
today. In this podcast, we're talking
about how to measure anything in project management.
Yes, you did hear me write anything.
It's the title of a new book that is co-authored by Douglas
Hubbard, Alexander Budzia and Andreas lead.
I have the pleasure of being joined by Doug and Andreas Now.
Doug is founder and president of Hubbard Decision Research, is a
management consultant and the author of a series of books
including the brilliantly titled How to Measure Anything.
Andreas is head of Data Science at Oxford Global Projects and is
a PhD Fellow at AAHSA University in Denmark.
Be prepared for mind expanding conversation about project
management and how any aspect of it can be measured, including
those things you thought might be immeasurable and why we
should all be engaged in the most important project of all,
the meta project of understanding how to manage
projects better. I'd like to begin by welcoming
Andreas and Douglas to the APM Podcast.
So thanks very much for finding the time to speak to us today.
Thank you for having us. You bet.
Let's begin by you telling us a little bit about your area of
work and interest within project management.
So Andreas, tell us a bit about what you're what you're working
on right now. Yes, Sir, of course.
So I'm, I'm Andreas lead and the head of Data Science at Oxford
Global Projects. And Oxford Global projects is a
small consultancy which is really about analysing vast
amount of vast amounts of project data and and using it to
to have a better decision making around projects.
Another of what we do is about, you know, debiasing estimates
and kind of assuring estimates, realistic, basing estimates on
historical data rather than bottom up type approaches.
But it's also kind of bigger than that.
So as the head of data science, I did some of the more
quantitative engagements, things like using AI for machine
learning for forecasting project outcomes or doing kind of
econometric cost driver analysis.
And the big goal with all of this is, is to better projects
because after having looked at all the project data we have
access to, which is now about 20 data from 20,000 completed
projects, the the big conclusion is that rojects aren't erforming
that well. O we'd like to understand, you
know, what leads to success and how can we imrove roject
erformance? Thank you.
So that's not necessarily a controversial finding, is it,
unfortunately. But good to know.
We're here today to talk about how we can make perhaps make
projects more successful through the way they're measured or what
what is in fact measured. So Douglas, tell us a bit about
your area of interest and and actually tell us how the book
came about as well, because that's always good to know.
Yeah, sure. So I'm Doug Hubbard.
I'm the president and founder of Hubbard Decision Research, and
for the past 36 or 37 years or so I've been in some form of
quantitative management consulting.
So all of my work, including all my books, this was my fifth book
actually, They've all been on some topic related to decisions
under uncertainty and difficult measurements.
So that includes risk analysis, making big bets, and lots of
different areas of forecasting problems.
And of course, that applies to projects in general.
My previous four books were not about projects specifically, but
certainly a lot of my work was about risk return analysis on
major new investments, which were usually projects in regards
to how we met. Actually, I'll let Andres
describe this because initially 2 of you called me.
Yeah, so. So I was writing an article
together with a colleague about cost, benefit, cost benefit
analysis. Now kind of alternatives to, to
typical inclusions in cost benefit analysis.
I, I also teach cost benefit analysis at the Department of
economics here in all who's where I'm based.
But we found that Doug's work was really interesting.
So we, we asked him for an interview and kind of some ideas
around what should we include in this article.
And I guess it was, it was a good match.
Doug was looking for his next book project and project
management was one of the topics he was considering.
And so because we had access to a lot of data and a lot of
analysis that kind of fit in really well with the book.
I think that's fair, isn't it, Doug?
Oh yeah, absolutely. In fact, I I chose this book
title based on analytics. I decided to go after How to
Measure Anything in Project Management because it had
previously tied as one of the top 2 with How to Measure
Anything in Cybersecurity Risk, which I've already wrote now.
So I wrote that one first. And what tipped the needle in
the in favour of the first book of the the 4th, my 4th book.
But the previous How to Measure Anything in Cybersecurity Risk
book was that I had a a well qualified, enthusiastic
co-author that said, Hey, I'll write this with you.
And I said, OK, well, then let's get started on that.
And so I was really looking for the right people.
I actually talked to a bunch of other possible candidates for
co-authors for how to measure anything in project management.
And my strategy is always to find somebody who's a
specialist, an expert in renowned in that field because
I'm always the outsider, I'm always the outside quant
analyst, applying it to different kinds of problems,
etcetera. So what I wanted to do a spin
off of my first book, my first book was just how to measure
anything. So that and I meant it
literally. So how to measure anything.
And I decided to do a series of spin off books like that.
And so I waited for the right co-authors and when I learned
about Oxford Global projects, I thought, well, that's ideal.
They've got all this data. They, they already have these
guys that are, that have been studying projects and all the
problems with projects for many years now.
So it seemed like the ideal co-authors.
That's really interesting. So Doug, can you measure
anything? Sorry, I've got to ask you.
Yeah, absolutely. Including the hardest most
possible things you've ever heard of.
Can you, I'll think of some ideas as I as as we progress
about what might be really hard to measure, but it sounds as
though the book then is a really nice mix of the quality of
quantitative, someone who's very deep with people who are very
deeply involved in the profession and then more of an
outsider. And I think that always adds a
really valuable perspective on the same topic.
I'm going to dive in there and ask you a really hard question
and probably very hard to answer in not very much time.
But but what's new about what you're saying in project
management and how it's measured?
Because I've read in the forward something very ambitious that
you wrote that was that you don't just want to challenge
traditional project management practises, but replace them with
something more of rigorous, transparent and effective.
This sounds really interesting. So.
So tell me more. I don't know who'd like to go
press. Doug, would you like to share
that with us? So I approach everything as a
sceptic and in every field I've looked in, whether it was
cybersecurity risk or operational risk management or
portfolio prioritisation methods that businesses use or just
decision making in general, I tend to find that there's a lot
of, a lot of assumptions they make about what works and what
does it. They don't really know, they
just assumed it always has worked.
They they never really measured it and they're often surprised
when they do measure it. So it's the sort of thing where
it's not really obvious though, like if you're, if you're
golfing, it's going to be obvious whether or not you're
good or bad at it, right? You get your score right at the
end of things, right, other kinds of sports, etcetera.
But in a lot of management decision making, you have this
really inconsistent and highly delayed and ambiguous feedback
cycle. So it's not really necessarily
obvious that what you're using is really one of the better
methods. There's probably many other
methods. People tend to be very
overconfident about their own judgement.
We know this from a lot of other research that's been, you know,
spearheaded by many others. We also know that they're highly
inconsistent, that they're influenced by irrelevant random
external factors, and they dismiss too many things as
immeasurable when all the research shows that you're
better off doing the math on a wide variety of problems.
Even relatively simple statistical models seem to
outperform human experts in a variety of different fields.
You can't trust the humans. Well, you can't trust the humans
for some things. It's really important you figure
out how to use the humans, what to use them for, and what not to
rely on them for. Quantitative methods in
mathematics are really important.
Human invention, slash discovery.
All right. As important as language itself.
OK. And it solves a lot of problems.
It's a tool that we should use more often.
We we revert too frequently to just rely on our own judgement.
We're having too much trust in other human beings when in fact
the data shows that simple statistical models are
outperforming them. So does that pass the argument
in the book? Oh yes, absolutely.
We, we take a sceptical view of, of a lot of the methods that are
used in project management and Andres can talk a little bit
about that. But we did, in addition to the
analysis of the 20,000 or so projects that they had in their
database, we did our own original survey of little over
200 project managers and projects and pretty extensive
literature review, more than 100 different, you know, peer
reviewed articles, standards, other sources and so forth.
And we were looking for what's the evidence that some methods
are measurably better than others.
And that was one of the surprising things.
I'll let Andreas talk about some of that.
But really, what works doesn't is will be a surprise to many
project managers I think. Well, what?
What were these discoveries you made, Andreas?
So I, I might just, I might just take a, a step back because one
of the first things we did when we approached the book was,
well, can can we say something about some of the, do some
projects perform better than others?
Do projects use specific types of methodologies or frameworks
or different types of tools for estimating how do they perform
compared to others? And so the the first thing that
we actually did was to look at our large database of projects,
which goes back to we have some historical projects from the
1800s. And what we see there is kind of
a very consistent trend in terms of projects not delivering to
expectations in terms of cost and schedule and also benefits.
And and that's despite the fact that since at least since the
90s, we've gotten more and more kind of methodologies and
frameworks and all these things that supposedly help us with
projects. So we, if these frameworks and
these tools we've developed works, we would expect to see,
you know, budget overruns have gone down over time.
We, we don't see as big delays. We are better at getting the
benefits that we, we expect we will from projects.
But when we look at kind of project performance over time,
we don't see that a performance improvement since 1990s despite
all this, all this work and improving projects in terms of
methodologies and certifications and all this.
So that was kind of the first clue.
Afterwards, we, we launched this survey where we looked into,
well, what, what do projects use?
And can we see some kind of statistical differences between
different types of approaches to delivering projects?
And, and I think the, the evidence, what we found was was
very contrasting because project management is a field where
there's just a lot of claims, There's lots of methodologies
that promise, you know, if you use this approach to deliver
your project, you're going to get twice the benefits in half
the time. But we didn't see that for any
methods. We didn't see any methods really
outperform others. We, we saw very few actually
effects of applying any kind of methods at all to projects.
I think that the one very consistent finding we had was
that projects have spent more time in the front end
development of projects. They they tend to perform better
and and secondly projects that apply more quantitative and
evidence based methods. They also tend to perform better
in terms of adhering to their estimates compared to projects
that use more qualitative and subjective estimating
techniques. One of the things I thought of
as I was going through the book was that I thought listeners
might be interesting. Was was I, I?
What can be measured in projects?
Can everything be measured and what are the things you should
be measuring? So the methodologies you're
saying out there, many of them don't deliver any notable
results. And is part of that because
they're measuring the wrong things or what do you do with
the measurements once you've got them?
So big, a big couple of big questions there.
So I guess if you want to take, can you measure everything in
projects and then which are the ones you should be measuring?
I think that would be really interesting.
Well, the simple answer is yes. There's literally nothing,
literally nothing that has any consequences at all for the real
world. Every project that anybody has
ever proposed had some expected consequences, right?
And every consequence is necessarily observable.
Otherwise, it wouldn't be something that everybody cared
about, right? It would have to have observable
consequences. No matter what you're thinking
of, it's really hard to measure if you have a project that's
supposed to improve collaboration or innovation or
the environment or customer satisfaction, or some sort of
social welfare or children's education, it doesn't really
matter. In each of those cases, there
are expected observable outcomes.
Now, you might be highly uncertain about them.
That's a different issue, but you can identify the
observations first, then the rest is about observations and a
little bit of math to reduce your uncertainty.
Does that mean you need to know what you're observing before you
start measuring? Do you know?
Do you have to have an expectation of what there will
be to observe before you can start thinking about measuring
that thing? You don't have to, but you won't
know the value that measurement until you know the reason for
it. You can have purely exploratory
discovery driven observations. A lot of science is this way.
You you don't necessarily know what you're looking for, right?
If I go to a, a, a, a new city and I'm walking around downtown,
I don't know what I'm looking for necessarily, right?
I might discover something. But if I'm going to spend a lot
of time and effort measuring something in particular, I have
to know what it's for, right? And all measurements are meant
to reduce uncertainty, to make better bets.
You're making better decisions. That's what all of this is
about. Every project manager, what
management means is all about making decisions before and
during a project, sometimes even after a project.
So you can think of projects, project decisions during the
project as sort of intervention decisions.
So what are they measuring if they're not going to do anything
with the measurements during a project?
That is, they can't think of anything they might do
differently at all. If it was surprisingly high or
surprisingly low, well then that has a information value of 0.
This is a computable value, by the way, the value of
information. So it's sort of like the cost of
being wrong times the chance of being wrong.
It can get a little bit more elaborate than that.
But generally speaking, if there's, if you can't think of
any consequence that would be different at all, or if there's
no uncertainty about it already, well then there's no value to
that that information. Do you think that project
professionals. Oh, know that deep down that
they do question why they're measuring something or is it
just we're just measuring these things?
So it's there's enough thought given to or consideration given
to what's being measured and why, what the outcome you want
from that measurement to be Andreas?
So I think there's a lot of culture and project management.
We typically measure the things we've measured before or that
that other projects measure. And it's actually one of the
things we go into that, you know, there's all these standard
setting bodies, there's all these kind of influential
organisations in, in the space of project management.
And a lot of them just kind of repeat.
This is how we've delivered projects before.
This is how we've done project management and how we've thought
about, you know, quantifying risk in some way.
But they, they typically don't really distinguish between what
are the methods that are proven to actually give you better
results versus the ones that in, in the best case don't do
anything, but at, at worst, they might add uncertainty.
We use kind of a risk matrix as one of the examples of, so a way
of we, we call it analysis placebo in the book, but it's,
it's the idea that you, you feel you're kind of measuring a risk
by using risk matrices to kind of quantify the impact versus
the likelihood. But there's multiple problems
with, with that type of more subjective scale approach where
all the research from, from not just product management, but
also different fields show that these types of quantifications,
they, there's a lot of error with them and they don't
actually contribute anything good.
But in projects, we, we apply some of these tools and because
we've been applying those tools, we feel that we've actually
we've done some kind of assessment of a risk even though
we haven't really, we call it analysis placebo that because
we've spent time applying A structured framework, now we
feel better about the situation even though we haven't reduced
uncertainty at all. So, so I think that's a lot of
that kind of culture. But another thing that's we
found was pretty surprising in the book was that projects say
spend a lot of time measuring things that have the lowest
information values. If you want to stick to that
framework of the value of information in terms of reducing
uncertainty for our decision making projects, they spend a
long time kind of estimating or measuring the CapEx cost, but
they don't spend as much time measuring the OpEx cost of the
kind of the maintenance of the the infrastructure, the project
that they build afterwards. Where typically the
uncertainties is typically in the latter and the Opecs the
same with benefits. Often times big projects,
especially big public projects have wider societal benefits and
those are really uncertain. But projects they don't spend as
much time trying to measure and reduce the uncertainty about the
benefits side of the project as they do cost because they're the
cost is the big typically it's, it's the big component that they
could ask questions about. They have to deliver too.
So should project professionals be focusing more on the benefits
that will come from a project and measuring those?
Yeah. Actually, when we compute the
value of information, and we've done this many times on many
different types of investments, including lots of different
categories of projects, we tend to find the high information
value variables. So I mentioned that's an
algorithm you can actually run. These are more sensitive
variables. They change the outcome of the
decision more and they're more uncertain, right, than other
variables. The high information value
variables tend to be the things they haven't measured before,
and that often includes benefits.
They spend more time historically spending things
that spending their time on, things that have lower
information values, things that are statistically less likely to
actually improve decisions. So we call this the measurement
conversion in our books. I think this is a pervasive
issue in every industry and projects and also outside of
projects. I I don't know how it couldn't
affect the GDP of countries. It would kind of have to.
Why have we ended up this way? What?
Why are we not measuring benefits better or measuring
them at all? There's a general, there are a
few different reasons I think and Andreas, you can comment on
this. But First off, I think there's a
habit of when you label something as intangible and
immeasurable, it's never considered again.
Once it's labelled intangible or immeasurable, no one ever goes
back and ask, is it really immeasurable?
Are there really no observable consequences at all that could
reduce our uncertainty about this?
Have we even figured out what it is that we're measuring right?
Often things seem immeasurable because we have an ambiguous
label. We're using terms like
collaboration or innovation, but there's observable consequences
that you can't identify out of each of those things.
So there's that. Things get labelled too quickly
or at all as immeasurable. That's one problem.
They're all measurable. The other problem is that people
are just as Andre has pointed out, it's just a matter of
culture. They measure what they know how
to measure. They don't really look at a
problem and say, what should I measure given this problem and
then learn how to measure those things.
That's not the approach. The approach is I know how to
measure these things. That's what I'm measuring.
So what needs to change to make projects delivered more
successfully or deliver better benefits?
Is is part of that looking at benefits and realising that
these can be measured in a useful way?
Yes, that's part of it. The first big project management
decision is approval and prioritisation.
Should you even do the project in the first place?
In those cases, even the decision of doing the project in
the 1st place is. And that first big decision,
modelling it quantitatively, is what's needed.
Because when we start looking at that problem by itself, we find
out that really most decision makers probably would not have
accepted most of their projects. Seriously.
Oh yes. So when you look at the data,
there's a way to quantify risk aversion as well.
And when and when you look at all the research on what
executives say about how risk tolerant they are, how risk
averse they are to, and then compare that to the actual
uncertainties and risk of the big projects, they wouldn't have
accepted almost any of them. So fewer projects, bigger wins.
Is it one approach, right. And also spending more time on
that early selection, you know, if somebody says, hey, this
project is a good deal because we make more than our cost of
money, that's not nearly good enough.
If your return is just a little higher than your discount rates,
you say, well, it's a good public project.
You know, we're going to make the the cost of this big
infrastructure project is justified or the cost of this
big software project or Rd project is justified because we
expect these returns. If those returns are modest,
like modest improvements over your cost of money, your
discount rates, that's not nearly good enough.
If a project is expected, If we look at just all the projects in
their database, if a project is expected to have a lot twice the
benefits as costs, and this is present value discounted to
current dollars, current units of monetary units.
If you expect twice the benefits as you have cost, there's still
so much uncertainty on benefits and costs.
There's about a 14% chance of losing money still.
In fact, there's even a chance of losing more than the money
you put into it. And that's.
Quite something, yeah. Yes, if you spend, if you spend,
you know, £100 million on some big infrastructure project, is
it possible you could lose more than 100 million?
Yeah, actually there is. One of the cost that are not
typically included in cost overruns is other process
disruptions. So you've seen that's when
there's a big infrastructure project, there's a bridge being
built or a new road being repaid for something like this, and
then there's a traffic congestion for a while because
traffic has to be rerouted. Well, the project takes longer
than expected. So that cost is the congestion
cost is longer than expected, maybe even greater than
expected. That's how you actually have
cost that aren't in your budget. There's a other disruption cost
in software cases. There's been cases of software
where a lot of money has been spent to implement something.
And then they figured out that this software that was supposed
to improve productivity made things worse.
So they had to unravel it. They had to undo it and take it
out, go back to the old way. That's more that cost more than
the cost of the project to do all of that.
Now that doesn't happen every time, but it happens frequently
enough that it's a risk we should consider and it's often
not even acknowledges a possibility at the beginning of
this approval process. So that's one big thing.
Fewer projects, bigger wins, you know, a 20 or even 30 or 40%
return. Not nearly good enough.
You got to go bigger, spend more time in that upfront planning
and look for better options. Keep exploring.
There's one of the issues we talked about in the book is
exploration versus exploitation. So this is kind of a classic
measurement problem. It's a operations research
problem. How long should I generate
solutions before I pick the best one so far and and build on that
one? Because there's a chance the
next solution you generate is going to be better than any of
the previous ones. On the other hand, you keep
deferring the project, which has benefits itself presumably,
right? So that's something that isn't
typically looked at and it's something you could spend more
time on in front end loading is you could spend more time
generating solutions. It appears that people converge
too quickly on a preferred solution.
They should think of many more solutions and they should
generate many more decision models and options to be used
during the project. So it's not just the solutions,
it's not the just the various designs of bridges, for example,
that you're looking at. If you're also designing the
measurement programme during the project, you're also designing
the intervention options. These are decision models that
run during the project. You're designing things.
It's like what method should I use?
Should this be an agile project or some modified version of
that? That's all part of front end
loading. And it's usually those things
are they're not only not spend enough time on they're, they're
just ignored as choices. They're just assumed that we're
going to deal with this project with this way.
We're going to follow this particular standard and we're
going to measure these particular things and maybe
after looking at a few solutions, we're going to pick
that solution. That's a big problem.
Just to add, when we talk about solutions, we mean solutions in
a much broader kind of definition of that word.
So when we talk about iterations of bridges, we really should
spend more time thinking about alternatives to bridges as well
that fulfil the same kind of requirement for the bridge.
So thinking in fairies or tunnels or other types of
solutions, not just iterations of bridges, because that's one
thing that we saw too, is that we we fixate on a type of
solution fairly quickly in projects and then we do
iterations of that kind of design.
How should that bridge look or how should that nuclear power
plant be? Whereas we could consider much
different types of solutions as well.
I don't know where to start with everything I want to ask you,
because that was quite macro. It's quite philosophical about
what? What do we do and how do we do
it in other alternatives, let alone the complication of
political feeling or government. When you hit government mega
projects for example, where the politics behind it can make it
something happen without really being questioned why or is this
the right thing to do. So for people listening, what is
the how can they imagine someone just in a how I get a job
working on a project? How can you begin to deliver,
execute, think about your projects in a in a better way
along the lines he suggests. And are you saying, let's look
at the data, that let's make data more of a priority than
these humans who might be too might be prone to optimism bias
or and then with the rise of AI is, is this going to nudge
projects more towards the way you're thinking that that if
it's not measured, it should be measured, Everything could be
measured and we should listen to the data and see the evidence of
what actually works and be more objective about it.
I think this is an interesting, interesting thing that comes up
quite a lot. So many might say, but what
about somebody who's working on a project right now?
I think that's already too far down in the weeds because just
organizationally that person doesn't really have the position
or authority to make some of the changes we're talking about.
We have to start much higher and much earlier in the
organisation. We have to talk about
stakeholders changing the way that they approach things,
right? So it's stakeholders that are
making ultimately making choices about where they should allocate
funds in the first place. And something has to be
communicated there that it's to their benefit to spend more time
in this analysis before you make big bets.
You know, if you're buying a house, you look at a few houses,
right? If you're buying a car, you look
at a few cars. You don't just pick the first
one you see, right? And I think decision makers
often are too quick to say we got to start coding or we got to
start pouring concrete, right? I, that's a, that's a, this is a
different way of thinking about it.
We have to spend more time in that upfront analysis, analysing
our choices. Again, fewer projects, bigger
wins. That's the, the optimistic part
here. I think it's the bigger wins
part there. There's room for really
impactful mega projects. That's where we should be
thinking those really fundamentally life changing
kinds of projects, socially changing kinds of projects.
Like like what? Such as?
Oh well, we have a couple of examples in the book, but at
least in the states here, is it possible?
Ideas have been proposed and it should be analysed.
Is it possible to stop hurricanes?
That's a hypothesis. And even if there's, if you work
out the information value that if there were only a 5% chance
of that being true, it'd be worth investigating just because
it's so impactful. The cost of hurricanes is starts
to compete with the cost of our entire aircraft carrier fleet,
you know, in the US, right, or the entire Interstate system of
expressways in the US. All of a sudden, when you add up
a few big hurricanes like that, you say, well, yeah, actually,
if it were even possible to even slightly reduce of the impact of
hurricanes, what are the different ways we could do that?
That's a big project. That could be a very big
project. It might not even be a big
project, Might be a bunch of little projects, by the way.
But also, you think about AI itself.
We talk about AI in the project and we tell people that you're
single most important project, and this is the biggest project
of all of them. By the way, the single biggest
project the world is facing right now is how to do projects.
That's the single biggest project.
That's the most important mega project that any country or
company is dealing with is how they should do projects.
And this is a key message in the entire book.
There's a lot of ways to run projects and a lot of ways to
measure projects, and we're just not spending enough time
figuring out which ways work better.
Well, some are better than others, and we're still stuck on
some, you know, habits that don't work as well as other
things. Thanks Doug.
Andreas, what? What are those better ways of
running projects then what? What actually has does give some
success. So we do have a lot of, I think
one of the positive messages of the book is that there is a lot
of potential, there are a lot of things you could be doing to
improve the projects you're doing at the moment.
So far we've talked a lot about, well, if we, we're able to
reduce uncertainty about the bets that we're making, we would
be making better bets, probably fewer bets, but the portfolio
projects would be entirely different to what we're
investing in now. And it would probably be better
because a lot of what we, a lot of the projects we invest in now
are the ones that look best on paper, but aren't necessarily
the the best projects in in the book.
Obviously the book is about measurement.
So we talk about like what kind of measurement should you be
making? Why are you doing?
Why are you measuring in the 1st place?
How should you be doing measurement?
And one of the things when it comes to information value is,
is often, if you don't know anything, almost anything will
tell you something, right? So if the uncertainty is really
high about something you haven't measured in your project, you
don't need to collect a lot of data to reduce that uncertainty.
And that's also part of the measurement inversion we talked
about. We spend a lot of time fixating
on the cost of the project. And sometimes to marginally
reduce the uncertainty, we have to do a whole whole bunch of
engineering in a project that might be really expensive and
there's still a lot of uncertainty after we've done
that detailed engineering. Whereas when it comes to some of
the benefits coming out of projects, since we haven't
observed it so far, we, we don't really need to make a lot of
observation to be in a better place than we were before, to
have reduced the uncertainty of what can we expect to get out of
it and how can we manage that better.
So we, we talk about, you know, you probably need less data than
you think you need and you probably have more data than
than you think you do. And that's generally going to be
the case for organisations and for projects, especially on the
things they aren't measuring so far.
What what other things we talk about that you should be
measuring as well as the the outside environment of projects.
You mentioned politics as as an example of how it can add
uncertainty to projects. And obviously that should be
something if you're doing a big project that runs over multiple
years, you should be measuring kind of the political context,
political support of a project, because that's going to add
uncertainty and it might mean that you're building in more
decision options along the way of a project.
But that's also one of the recommendations of the book
that, you know, the projects, they aren't in their own bubble.
They they do engage with and they are impacted by the
external environment. So that's also some other things
you should be measuring that you haven't been before.
Thank you. Could you give an example of a
project that perhaps does take on board the things you're
telling me the recommend, the recommendations that you make in
the book? Is there anything out there that
people could find out more about?
I've run into a couple of examples in in industries that
do some of the things we're talking about.
So for example, the option to cancel a project, that should be
an option that's defined very well in advance.
It's a part of a decision model that's run every day.
Now nobody's doing quite that just yet, but in the
Pharmaceutical industry and some related industries, they have a
stage gate process. So only about 8% of all
pharmaceutical products that are investigated ever make it to
market. So how do they make any money?
Well, when they quit, they quit early.
And when you look at a lot of big projects that cancelled, and
a lot of them do, they cancel with nothing to show for it
after spending a lot of cost and having a lot of external
consequences that are costly after.
If you look at those kinds of projects, you often find out
that they could have and should have cancelled it quite a long
time ago. It they had all the data that it
would would have justified cancellation quite a long time
ago. It was just running on inertia.
But also there's that feeling of, I think is it called sunk
cost, sort of sunk cost of allergy where you've put all
this money in already so it's too late to turn back.
Right. It should always be on the
forecast of the future. Here's what we here's our
uncertainty about what there is remaining to be spent given what
we've learned so far about the complexities of the project,
because that might update our uncertainties about what's
remaining to be spent. And we're comparing it to the
outside world. It might not just be the
project, it might be that something in the outside world
has changed that changed the benefits of this thing.
If I'm. 35% of the way into a project for some big
infrastructure project and AI at the same time is improving
because the and it will, because these are multi year projects,
right? So there will be changes in not
just AI, but materials and things like this and fabrication
methods during the project. At what point during the project
do we say there's enough that's changed about the outside world
that I should now re engineer something about this project?
That's a intervention decision. That's an intervention option
that we should define in advance.
So the pharmaceutical companies are one, but also aerospace does
a little bit of something that we talked about where they do
anticipate changes in technology.
Because if they're developing a new crew re entry vehicle from
the space station or something like this, they start designing
it without necessarily assuming that the only materials they
have are the current materials. Because it'll take a decade or
more to design these things. And during that time there will
be technological improvements. They're kind of counting on them
in some ways. So we should take that into
account. If we have a big infrastructure
project, should we be saying asking questions like if AI
could have changed this design if only I waited 2 1/2 years
from now, what would I do differently?
Is is there a point at which it might be worth waiting?
There's something we introduced in the book called Technology
Regret. Technology regret is includes
things like you invest too soon in a technology that's rapidly
changing, so you fixate on a bridge design or fixate on some
software features at the same time while you're pouring
concrete and writing code. AI is getting better.
Not only AI, but materials and fabrication methods, 3D
printing, hole bridges and so forth.
That's getting better during this.
Now AI is actually, I think improving faster than the other
two items I mentioned. So we should at least be looking
at there. There's not too many big
projects that are going multiple years that shouldn't also be
tracking what's going on with AI and asking questions about.
Is there a point in time I should consider the possibility
of right now and design an intervention option where I'm so
far into the project and something the outside world has
changed? And given how far I am into the
project, and given the benefits of switching gears at that point
in time, does it make sense to change something during the
project? And you're saying that the
answer could be yes, it could be, it could be beneficial to.
So, so you're, you're planning in uncertainty, you're kind of
planning that. That's quite probably quite a
mindset shift for lots of people working on projects like this.
Yeah, I think people are used to the concept of options, though
some are. They're used to the concept of
real options too. You know what's the what's the
value of keeping an option open, right?
Actually, even things like the option to cancel a project.
This surprises people, but it de risk a project.
Risk of a projects go down when you build in intervention
options because in the case that you do need to cancel a project,
you'll at least be able to cancel it sooner.
So your expected losses would be lower if you do cancel a
project. Now I've had lots of clients
that would say, yeah, I've seen projects get cancelled, but
prior to us developing decision models for them, they never saw
a chance of cancellation on the business case for a project.
How often do you see that right, if you're if you're presenting
to a government agency or someone else in the corporation.
I've got this idea for a project.
Should I include the fact that there's based on historical
data, there's about a 15% chance of cancellation with nothing to
show for it. Some we've heard this, Andres
and I were just talking to a group of the last time I was in
the lending group of, you know, project managers and stuff like
this. They'd say, well, you know,
things wouldn't get approved if that were the case.
If you're I said, well, if that's what you're looking at,
then just leave out half the cost too.
Why not? So the no, it's, it's about
being transparent about these issues.
Also, I think there's a fundamental conflict of interest
here if the person opposing the project is the only one building
the decision model for it, right, and the only one
presenting it to decision makers.
I do think there's room for some independent body of auditors.
Actually, Oxford Global Projects does some of this right now
because they audit people's projects, project plans and
costs and schedules. That's part of the services that
they do. And one of the values of having
an outside consultant come come in, in general is that, hey, we
don't have any skin in the game here.
We're just going to do our analysis and give you an honest
representation of the outcomes. Because usually we're working
for the check writers, the people who were allocated funds,
not necessarily the project manager or the person who wanted
to promote the project. We're working for the people who
have have to commit to the funds in the 1st place.
So they want an honest answer and I think that's part of it.
We have to think about some built in conflicts of interest
in the way that we approach even proposing projects to begin
with, but and then everything else that we just talked about.
South more rigour what? Andreas, what would you like to
add to that? I love this conversation has
been about you know what, What type of projects should we be
making? How can we plan projects better?
But we do spend quite a long time in the book talking about
different types of measurements for projects that are already in
flights. What should you be start
starting to measure and how can you make, how can you make kind
of metrics more, more decision driven?
So we, we talked a little about, we, we have specific chapters
about, you know, cost and schedule and benefits progress,
typical things that you need to measure on, on projects and that
it's really useful to measure. But one of the things we
observed was that a lot of the metrics that are currently being
used and on all sorts of dashboards, they're very kind of
exploratory, as Doug would put it.
So they're there because decision makers are project
managers, are risk managers. They feel that once they know
that once they look at the dashboard full of metrics and
they see like the right combination of different figures
and charts and, and metrics, they'll have some kind of
they'll know what to do in that kind of instance.
They'll know how to react on the metrics on the dashboard.
But a lot of the time what we discovered was that a lot of the
metrics aren't geared for decision making really, and not
about, you know, what intervention decision or how
should we react if we see a specific metric.
So the book spends a lot of time thinking about, you know,
progress metrics like cost performance indicators or
scheduled performance indicators and so forth, and thinking
about, well, how could we make it reframe that into something
that is easier to understand. And a lot of that is reframing
it into, well, how much productivity increase would we
need to have to actually catch up on our delay, for example?
So we do give a lot of advice in terms of how should you be
measuring, how should you be presenting metrics?
And I think the other thing in the book that we do is we be
sure that because this is a book about measurement, there is some
maths in it. But we do show that, you know,
all this maths isn't complicated.
So alongside the book, we have a website where you uploaded a
whole bunch of spreadsheets to show that all these things.
We talk about the technology regret example, we have an
example of using machine learning for predicting
outcomes. We have different types of
simulation exercises and we run it all in Excel, which you can
download and apply it to your projects just to show that.
It sounds pretty complex if you're used to using more
subjective scores and scales and risk matrices, but it actually
isn't that difficult and there's a lot of already built solutions
out there that you can just apply.
If there's one thing you'd like readers to take away from your
book, what what might that be, Andreas?
So if, if I can positively frame it, because you've talked about
issues a lot on this on this podcast so far, is that there is
actually a lot of evidence, even though there's in the industry,
there's a lot of claims around or if you just apply our methods
or our tools or our software, you're going to better projects.
That's typically what we see. If we go out to conferences and
we look at like vendor stands, it's all the same software
companies with some kind of reframing of if you use our
products, you're going to deliver to cost into schedule,
right? That'll be the typical
experience. But there's so many claims out
there can be quite difficult to find out what works and what
doesn't. And what we've tried to do is
actually just to look at the data.
The research has already been done.
We've done some of our own research and we've identified
that, you know, some things are better than others in terms of
measurement or estimating or delivering projects.
So that there's there's an opportunity to apply the things
that work and kind of toss the things that don't work.
Fantastic. That's very well put.
Thank you, Doug. Any any one message you'd like
to get across to to listeners? Yeah, we, we refer to the the
most important project, the one that is about making projects
better. We call that the meta project in
the book. And everybody ought to start
thinking about their meta project.
You know, no matter how big your project portfolio is, no, no
matter how big the individual projects are, some time is worth
being spent on the meta project. Your most important decision is
how to make decisions. By the way, that's in life and
business and government and everything.
Your most important decision is how to make decisions.
So we should apply that here. And so the meta project is, is
the project about making all the other projects better.
What's a great way to spend 1% of a whole project portfolio
trying to optimise the other 99%?
That's the best way to spend 1%, right?
And it might be about that much for a lot of project portfolios.
If you look at some of the infrastructure portfolios, you
know, even spending 1% of those portfolios is a significant meta
project where a lot of analysis could be done.
And so we recommend having a project called the Meta Project
and Everybody's portfolio. Thank you, Doug.
I feel as though we've just scrape the surface of what we
could talk about in your book. I feel like I might have to do a
follow up podcast sometime soon. But thank you very much for your
time. Thank you for your thoughts.
I wish you best of luck with the book, but it's been really
fascinating talking to you and I'm sorry we couldn't cover much
more of what you've what you've put in the book, but thanks very
much for your time today. You best text.
Thank you. Thanks again to Doug and Andreas
for joining us and to you for listening to the APM podcast.
I hope you've been left with much food for thought, so don't
forget to look out for more episodes, alterate and reviews.
Wherever you get your podcasts. We'd welcome you to get in touch
with your comments, feedback and suggestions by emailing us at
apmpodcast@thinkpublishing.co.uk. Spotify and YouTube users,
please do also leave us your comments.
This podcast has been brought to you by APM, the chartered body
for the project profession. For more information on APM,
visit apm.org.uk.