Welcome to the APM podcast. APM is the childhood body for
the project profession. My name's Emma DaVita.
I'm the editor of Project APM's quarterly journal and your host.
In this podcast, I'm speaking to Daniel Armanios, BT, Professor
of major programme Management at Oxford University's SCII
Business School, and Zach Swafford, Co founder of Dot, a
San Francisco based startup. He's a project management tool
uses AI to assist with brainstorming, road map planning
and report generation, among many other tasks.
We thought it's the perfect time to cast Sari over the rapid
developments in AI over the past year and speculate on what 2025
might hold with two guests with two different perspectives on
the world of work. Listen on to hear their thoughts
on the exciting potential of AI, their worries about the
technology, and their advice on what you should be doing to
practically get to grips with this work revolution.
What's really valuable, I think, is their insights into how AI is
being used in projects right now.
So do listen on. I'd like to welcome both of you
to the APM Podcast. Thanks for giving us your time
today. Let's begin by finding out a bit
more about your work and your area of interest in AI.
So perhaps, Daniel, do you want to tell us a bit about your work
and your area of interest when it comes to AI and project
management? I'm Daniel Armanios.
I'm the BT Professor and Chair of Major Programme Management
here at the Site Business School, University of Oxford.
I think my work is at the intersection of civil
engineering and sociology. So essentially I'm interested in
how organisations coordinate to underpin really big scale
initiatives. And so my interest particularly
around the AI facets, is thinking about where do you
deploy what kinds of algorithms in the organisation.
And so when you have a project with several organisations,
several stakeholders, where best to place what kinds of
algorithms. And so I've been spending much
more time on the organisational design programme, project design
kind of perspective to look at where you place these kind of
algorithms, knowing that AI is not all one thing, right?
It's been multiple things. And so that's where I've been
spending a lot of time. And the second is where is the
assistance of AI really good at processing a lot of information,
but also where do we need to build guard rails and caution
around the hallucinations? The ability for AI to shape the
informational landscape in ways that might not be fully
revealing of the truth. And so take its benefits, but
also be mindful of its places where it can go off, off the
rails, if you will. Thanks, Daniel.
So by what you're talking about, do you do you tend to focus on
mega projects, bigger projects? Usually large scale projects,
Yeah, usually large scale projects.
But I think often how it's being deployed in smaller projects can
be a harbinger of what could happen at scale in a larger
initiative. Is that how it tends to work out
in the real world, that that organisations will will
experiment with a smaller project and naturally that will
scale up or not depending on how successful it was or lessons
taken from that and shared? I mean, I'd be, I'd be
interested in Zach's corporate experience around this in terms
of what they're doing with this, the start up at Dart.
But I think what I would say is, is that ideally that's what
you'd like, like some experimentation on a sandbox.
But I think matching with, I think this year was, was really
a year of, Oh my gosh, we need to get on this.
And so what's happened, there's a lot of organisations that
feel, Oh my gosh, we just have to get everyone to do it.
And no matter what, deploy it. And so you would hope it's done
with strategic intense, you know, sandboxing, etcetera.
But I think there is a bit of, Oh my gosh, we're behind the
curve, let's just deploy it. We need to show that we're doing
something with AI. And so in some cases they're
just experimenting and you're seeing this with AI itself.
I mean, a lot of these products are being disseminated without
necessarily full on validation whereby it's like what we're
going to know what's the issues or know what is by people trying
it. And so I mean, even open AI when
they launch ChatGPT, other things, it would just let's just
see what happens. And you know, there's different
perspectives as whether that's OK or not, but I think it's a
mix of both, right? Some are sandboxing, trying
this, figuring out the kinks where others are just like,
let's just get it out there and see what happens.
Each has their pros and cons, if you will.
So a little bit messy in real life.
Yeah, unfortunately. Thank you, Daniel.
Zach, tell us about about your work and a bit about your
experience with AI. My name is Zach Swafford and I
am a Co founder at a start up called Dart.
We do project management with AI, so pretty relevant and we
got started a couple years ago basically because we saw in, we
saw a big sea change here basically in the project
management space with AI. We saw this coming a little bit
and wanted to build really from the ground up a new project
management tool that would incorporate AI and really
specifically when I say I, I mean LLMS on into every facet of
the project management experience.
Can you spell out what an LM is just for people listening?
You might not know. So an LLM is a large language
model and this is something like Jet GPT or Gemini is a Google
One Co pilot and Microsoft Claude is another one.
Those are some of the biggest models and those the idea is
that they use a lot of language, a lot of text and other
information from the Internet to create a chat like experience.
But really thinking beyond chat. I think in the longer run, these
are the models that will act more like a like a real AI like
a real Jarvis or whatever you want to think about.
You know, these these ideas of big AI models that can do almost
anything. These LLMS are the technology
that starts to enable that more so than traditional machine
learning, which can be great at answering specific questions and
is an excellent tool for lots of things.
But the LLMS are the side that I am the most excited about right
now. So there's lots of interesting
stuff we can talk about there, get into that.
We have tonnes of really interesting features in Dart.
But to answer sort of the question of how are, how are
organisations building up to this?
I've seen it, I've seen it all sorts of different ways.
I've seen, like Dan was saying, really diving right into a big
project and kind of jumping off the deep end, which can be
risky. Of course.
I've also seen really successful implementations where folks are
trying something on a smaller project and building up to it.
And I think there are there are pros and cons and there are also
different levels of risk that different organisations are able
to accept or willing to accept. And depending on the industry.
At DART, we work with a lot of different industries.
And I always say that some of our marketing agencies, right,
can accept a higher level of risk and a higher level of
incorrect answers or hallucinations or whatever than
a team that might be working in healthcare or, or construction
or something like that. So that's another perspective
on, you know, maybe different industries can move at different
speeds here. As well.
And Zach is have you had a background in project management
before? Yes.
Great question. Yeah, sorry, I should have,
should have answered the background question.
So, yeah. So I studied AI actually
specifically at Stanford a few years ago and then I went into a
company called Plenty and we did indoor farming.
So at Plenty and I worked on the hardware and software sides.
There are LED teams in AI software and hardware actually.
Would you mean by Intel farming? Is that literally what?
Oh yeah, what do you mean by that?
Absolutely. It's what it sounds like, Yeah.
So. Farming inside.
That's right. Yeah.
So you can think of it like a big automated greenhouse.
I think that's the simplest way to think about it.
We grew leafy greens, lettuce and arugula, kale, lots of fancy
herbs and the indoor farming process is.
The big idea was using automation and AI to really have
the perfect environment for plants and grow them really fast
and well and healthily. And when you have an indoor
farm, you can also place it right next to where it's going.
So you also don't have to deal with the supply chain logistics
and you get a lot of fresher food.
So yes, very interesting work and, and plenty in particular
was a really fast growing company.
So we, we grew a tonne and that meant that we had all sorts of
scaling problems, all sorts of project management needs.
So I filled the gaps in engineering management, but also
in implementing project management, both for software,
which is really my original background, and then also even
building Gantt charts and helping the construction and
then the builds we were, we were another startup.
So there was just a lot going on and, and I filled some of the
gaps there. And that's really how I got
started with project management formally.
OK, well that's really interesting.
So you 2 have very different perspectives.
Daniel Moore, the organisational take in the bigger picture and
Zach, really nitty gritty of making this work day-to-day.
I think there's the thing that I'd really like to hear about is
looking over 2024 comparing where the world is now compared
to a year ago. What would you say have been the
the biggest developments when it comes to AI and project
management specifically? Zach, maybe on the ground, what,
what would you say? Jump out at you?
I think that in the past year I've seen of new tools that
allow people to chat directly with their projects.
So this means asking questions about are there any blockers?
Are there what, what are the, what are the critical path in
this timeline? Those sorts of things.
Or sometimes the questions that project managers or potentially
even other stakeholders might formulate and then might want to
ask, but maybe would need to go through.
You need to have a bunch of messages about, about that and,
and go through a whole process to figure those sorts of things
out. Those are the kinds of questions
that if you have your project management tool or your system
in place to have all that data in good shape, then you know a
random stakeholder would be able to just ask the question what's
our critical path right now? What's our timeline and get a
chat based answer, which is particularly nice actually for
non-technical folks or people that aren't steeped in the
project management world like we are.
OK. That's great.
Daniel, you agree with Zach. Are there any other developments
that you've noticed that have been on your radar?
Definitely one of the trends I see very much in agreements with
Zach. So the ability to take large
amount of corpus of text because if you think of projects, they
fundamentally produce a tremendous amount of language
data, text data. And So what DART is doing other
organisations and plan others, they're really enhancing
drastically the processing capability.
So I mean, you can it's get, it's quite revolutionary project
world. I mean, the idea that you can
take a request for proposals, run it through these LLMS and
come up with an addition, an initial Gantt chart, a racy
diagram just from the data to be able to ideate things that you
know are doable, but usually took, you know, burned a lot of
time. And if you think of projects,
often the issues are like death by 1000 cuts.
You know, if you can save 30 minutes here, an hour here doing
this, the fact that you can see critical pass in your data run
last planner models. And this is where the Omni
models get into the way you can ensemble different models
together. Perhaps even now there's a lot
of interesting trends on, you know, reference class
forecasting, benchmarking, for example, right, how you come up
with the reference class, There's 20,000 different ways
you can come up with a reference class that can be done back end
automatically with some of these generative AI large language
models. So that's one dimension I think
is really interesting. The ability to take all of that
data and make sense of it, that would take hours, months of
really specialised time. And even if it's not perfect,
just getting you started, you've already saved an infinite amount
of time. So the ability to ideate on this
is, is is really revolutionary. I think where another trend is
happening is the visual turn. So there's now models that can
translate what you're doing or what you're thinking in text
into image. So there's some organisations,
some companies like Icon for example, that they, what they're
planning to do is take your ideas and then translate them
into initial construction documents.
That's an amazing turn and then schedule what would look like
that from that. So I think the visual turn, even
using video to do this, be able to OK, I have this idea.
Let me generate a video of what this could look like and immerse
someone in it. So the connections between large
language models and visual or even video motion kind of
imagery that could then be tied into immersive reality, other
things. So it's not just the text and
the visual, but the modality is really quite fascinating and
really exciting because especially at the early stages
where if you could plan and see much more of the information
around you both in Word but also image, that could save so much
cost at the back end because you've already kind of stress
tested. Red teamed a lot of different
scenarios that would have taken a lot of costs to do.
Because if you have to do that without this kind of assistance,
the issue becomes I have to then think, is this time to build
this scenario, to run this chart, to run these matrices?
Is that really worth the time to do so?
Then you have to spend a lot of time being very careful about
where we're going to search more, where we're going to plan
more. And with these tools, you can
run many more trials than you would otherwise.
And it has huge implications. I think it's just really
exciting to see that strain both in the text base term large
language models, but also now the different modalities, text
to visual. And we could talk about what I
think is happening more recently, which is even the
ability to train on really pretty interestingly accurate
synthetic data too. So it's not just looking at your
own prior project data and seeing it running synthetic data
that because of how they're trained, it could be quite
accurate. It may not be, but at least as a
starting point could be quite fascinating.
I mean, for example, in spatial computing, which is a lot of the
back end Nvidia's now training, you know, if you have a project
on autonomous vehicles or other things, they're actually
training on synthetic data because the lighting and stuff
now is so good. So autonomous vehicles, for
example, are being trained on synthetic data to some degree.
We get into the final one, I would say, and then I'll move
on. Is this.
So we have we talked about textual, we talked about visual,
but also the ability now to artificially or with AI create
kind of interactions of social interactions between people.
So we can take the next step and say if these different project
manager are working together, how would their interaction look
like? What pitfalls can we worry about
simply by embedding rules into simulated managers or agents?
This would be called deep reinforcement learning.
And they're remarkably adapt. So for example, they can run
scenarios like what would you know, birthday parties and who's
invited? I mean, the social interactions
you can do with just very basic rules.
So I could see in the future, and even now it's happening, run
different managers with very basic realistic assumptions what
would happen when they work and interact in this way over a few
months, what could be, what could get, what pitfalls could
we worry about, what not? And you could do that all at the
early stage. So it's really, I mean we're
only scraping the very initial surface and I think that's why
Dart and plan icon. These are just going to grow
because of their ability to really ride that initial wave.
And so with that comes amazing power.
And also, you know, we could discuss the perils later, But in
summary, I think three things. One is the massive ability to
read large scale text, the text to visual and motion right, It's
like video and so forth. And then the third one is even
to simulate what interactions of individuals in these projects or
large scale programmes or even small one could look like.
Is it realistic? Not necessarily, but it is at
least a really good way to ideate in very different ways at
scale. So what you're explaining there
sounds very exciting and we're on the cusp of big things.
I'm just, I've, I haven't heard anyone talk about the, the kind
of project manager interfacing with another project manager.
Do you mean that you could get AI to understand the characters
of individual people and how they might work together?
Is that what you're saying? Yeah, I think so.
So there is, and we can talk about the integration of how
these different algorithms work. But one set of algorithms are
what's known as deep reinforcement learning.
So essentially what happens is you populate with some very
basic rules so you can look at your prior work as a project
manager, otherwise say what are kind of heuristics, rules of
thumb or ways we operate that we can embed into an agent, right?
And then you say, and then how you, the learning aspect is when
it does really well, you say, yes, that's great.
When it kind of goes off the rails, you say, no, no, no,
that's not what I'm looking for. And as you train these agents
with these rules and let them kind of code another and deal
with the data, you could potentially simulate
interactions. There's some interesting work.
It's funny, Zach mentioned Stanford.
I graduated from Stanford as well.
There's a really interesting study at a basic level out of
Stanford where they developed this kind of, I don't know what
you call it, like a kind of, it's almost gamified, but it's
an interaction called Simorca. And essentially what they do is
they embed agents with kind of rules.
Now in that case, it's like we're hosting parties or general
social interaction. But if you use your insight and
acumen as a project manager to understand here's how I break
down sets of work, here's how I interact with with suppliers,
etcetera, you can simulate that interaction.
You can even build avatars to practise.
So for example, let's say I'm going to have a difficult
conversation with a supplier, but I want to practise what that
would be like. You could interact with an
avatar who has these kind of rules to help you train for this
difficult conversation I'm going to have about procurement.
Right. So the, the interaction ability
and that's only just started maybe within the last couple of
months, I would say in the project management space.
It's been longer in the CS space, but you're starting to
see that ability in the project management very recent.
So it's not just taking my data, rendering it, visualising the
Ida, but even practising how we even do the work is possible now
I think. OK, So what I think I'd like you
to do is if I take you away from from the, you know, what ifs and
the excitement about what could happen to actually bring us back
to how AI is becoming a facet of work life and projects these
days. So kind of practical examples.
So there are honestly lots of stories of success that I'm
familiar with. And I think one place to start
is where the newest technologies are at their best, where they
really shine and thinking about how that can be applied to PM as
a discipline. So one of those Dan just
mentioned is acting as a smart, I like to say sometimes a smart
intern almost giving you an 80% version of what you need to.
And maybe maybe it's almost like a team of interns where you can
send them out and proactively get all sorts of different first
stabs at a project or at A at any given task really.
So one place where I've seen that really successful a lot is
in initial phases is in brainstorming, ideating, trying
to come up with an initial plan. A lot of this happens in
software, maybe more so than construction, but at the
beginning of a software project we might be thinking about what
are the different ways we can tackle this.
Here's a bunch of the user feedback that we have.
We have a long list of tonnes of documents about how users act
and behave. And now what we need is some
solutions, some some ways to tackle that problem.
There are great tools really tactically, I might just put
that as a first stab. I would always say just put that
into Cha Chi PT. Just put all those documents
straight into Cha Chi PT if you can, if your organisation allows
that sort of, you know, security wise, but put all that straight
into the nearest large language model and see what happens,
right. And, and see, get some of that
back out. Ask, ask it, Hey, can you help
me solve some of these problems? Can we ideate about how to solve
some of this? And probably what you'll find is
that some of the answers are good, some of the ideas are
good. A lot of them lack the context
and awareness of your organisation, of your practises
of kind of what what you actually need to do.
You'd be surprised sometimes how that's not totally necessary.
But in a lot of cases, then you might need to iterate and start
to provide more of that context and start to think about more
advanced tools that would allow you to include that context.
So that's one way. Another way I would say is,
yeah, I mean, I guess I think an important thing that I wanted to
emphasise was figuring out the ways to work with your
organisation to have the right security posture to allow you to
use those tools. Because it is really important
that people at all levels of an organisation are able to
experiment with this stuff, in my opinion.
And there are ways now to allow you to not trained on any of
that data, right? You can get an agreement with
open AI to originally this was not the case.
And so people were worried about providing information to LLMS
and there was kind of a let's let's take this slow sort of
approach. Now.
I would say it's a lot easier for anybody to get the correct
kind of agreements in place to say, OK, our data isn't going to
be used. We can treat this more like any
other service provider or cloud service provider and let's you
know, let's get chat be set up in our organisation so that we
can start experimenting with this.
Do some of that brainstorming. Do some of that initial get.
Get some of the initial takes on how we can apply this.
Provide the documents, provide the context, start to get back
some of the answers. So it's really about using
another resource to make connections.
You might not necessarily. It's like having a whole nother
team of people in the room who might come up with some really
terrible ideas, but some of them might be the valuable ones that
you may never have picked up on in the first place.
That's right. Yeah.
And I think that that's an awesome opportunity for or folks
to take away some of the, the hard work, the brainstorming
work, the grunt work of just building, building a timeline,
building a Gantt chart, let's say, outsource that to some of
these tools. And then free you up as as a
honestly, as APM contributor at any level really to do some of
the higher level work. Even if you're junior in an
organisation. Now if you end up with a team of
interns that can come up with great ideas for you, then you
can start to build the muscle of discerning what the right course
is and then escalating that as your your work essentially.
OK. And what about a more
sophisticated level or a more senior level cross projects
where AI is taken more seriously beyond the kind of initial
experimentation as you've described?
Where do you see that happening right now in the real world?
Yeah. I think there it becomes more
about, I see a lot of the challenges at that level as more
interpersonal challenges of getting stakeholder alignment
and you know, you've got a plan or you've got some candidate
plans and you need to, you need to collect all the resources and
get everyone focused around a common goal, a shared goal.
That's the kind of thing that AI doesn't do as well right now.
So you can, but there are still ways that you can really apply
those tools. Daniel mentioned earlier the
ability to run scenarios right to you can you can say, hey, act
as this stakeholder. Let's, let's I, I need to
practise for a stakeholder meeting here where I'm going to
be delivering some tough news or I'm going to be pitching this
plan. Help me give me some feedback on
the ways that I could improve this project proposal, but also
just my demeanour and the way that I the way that I present
myself and present this project. That would be a very concrete
case and and anyone could just have that chat with ChatGPT
right now to to practise for a meeting.
You don't need some super hyper specialised tool.
Once you try ChatGPT and you see where it fails, you can move to
a more advanced tool for something like this specific to
the project management domain. But I would encourage everyone
to just give that a try and see if see if you can learn
something from it. See if you can get a takeaway
there that can help you to help even improve your your inner
human interaction. That's I haven't that's not
really something that we've covered much before around using
AI to improve your human skills. That's you always hit a negative
and how it's taken away from the world of humans.
But this is absolutely putting that on its head.
Daniel, I can see you nodding your head there in agreement,
but what have you seen AI used across projects, programmes over
the past year that would have been successful?
The way I generally characterise it, I think Zach is really spot
on. Maybe maybe to take a
background, I think it's important to understand how
these language, large language models work so that you can
understand what it's useful or not.
Essentially the best analogy I have is let's just use
generative AI large language models as the example.
It's kind of like a stochastic parrot.
So what does it do? It sees some words in a question
and it says, OK, I've seen these words Co located with each
other. I'm going to go to my training
data and model that I've developed on that training data
and respond back with something that seems probabilistically
related, right? So if I asked certain kinds of
questions, it knows I've seen that question before, I've seen
these words kind of matched together.
And so I can give you a response that would probably seem
similar. Now, why do I say this is you
have to realise that ChatGPT others have been trained on
essentially the realm of large scale corpus of data like
Internet level data. So why do I say this is that
it's good to start where I've seen it useful is good to start
for like generalist things to start things that you know are
very common in project management and are very like, I
would start with the thing What I've seen is you start with a
very mundane things that everybody with a project has to
do, a Gantt chart, a racing matrix, you know, like that
level and stuff that you have that you're willing to share
publicly, right? So you start there and you get a
sense of, and I think that's where the biggest, easiest low
hanging fruit is mundane, painful things that you have to
do. You want to have 80% trial like
Zach said. And then that way I could spend
my time finessing the details as opposed to building that up over
hours. I can just start there.
Where it gets like the added value is now if you have
something that's your specialist sauce as an organisation.
Now why did I mention all of this stuff about large language
models is they're chained on generalist data.
So now the question is, how do I get it to understand better my
refined thing? Let's say I'm doing healthcare
work and I'm looking at a specialised disease just using
ChatGPT, etcetera. It's going to give me everything
about medicine, but it's not going to have specialised
knowledge. And sometimes you want it to
train on smaller data. So there's two ways Zach brought
it up about this is that some companies, their large language
models, you essentially have to ping the server of that model
and that's how it works. Other ones, you can actually
download their neural model, bring it into your closed kind
of space, give it data that's only to you in your closed
firewalled environment, and essentially take the basis of
the algorithm existing and train it with more specialised data.
So now if you're taking the next step and say, well, I get a
sense of how I want to do this Gantt chart.
I get a sense of how I want to do the race, like general things
that just are pain points that are mundane.
And then I take the next step and say, hey, I want actually
more targeted knowledge for which we have the secret sauce.
You can train that model uniquely to your data.
And so that's called transfer learning essentially.
That would be the next step. Now I can get the really
specialised know how to develop it.
That's just for me. Now the point Zach made.
And I think there's concern that, Oh my gosh, if I use this,
the human will be displaced, etcetera, because AI can
hallucinate. You really need to have expert
knowledge of what is sensible and what doesn't make sense.
So there's a little bit of danger if you're trying to use
it to explore in some newer area you're not familiar with, you
need to make sure you're triangulating it with other
things and, and stuff, you know, it's your expert knowledge.
In other places where you're using it to experiment, you're
going to need to look at other sources just to make sure.
OK, that sounds interesting, but let me Fact Check it.
And there's ways to, there's settings actually you can do to
change that. There's something called
temperature. So if you actually make it, I
think it's really, really low. It's saying, I don't, I want you
to be absolutely certain about this.
I don't want you to be creative versus I want you to really
think crazy. You can set the temperature
really high and say, come up with the most creative, unhinged
thing. And even if it hallucinates,
maybe it's interesting, right? So it just depends on the nature
of the problem. But very quickly, generalist,
mundane pain points start there. As you build acumen, build
experimentation, you want to do something more specialised, you
can take in those models and train it specifically on your
data in a firewalled way. There's ways to do that now.
What benefits is AI bringing right now to to projects that
you've seen, Daniel? I think massive time benefits
already immediately, right, in terms of saving you all the
times you have to deal with things.
I think also it's saving you on cost later on because if you're
doing so much, the the theory is if I'm experimenting and
trialling things on several scenarios and getting a bunch of
these 80 percents and looking at it, in the long run, it's
helping me avoid issues that I would have that I wouldn't have
been able to recognise if I didn't try that experimentation
up front. But I think the most immediate
for me is the time benefits, things that used to take really
long time that have to be done are getting you most of the way.
And I think to Zach's point, as you use, if you do it with a
variety of LLMS, each trained in different ways, like Claude,
like Gemini, like Co pilot, like ChatGPT, you also can
triangulate pretty quickly as well.
And so I think there's there's like triangulation robustness
that I think in the immediate saves you time.
That's immediate. I think the cost stuff is, is
the things you would say based on comparison of what you had to
do in the past of mistakes you can avoid now because you've
you're able to experiment far more widely.
But I'd say immediately is the scheduling and time facets.
As a leader, so as a project leader, like one level up, how,
how do you lead on AI within, I know your department or you
programme or an organisation And how do you, I think, Daniel,
this is a point you, you made around making AI explainable in
terms of what it does and what assumptions it makes about
project delivery and outcomes. So, so to start asking people,
your workers, your Co workers to have that discernment, not to go
into this blind. I mean, it's pretty nuanced kind
of sophisticated stuff. If all you've been doing is
trying to experiment a bit with ChatGPT, you might not get all
of this stuff. So as a leader, as a project
leader, how should you talk about this?
How? Where should you be about the
language you're using and what you're asking people to to take
from what AI produces? I think you're really hitting a
profound nail on the head, Emma, because if you look at the
surveys with CEOs, executives, more senior leadership and
projects and organisations like Price Waterhouse Coopers did a
very nice one. And they basically said 61% are
like thinking this is going to solve a lot of issues.
They're going to increase productivity rapidly.
And then if you do, there's another interview by Upwork on
the day-to-day worker. Look at the, I call it the front
line. It's not a great word.
But let's just say the, the those who are nearer to the
locus of work and they find something like, I want to say 77
percent, 75 feel it's actually making them worse off and
burning them out. And what's happening is, is
you'd be sad. You know, we joke about the, I
mean, Zach mentioned the experimentation.
You would be surprised how many senior level people are just
saying this is going to solve everything without experimenting
with it themselves. And So what I think is
happening, the gap is, is an over expectation of what this
will achieve and how much judgement will be needed.
This is why I think it's really, really important.
First and foremost, senior leaders need to experiment with
it. And if you are actually, they
need to experiment this so they can calibrate their expectation
in judgement because they're looking at the kind of what they
see or hear or perceive and using that to drive.
Oh, yeah, yeah. Just do it for everything.
And that's creating a lot of burnout.
So that's one aspect. So the question is, and there's
different ways to think about it, Do you treat it like,
there's a very nice analogy by Ethan Malik at Gorton.
He argues, do you look at this as cyborgs or centaurs?
So the idea is, do you look at it as I even more pragmatically,
do I outsource bits and we look at it sequentially where AI does
something and passes it to me, etcetera?
Or are we doing it in parallel where you can't even see where
the human and AI end and there's different kinds of ways you can
use AI in your teaming to do that.
Do you treat it as another team member that you outsource stuff
and get back and sequence it? Or are they part of the ideation
where you don't even see where the the team comes and goes?
That's more like pragmatic. I think on terms of your
question about at the end of the day, we're dealing with a black
box where we don't know all the latent features.
There's a real push right now to get a sense of what is going
into the algorithms, how representative they are, how do
they change the data when they go in.
And there's some discussion about creating like prescription
labels for AI or nutrition labels with things like how
representative is the data? What data was this trained on?
Where is it vulnerable? How long is this?
How long is the insights I get from this algorithm worth useful
or not? Does it take?
Is it months? Is it days?
And I think even those simple things would really help people
understand. Yes, it's a black box, but given
what I know is going in and what's coming out, it gives me
enough kind of insight to say, OK, maybe I should try these
things versus another. To give a really tangible
example of management consulting you brought up, Emma, is that
they find they did a really interesting study at BCG and all
it was was literally cutting and pasting ideas into the into the
algorithm and there was a massive boost in performance.
I want to say like not massive, it was like 17% increase just
from copy and paste. But when they gave things to the
algorithm for which it was not equipped, sometimes these
algorithms have trouble triangulating data or even basic
math. They actually experienced a
percentage loss in performance. So part of the trick is they
have to understand what to give the algorithm and what is not
good to give. And so if we have these kind of
nutrition labels, prescription labels to say basically, yes,
these are the kind of things we're excelling at ideation,
brainstorming, recombining information.
And here's where it's not good, like triangulating between
spreadsheets and other things in interviews or even sometimes
basic math, then we're going to suffer.
So I think that and that's why it really it depends on senior
management experience. It would be great if they
experiment with their team so they can work on how they would
negotiate, how they would bring it into their protocols.
And from that they can learn more quickly what to share and
what does it get completely off so that you get the performance
benefits without the performance detriments, if you will.
That what are you talking about there is learning what
algorithms work for which data and that's the sort of thing
that. On a team, people individually
might be going off doing their own thing and not thinking
necessary strategically. So have you got any advice
around if there's a project team and everyone's doing their own
thing and they know they should be doing and it isn't the most
valuable thing there to share what you've learnt?
It is. And so as a leader, that's
something that you should be building into, building time
into regularly to make sure that everyone knows what everyone
else is doing and what works and what doesn't.
And somehow there is some kind of process around that so people
aren't wasting their time and everyone's reaping the benefits.
Because it's like you're very much working in, you know, with,
I imagine people who aren't that used to using this kind of stuff
in their projects or they're excited about it or, you know,
what advice do you give them? How do you see people use it and
experiment with it? And what advice would you give?
Yeah, absolutely. Yeah.
Well, I, I completely agree with all of the points that we've
talked about so far. I think that socialising, what's
working and what isn't within an organisation.
I'm personally a big believer and my background in startups
and perhaps show some of my bias, but I'm a big believer in
letting teams be independent and experiment with different things
and even in a bigger organisation and then have the
having the the successful strategies, the successful
projects rise up and then be disseminated.
And so I would encourage the way that I would think about it
would be to encourage teams to use different tools and
experiment and try, try out what the the new, the new breadth of
technologies, because we don't know what's going to work in the
long run. But we're still we're still in
the middle of this transition or even at the beginning.
And so there isn't one right answer that's going to at least
most organisations, all but the very luckiest probably are not
going to set one course through this storm and maintain that the
whole time, right. So the key instead would be to
develop systems, as you're saying I'm a really where the
best practises and the best new ideas coming up can be
socialised and spread more broadly and spread throughout an
organisation to have everyone on all the different teams follow
those best practises as they evolve.
I think it's really important to ask both of you where what you
think 2025 will hold. I mean, I know that it's hard to
look so too far ahead with this. And equally it can be fun to get
carried away the excitement to see what, what, what are the
possibilities, Zach? What, what would do you hope
2025 might bring? So what do you feel optimistic
about? And equally what what might keep
you awake in the middle of the night around AII?
Think that there's a lot to be excited about.
I, I personally believe that AI will be a paradigm shift across
industries. So in the long run, I think it's
of a similar scale to the Internet or personal computing
or these sorts of things where every industry, every vertical,
every specialisation, including of course PM is going to be
changed. And so I think that whether
that's in 2025 or in the next, in a few years after that, it's
a different question. But I do think that stay, I
mean, it's so exciting because there's so much possibility and
there's so much value from a business perspective to be
created and added and so much time to be saved and so many
more projects to be done. Just straight up, we can do so
much more. If we if we can work faster, we
can do so many more things, which is really exciting on a
human level. I think that just more and more
the ability to have AI agents are sort of these both somewhat
more autonomous tools that can have some long running context.
One of the biggest problems with some of our chat chat based
interfaces right now is that they lose that organisational
context. Today.
They don't know if you start a new chat, you maybe you told it
about some of your best practises and and that kind of
thing, but you have to start over if you start a new chat.
So more and more I think we're going to see specific dedicated
agents is sort of the term of art to have that long running
memory of here's how things work, here's how we do things,
here's how we did the last project even right, which is
something that short chat session would not have.
So an agent that has that long running context and they can
also bring to bear some of the general knowledge and general
skills to kind of get something done.
That's something I'm really excited about.
On the downsides, I think that we've already talked about it a
bit, but just there is risk with any, any paradigm shift this big
that some people in certain roles and even some entire
organisations that don't, all of which maybe don't adapt fast
enough to the, the new changes will be left behind.
And so I think that the biggest risk, I mean, the lucky thing
about this risk that I'm flagging is that we can all
still fix it. It's not the, it's not too late
to upskill yourself and your organisation and to adapt to
some of these tools and to keep adapting over the next 5 or 10
years and, and to, to weather that storm and learn what you
need to learn and build a more valuable and more powerful
really organisation in the longer run.
So there are risks there, but I think that we're it's still
early, so we can all we can all still weather this storm.
Thanks, Zach. That's so interesting.
Daniel what? What would be your take on what
what we can expect in 2025 and what might keep you awake at
night around AI? Yeah, it's hard to think of 2025
because, you know, if you talked McKinsey five years ago, they
were thinking, oh, it's 10 to 15 years from now.
We're going to have, you know, generative AI and language
learning. And then all of a sudden it
came. So I'm not sure when it will
happen. But my, my take away from this
is we got to think things that we thought were 5-10 years from
now could actually be next year. So I'm just going to say the
ones I think are interesting. I think there's a couple of
things. One is the powering of AI.
The amount of energy and water that's going to need to upkeep
the data is massive. So I'd be looking out for what's
going to happen in fusion energy, what's going to happen
in small modular nuclear reactors.
Because for example, the projections are one study out of
Yale basically said by 2026 even that the amount of energy that's
going to be needed for the servers is the same amount of
annual consumption in Japan. So we're talking massive energy.
The other one is cooling currently with just ChatGPT 3.
The the study I saw it uses something between 4.2 to 6.6
billion cubic metres of water. That's half the entire
consumption of the UK. That's nuts.
So look out of the technologies in terms of the cooling and the
powering. So that's one.
The other trend is how do I make AI work on small data?
Everyone's assuming big data, so I think you're going to see a
lot more on transfer learning. How do I use parsimonious data
that doesn't need an insane amount of energy?
So that's kind of 1, and I have some others, but you know, you
wanted to ask a question on this.
Yeah. No, no, no, that was that.
That was what I wanted to follow, follow up on.
What were the other things do you think are coming our way?
Not necessary, not necessarily for 2025, but on the agenda at
some point? I think an interesting one is
the developments in quantum. So machine learning right now
runs on what they call binary logic, right?
0 ones, typical computers, Quantum is game changing in this
regard because now assume instead of just having zero and
one, I'm looking at the probability of being zero and
one, now I'm thinking of a continuous distribution that
used to be binary and there is massive power that can be
developed from that. Now I think it's a ways away,
but look, if LLM happened earlier than we expected,
there's going to be interesting things of quantum.
I've already seen some of those methodologies used in marketing
etcetera. I think one that's more
immediate is AI going into the contracting or procurement
space. So the notion of smart
contracts. So imagine now instead of human
to human interaction or project, it's machine to machine.
So let's say company in my project, I'm a supplier A and
I'm the PMO and we've agreed that I want, I don't know, 100
widgets. I have an A sensor on the
machine building it that the minute that the widgets are
built, it immediately enacts the contract and delivers it.
It's entirely self governing. That's going to be interesting.
So you're going to see some interesting work on the
procurement space with smart contracts potentially on the
downside, the ability to discern between what is fake and what is
real is becoming really, really hard.
So there's going to be, imagine a world where someone has a
crisis, uses audio from a influential figure in politics
or corporation. You can foment crises quite
quickly. So we're going to have to build
rapidly our ability to better discern fake versus not.
And so looking at the regulatory space is going to be
interesting. I think the US, the EU and even
the Chinese are thinking differently about this.
I think the US is a market LED approach.
They want to see what the market does and let that kind of run.
I think the EU is taking a more safety approach.
They're worried about validating the data quickly.
This is why you see the EUAI Act versus how it is in the States.
And I think in China they're thinking about the integration.
They realise that some of this is going to be difficult, but
they realise they have to build their capabilities to discern
what's real or not. So I think the regulatory space
is going to be interesting to see about how we handle the, you
know, the proliferation of this stuff at scale, but also
recognising our inability to recognise synthetic versus not.
In some cases that may not be a problem.
In some cases you can obviously imagine it could be a very big
problem. I'm going to try and wrap this
up now, but are there any final thoughts that either of you want
to share? Or maybe I can ask you how
optimistical, pessimistic you feel about this right now?
I'm very optimistic. I think my my final takeaways
would be just to embrace, embrace this paradigm shift if
you can, as an individual contributor all the way up to an
executive. Anyone who's working on PM
problems, I would say just embrace, Embrace the change and
see how you can do your best to see how you can apply it to your
organisation over time. That's all that anyone really
can do. Daniel, do do you want to give
us your your final thoughts? How optimistic are you feeling?
Yeah, sure. That's, that's a hard question.
I would say I'm in the middle, right.
I'm measured about this. I think, I think there's a real
lot of potential. But we're also at the same time
really as a society, very ill equipped with the capabilities
to really harness that potential and where it where to understand
where it goes wrong. So my encouragement is very
similar to what Zach said, which is the need to experiment.
Everyone needs to experiment at a level of which to do this and
not to try to outsource it. The temptation is let me
outsource it to a technology unit.
Let me outsource it to my lower end.
You need to experiment it with yourself to calibrate your
judgement. And I think hopefully with that
as a society, if we can build the right labelling, the right
guardrails, I think it could really harness that potential.
But it's still early days. So I would say, you know, it
depends on what day you get me. Listen, it just needs to say
thank you to both of you for an interesting conversation that
could and should go on for another couple of hours.
So thank you again so much for your time.
It's been absolutely fascinating.
And I'd love to catch up with you again in the last six months
and see how far it's all moved since then.
So, so thank you very much for your time.
Yeah, thank you. This was a fun conversation.
Thanks, Ella. Thanks again to Daniel and Zach
for joining us and to you for listening to the APM podcast.
I don't think I've ever recorded such a mind expanding podcast
before, so I hope they've given you much food for thought and
practical advice on how you can use AI in projects and how to
lead on experimenting with it. But anyway, don't forget to look
out for more episodes or to rate and reviews.
Wherever you get your podcasts, we'd welcome you to get in touch
with your comments, feedback and suggestions by emailing us at
APM Podcast at syncpublishing.co.uk.
This podcast has been brought to you by APM, the chartered body
for the project profession. For more information on APM,
visit apm.org.uk.