Hi everyone, My name is Patrick Akil and if you're interested in
how they build AI features over at Microsoft Copilot, this
episode is for you because I'm joined by Stephanie Fisher,
Senior Product Manager over at Microsoft Copilot.
And a lot has changed the expectations for engineers and
data scientists, the focus on delivery as fast as possible and
doing that through many, many experiments.
I really enjoyed this conversation.
So enjoy. I was wondering because you you
mentioned that you were PM before anything copilot related.
Yes. What has changed now that
Copilot is there with regards to PM?
Everything, literally everything is start.
Yeah, such a good start. I think like what has changed is
what makes a good product, therefore what has changed it,
what makes a good product manager, right?
Because you have to adapt your skills to whatever you want your
product to be. And then also just, I think the
variety of things that you're now doing as APM is so different
than it was before. Yeah.
So I don't know which aspect. Why?
Why did the product change that much from your perspective?
So, so I was in, I was building office products before, right?
I'm a product manager for worst copilot right now.
Before that I was a product manager for like other office
applications in general. And I feel that back then
everything was about like a great product was a great UX,
right? Really good user flow.
For example, one of the products that I worked on just before
Copilot's game was, for example, multilingual proofing, which
meant if I'm in Outlook and I'm a multilingual speaker, like I
am like, that means that I'm sometimes writing in English.
That means that I'm sometimes writing in Dutch, right?
And it's really annoying if my spelling suggestions, if I'm
writing a, a e-mail that's mixed English in Dutch.
So that's squiggles of the Dutch, for example, you get red
lines everywhere. So I was building a product
related to that multilingual proofing.
And then like the thing that I thought through as a product
manager was just simply OK, what's you know, how will a user
use it? What is a good user flow?
What is the best UX? You know, what does the system
have to do? We're talking pre copilot and
pre AI about deterministic systems as well, right?
The system just had to work. And as a product manager, my job
was to create an excellent spec. And in that spec I had to take
all the edge cases into account. And that's what I did.
And then a good product was a great user flow, but also a
product that followed the spec to AT, right?
Right now, I think in AI we're moving towards outputs like it's
all about output quality as opposed to UX.
So we're moving kind of from pixels to tokens, right?
And no good UI right now in AI roles can make up for the fact
that your output quality isn't good.
So a great product now is not a great user flow, but a great
product is, is your output accurate?
Is your output relevant customer satisfaction?
And that also means that as APM, your job becomes really
different because I'm going to have to constantly test the
prompts that my scientists are using, are building on top of
the models and see, oh, how's the output quality on these
types of documents? How's the output quality on
other types of documents? Like, And then I communicate
back to my scientists and say, hey, listen, guys, like this is,
this is kind of the golden standard for looking for this
type of outputs or this is really bad, right?
But then again, like the systems right now we're building are non
deterministic like like we can't plan for the outputs, right.
We can build great problems if we don't know what the output
will be ever. So just kind of that whole
testing, experimenting mindset is something that's we never did
before AI world. Interesting.
With deterministic stuff you could aim for like let's say
your idea of perfection and then it works over and over and over
again. Do you think you can still do
that in this non deterministic way or will there always be edge
cases that you just can't accommodate?
For I think there will always be edge cases you can't accommodate
for like you can make a great problems, obviously.
And we have all these evaluations from the science
side as well where we can measure, oh, how well is this
model or this problem doing in terms of accuracy, relevancy,
all these types of things. But then again, like you never
know indeed, because it's non deterministic.
So yeah, so, so, so you can't, can't plan for everything.
How do, how do you decide then if you're experimenting with,
let's say a new AI feature, when to go live?
Because the only experience I had was we're working towards an
outcome and the outcome was making people more productive.
And even if our AI solution was only X amount of percent
accurate, I was like, let's do user testing.
Let's measure if we actually contribute towards that outcome.
And we also have to test kind of the engagement and adoption and
stuff like that. But even if we're not 95%
accurate, but we're hitting 80% and people are already
significantly more productive, we can go live and like release
that. What is your thought on
releasing new AI features? I think it's a good point.
Like as a like I do have together with my scientists,
just certain baselines that we want to hit in terms of metrics,
right? Like we do want the outputs to
be above certain accuracy. However, like the point that
you're touching on is really interesting because what you
also don't know now in using AI products is how people are
actually going to use it, right? And the only way that we can
know that is by bringing it out to users as fast as possible.
So I think as BMS, we're really and like as feature crews in
general, we're really going to have to switch towards a more OK
experimental mindset. Like my entire work right now is
building hypotheses super quickly, trying to validate
those hypotheses and like throwing them away and failing
fast and like modifying those hypotheses.
So for me the process or or or the time towards shipping at
least to internal users has become much smaller just because
I want to see, OK, but like we can test everything within our
own environment, but then we we can see all the accuracy,
relevancy, all those kind of metrics.
However, I've no idea how this will behave in a real world
indeed because of the non deterministic aspect of it.
So, so like my whole mindset of me mindset of me and my crew is
to ship as soon as possible and then see real world usage
because only then also you're going to encounter those those
edge cases and moments where it doesn't work, right.
Yeah. Do you always ship kind of in
this phased approach to internal users first before releasing to
the wider public? Yes, yeah, yeah.
That's how we've always done it at Microsoft.
So we have, yeah, First we ship to internal dog food, dog food
users. So those are users who signed up
for. I want the earliest earliest
features. We call them dog food users.
We call them dog food users, then we shift to the interactive
Microsoft and then we start shipping to worlds towards
external users. But what is really just like
anecdotally, what's really interesting as well as before
AI, we yeah, yeah, yeah, yeah. Before our AI world, whenever we
were building a product like before we would even release it
to those dog food users, we would have a certain confidence
in a product. It was almost ready for external
users as well. So the only reason why we would
then ship to internal users is just to, you know, catch some
bugs, etcetera, you know, see if there are certain parts of the
products that aren't working. But at least back then, we had
almost like a 9095% confidence, OK, the product is good enough
to ship to external users. However, like talking again
about that experimental mindset, that is not something that I
want to do anymore. I don't want to kind of siloed
together with my scientist, you know, try to optimize before
we're even going out to actual users.
Instead, I think we have to go to actual users as soon as
possible. However, like imagine the shift
within our organization that needed to happen.
Like, I think a year ago I was working on a pretty
controversial AI product, and my idea was, OK, you know, let's
just ship it as soon as possible just in order to validate our
riskiest hypothesis as soon as possible.
But imagine you're a dog food user and, you know, you're used
to seeing stable products, and you're used to seeing products
as well that are meant to, you know, shift to actual customers.
And then suddenly you're getting something that is like, to be
frank, pretty unfinished. I was discussing with my
software engineers, okay, you know, what is the bar?
What is the bar in this new AI role to ship to dog food users?
And we called it the non embarrassment bar.
So the moment that there are not like it doesn't have to be
great. It absolutely doesn't have to be
great, but if we're not embarrassed, let's just ship it
to users, internal users and see what's happening.
Okay, how was the feedback? In the.
Beginning. Yeah, yeah, I don't see was
yeah, yeah, yeah. Because you mentioned you're a
user and you think, oh, gosh, these products are actually
going out to users, you know, to external users because that's
what you think is happening. Interesting.
Yeah. So I got so much negative
feedback of people being really concerned.
You know, you can't shift this to to user.
And I'm like, yeah, yeah, yeah, I know guys, you know, but this
is just part of the new experimentation products.
Yeah. The process.
I, I had Sam Bacon and he's the CPO of Veed, which is like a
really big, I think it's the biggest online video editing
tool out in the market. And he says what distinguishes
us, and we used Google as an example, is that I can talk to
customers like this and we try and do that every week.
How easy is it for you to not just talk to dog food users but
actually customers in the end? Real customers I, I think the
great thing about working at such a large corporations that
we have a ton of structures in place where we can actually talk
to enterprise users like real life enterprise users.
So we have programs in place where a ton of enterprise users
from enterprise companies sign up and they say, hey, we want to
develop these AI products together with Microsoft.
So you know we're delivering features to them really early so
it can they can give early feedback.
So in that sense it is pretty easy.
At the same time, like the hardest, like like again, the
hardest thing is when I was building products before AI and
again, it was very much based on user flow and UX, etcetera.
I could go to platforms like user testing.
There are platforms that are user testing, right?
And just people all around the world can sign up.
It was very easy because you could just like show a flow and
think, man, people would comment on it, right?
And then you would have at least a sense on how it would resonate
with users. However, right now, in a world
where it's so much about output quality, you can't do that
anymore, right? Just because people can't.
Yeah, yeah, yeah. And people can't really test it
out anymore. So.
So it so it is hard, but there are avenues simply because we
have the benefit of being a large corporate and with our
name, Yeah. I.
Think it makes sense. Earlier you, you talked about
kind of this change in behaviour, right?
First delivering very high quality and also going to your
dog food users with the assumption that this is
basically a formality and it's already there.
We're just trying to iron things out.
Yeah, in case we miss things. Yeah.
But also from your perspective as APM it it sounds like you
made this shift like no problem at all and you're having a
blast. Which is amazing.
I grew up in, I, I don't have much PM experience and I did all
of a sudden become responsible for an application and that also
had a Gen. AI component.
And because I think I have this engineering background, I always
like just shipping and then seeing, OK, do we meet our use
case? Do we meet whatever we aim to
achieve? And if that's with 80% or if
that's with 100%, I don't mind to be honest.
And we can move on and deliver value in other aspects.
So I've always kind of had that. But how is that shift then for
you? I think I also maybe had the
benefit that I was a software engineer before being APM and I
was only APMI think for a year, 1, 1/2 year before the whole AI
stuff hits. So I didn't have like a ton of
habits or something like that, right.
And what I think is kind of ingrained in engineers, which I
noticed with myself and also with my team, like, yeah,
shipping thinks it's fun, but experimenting is also like so
much fun, right? So I could very like, I could
very much like, easily tap into that by myself, but also like
getting my engineers excited about just running experiments,
looking at scorecards. And that data came pretty
naturally, to be honest, just because, like, it's, it daps
into some inherent curiosity, I think, which worked really well.
That's amazing. Yeah.
Yeah, yeah. Has your expectation of software
engineers, let's start with software engineers first changed
throughout kind of this experience?
Yes, yeah, very much I think. I think as a like a, a good,
this has always been my vision about product management.
I think very traditionally, like really back in the days, it used
to be pretty hierarchical in terms of how you build products.
So the product manager sits in like an how do you say the ivory
tower and you know, defines what needs to happen and writes A
spec and then throws that over to design and design makes some
sort of engineer design makes some sort of design, and then we
throw that to engineering and engineering just needs to
execute, in my opinion, from like PM.
This is like not how you shoot PM anymore.
I really see myself as APM that's facilitating all the
other people in my team to be brilliant at what they do.
And I really think product building is like an equal job
between design, engineering and products.
And now with the whole shift towards AI, where there's indeed
like tons more experimentation, etcetera.
I always also expect from my engineers that they step up
their game in terms of experimentation, but also step
up their game into what are those experiments that we can
run, AKA what would resonate for a user in your opinion as a
software engineer, right? So I'm really trying to
stimulate them to also think about the user and the product,
etcetera, because I also don't think that should be my role
anymore. I have a pretty large team,
right? I think I have like 30 software
engineers, scientists, designers.
It's really big. Yeah.
OK. And that means I, I can't make
all the decisions myself. You know, I really, yeah, really
scary. Yes.
And I also don't want to do that because, like, it's not the
product. It's not going to be the best
product if I'm by myself going to think through what they, what
the product should look like. So I'm really trying to
stimulate my engineers and my designers into product thinking,
into users thinking. Thankfully they are really great
product thinkers. Like all my engineers have a
really good understanding of customers, etcetera.
But there's also some work to do.
So I invite them over to user interviews and those kind of
things just so that they can get a better product sense.
But because, yeah, I think they're going to need to step up
in that regard as well. And they're for for everyone, by
the way. I think PMS are going to need to
learn how to code better and design better.
Designers are going to need to learn how to code.
You know, engineers are going to need to learn how to PM.
Like, it's becoming so much more.
Don't worry. So much more intertwined
everything. Yeah.
Yeah, what, when is the right time to involve engineering?
Because I, I only have experimented with this.
I did a kick off new milestone, engineering stakeholders all in
one room. Let's discuss kind of the pain
points and the problems. Yeah.
And the feedback I got from some people, really enthusiastic,
really excited from the beginning.
Other people like this is too vague for me, like there's no
structure yet. We don't know what we're doing.
Yeah. I feel like it's a waste of kind
of my energies and effort. Yeah, interesting.
So then I'm like, OK, next milestone, I, I just do a raise
of hands who wants to be there from the beginning?
But then I think it's valuable, but I don't want to put this on
people like it's always been this balance for me.
Yeah, I think it is a balance like, but when I was a software
engineer, one of the things that I used my PM for most is really
understanding, oh, why am I taking up this user story again?
Because like, if you're an engineer, you're involved so
much in the reason you're kind of forgetting the big picture,
right? And I think that is the main,
the main difficulty that my engineers have.
And I constantly have to remind myself like, oh, I need to
communicate to my engineers like, why we're actually doing
this? Like what are we contributing to
and what are they directly contributing to, not only in
terms of product, but in terms of success, You know, so we
obviously want to increase engagement, activation, like all
those sorts of metrics, like, and how does that trickle down
to them? Because it gives them kind of a
sense of, oh, This is why I'm working so hard on this.
Yeah. Purpose.
My sense is I'd love to involve engineers from the beginning
when I have an idea of, hey, this is the user problem, right?
So I've talked to a bunch of users and kind of consolidating
all of that information. Then I'm presenting that out to
my engineers. I'm like, OK, this is the
direction that I'm thinking in, but I'm also open towards
thinking like what you guys think.
And the same thing with designers and scientists as
well, by the way. I think we didn't work with
scientists so much before, but they're so important right now
in our work. So it's an entirely different
type of colic that we're going to have to involve from the
start. Yeah.
Are they ingrained and kind of
part of the engineering team? Yes, it is a separate team with
like a separate manager and things like that.
But I usually work in we call ADRI Trios.
DRI stands for designated responsible individual,
something like that, whatever, another three word acronym.
Anyway, it means there is someone responsible for every
discipline. So I'm obvious, I'm whatever
product I'm building, I'm always putting together APM.
That's me, a scientist, scientist and engineer and a
designer. And with the four of us, we're
responsible for the products. Like doesn't matter.
Like I don't care if you're an engineer, like if something
falls into your place and I'm not there or I'm busy or
whatsoever, you should solve it. Like we're really like with the
four of us responsible for the products.
And all those the same people kind of over and over again.
Or do you switch with regards to the people in those little
teams? Yeah.
So it's a, it's a good question. So as I mentioned, I have a team
of 3035 engineers approximately and within that we're building
separate like all different products, right.
So it's usually like well 35 / 4.
So that's about like 8-8 different products are being
built or something like that. So it's like constantly like
different people and working on. Different products and you're
the only product person that's there, yeah.
Wow, I wish I had to help but I don't.
Yes, that's super. Interesting.
Like I've been in organizations where there's teams to the
smallest level, 3-4 engineers, then there's also PM there.
And then there's this hierarchy also of PMS, and they have one
person that's product manager, and they also use terminologies
of like product manager and product owner, kind of, Right?
Yeah. And then in the end, there's one
product manager without a team that just has a team of product
people and they have smaller teams and like, how does this
work? Yeah, Yeah.
That is interesting. Yeah, we don't have that.
And again, like I'm, but I'm outsourcing a lot of my
engineers as well. I think like the PM role is very
confluent as well. It means so many different
things in many different organizations even need like
product owner, product manager, project manager, etcetera.
For example, as APMI personally don't run ADO boards or
something like that or sprints or whatsoever.
Like my engineers are responsible for that.
They can do probably much better than I can.
I don't add any value to that personally.
It's not like within my skill sets.
So I'm very consciously trying to think like, oh, what can my
engineers do? What can my designers do and
what do I do? And that's always kind of
through the lens of. Where do I add unique value now?
What are the decisions that you do take part of?
Do you have cycles of retrospectives and plannings and
forecasting or like OK ours, stuff like that?
OK, ours, yes, that's a big one. That is a big one.
Yeah, yeah, obviously we do make OK ours I think every half year
or something just to know as an organization or as like a a part
of the product, you know, what are the metrics that I want to
move? And then obviously I have a like
within Word copilot, I'm responsible for making people
understand documents better. Like there are different jobs to
be done whenever a user is opening a Word document, right?
Like it is understanding a document or writing a document,
one of these things, usually I'm responsible for that
understanding part of it. So, so like I'm responsible for
the road map within that part. So part of my job is just
thinking about, OK, you know, what should AI help users with
with this regards like upcoming year?
So I've kind of a a backlog or things that I want to jobs to be
to be done that I want to improve.
And what were you asking again? What?
What kind of cycles do you take part in with the team
interaction? Yeah.
So I make OK Rs for that. And I have like a road map in my
head. And usually within Microsoft, we
had planning cycles every three months or something.
So every three months we would decide, oh, this is what we're
going to work on. After months, that doesn't work
anymore. And I really like literally like
it's too long. It is way too long.
It's way too long like. Three months already is too
long. Yes, yeah, three months already
luck, you know, like I mean you can make all the plans in the
world that you want and then next week there's a new model
being released or and and then you're done.
I mean, I love this because always when people are like,
where do you want to be in five years?
I have no clue. Yeah, this is no clue.
Insane. It is insane.
Like, yeah, like big companies, you know, they're just like
created out of the ground and then they're like next week
they're acquired, next week there's a new model.
It's like insane to keep track of, like what is happening in
the industry. So you can't plan for three
months. So we're trying to plan with my
future crew every month. Yeah.
So every month I'm kind of looking through whatever
priorities in my opinion to work on.
Then we're just like prioritizing them.
And, and again, it might happen that during that month we're
switching gears just because something else became more
important. Yeah.
I was wondering how you deal with bias because if I'm
responsible for kind of understanding a document, I do
that on a day-to-day. I open Word, this document, I
have to understand it. I've done this so many times,
so, so many biases I feel like. And then also when you have 40
other people that contribute towards making this a better
product or creating products and like improvements as part of
that, how do you deal with bias and kind of remove that from
people's way of working thinking?
Well, I think we should do this. And that really anchors people.
Yeah, because that's such an interesting question because
what you don't want as a culture within a company is that you
always have like these visionaries and innovators,
etcetera, and they can talk really well and they've really
great ideas, right. So it's really easy for them to
influence the product just based on their gut feeling or like
their visionary thinking. But that's not how we should
build a product. Obviously in every product
there's a bit of a bit of gut feeling, but that's not the way
that we need to make decisions. So coming back to that
controversial product that I was talking about, a ton of people
didn't like that I was building it.
Like absolutely not all the way to our leadership team.
A ton of people didn't believe in it.
The only way I could prove that it might have been a good idea
was to experiment. So like purely data-driven,
looking at the metrics that we care about and only
communicating from that angle. So, so that's the best way to
remove bias in my opinion. Like everyone can have opinions
and that's great. I also have opinions and biases
and those have been disproven so many times through running
experiments where you feel like, oh, this is for sure going to
move the metric this or this way.
And it doesn't at all. It does the opposite thing,
right? And that is super exciting.
So we're doing that and we're actively communicating within
our organization as well, whatever our findings are, how
we're experimenting, etcetera. I like that a lot.
I thought you were going to say because people that are
stubborn, they like their own ideas, right?
And they fall in love with their own ideas.
So then when metrics and data comes and disproves it, it's
like sad and disheartening. And the next one would be even
harder. But I like you're saying it's
super. Exciting, yeah.
That means we can do many more things.
Exactly. Go ahead.
Yeah. Like I have so many like dailies
with my engineers. We're looking at scorecards and
we're like, oh, wow, this is like completely opposite to what
we were thinking. And then we're discussing like,
how come that we had that gut feeling, right?
And what, what might these numbers actually say?
So what do we have to rewire in our brain?
So it's a really cool culture to build, kind of this experiment
called experimentation culture. Yeah, Yeah.
I can also see that it's very challenging because quantitative
data is one thing. Yeah.
And I think it's great to rely on that.
And there's also then the qualitative part of it, like if
you're doing things with AI, you're super trailblazing.
So even if the data might say things yes, like talking to
people might say otherwise, it's very hard to balance both
feedback points. You're yeah, you're totally
right. Also because like users don't
like users usually don't really know what they want, right,
Obviously. And especially in AI world, it's
super hard. Like what do I have to build for
you next week? That week?
I wouldn't even imagine. I would have to see something
and be like OK, just. Click exactly and you're like,
ah, this works or this doesn't work, but it's so hard to
rationalize. So the way that we now interact
with like I'm finding those user problems, finding, finding the
things that we need to work on really changed as well.
Because like pre AI, again, I gave that example of
multilingual proofing, right? Those were established.
Like Outlook, for example, was an established product and we
had like we had a backlog of users, like of of things users
wanted us to work on, but that was all documented really
nicely. You know, customers would reach
out to us through different platforms.
You know, we knew which features they wanted.
It was just a matter of prioritization basically.
It's not even that long ago, and it feels like so it.
Feels so peaceful. Yes, yes, it feels like
prehistoric time, you know, but now indeed, like you can't
anymore. So so that also really changes
how we're engaging with users. So what we now have to do is
actually literally sit next to those users and asking, OK, just
tell me about your workflows. Like what are you doing with
words? And then just observing really
well and then understanding like, oh, in these kind of
moments in a workflow, we can inject AI in a very natural way.
And that's the only way that we can kind of have that discovery
process now. But it's super hard, so
difficult. Yeah, discovery teams, I think
those are going to be more and more at the forefront.
I think so actually. Kind of pairing with users
observing, users interviewing, investigating.
And. Everything is still an
assumption and then indeed testing.
And then we're just experimenting.
Yeah, exactly. And that's also, I think, a
really big part of being APM. Like you can't get away with not
deeply understanding a user anymore, right?
You really have to deeply understand your customers at
this point. Yeah, yeah.
I get that. When everyone, or at least when
you're in an organization where there's a lot of PMS and
experimentation as a culture is kind of coming to the forefront
because we have to, we have to deliver and then validate and do
that over and over again until we have kind of a proof point of
value. Does data become an issue?
Do you have, for example, enough data or access to data to
validate those assumptions? Or does it become a bottleneck
because everyone needs it? Everyone needs it.
Yeah, it is a bottleneck in a way that we just, I mean, we
have so many users, but still it's not an infinite amount of
users, right? So we can only run this very
many experiments in parallel. And that's my main bottleneck
right now. I've like 12 experiments that I
want to run like at the same time, but my user base isn't
that big. Like I count AB test 12
different experiments, I still get statistically significant
results right? Because we only care about when
it's statistically really proven.
So yeah, the bottleneck right now, which is a luxury position
to have though like to want to run so many.
Experiments I want to do 12 yes. Yeah, yeah, yeah.
But right now, at this point, I think we're even that far in our
experimentation culture within my team that if I didn't have
this as a bottleneck, I would be worried.
I'm like, oh, Stephanie, are you really experimenting?
You know enough? Yeah, yeah, yeah.
At least this tells me, oh, there are so many hypotheses
that we're trying to investigate so that that that's cool.
Yeah, that's so cool. Yeah.
And you mentioned very strongly that you don't want to be the
one person that comes up with all these experiments.
So all these experiments are like a combination of all the
disciplines coming together to try and solve this.
Problem. Yeah, yeah, yeah, for sure.
It's super cool. It's it's on every angle as
well. It is designers wanting to try
like a few things and see like how slightly different elements
and design changes engagement, right?
These engineers who are, for example, we're now very deep
into chaos testing. So engineers who are trying to,
I don't know, mess with our house booth and see how that
influences engagement. This is me trying to experiment
with different like features or or whatsoever and science when
they can do things. So yeah, it's a really
combination of things. It's still I think as ABM
responsibility to kind of create order in that chaos of like of
excitement and and think through, OK, this is a priority.
This isn't priority. But yeah, it is sourced from
everyone for sure. Have you ever had an experiment
where there's a really good outcome, like things make sense,
but from your product vision you're like, well, it's just not
a good fit? Like it's not what we're trying
to achieve with this. I've had that and therefore it
is really important as well before you run an experiment
like, OK, what am I going to do with this with the outputs?
Because it's also very natural to be like, let's let's
experiment of everything, you know?
But if you're not going to take an actionable result based on
that experiment just because indeed, like even if the outcome
is this is great, it just goes against your product feeling
that you just like, why are you running the experiments in
general? It might be to gather
information, and that could be valid, yes.
But then it's something that we're discussing upfront with
the engineers, like, hey, we can run this, but we're not going to
make any decisions based on this.
It's just to gather data and information.
Yeah. So that happens.
Gotcha. Yeah, I was wondering because
this is mainly on the metrics level.
I was responsible for this Gen. AI part and then it was like,
OK, what are we aiming for with regards to accuracy with regards
to like time reduced, I can have kind of a guesstimate metric,
the dot on the horizon. So that's what I did.
We're aiming for these things and then continuously we're
evaluating if those metrics still make sense.
How do you put your metrics kind of on the table?
Because you mentioned with together with data science, we
do have a few metrics that we're like striving for, but how do
they originate? How do they originate So, so it
kind of boils back maybe to like OK Rs.
Like, I mean, I don't love OK Rs in terms of like writing them
simply because it's hard, right? But I think it's also like a
tendency to like, it's very easy to say I'm writing, OK, ours and
let's just let's just increase engagement and that's it, right?
But you really have like what you have to do is get back to,
oh, like, what am I really trying to do for a user, right?
And like a user's not necessarily helped with them
clicking on something more, right?
They're trying to achieve something else.
So kind of writing that down kind of like and you said time
to what, what were you trying to optimize time to?
Making people more productive, so we're losing time spent.
We're losing time spent. Yeah, exactly.
But then so that is a really great, I think overarching goals
which you shoot right down and then kind of distilling it back
to, oh, that actually means that is obviously the hard part.
I found that it really depends on the maturity as well of your
team on how, how good you can or, or which metrics you can
define. So for example, at the beginning
of Copilot, we're really deep, like diving deep on those
metrics for the first time. To be honest, in Copilot in the
AI world, just because it was starting, we were only looking
at, yeah, excavation and engagement, right.
But now like 2 1/2 years in or two years in or something like
that, we're starting to see more like, Oh yeah, OK.
We're we can't go for like optimizing productivity to
engagement. What is like, what is in between
and how are we, how can we define that?
And it's just a brainstorming exercise, to be honest, between
ton of people, usually between PMS and managers, like
engineering managers, but it's hard.
It's such a hard thing to to create.
Yeah, I feel like. Listening to your story and kind
of the the scale of the operation you're in, you're at
least engaging with a lot of people.
How do you build kind of those relationships to have trust in
the other people that they have kind of the same intentions or
build the same level baseline of way of working and also
communication cultures might be different completely.
I feel like you need relationships to thrive.
How do you build? Those.
With so many people. It's, it's a really interesting
question also because like you were chatting before the
podcast, like all my engineers are in Serbia, right, Which is
they're physically distant from me.
It's like an entirely different culture.
I've personally, and, and then like I'm also working with a ton
of folks from the US, like, like none of my colleagues are in the
Netherlands, right. So there's always.
Like the only one. I'm literally the only one.
Yeah, literally the only product organization in the Netherlands
sits here. All of it, all of it.
Like I personally always come from deeply understanding the
person up so well. So like eventually your
colleagues and like the the engineers you're working with
are also kind of your customers in a way, right?
You can see them like that. And just like you're trying to
deeply understand your customers, you have to deeply
understand your engineers as well.
So I'm always trying to come from kind of the human and go
things like, oh, why are you like, why are you here?
Why are you working in Microsoft, right?
Like what are you trying to achieve as an engineer?
And how I also see responsibility for me as a PM,
like how can I help you as an engineer reach your goals,
right? And that means, so for example,
I've had engineers that I knew had a passion in something that
I couldn't execute it well. That's great.
And it's like on me to make sure that if a project like that
comes along, I can actually put them on that, right?
Or Hyatt engineers reaching out and saying, hey, Stephanie, I
really want to improve my product thinking.
And I sometimes have ideas, but I feel afraid of speaking up in
a large group. So can you help me with that?
And then we have weekly calls where I'm just listening and
they're pitching product ideas to me.
And I'm saying, oh, yeah, you know, you did this work as well.
This doesn't work well. Oh, maybe you can do this.
So bottom line is really deeply trying to understand people
helps building trust so much, I feel, because then you kind of
mutually understand what they're coming from, where they're
coming from. And especially also in
organization of Microsoft. Like I, I work in words Copilot
and then there are like 100 different products that
Microsoft is building and I have to communicate with and, and
align with those kind of product managers as well from different
products. And sometimes obviously you see
clashes, but those clashes always are coming from the fact
that's what that person wants to achieve.
It's just simply different from what you're trying to achieve.
However, do I really understand what you're trying to achieve
the other person and why that is?
And if I understand that, then I can maybe find an angle or a
way, you know, that's like that, that that's going to help both
of us. But if I don't ask you, if I
don't deeply understand you, I have no idea.
And that's where you're getting these, like, surface level
crashes, right? Yeah.
I really recognize what you're saying and it's it's mainly what
I see, what I'm just observing when I see two people trying to
push their opinion on the other person and they're just going
past. Each other.
Yeah, and they're not noticing. No, no, exactly.
And they're not listening either.
No one's asking a question. Yeah, exactly.
And there's probably like a base of understanding like they're
shoot, there's somewhere a base where they can understand each
other, right. But someone communicates it like
this and it doesn't resonate with the other.
So indeed, there's push back, right?
Even though those people would step back a little bit, we would
like find some common ground from where we can then, like
collaborate together. Yeah.
I feel like the people that are curious kind of innately also
not just about products, but also about people where they're
coming from, behaviour. You even mentioned kind of
ambitions and goals. I think that's fantastic,
especially as a person that's responsible for product and kind
of work that gets done. You are equipped to put people
in positions to thrive and if that, with their ambition and
that's perfect. Yeah, yeah, exactly.
And like eventually that creates a better product as well.
If I have happy engineers working on the thing that
they're so passionate about, that's only good for the
product, right? So it is in my best interest as
a product, as a person responsible for the product as
well, that my engineers are as happy as possible and know what
they're working on, they're excited what they're working on.
It might be the most kind of undervalued statistics.
Happy engineer. Yeah, Yeah, genuinely.
Yeah, maybe it should be an OK, honestly, I think generally like
people's skills in that way and deeply understanding people's a
very unvalued skill. And like we always think about,
oh, you know, like is it product manager or even an engineer or
you're, you're as good as like the lines of code that you
write, etcetera, but you're not. I was a when I got hired as a
software engineer. I had absolutely no software
engineering experience in real life.
In in real life roles obviously like coding by myself.
Otherwise I wouldn't. Get a job, otherwise I wouldn't
get a job. But not in any real life
systems. And, and so I got a software
engineering job at Microsoft, right?
And it wasn't because of like the brilliant code that I wrote,
but it was because I, I was a customer facing software
engineer. It was because I was good at
understanding a customer and really trying like trying to see
where they were coming from and what we needed to build and kind
of, you know, build a bridge between the customer and our
engineering team and not because I was like a the 10X coder, you
know? Yeah, yeah.
Yeah, I feel like that is such a life skill.
Like you had that as a person, you got a job as a software
engineer because of that. You're completely leveraging
that and the role that you now. Have.
If there's ever another role, it will still be valuable.
Yeah, yeah, yeah, for sure. Yeah.
It's like those transferable skills, right?
Yeah, absolutely. As APN because you're also the
only person in the Netherlands who challenges you like from a
craft perspective. That's an excellent question.
All my engineers, honestly. Yeah, yeah.
I love it when I'm like kind of building a first version of a
spec and I'm only building fee 1 specs at this point just because
like I know that whenever you do, you get that feeling
sometimes that you're building a spec and you've put so much love
and effort into it and you're like, this is perfect and you're
sharing a doubt and you get so much feedback that yeah, so many
holes that it almost feels personal, right?
I cannot do. That.
Yeah, really bad at that. Exactly.
So I'm like, I'm just like going to make like OK, fee 0.1 specs.
I'm just going to send it out to all my engineers and my, and,
and what I mentioned I'm very lucky to work with engineers who
are so product and customer focused.
They constantly challenge me. And then also kind of my my
direct people that I work with are the engineering managers and
the science managers. I have like often more than once
a week I have meetings with them.
We're just discussing products and ideas, etcetera.
Yeah. So those are the people.
Yeah, I heavily rely on them. Yeah, yeah.
Some organizations that I've seen, I go in and out have like
this, OK, you are part and responsible from your team, but
then you have alignments with people like from the same craft.
So I thought you would have a structure like that as well.
But I really like that the challenge like on a day-to-day,
I feel like it should come from a team.
I think so, to raise the bar together.
Yeah, exactly, exactly, exactly. And it doesn't become so siloed,
right? You make like everyone
accountable for what you're doing.
And obviously I have other PMS, but I'm not, no, I'm not
necessarily, they're not necessarily challenging me on a
daily basis. It's not the same.
Yeah, they also, they're also not so deep into like our aspect
of the product that we're trying to build.
And my engineers are right, yeah.
That's super exciting. Yeah.
You said you were lucky that you have people like that.
But I feel like, I mean, if you were my PM, kind of the way you
think about product and also the importance that you put on kind
of feeling that other people need to think alike and think
about the product. It's kind of also the culture
that you've cultivated. Thank you.
Yeah, yeah. At least that's my assumption.
Yeah, that is it is. It is like I'm not trying to
like my own horn, but I think it is something that I get back
from my engineers and because I'm not like I've been product
managing for not long, right? Like a few years, maybe 4 or 4
1/2 years, something like that. So on paper, I'm for sure not
the best PM that's out there because like I am, I work with
people who've been PM ING for 15 years or longer, right?
So they're probably better in kind of the, the technicalities
of PM ING. At the same times, my engineers
tell me that they really enjoy working with me indeed because
of that reason, because I set a, I think someone even said at a
certain point like, oh, you really put back the kind of the
enjoyment in my work, right? And I think that's indeed what
can really make you stand out as APM, just creating that culture.
That's a beautiful compliment. It was such a good compliment.
My job is done, you know, I can retire.
Yeah. Are there because you went from
software engineering, now you're PM and now you've dealing with
kind of a lot of disciplines. Is PM like the thing for you,
the thing that makes you most happy?
Thank you so much. I love Yeming.
Yeah, it's so good. It's a bit of everything.
It is a bit of everything, yeah. And even before, like my road to
here has been super weird, right?
I studied hospitality management.
So I come from a hospitality management background.
I worked in hotels before moving into tech.
I moved from hospitality management to tech by doing
sales. So I was first doing sales and
software engineering and PM ING. So I've seen like a whole bunch
of weird stuff. And I think this is indeed like
the perfect combination of like what I love about hospitality
management is the whole customer focus, right?
And I can totally use it here. What I love about engineering is
that we're actually building products.
I can totally do that here as AVM.
I write marketing and stuff like that.
So kind of the sales element isn't there as well.
Like it is such a perfect blend of everything.
And then it makes me so happy to work with people who are so good
at what they're doing, right? There's engineer scientists and
designers. And as a PM, just like, wow,
this is amazing, you know, and we can build cool stuff
together. So yeah, this is like, I'm
staying here. Nobody's kicking me out of PMA.
Yeah. I can truly relate to that
experience. Like as PM, I feel like still
you're kind of in a leadership position.
And I truly believe that the way you behave, the way that every
leader behaves, like it trickles down into whomever you engage
with kind of in expectations behaviours.
Do you own up to the problems that you've caused or like the
the things that you've done, or do you shy away from them?
Or do you like law or do you like, are you integral?
All of that matters. In the people that you engage
with, Yeah, yeah, you're right. Like you're responsible for the
culture. Like not only for the product,
but for the culture as well. And the culture trickles back
into the product to which is really cool.
Yeah. And like, the great thing about
PM ING right now is like, it's so exciting to be responsible to
think of. Oh, you know, what should AI be
like in five years? Indeed, as you were saying,
like, it's trailblazing. Nobody.
Nobody knows the answer. And it's so cool to be
responsible for thinking through that.
Cool. Yeah, I have one last question.
Yes, and that's more from a tooling perspective.
Are there any tools that you've used AI related that make you
more productive that kind of yes help you in your product?
Yes, absolutely. Which ones do you love?
Which ones do I love? I love Cursor, to be honest.
Yeah, I don't do a ton of five coding just to like make designs
and prototypes really quickly because I think that's such an
easier way to communicate your idea.
Like back in the days as well, when I had a feature idea,
right, and I needed to convince people I would probably need to
engage with designers. Designers wouldn't need to make
time to kind of get my thinking on paper.
Yeah, back. And forth is not right.
Back and forth exactly. Maybe you would want to engage
in engineers, make like a quick prototype or something like
that. Well, how long would that take?
Like maybe 4 weeks, six weeks or something like that.
Now as APMI can just sit, sit behind my screen, five goes
something in a day, shared up with people and people are like,
oh, this is a good idea or not a good idea.
Like the velocity is so much faster, which is really, really
cool. So for short cursor, for five
coding, I've tried obviously like lovable V-0, like all these
kind of things. SOPM it's also your job to try
out like ton of things play around.
Yeah, obviously I'm loose using ChatGPT a lot as well.
Also for consolidating user interviews, doing research, you
know, all of those things. New entropic opus model is
really cool. I'm not sure if you tried it.
I I read about. It it's so good, yeah, it's so
good as APM. You're also like, you have to be
so model forward thinking, right?
So you have to understand what each model can do and like how
to grow prompts and how to get the most out of your models.
So whatever works there, like can be, it can be claws,
whatever. You're also like super pragmatic
with regards to getting stuff. Done but.
All these responsibilities and now also data science kind of
coming on your plate. Is it too much like from a
product sense? Do you think it's getting too
wide with regards to things you need to know or kind of need to
at least have an idea of? One interesting question I
think, I think I had to, I had to shift a ton of my
responsibilities to engineering and designers like making like
small decisions, etcetera. I don't want to be responsible
for them anymore. I'm just kind of observing and
if it goes completely wrong direction, I'm kind of steering
them. But like small decisions totally
up to them. That makes that I have much more
space for kind of experimenting and like vibe coding and
understanding models etcetera. So it's indeed about being super
deliberate with your time. And like what I mentioned
before, thinking about, oh, where do I add unique value,
what is super relevant of my role and that is understanding
the industry, the mark, the models, et cetera.
And then all of those product decisions, I just trust other
people to do that because I've created that trust with the
team. Yeah, that's amazing.
Yeah, I've really enjoyed this conversation.
I think you have super cool energy, super pragmatic,
passionate about what you do. Thank.
You so much, I really enjoyed it as well.
Thank you for inviting me. Appreciate it.
Of course, we're going to round it off here.
Let us know in the comments section what you thought.
Like the episode? If you did, it's free, so click
the like button and we'll see you next time.