Hey, a quick message, for those of you who are listening to this
episode on Spotify, I have a small favor to ask Spotify.
Now allows mobile users to rate podcast, I would really
appreciate it. If you can take a quick, pause
to go to the technique Journal podcast page, and leave your
favorite show. Your best rating on Spotify.
It will help me a lot to get this podcast to reach more
people on the platform. Thanks a lot.
I'm going to take this from Elizabeth Hendrick since one of
my favorite quotes testing is an activity, that That happens
throughout. It is not a phase that happens
at the end. So if we truly believe that
testing is an activity that happens in the very beginning.
When we first see that feature, start thinking about the risks
at the very beginning and start thinking about, how are we going
to mitigate that? And a lot of that litigation
involves testing Hey everyone. My name is Henry Surya with
Robin. And you're listening to the
technology, you know, podcast the show where I'll be bringing
you the greatest technical leaders practitioners and
thought leaders in the industry to discuss about their Journey
ideas and practices that we all can learn and apply to build a
highly performing technical team and to make an impact in your
personal work. So let's dive into our Journal.
Hello to you, my friends and my listeners out there, welcome to
the technology. Now, podcast the show where you
can learn about technical leadership and Excellence from
my conversations. With great thought, leaders out
there, and welcome to the episode number 92.
Thanks for tuning in and listening to this episode.
If you're new to technology, you know, don't forget to subscribe
and follow the show on your podcast app and social media on
LinkedIn. Twitter ER and Instagram.
And if anyone wants to contribute to the creation of
this podcast, support me by subscribing as a patron at
technology. No, dot f /.
Patron testing is a critical part of any software and product
development. However, there are still many
teams that consider testing as an afterthought or even making
quality and testing. As the responsibility of Silo
teams who are quite detached from the product development
life cycle. I've been wanting to invite
guests who can share About how we can do better testing as a
whole. And I'm so excited to share this
conversation with you. All my guests.
For today's episode are the lovely duel Janet Gregory and
Lisa Crispin Janet and Lisa are the causes of several books on
HR testing and the cofounders of agile testing fellowship and
they are very well-known and influential champions for better
software testing approaches in this episode Janet and Lisa
shared the agile testing concept and mindset with an m.
Emphasis on the whole team approach.
Our discussion was then quickly followed by an explanation of
the holistic testing concept with a complete walkthrough, how
we can use the holistic testing approach in our product
development cycle, including how continuous delivery fits into
holistic, testing Janet, and Lisa also describe some
important Concepts in agile, testing such as the age are
testing quadrants that we can use to help classify and think
about variation of test cases and the Power of three also
widely known as The Three Amigos towards the end.
Janet, and Lisa, also shared their insightful perspective on
conducting better exploratory, testing and demystifying,
testing and production. I really enjoyed my conversation
with Janet and Lisa learning about the concept of a jar and
holistic testing and how those approaches can help us to create
better software and product. I especially like their Dynamics
in our conversation which made the discussion Lively.
If you also enjoy this episode, please share it with your
friends and colleagues who can also benefit from listening to
this episode. Leave a rating and review on
your podcast app and share your comments or feedback about this
episode on social media. It is my ultimate mission to
make this podcast available to more people.
And I need your help to support me towards fulfilling my
mission. Before we continue to the
conversation. Let's hear some words from our
sponsor. Today's episode is At least
sponsored by skills matter. The global community and events
platform with more than 100,000 software professionals here
members. Can organize their learning
experiences around the technology topics.
They care about most you get on-demand access to their latest
content thought, leadership insights, as well as the
exciting schedule of tech events running across all time zones.
So whether devops our data science is your bus or you are
fan of functional programming or all.
All things Cloud you can make real connections with people who
share your interests head on over to skills method or calm to
become part of the tech community that matters most to
you. It's free to join and you will
find it easy to keep up with the latest tech Trends.
Are you looking for a new cool swag taglit Journal?
Now offers you some swags that you can purchase online these
wax are printed on demand based on your preference and will be
delivered safely to you all over the world where shipping is
Available. Check out all the cool swag is
available by visiting technology.
You know that death / shop and don't forget to break yourself.
Once you receive any of those threats, hello everyone.
Welcome back to another new episode of the tekhelet.
You know, podcast today. I'm very excited to have the
testing Duo very much. Well, known in the software
industry. We have Janet Gregory and Lisa
Crispin in this episode. So, for those of you who don't
know, both of them, they are actually the author of a gel.
Testing book which was published in maybe 2009.
So it was kind of like a revolutionary during that time
when all a gel is in the bus and also try to improve testing
practices and they wrote another book, which is titled more agile
testing, this was published in 2014.
Maybe there was something that they want to improve from the
first book. And recently, they also publish
a book which is a kind of condensed version of both books
called agile testing condensed. So as you can tell today we are
going to talk a lot about a gel testing and some Testing related
stuffs for sure. So Janet and Lisa really pleased
to have you in the show. Welcome to the technology, not
podcast, thank you. Thanks.
It's really nice to be here. Thanks for inviting us.
I always like to start with asking my guest to introduce
themselves. So maybe, if you can take turns
and introduce yourself telling us more about your highlights
and turning points that were to share.
So, I had a little bit of a different start because I worked
right out of high school and then I got married.
We'd had kids did all of those things, did some traveling lived
overseas for a couple of years, Singapore, in Jakarta, and then
when we come back I decided I needed to do something with my
life. So I went to University and I
graduated computer science and became a programmer but this was
all around the age of 40. And then I started programming
for about, I guess, six years that I like to tell people, I
saw the light and I went into the testing.
Was a QA manager for a few years, but my first job was not
very satisfying because it was in a very chaotic kind of
environment. Being a QA manager, all about
process and nobody was listening to me, and I was so frustrated.
And when I quit that job, I was quite lucky because I ended up
in an agile team as a tester. I just kind of washed my hands
and started all over again. That really truly did change my
life because that's when I met Lisa.
Sup, because I had this moment of panic thinking what does
testing in an agile? I don't know.
So I met her because she was writing her, very first book
testing XP. So I had the opportunity to help
review that book and learn lots. And then, of course, that Lisa
and I have had our own kind of separate careers, what very much
bound together. All the way through I was tester
/ coach because I was one of the few people that actually
understood how to test in an agile team.
Until our first book was written, I should say.
And then I started Mark Consulting and then I've been
Consulting ever since Lilly helping teams to understand what
it means. So that's the my career in a
nutshell. When I went to college you
couldn't get a computer science degree when the thing you had to
get electrical engineering, but my original degree is in animal
science with a focus in beef, cattle production and then I got
a Masters in Business Administration and was had a
research job. But at some point I needed a job
and wandered into the University of Texas at Austin employment
office, and saw a sign that said, programmer trainees
needed, no experience necessary. And I said, well, that's me and
I'm fortunate that a, they hired me and be, they hired me for my
domain knowledge. They want to people with disis
background, they had a great idea, the train programmers
themselves so that we all could have Collective code ownership
because we all coated this, a way we work together.
We collaborated that was back in the day when programming was a
low pay low status job. So there were plenty of women.
A lot of fun eventually. I went to work for a software
vendor and kind of accidentally as most people do got into
testing. Oh, our customers are really
upset that all these bugs go out with our releases.
What if we tested it? And so that was in the early
90s. When I switch from programming
to testing, I've just been fortunate.
I've just been on so many great teams.
Most of the time. In my career, I've been on some
bad ones but mostly great ones, I was able to jump my first
extreme programming payment 2000.
It was difficult for testers because it got all the
publication's of the time about extreme.
Program, maybe they were great. It was all about quality is all
about testing. It was all about people but
nothing about testers and what they might do on an extreme
programming team. So you know, my team and I were
trying to figure it out through the extreme programming, mailing
list, I met Janet. So we were all trying to figure
this out. Together Co wrote my first book
would tip house because we want to share what we learned so far.
Because we knew there were other challenges.
So it's been all about sharing experiences.
And I've been very fortunate to collaborate with Janet since
then not only on the books. But on conference time Since and
read other articles and training courses.
It's just been great to reach out to our software.
Can be maybe testing Community around the world, get everybody
to share experiences, and we kind of channel that out and
share it to everybody as much as we can.
Thank you for sharing both of your stories if I can pick some
of the interesting things. So I didn't know you have stayed
in Singapore and Indonesia China lesion by the way.
So that's kind of like similarity from both of us.
So, both of you studied programming but then went into
testing. So tell us a little bit more
white Testing, apart from so many other areas in software,
engineering. Okay, my very first job in
programming. I work for the Vancouver Stock
Exchange and so things had to work.
I was very fortunate that my boss every day would come in and
sit with me and say, so, Janet, what are you working on today?
There was no pairing. We all set with cubicles and
things. So I would tell him what I was
going to be working on. And then he'd say, so how are
you going to test that and in at the end of the day, he would
come. And you'd say, so Janet, what
did you do today? How did you test that?
So I learned to think about testing right from the very
beginning for my first job. And I didn't know any
difference. So, when I went to my next job
which didn't do testing on their things and was working out just
as critical application as Stock Exchange, it bothered me.
I coded for a couple of years there and then my boss come to
me and said so Janet's. We think we need to have a test
team. Mmmmm.
You are the only one who was constantly complaining about our
princess. I would like to be our QA
manager. I knew that I didn't want to
program forever. I knew that from the very
beginning and so I actually jumped at the chance and then I
took everything I possibly could learn about testing and started
there, so it ended up not working out for me there, but
that's how I got into it. I've never looked back.
That's kind of like very interesting question.
How do you test that? Because I think many software
Engineers even don't think about that.
So they just build the features and finish and maybe pass it to
Q&A. So, how about you Lisa?
Do you have any interesting insights, why you went into
testing world as well? That's a pretty good story.
So I was working in tech support for a software vendor a pretty
big software. Vendor, this is back in the days
when Microsoft and Apple were still small.
I think we were bigger, we're frustrated because that was
back. When you sent everything out on
paper to the customers, These are Big customers that was
mostly with database software. We do a release, the tapes would
be sent out. We had nothing to do with it.
This was back. When people just called on the
phone, no faxes. No email.
Just had an angry customer on the phone.
Who said, I cannot believe you didn't see this giant bug.
We hadn't even seen it. We didn't have it yet either.
So it occurred to us to ask the developers.
We were in Colorado and they were back in Germany.
And we said, could it be possible to send us maybe a
preliminary version before you send it to the customers?
And we could just Start trying it out.
So they said, yeah, we could send it like a week ahead, so we
could install it, we can start testing it, we could find bags
and we can start working on patches so that when they angry
customers called basic, don't worry.
We're working on a patch and we'll have that out to you next
week and we did it on our own and then our managers were like
Sting, what an interesting idea. What if we have a department
devoted to people who do testy and coordinate their releases
and cut the released tapes and they will the customer Maybe we
should combine all these things and so I put my hand up like
that, sounds fun, and like jamming.
I've never looked back. Wow, so both of you took the
chance jam into the opportunity and went ahead an till now.
So which brought you into the concept of a gel testing.
The first book was written in 2009 if I'm not wrong.
So what is agile testing? Maybe if you can give the
definition for all of us here who maybe have not heard about
it yet? Our official definition
collaborative, testing practices that occur continuously from
inception to delivery and Beyond supporting frequent delivery for
our customers testing activities focus on building quality into
the product using fast feedback loops to validate our
understanding in the practices, strengthen and support, the idea
of whole team responsibility for Quality, we spend a lot of time
trying to put that together but really what it means is play
nice in the sandbox. Work together, think about
testing from the very beginning when I think about this and we
might talk about a little bit later is that definition could
be applied to the term were using these days, which is
holistic testing. And by the way, that was a
community effort. We got a lot of inputs and
people in the community, we've written a couple blog posts on
as we were developing that and so you get more details on our
website at altestore.com slash blog and search a room for that.
But yes, Janet that's a really good point that the definitions
don't. Works, even if we use a
different label. So if I read the descriptions,
definitely first, it's like, there are so many things mention
there. But there are a few key things
that I observed. The first thing is to mention
about this concept called whole team.
What do you mean by whole team? What is the relation with
testing as a whole team and our experience being successful at
delivering valuable software products to customers is an
effort that requires everybody on the team to be committed to
it and work together. We've seen over the years that
Saying at the end only by tester doesn't work, even when I work
in waterfall projects, everybody still did testing the whole
time. Wonderful doesn't mean you have
to do a bad job and having that mindset where we've all talked
about that everybody on the team.
Regardless of our specialty, here's a global equality.
We want our software. This is what we want to get
through. This is our goal and we're
committed to getting there because we know, it'll be hard.
We know where run into a lot of obstacles, will have a lot of
hard problems to solve but together with all our different
skills, And experiences will be able to do it because we're
committed to it. We're not going to just throw
our hands up and give up. That's what it takes.
If it's just part of the team, even if some of the developers
like, oh yeah, I really care about quality.
Alright, some unit tests, but if that's all they do, and they
don't involve. But testers or the testers go
off and try to do all the testing on their own.
We just seen it doesn't work. The state of devops survey has
really supported that with hard data that was damaged in my
experience over all the years. But then we saw data from that
survey supports It when developers own the testing, when
they own the automated test, yet they created them and maintain
them along with the testers. And the testers help them with
all these other activities, like exploratory kissing.
That's what correlates with high performing teams.
I know as humans we don't pay attention necessarily the hard
data but we have the hard data to back up our views and it
starts from the very beginning. That question.
How are we going to test this? If we start that at the very
beginning that drives the testing all the way through.
So I think I agree. It's a very important
distinction, having the whole team, really taking
responsibility of how to test the application or the system
that they're building. Because as you can tell in the
industry, I'm sure throughout your Consulting experience as
well. There are many teams still have
the silo responsibility developer just build the
software. While there's another team
called the testing team, QA team, quality engineering,
whatever you call it. That's the testing part.
So what in your opinion should change in order to make it more
like a whole team concept, all this Still kind of like a common
Trend in the industry. I can tell at least from my part
of the world. What should change in order to
go towards this whole team concept.
Well and I'm going to take this from Elizabeth Hendrick since
one of my favorite quotes testing is an activity that
happens throughout. It is not a phase that happens
at the end. So if we truly believe that
testing is an activity that happens from the very beginning.
When we first see that feature that very first feature starting
to think about what are the risks X to be able to do that.
That means that who's ever thinking about testers.
If you have testers on the team, if you don't have testers on the
team, start thinking about the risks at the very beginning and
start thinking about, how are we going to mitigate that?
And a lot of that mitigation involves testing it starts
there. And so moving, both testing
activities, thinking about the early, I think that's how we're
going to change it, don't bring in testers when you've got code
to test. The man early to start thinking
about those risks and really talking about that level of
quality because that's how we start it.
And I think that's has to change its the mindsets of fatigue and
individuals because it's how we think about testing.
So I don't use software tester. I don't use that term for myself
because I think it's more than software.
We test ideas, we test assumptions.
We're testing many things. And so it's not only the
software. I think we test the Product and
the product is the whole. Yep, which you mention is, like
a mindset, right? So in the book, you mentioned
agile, testing mindset. So I can also see, observation
from at least my part of the world.
So testing is always the Miss. Like, maybe lower part of
engineering, right? Where people assume they just do
manual testing during the reproductive jobs.
I know that some testers are like good programmers as well.
They're right great Automation and all that but still
unfortunately, there are still these misconceptions also This
term called quality engineering quality assurance, so people
think they added efficient that is responsible for Quality.
So I think all these, maybe need your advice, how to change this
mindset, so that we can move towards this, a gel testing
mindset. Again, where everyone will
involve in the testing and really cares about quality since
the beginning. Yeah, Ted.
And I were just chatting about that this morning, because it's
not easy, and I'm just trying to think back that had in mind,
mindset changed. I remember the first first
two-week iteration on my first extreme programming team.
I was working for a start-up consulting company and most of
the first two weeks, I was helping another client of sight.
So I came back just like the day before we're going to demo to
the customers or a very first iteration.
One of the things I did was okay, I'm going to start up the
server and we got this little application going and I'm going
to log in as two different users and the server crash.
And I was like, what? We can't show this to the
customers. It's terrible.
You can even have two people on it.
Fortunately, an extreme programming.
You have a coach part of your team.
Archive said. Now.
Lisa, we don't have a story in this iteration for supporting
more than one user. The reason for that is our
customer is developing this to show people the potential
futures of their product. They need funding and I'm going
to use it, they're just going to demonstrate it.
They don't need to have two people logged in.
I was like what, what what oh that's mind-blowing to me.
Quality was the server says up its reliable.
Thing works. But now the quality is what the
customer needs. We really had to focus on that
so I really think it's an ongoing process, I think it
needs training. I think it needs daily coaching
from somebody who's done it before, who knows what they're
doing to help the team Janet. And I like to say, if you
haven't already been on a high performing, a dollar whole team
approach. Team, you can understand what
the unicorn magic of that is, you really have to experience
it. And so, you know, if people can
get somebody hire somebody at least temporarily, Like who does
know, is who does understand it and can help the team get over
it, because mindset switches are hard.
It's a cultural change. As Janet says, you know, we're
going to focus now on preventing bugs not catching bugs at the
end. We're not going to be the
quality police because it's not our job to determine what
quality is. It's the customers now we've got
to find out. What does the customer want?
That's big part of our job. As testers this, to help
everybody understand, get a chair and understanding of what
the customer really needs. That's a really good story Lisa.
I wish I would have heard that 20 years ago because That's
exactly it. From a testing perspective, the
hardest thing is to understand that we are testing this small
slice. We are making sure that this
small slice works right now, doesn't mean that we're going to
give it to the customer right now if that's what they need, if
they can use it. Yes.
But from a testing perspective that is the hardest thing is to
understand that their testing is narrow for this particular
Story. Once testers get used to that
they realize Much easier it is to keep testing and adding
complexity. You have that first story and
then that works. You add complexity and you can
test that and that works. You just keep wrapping that
complexity around and you get a solid feature.
And I think that makes the world of difference.
But it is a hard mindset to think that we have to do that
narrow. It's funny.
I have to laugh here. I just going to add a little
story because I can't talk without my hands.
So, and Gasps, you have to visualize that I'm holding my
hands up and I'm doing narrow Focus.
That's true. That's the thing in my early
days or even for the first two years that I was on agile teams,
my teammates had to keep reminding me, Lisa, I know you
just think you found a bug could you just make sure the happy
path works first so that we can be confident that we have at
least started in a good way because, you know, I was always
going off in the weeds. I bet there's a bug over here.
No. Lisa, we think it works.
We want you to help us. I know that it works driving
development with those tests, I was like, creating huge numbers.
I was with the product owner, we're creating these huge
matrices of test cases and expected outputs and given that
the developers, okay, here's what you should do.
And they're like, oh my gosh, I can't see the forest for the
trees. So can you just give us the
happy path? Let's all make sure that works.
Then let's add on like one unusual scenario and just do the
step at a time and that was something I had to learn because
I had been more used to even though I was trying to always
collaborate with Developers and Tessa still, we were doing big,
bang things. So I was getting a whole bunch
of stuff to test that once and that's a quite a different
experience. Yeah.
And I think when I've taught a course hundreds of times and I
think that's the biggest takeaway most of the time from
the whole team because we teach it to the whole team is that
they can do that once. Well piece and add complexity as
they go and test it. And I think that's a real
showstopper for a lot of people just going.
Whoa. And she said, I was laughing
when you shared that because I wasn't even Being about
application, that should just work for one people.
We always assume that it could work for multiple people.
So you brought a point where sometimes you have testing is
hard because you don't know what kind of slice that you should be
focusing on. And I think you mentioned the
key point where we should test within a smaller scope and
complexity afterwards. And it seems like this kind of
like a line to your concept of holistic testing where you have
this short cycle from build test deploy and all that.
So maybe this is a good time to actually introduce.
What do you mean by a holistic? Testing that You are talking
pretty lately. I'll take this one, because
there's a lot of talk about shift left shift, right?
And when I hear shift, left shift, right?
I think of a lateral line, like a horizontal line, I'm thinking
that software development isn't like that.
So the devops loop started and I think that's great for some I
had seen that was in discovered to deliver Island Goddess Diana
and Mary Gorman's book. They had this nice little loop,
the devops loop, and then Dan Ashby, and one of his blog post
talked about Newest testing, he said the step up so it was
great. But where does testing?
They always had this choke phase for testing and it didn't work.
So he made this we test here and he put all around the loop.
We test here and here and here and everywhere.
And I thought, perfect and I use that for a long time but still
something bothered me about the devops loop.
So Lisa, come up with one iteration and then we just kept
playing with it. And finally, I kind of had this
little mini light bulb moment and took the the loop as it is
now, Now and wrote a blog post on it showing that testing
really does start from the discovery.
So when we start thinking about Discovery, that's a very
beginning thinking about the risks and planning for it going
through the building and those are all stages that we do,
whether we go really fast through them because we're doing
a small story, we discover we understand and we deploy and
test, we might put it into production but one way or the
other, we have an internal release it.
And if we don't put it too, Action and if they gave out what
we learn from it. So it really is looking at that
whole cycle, thinking about testing holistically, when I
read that definition, and I hadn't really thought about that
definition agile, testing, and it's very applicable to holistic
testing. But I'm going to let Lisa talk
about why we call. It holistic, testing versus
agile testing these days. I mean, obviously, it was Janet
that came up with this idea. But part of the problem was Dan
Ashe. We use continuous testing, which
made a lot of sense to us. And then somehow people took the
term continuous testing and co-opted it to only mean the
automated regression tests that run continuous integration.
And so when you s taste testing, that's what people thought of it
was like that's like a teeny tiny little part.
At that thing, it's an important part.
What describes it better. So Jenna came up with the word
holistic, and, yeah, because whole team approach going
through the whole cycle. The testing activities are in
the whole cycle and it really sums it up.
What I found interesting, isn't it?
Just recently, I was still a hint on the tester on a future
team and we're going to company with several different teams.
I shared Janet's blog post on our slack.
One of the really experienced testers on another team was like
and he had a light bulb moment to, he says, this model is
great. I can finally explain to people
what it is. We do because it really summed
up when there are well, performing teams that are doing
a good job of the holistic approach.
And there's a tester on that team.
Kind of acting is a testing consultant.
Guiding them. Leading that effort now, he can
take it explained. The other thing, this is what we
do. I have a way to show you now,
other people have also come back with that kind of feedback.
One of the ways that resonated with people who are already
doing it, recognize it. So, we hope it'll help other
people than be able to achieve that holistic approach in grow
that in their own teams. So maybe if you can walk us
through in terms of practical sense, right?
So you have this cycle, there are multiple stages in the
cycle. So I just pick a starting point
Discovery as you mentioned that you have planned.
Understanding building deploying releasing observability and
learning throughout these stages.
There should be some level of testing involved, so maybe if
you can walk us through in Practical terms day to day,
maybe in the spring, how do you run this?
Do we create a story for each of the stage, or do we have one
story where each has its own stage.
So if we think about it, say The Discovery, I would say we have a
story. Let's take the simplest story
that we possibly can one person can log in using leases.
I think so that story is the happy path.
Lisa can log into the system, what does that take?
And so we would take and we would think about what risks
there are. Maybe the risk is a servers and
up, she can't get validated, whatever it is.
We think about the risks. We might share some acceptance
tests. High-level acceptance test.
We might do example mapping if it needed to be, so that's part
of the understanding. So you're planning at a high
level. What is it take?
You understand that. All right.
And every story will be different depending on how well
loan it is. What are the risks?
So, then you have some high-level acceptance tests.
You were two examples and the whole team has an understanding
of what they are going to build and then they build it.
Of course, the building will include automating it.
Include exploratory, testing on that story and includes tdd for
the programmers. It includes whatever it takes to
make that code into a Deployable piece.
So you have that code. Deploy it your automated tests
are already running. You can run them on your system.
You can do any other human Centric, kinds of testing, like
perhaps you have to do some performance testing or something
like that on a deployed system. So all of these little testing
activities happen and then if you really sit with us
internally or externally to a customer, you're going to make
sure that happens and so testing in production, every time I say,
testing and production is that back 20 years ago?
No, no That's not what we mean anymore.
But you can do that. And then you can have your
observability and you're monitoring happening, but that's
all about learning. How is your customer using it.
If you've actually put it out to production if they're using it
and then you learn and you go into your next story.
So that cycle can be very fast. If you doing something very
simple like logging in with a valid user, we're not doing any
extra step, we're just doing that happy path.
The next story might be validating all.
The usernames adding complexity. Maybe the third story is
something else, but you adding complexity all the way along and
I think that is recognizing that Loop is just a continuous cycle
constantly adding on. One thing I particularly like
about the loop is haven't this part where we're going to learn
observing production that's one of these past few years.
My mission has been to make sure that testers get involved with
that because our products are so complex these days were Is the
cloud we're using all these Services microservices we need
to start as we're thinking about what code we're going to rate
for that little story that Jan has talking about, what
Telemetry do we need? How should we instrument that
code? What events do we need to
capture so that we can create our monitoring dashboard so that
we can create our alerts, if something goes wrong in
production. So we have all the data we need
for the things, we didn't anticipate and didn't create
dashboards or alerts for, and we can learn very quickly what went
wrong and fixed it and also we're We're so lucky these days.
The analytics tools that are out there which we're done for
marketing and product so they know what customers are doing.
So valuable to us as testers, we could go out and look at Sea,
power integers, using our product, you know, some of their
tools and I'll show you a screen cast that what somebody did,
which is kind of creepy. But it's very educational when
you see somebody clicking on something that's not clickable
or just moving the mouse around because they don't know what
they're doing this. So helpful, to improve our
product to know where to focus. Our testing to know how to help
you. Hers, we need the whole picture.
We can't just be people say, shift left shift, right?
We can do one of the other. We've got to be holistic.
We've got to go through the whole cycle and be involved all
the time. Using our testing skills to
identify risks. It's a spot anomalies to spot
patterns to help the theme. See those, I was just actually
listening to your last podcast with Liz found Jones.
She was talking about observability, one of the things
that she said and I went this is the way to think about it
because it hit home. The way she said it was, can you
understand what's happening in your system and why without
having to push additional code slicing and dicing data from the
Telemetry signals. And I thought, what a wonderful
statement that is about observability.
It sometimes I well, because I remember somebody paid for years
of oh God, we cannot figure out why it says 500 error happening
with our web server. Okay, well let's add this
Telemetry and let's redeploy. A little that didn't do it.
Let's have this very good boy. No, that lives, this one of my
sheroes, the people in that space and in observability and
site reliability, engineering. They understand testing, and
they understand the importance of having this data.
We all need to be aware of that. And not think that some separate
area of expertise that we don't need to know about.
Lisa said, that's one of the testing activities that happen
in that understanding is what do we have to put into the coat?
Having heard what, you just described both of you in detail
in depth including an example how we can practice it?
I think It totally makes sense. I can see why in the industry,
sometimes we all work in silos, most of the times developer just
read, okay? This is the story requirement.
That's it. I don't think about Telemetry, I
don't think about how we can measure user success and all
that. I just built based on the
specification has this probably also follow.
So, I think the whole concept of having this cycle for every
story, you can be bigger scope. It can be small scope.
But having all these cycle covered in a story actually,
really good, because you can have a good understanding in
depth from the start from the discovery up.
That use that themselves how we can monitor and make sure that
the story itself brings value to the user.
Thank you so much for explaining this concept.
One part that I think with the mention because you assume that
the story can be deployed to production where you have the
release deploy and observability but so many people still haven't
practiced this, which is continuous delivery.
How important is continuous delivery.
In this holistic testing, well I see so many different contexts
to make continuous delivery is Is something to think about and
to strive for, even if you live or do it, a lot of things are on
three months, release Cycles, delivering to the customer
between month, and it might be because their customer chooses,
not to take it. It's a requirement that a lot of
customers do, they? Don't want almost little fixes
for many reasons, but if you have that goal and you're
thinking about it, then that means that your stories are
going to be testable. Your stories are going to be
small enough. You can test and you will get
those good habits. Even if you choose only to give
it to the customer who wants to. Well I encourage teams to
practice continuous delivery to go for that even if it's on an
internal server and they're practicing it on their staging
environment, they can still get to that even if they don't push
it to the customer every time, it's a great way to get all of
those practices in place. So it's not going to happen
overnight because a lot of things Don't even have
automation yet. How did you get there?
That's just a series of small steps understanding.
That every team has a different context.
So we talked about it as everybody does but they don't
they don't there are still teams out there that don't even have
continuous integration. I remember when I told Lisa that
was probably 10 years ago or 15 years ago and she goes what?
Because every team she worked on problems and she couldn't see
working any other way. But they're still teams without
continuous integration, which, you know, it's kind of sad, but
I know they exist. It's puzzling to me because
like, it says everything I've been on.
I had it because it only took us a day or two to get it going.
So, what is the problem? People, it's not hard, but I
like Janet says, before we even had the words continuous
delivery, which I first learned from Jazz humble, and they
Farley's excellent book of the same name, which is a book about
testing. My team, took us a few years
because at first, we couldn't even get a passing build with an
artifact. We could deploy to production.
In two weeks. And then we tried to get one
everything we set goals, okay? We're going to have a passing
build every day and for the days we don't have that.
We're going to have a calendar where we put colors to remind
us. So everybody in the company can
see including the executives and start to understand the
importance of having this passing.
Both everyday just build it a step by step.
It took us years to get there but then we always had a
Deployable artifacts. We could deploy to production in
our business domain customer didn't want new changes every
day so we only did it every two weeks but we had the ability at,
that's the goal. One thing I tell people that
don't have anything, they don't even have CI, you have a
deployment pipeline changes that you make in your software
product gets production. Somehow there are steps that you
take to get there even if their manual and it's really important
that the whole team understands all those steps.
So we tell people sit down together, but use your virtual
whiteboard and virtual sticky notes and draw your pipeline,
even if it's manual, or even if it's some automation, some
manual, understand what you do. And then try to pick the play
sets. May be your biggest bottleneck
where it would help you to automate it.
Get that faster feedback, make your process more robust so that
you do have something solid you could get to production but you
have to build a step-by-step. I learned from Janet years ago,
the first step in solving. A problem is make it visible.
So is your problem is you struggle to get anything to
production because you're still have good processes.
Start by visualizing what it is now and then decide how you want
to improve it. I will be surprised if many
people Still, do not practice, continuous integration but few
years back I could even see some teams do not even have Version
Control or maybe their Version Control even just locally.
So I think this is also a bad practice.
I think we should all avoid. But by now, I hope all the
listeners who have practiced at least Version Control and
continuous integration. So in continuous delivery, we
know that you have many stages in your deployment pipeline.
Obviously, many of those stages will be automation test.
So I know that you have this concept agile testing quadrants
where you have the garage. Has different types of tests.
Maybe, can you give us an overview?
What is agile? Testing quadrants and what are
the representations for each of the quadrants in actually wear
red petticord in myself and Brian Merrick and it was one of
those conference late night. Chats, we were talking about
this and Brian was explaining these quadrants so he had a
napkin it's one of those stories, right?
Say had a napkin he's drawing on it and then Brett and I booked
him for months until he finally wrote it up in a blog post, so
then we could use it. We did ask his permission to use
it but the quadrants is a way of classifying, the tests four
quadrants so the left side is about guiding development tests
that you right before you start coding the right hand side is
about critiquing the product. So test that you do after coding
is complete, the top bath is business facing tests, so those
are tests that your business would be able to look at would
be able to understand would be Able to do feedback on and the
bottom half is technical facing, which just means that the
language that those tests are in a something, the business
wouldn't look at, for example, Performance testing, they care
about the results, but they would never go look at the
actual tests. So the four quadrants, and
they're numbered for ease of use only technology facing tests.
That guide development. 101 and those are kind of unit tests.
So the programmers are mostly responsible for guiding.
Development with the unit test quadrant 2 is business facing
tests that guy development. So this is where when we do
acceptance test-driven development or behavioral driven
development thinking about how are we going to use those
examples to guide development? What tests can we write before?
So things like simulations or those sorts of things.
Those are all things that help us quadrant 3 is business facing
that critique the product. So generally we put in
exploratory, Retesting in there or user acceptance, testing
business facing tests. We need to do them because
there's a notes, what didn't we think about?
And then Quadrant 4, which was the quadrant that made me really
like this. Because at that time and that
was in 2003 that conference, every time I would talk about
testing in agile projects, every tester.
I meant would say it won't work. They talk about customer tests
and programmer tasks. But what about all those other?
Chests. Quadrant 4 gives us a way to
talk about all those other tests performance reliability in all
the quality attributes that we know we have to work with.
So the quadrants are a way of visualizing your testing.
Again, we encourage teams to think about the test, they do
and then kind of put them in the quadrants to think about why are
they doing them? Are they doing it to support?
Because the left hand side supporting or guiding, that's
about, About preventing defects in code and we test our
assumptions. Can we test our knowledge before
we ever write a line of code? The right hand side is about
finding defects. What didn't we think about?
So it's a way of classifying it. So it's a way of visualizing the
kinds of tests and where do we actually want to do this?
Thanks for walking us through the Adele testing quadrant.
So maybe for those of you who cannot visualize what Janet is
saying just now, so maybe I'll put it in the show notes.
So what the diagram looks like, But there are so many categories
of tests. Depending on the left side, top
bottom, maybe you can have a look.
But importantly, there are so many types of tests for one
particular story should be built.
So many of those tests, what is the ratio here?
How should we build these test cases?
Because I also notice that you have this good practice which is
called The Three Amigos or the power of three.
So maybe walk us through a little bit on how we should
write all those tests and how I wish it ran all the tests and
all the quadrants together with each feature.
Sure, that your team's going to work on and with each story that
you work on, it depends for like a web application.
Maybe you're doing something with a user interface and then
it has a back-end server and the database have all those pieces,
we often will start in that quarter to that business facing
tested guide development, with prototypes and wireframes.
And what is rui going to look like?
We can test those, maybe get involved with designers.
When we say that the power of three these days, I think you're
going to need four or five because you often need the
designers or We have experts or aberrations, experts, it could
be anybody. So we make sure that testing
those feature ideas is Janet mentioned earlier.
Once we've written sliced up this testable stories and start
working on them down. In that technology phasing
quadrant one that guides development.
The developers are the programmers are coding test.
First are doing test-driven development at the unit level,
that's going to be something that the programmer Zone because
that's not a testing activity as much as it is a code.
An activity, but it does help us design, testable code, and
operable code, all those good things.
Then we're going to maybe go back into that quadrant two and
do our story testing. And again, it's great to have a
programmer in a tester pair up on that or attachment a product
owner or even better. For my point of view and
Ensemble with several people in different roles testing it
together. Well, I guess you can do that.
So we set the story Little know we were for exploring later on
but Beverly, you could do except this testing at the Level a,
I've done that a lot doing the show me that jam.
It talks about the programmers, done thinks they're done with
the story and they asked, pester hey, can I just walk you through
what I've done and just a few minutes.
You may have had that. Aha moment of oh I didn't
understand that quite writer. Oops I forgot a piece.
You've prevented a bug. You haven't even checked the cut
into the source code control yet.
So having that close collaboration.
It's like this iteration testing coding, testing coding.
When we really think the stories ready to go and we get enough
stories in a ER now we can start into that quadrant three, the
critiquing the product but still business facing exploratory,
testing usability, testing things.
We really need for code to have been written and deployed.
Again more collaboration involving more people.
One of my recent jobs, when somebody thought they had a
feature that was pretty ready to go, we just opened it up to the
whole engineering organization. So hey, we're going to do
another sample. Testing session exploratory
testing on this feature for 30 minutes at this time, please
join us. We have people rather tame join
fresh eyes. I've never seen the future
before that's so important. And just as such short time, we
would flush out anything. Any bugs that were there any
hidden assumptions that we hadn't thought about getting
that collaboration even from people with them teens and then
the technology Pages stuff you might do at the beginning.
Maybe we've got some new piece of a product that we want it
needs to support a large number of users at the same time, it
had really good performance. Well we have to think about the
architecture and how do we know that?
Architectural scale. So maybe then we Spike the code.
We don't do the careful thing. We just do some throw a code
with that architecture and now we can do performance testing on
it or load testing on it. See if it's scales and if it
does we throw away that code and we start running the real code.
If it doesn't we Spike. Another architecture those
technology facing tests may be done at the very beginning if
that's the most important quality attributes who are
business and those become guiding development.
So the key thing that I observe when you Lisa is that is still
collaborative, right? It's not just one person or one
team who is responsible to come up with all these tests it could
be from product manager, it could be from tested, it could
be from software developer designer and whatever the role
is in the team. And again this the feedback,
right? So the test sometimes drives
back the development and also by zebras are so, I think I like
this concept of holistic starting already.
Thanks for sharing this concept. So you mentioned a couple of
times about exploratory testing maybe this term, I think little
bit vague for some people. Well, some people think it's
like okay we just do any random test.
See whether the system crash maybe a little bit of Guiding
Light here. What do you mean by exploratory?
Testing. Well first of all I would
recommend everybody read Elizabeth Hendrick since book,
explore it. I like the fact that she's
actually got a chapter in there for programmers, how to explore
when they're testing themselves. She has so many good ideas.
One of the things that I really liked and I use to guide me is
the format to use Explorer. What am I going to explore with
chooses a word resources? I think variations.
But what is it that I'm exploring with and then to
discover. So when I put a exploratory test
Charter together I use that format because it helps guide
me. It's a mission statement, it's
not necessarily easy to do because you don't want it too
small. You don't want it too big.
What is it? That I'm going to explore and it
might be a specific risk. We think we have some security
issues in this area. So, maybe I'm going to be
exploring certain kinds of exploits or something, but
maybe, I want to see something. Say, we're adding this form and
we got people's names. So, if I use the agile testing
Fellowship, our website that we have.
One of the things we did was we wanted to make sure that we
could use any names. So, for example, your name
doesn't have any funny characters but if I put in
somebody from Thailand, they might have some different
characters or from Sweden and Norway, they have those
Different kinds of characters, will it supports every name?
And so we just put an exploratory test Charter to do
that. I also like to think about it,
as if I took that Charter and I tested, I would have my own
notes, I would have my own findings.
If I gave it to Lisa, she would explore differently.
She would use different examples.
She would have her own. So there's enough leeway that we
would possibly test differently, but we're still Exploring the
same thing and so it just takes practice, but there's a mission,
not just randomly hitting keys. I want to look for this specific
thing. Yeah, they're clearly be value
and ad hoc testing a lot of teams do bug bashes for they
just get a bunch of people and they start hammering on whatever
they wanted release that's fine. They'll find back to that way
too, but the exploratory testing is designed more to try to find
those unknown unknowns before. They bite you in production and
be Organized. I actually got a great tip from
a programmer that I work with few years back.
When writing the charters, I like to start thinking of what
resources I'm at test with first and that helps, you think of
that mission. As Janet says, what haven't got
my disposal that I can use to test this?
Maybe I've got some API endpoints I could use or maybe
I've got some production like test data, I could use or some
personas that the marketing department came up with that I
could use. So think about the resources, is
one help because it's hard to learn, right?
Those things and I would also recommend pairing was already on
it. I would also put all those
Charters as stories in your project, tracking tool in your
backlog. Along with your feature stories,
they have to be visible. Everybody needs to know that
those have to be done along with all the coding and all the other
testing. We need to do that exploring as
well. Thanks for clarifying this
concept because many people still just assumed exploratory
testing. Yeah, okay.
You just go ahead and try to break the system or do random
stuff. So I think there should be a
path. Loud chatter Mission, you keep
mentioning that. I think that's a very good
explanation. One more misconception with
Janet mentioned is about testing in production.
Again, it's got common misconception, people think oh
maybe we should not write tests, we just deployed, maybe let
people figure it out. Maybe a little bit of Guiding
Light here as well, but I remember listening to a keynote
years ago at a conference and they said testing is dad, let
the customers test. I wanted that person in the
worst way and I kicked myself now for not putting my hand up
and saying something, but I was hoping that person would say in
my context because that is so important.
So testing and production, if I'm doing a Google Maps, yeah,
let the customers figured by the bugs and report him, but if I'm
doing code for a heart monitor, for example, I'm going to have
way different risk profile. We're going to do all the
testing we possibly can before we put it into production.
And as such but testing and production and least I can talk
about it much better than I can most of the time.
But really if you have a safe way to do it, things like
feature Flags, right? So the customer actually doesn't
see it. We're not touching customer data
where we're testing and we have it turned off.
So there's different ways to do it.
I think Katrina cookies book testing and devops is still the
best one that I've come across for testing feature flags and
working within that construct. Yeah, right.
It'll guide for testing and devops that's available, van,
leave Heaven. You Can Get It Free.
She doesn't require you to pay although it would be nice to pay
because it's one of my go-to books for sure.
But it's so important because as good, a job.
As we do have testing, we know our test environments, don't
look like Productions. We're not going to test
everything. We're going to miss things, and
so we need to be able to put things in a production
environment. You some kind of really strategy
that. Let's just do that safely.
It takes build an infrastructure.
And again, that's an effort that your whole team needs to get
involved in. And that's when you Need to have
operations specialist site, reliability, Engineers
platforms, and ears in your team are working with your team to
help you with that. I think you just said something
that really struck with me there.
Lisa, you said, in the production environment, not
necessarily with production data and I think that's the key is
the production environment, not the customers data, my last
full-time job that I had. We could not test and production
in any way we just could not As if I as a little Services
application, there was no way to do it.
If we had tried to do it with a got in trouble.
And so then it became really important to be able to use our
production data observability, what our customers doing power,
its does think performing. And also the analytic tools,
what are the individual users? We could actually see what they
were doing. Like.
Oh, people are complaining about this and we can't reproduce it
in our test environment. Let's go see what they're doing
in production. Thank goodness.
We have those tools now because if Can't test it in production,
you need some way to understand what's happening there context.
So important things will emphasizing that again context.
So for people who learn about testing production, it doesn't
mean that you just deploy your code without any kind of safe
mechanism like feature Flags or maybe robust monitoring and
observability. Because without knowing what
your system does, its kind of like risky, especially if you're
dealing with Finance or health and all that.
So unfortunately, due to time we have to wrap up.
It's been a really great Conversation about testing in
overall but before I let both of you go normally I would ask this
question called three technical leadership wisdom.
So I will leave it whether you want a combined effort or if you
want to go individual. So what will be your tree
technical leadership? Wisdom to share with us?
Yeah, I thought that was an interesting question and I think
my answer is different now, but it would have been a few years
ago. One of the things that we've
learned in the last part, 10 years now, but people don't pay
enough attention to. It is a prerequisite for
success. This fall software development
is psychological safety. We had to feel safe to
experiment and learn we have to feel safe to ask questions.
When you don't feel safe, you're going to burn out.
My advice is, if you're in a toxic environment where you
don't feel safe, where people are not safe.
Try to be an agent for change. I've really found a good help
from the books by Linda rising and Marilyn Mansfield has
changed. And we're Fearless changed
patterns to help influence people.
You can maybe influence people and if it doesn't work, if you
can get out, Out. Now I realize that's a
privilege. Some people they need to pay
their bills and they don't have any other options but also
realize it isn't your problem. No amount of yoga and meditation
is going to fix it for you. Don't blame yourself.
It's a problem with your organization.
So pay attention to that before you burn out and really suffer
the anger teams going to be suffering too, that's one of
them. I'll let you do one Janet to, of
my go right in with that, one of the things was small, f.
Can bring about big changes, don't underestimate your scope
of influence. The other one that kind of ties
in with you two, were talking about psychological safety.
Don't be afraid to try new things, keep learning many years
ago and this was in my pretty early days out of high school
was my second job out of high school.
I was in administer stencil administrative assistant and I
had a really lazy boss. Really lazy people kept saying,
why do you work for him? What I ended up doing was, I
learned because he had me do most of the budget
reconciliation and things because he didn't like to do it.
So I had to do it for him but then one day he was sick and it
was a very important meeting of all the directors and a
budgetary kind of meeting. And they were talking about it.
I walked in with all the papers and they said, where's Brian?
And I said he's sick and they said she would cancel a meeting
and I said, no, let me explain because I had You can on that
extra thing and done it. Learn that it just opened up a
whole new world for me. I was viewed very differently
than I had been the day before it paid off.
So sometimes don't be afraid but then take that jump take that
opportunity that is in front of you.
Don't be afraid to take it but that's only good if you have
psychological safety. As Lisa said before my second
big piece of advice, something I have followed my elf for a
change. I think my own advice.
The last few jobs I have done, is build relationship.
One of the things I got from Katrina cookies, book is about
relationships within your team, but also outside of your team.
I've gotten so much benefit from asking people on the platform
team. Could I have a one-on-one with
you for 30 minutes every couple weeks?
So helpful, I learned so much from them.
I got so much support from them. That help my team.
I just felt like it was a secret weapon.
Do start with people who seem friendly and seem like they'll
Your allies you got to pick them out the just forms a foundation
for so much. The kind of learning that Jim is
talking about and helping you get changed, because maybe
there's something you want to do, and you can't interest
anybody on your own team, maybe somebody on another team wants
to help you really beautiful. So, everything aligned together,
I can see the Synergy between both of you, right, as you can
tell from the books and all that.
So for people who want to follow up with you or maybe continue
this conversation outside of this episode, is there a place
where they I can find you online.
Lots of places, we're both on Twitter.
My Twitter ID is Janet Gregory chca for Canada because I'm
Canada. People get that confused.
Lisa. Your Twitter ID.
It's just Lisa Chris and I'm Lisa Chrisman on all social
media platforms that I'm on. We're on LinkedIn for any of our
websites. We all have a contact us.
My website is Janet Gregory .c a or agile tester, .c a or the
agile testing fellow. And I'm Lisa Crispin.com are all
connected together. If you go below it's pretty easy
to find and Lisa and Janet also has a limp up book so it's the a
gel testing condensed. If you ever want to have this
condensed Concept in one goal, I think it's also a good place
where you can get all these resources one last question.
One fun fact. So when I read your book, I
always see this dragon and Donkey.
I have my hypothesis, but I'll let you explain what all this
about dragon and Donkey. Well, the donkeys are kind of my
mascot because I have donkeys. So we have for donkeys here and
our little Farm in Vermont. I've taught me a lot over the
years about trust and Agility. I pitched them up to carts and
wagons and drive them and take them places and they be work
around the farm. There's sort of therapy donkeys.
I took him to senior centers and schools and things like that.
So yeah, it's just my passion. So three of those donkeys are
miniature donkeys. Yeah.
And then she's got one big one who protects the three little
ones and for those kind of cool. And the dragon is because I'm a
fantasy but file of fantasy stories.
My favorite series is a dragonriders of Pern, by Anne
McCaffrey. So, I like friendly dragons, not
like the Game of Thrones dragons, the friendly ones.
So it's just kind of something that appeals to me.
Thanks for explaining that really fun fact, for those of
you who also notice in the book that whatever you see dragon and
donkeys, this is what it means. Again really Pleasant to have
this conversation. Thank you so much for spending
your time Janet and Lisa goodbye for now.
Thank you so much for having us. It was a fun conversation.
It was Thank you for listening to this episode and for staying,
right until the end if you highly enjoyed it.
I would appreciate if you share it with your friends and
colleagues who you think would also benefit from listening to
this episode. And if you are new to the
podcast, make sure to subscribe and leave me your valuable
review and feedback. It helps me a lot.
In order to grow this podcast better.
You can also find the full show notes of this conversation on
the episode page at technology Arnold death website, including
the full transcript enter, Testing courts and links to the
resources mention from the conversation.
And lastly, make sure to subscribe to the show's mailing
list on pack leader. No dot f to get notified for any
future episodes. Stay tuned for the next
technology. No episode.
And until then goodbye.