1
00:00:00,000 --> 00:00:03,000
Hey, a quick message, for those 
of you who are listening to this

2
00:00:03,000 --> 00:00:07,500
episode on Spotify, I have a 
small favor to ask Spotify. 

3
00:00:07,500 --> 00:00:11,000
Now allows mobile users to rate 
podcast, I would really 

4
00:00:11,000 --> 00:00:13,300
appreciate it. 
If you can take a quick, pause 

5
00:00:13,300 --> 00:00:16,100
to go to the technique Journal 
podcast page, and leave your 

6
00:00:16,100 --> 00:00:18,800
favorite show. 
Your best rating on Spotify. 

7
00:00:19,200 --> 00:00:21,800
It will help me a lot to get 
this podcast to reach more 

8
00:00:21,800 --> 00:00:24,300
people on the platform. 
Thanks a lot. 

9
00:00:24,800 --> 00:00:27,000
I'm going to take this from 
Elizabeth Hendrick since one of 

10
00:00:27,000 --> 00:00:30,600
my favorite quotes testing is an
activity, that That happens 

11
00:00:30,600 --> 00:00:33,400
throughout. 
It is not a phase that happens 

12
00:00:33,400 --> 00:00:36,300
at the end. 
So if we truly believe that 

13
00:00:36,300 --> 00:00:39,000
testing is an activity that 
happens in the very beginning. 

14
00:00:39,600 --> 00:00:43,700
When we first see that feature, 
start thinking about the risks 

15
00:00:43,700 --> 00:00:46,900
at the very beginning and start 
thinking about, how are we going

16
00:00:46,900 --> 00:00:49,300
to mitigate that? 
And a lot of that litigation 

17
00:00:49,300 --> 00:00:58,000
involves testing Hey everyone. 
My name is Henry Surya with 

18
00:00:58,000 --> 00:01:01,300
Robin. 
And you're listening to the 

19
00:01:01,300 --> 00:01:04,599
technology, you know, podcast 
the show where I'll be bringing 

20
00:01:04,599 --> 00:01:07,700
you the greatest technical 
leaders practitioners and 

21
00:01:07,700 --> 00:01:11,500
thought leaders in the industry 
to discuss about their Journey 

22
00:01:11,700 --> 00:01:16,200
ideas and practices that we all 
can learn and apply to build a 

23
00:01:16,200 --> 00:01:19,700
highly performing technical team
and to make an impact in your 

24
00:01:19,700 --> 00:01:23,000
personal work. 
So let's dive into our Journal. 

25
00:01:28,200 --> 00:01:31,600
Hello to you, my friends and my 
listeners out there, welcome to 

26
00:01:31,600 --> 00:01:33,900
the technology. 
Now, podcast the show where you 

27
00:01:33,900 --> 00:01:36,700
can learn about technical 
leadership and Excellence from 

28
00:01:36,700 --> 00:01:39,200
my conversations. 
With great thought, leaders out 

29
00:01:39,200 --> 00:01:42,400
there, and welcome to the 
episode number 92. 

30
00:01:42,800 --> 00:01:45,300
Thanks for tuning in and 
listening to this episode. 

31
00:01:45,600 --> 00:01:48,700
If you're new to technology, you
know, don't forget to subscribe 

32
00:01:48,700 --> 00:01:52,300
and follow the show on your 
podcast app and social media on 

33
00:01:52,300 --> 00:01:54,300
LinkedIn. 
Twitter ER and Instagram. 

34
00:01:54,600 --> 00:01:57,100
And if anyone wants to 
contribute to the creation of 

35
00:01:57,100 --> 00:02:00,900
this podcast, support me by 
subscribing as a patron at 

36
00:02:00,900 --> 00:02:02,600
technology. 
No, dot f /. 

37
00:02:02,600 --> 00:02:07,000
Patron testing is a critical 
part of any software and product

38
00:02:07,000 --> 00:02:09,800
development. 
However, there are still many 

39
00:02:09,800 --> 00:02:13,800
teams that consider testing as 
an afterthought or even making 

40
00:02:13,800 --> 00:02:16,700
quality and testing. 
As the responsibility of Silo 

41
00:02:16,700 --> 00:02:19,900
teams who are quite detached 
from the product development 

42
00:02:19,900 --> 00:02:22,300
life cycle. 
I've been wanting to invite 

43
00:02:22,300 --> 00:02:25,600
guests who can share About how 
we can do better testing as a 

44
00:02:25,608 --> 00:02:28,200
whole. 
And I'm so excited to share this

45
00:02:28,200 --> 00:02:30,500
conversation with you. 
All my guests. 

46
00:02:30,500 --> 00:02:34,000
For today's episode are the 
lovely duel Janet Gregory and 

47
00:02:34,000 --> 00:02:38,700
Lisa Crispin Janet and Lisa are 
the causes of several books on 

48
00:02:38,700 --> 00:02:42,900
HR testing and the cofounders of
agile testing fellowship and 

49
00:02:42,900 --> 00:02:45,800
they are very well-known and 
influential champions for better

50
00:02:45,800 --> 00:02:50,000
software testing approaches in 
this episode Janet and Lisa 

51
00:02:50,000 --> 00:02:53,300
shared the agile testing concept
and mindset with an m. 

52
00:02:53,500 --> 00:02:55,400
Emphasis on the whole team 
approach. 

53
00:02:55,700 --> 00:02:59,100
Our discussion was then quickly 
followed by an explanation of 

54
00:02:59,100 --> 00:03:03,000
the holistic testing concept 
with a complete walkthrough, how

55
00:03:03,000 --> 00:03:05,900
we can use the holistic testing 
approach in our product 

56
00:03:05,900 --> 00:03:09,900
development cycle, including how
continuous delivery fits into 

57
00:03:09,900 --> 00:03:13,800
holistic, testing Janet, and 
Lisa also describe some 

58
00:03:13,800 --> 00:03:17,100
important Concepts in agile, 
testing such as the age are 

59
00:03:17,100 --> 00:03:20,700
testing quadrants that we can 
use to help classify and think 

60
00:03:20,700 --> 00:03:25,100
about variation of test cases 
and the Power of three also 

61
00:03:25,100 --> 00:03:28,600
widely known as The Three Amigos
towards the end. 

62
00:03:28,900 --> 00:03:32,400
Janet, and Lisa, also shared 
their insightful perspective on 

63
00:03:32,400 --> 00:03:36,200
conducting better exploratory, 
testing and demystifying, 

64
00:03:36,300 --> 00:03:39,900
testing and production. 
I really enjoyed my conversation

65
00:03:39,900 --> 00:03:43,600
with Janet and Lisa learning 
about the concept of a jar and 

66
00:03:43,600 --> 00:03:47,400
holistic testing and how those 
approaches can help us to create

67
00:03:47,400 --> 00:03:51,000
better software and product. 
I especially like their Dynamics

68
00:03:51,000 --> 00:03:54,500
in our conversation which made 
the discussion Lively. 

69
00:03:54,900 --> 00:03:57,600
If you also enjoy this episode, 
please share it with your 

70
00:03:57,600 --> 00:04:00,700
friends and colleagues who can 
also benefit from listening to 

71
00:04:00,700 --> 00:04:03,300
this episode. 
Leave a rating and review on 

72
00:04:03,300 --> 00:04:06,800
your podcast app and share your 
comments or feedback about this 

73
00:04:06,800 --> 00:04:10,300
episode on social media. 
It is my ultimate mission to 

74
00:04:10,300 --> 00:04:13,000
make this podcast available to 
more people. 

75
00:04:13,200 --> 00:04:16,300
And I need your help to support 
me towards fulfilling my 

76
00:04:16,300 --> 00:04:18,300
mission. 
Before we continue to the 

77
00:04:18,300 --> 00:04:21,000
conversation. 
Let's hear some words from our 

78
00:04:21,000 --> 00:04:23,700
sponsor. 
Today's episode is At least 

79
00:04:23,700 --> 00:04:27,300
sponsored by skills matter. 
The global community and events 

80
00:04:27,300 --> 00:04:31,800
platform with more than 100,000 
software professionals here 

81
00:04:32,000 --> 00:04:33,600
members. 
Can organize their learning 

82
00:04:33,600 --> 00:04:36,000
experiences around the 
technology topics. 

83
00:04:36,000 --> 00:04:39,800
They care about most you get 
on-demand access to their latest

84
00:04:39,800 --> 00:04:43,100
content thought, leadership 
insights, as well as the 

85
00:04:43,100 --> 00:04:47,300
exciting schedule of tech events
running across all time zones. 

86
00:04:47,700 --> 00:04:51,300
So whether devops our data 
science is your bus or you are 

87
00:04:51,300 --> 00:04:53,300
fan of functional programming or
all. 

88
00:04:53,400 --> 00:04:56,900
All things Cloud you can make 
real connections with people who

89
00:04:56,900 --> 00:05:00,900
share your interests head on 
over to skills method or calm to

90
00:05:00,900 --> 00:05:03,800
become part of the tech 
community that matters most to 

91
00:05:03,800 --> 00:05:05,900
you. 
It's free to join and you will 

92
00:05:05,900 --> 00:05:09,000
find it easy to keep up with the
latest tech Trends. 

93
00:05:09,300 --> 00:05:12,300
Are you looking for a new cool 
swag taglit Journal? 

94
00:05:12,300 --> 00:05:16,300
Now offers you some swags that 
you can purchase online these 

95
00:05:16,300 --> 00:05:20,000
wax are printed on demand based 
on your preference and will be 

96
00:05:20,000 --> 00:05:23,300
delivered safely to you all over
the world where shipping is 

97
00:05:23,500 --> 00:05:25,600
Available. 
Check out all the cool swag is 

98
00:05:25,608 --> 00:05:27,400
available by visiting 
technology. 

99
00:05:27,400 --> 00:05:30,900
You know that death / shop and 
don't forget to break yourself. 

100
00:05:31,000 --> 00:05:36,600
Once you receive any of those 
threats, hello everyone. 

101
00:05:36,600 --> 00:05:38,800
Welcome back to another new 
episode of the tekhelet. 

102
00:05:38,800 --> 00:05:41,800
You know, podcast today. 
I'm very excited to have the 

103
00:05:41,800 --> 00:05:44,300
testing Duo very much. 
Well, known in the software 

104
00:05:44,300 --> 00:05:47,000
industry. 
We have Janet Gregory and Lisa 

105
00:05:47,000 --> 00:05:50,100
Crispin in this episode. 
So, for those of you who don't 

106
00:05:50,100 --> 00:05:53,200
know, both of them, they are 
actually the author of a gel. 

107
00:05:53,400 --> 00:05:56,500
Testing book which was published
in maybe 2009. 

108
00:05:56,700 --> 00:05:59,200
So it was kind of like a 
revolutionary during that time 

109
00:05:59,200 --> 00:06:03,100
when all a gel is in the bus and
also try to improve testing 

110
00:06:03,100 --> 00:06:06,700
practices and they wrote another
book, which is titled more agile

111
00:06:06,700 --> 00:06:09,400
testing, this was published in 
2014. 

112
00:06:09,500 --> 00:06:12,100
Maybe there was something that 
they want to improve from the 

113
00:06:12,100 --> 00:06:14,400
first book. 
And recently, they also publish 

114
00:06:14,400 --> 00:06:17,200
a book which is a kind of 
condensed version of both books 

115
00:06:17,400 --> 00:06:20,800
called agile testing condensed. 
So as you can tell today we are 

116
00:06:20,800 --> 00:06:24,300
going to talk a lot about a gel 
testing and some Testing related

117
00:06:24,300 --> 00:06:27,100
stuffs for sure. 
So Janet and Lisa really pleased

118
00:06:27,100 --> 00:06:29,500
to have you in the show. 
Welcome to the technology, not 

119
00:06:29,500 --> 00:06:32,300
podcast, thank you. 
Thanks. 

120
00:06:32,300 --> 00:06:34,600
It's really nice to be here. 
Thanks for inviting us. 

121
00:06:35,200 --> 00:06:38,200
I always like to start with 
asking my guest to introduce 

122
00:06:38,200 --> 00:06:40,100
themselves. 
So maybe, if you can take turns 

123
00:06:40,100 --> 00:06:42,400
and introduce yourself telling 
us more about your highlights 

124
00:06:42,400 --> 00:06:44,300
and turning points that were to 
share. 

125
00:06:45,200 --> 00:06:50,700
So, I had a little bit of a 
different start because I worked

126
00:06:50,700 --> 00:06:53,200
right out of high school and 
then I got married. 

127
00:06:53,400 --> 00:06:57,700
We'd had kids did all of those 
things, did some traveling lived

128
00:06:57,700 --> 00:07:01,700
overseas for a couple of years, 
Singapore, in Jakarta, and then 

129
00:07:01,700 --> 00:07:05,600
when we come back I decided I 
needed to do something with my 

130
00:07:05,600 --> 00:07:08,300
life. 
So I went to University and I 

131
00:07:08,300 --> 00:07:12,700
graduated computer science and 
became a programmer but this was

132
00:07:12,700 --> 00:07:16,300
all around the age of 40. 
And then I started programming 

133
00:07:16,300 --> 00:07:20,100
for about, I guess, six years 
that I like to tell people, I 

134
00:07:20,100 --> 00:07:23,100
saw the light and I went into 
the testing. 

135
00:07:23,400 --> 00:07:27,800
Was a QA manager for a few 
years, but my first job was not 

136
00:07:27,800 --> 00:07:32,000
very satisfying because it was 
in a very chaotic kind of 

137
00:07:32,000 --> 00:07:34,900
environment. 
Being a QA manager, all about 

138
00:07:34,900 --> 00:07:38,700
process and nobody was listening
to me, and I was so frustrated. 

139
00:07:38,900 --> 00:07:43,600
And when I quit that job, I was 
quite lucky because I ended up 

140
00:07:43,600 --> 00:07:47,700
in an agile team as a tester. 
I just kind of washed my hands 

141
00:07:47,700 --> 00:07:51,400
and started all over again. 
That really truly did change my 

142
00:07:51,400 --> 00:07:53,200
life because that's when I met 
Lisa. 

143
00:07:53,300 --> 00:07:56,700
Sup, because I had this moment 
of panic thinking what does 

144
00:07:56,700 --> 00:07:58,700
testing in an agile? 
I don't know. 

145
00:07:58,900 --> 00:08:02,000
So I met her because she was 
writing her, very first book 

146
00:08:02,000 --> 00:08:05,400
testing XP. 
So I had the opportunity to help

147
00:08:05,400 --> 00:08:08,900
review that book and learn lots.
And then, of course, that Lisa 

148
00:08:08,900 --> 00:08:13,500
and I have had our own kind of 
separate careers, what very much

149
00:08:13,500 --> 00:08:17,000
bound together. 
All the way through I was tester

150
00:08:17,000 --> 00:08:20,800
/ coach because I was one of the
few people that actually 

151
00:08:20,800 --> 00:08:23,200
understood how to test in an 
agile team. 

152
00:08:23,500 --> 00:08:25,200
Until our first book was 
written, I should say. 

153
00:08:25,500 --> 00:08:28,300
And then I started Mark 
Consulting and then I've been 

154
00:08:28,300 --> 00:08:32,299
Consulting ever since Lilly 
helping teams to understand what

155
00:08:32,299 --> 00:08:34,500
it means. 
So that's the my career in a 

156
00:08:34,508 --> 00:08:37,400
nutshell. 
When I went to college you 

157
00:08:37,408 --> 00:08:40,000
couldn't get a computer science 
degree when the thing you had to

158
00:08:40,000 --> 00:08:43,600
get electrical engineering, but 
my original degree is in animal 

159
00:08:43,600 --> 00:08:46,400
science with a focus in beef, 
cattle production and then I got

160
00:08:46,400 --> 00:08:49,000
a Masters in Business 
Administration and was had a 

161
00:08:49,000 --> 00:08:51,700
research job. 
But at some point I needed a job

162
00:08:51,700 --> 00:08:55,400
and wandered into the University
of Texas at Austin employment 

163
00:08:55,400 --> 00:08:57,800
office, and saw a sign that 
said, programmer trainees 

164
00:08:57,800 --> 00:09:02,900
needed, no experience necessary.
And I said, well, that's me and 

165
00:09:02,900 --> 00:09:06,400
I'm fortunate that a, they hired
me and be, they hired me for my 

166
00:09:06,400 --> 00:09:08,300
domain knowledge. 
They want to people with disis 

167
00:09:08,300 --> 00:09:12,200
background, they had a great 
idea, the train programmers 

168
00:09:12,200 --> 00:09:15,100
themselves so that we all could 
have Collective code ownership 

169
00:09:15,100 --> 00:09:17,300
because we all coated this, a 
way we work together. 

170
00:09:17,300 --> 00:09:20,500
We collaborated that was back in
the day when programming was a 

171
00:09:20,508 --> 00:09:22,900
low pay low status job. 
So there were plenty of women. 

172
00:09:23,400 --> 00:09:26,400
A lot of fun eventually. 
I went to work for a software 

173
00:09:26,400 --> 00:09:29,700
vendor and kind of accidentally 
as most people do got into 

174
00:09:29,700 --> 00:09:31,600
testing. 
Oh, our customers are really 

175
00:09:31,600 --> 00:09:34,300
upset that all these bugs go out
with our releases. 

176
00:09:34,500 --> 00:09:36,800
What if we tested it? 
And so that was in the early 

177
00:09:36,800 --> 00:09:38,700
90s. 
When I switch from programming 

178
00:09:38,700 --> 00:09:40,800
to testing, I've just been 
fortunate. 

179
00:09:40,800 --> 00:09:42,400
I've just been on so many great 
teams. 

180
00:09:42,400 --> 00:09:44,100
Most of the time. 
In my career, I've been on some 

181
00:09:44,100 --> 00:09:47,300
bad ones but mostly great ones, 
I was able to jump my first 

182
00:09:47,300 --> 00:09:49,200
extreme programming payment 
2000. 

183
00:09:49,300 --> 00:09:51,300
It was difficult for testers 
because it got all the 

184
00:09:51,300 --> 00:09:53,200
publication's of the time about 
extreme. 

185
00:09:53,300 --> 00:09:55,600
Program, maybe they were great. 
It was all about quality is all 

186
00:09:55,600 --> 00:09:57,700
about testing. 
It was all about people but 

187
00:09:57,700 --> 00:10:00,600
nothing about testers and what 
they might do on an extreme 

188
00:10:00,600 --> 00:10:03,800
programming team. 
So you know, my team and I were 

189
00:10:03,800 --> 00:10:06,300
trying to figure it out through 
the extreme programming, mailing

190
00:10:06,300 --> 00:10:08,900
list, I met Janet. 
So we were all trying to figure 

191
00:10:08,900 --> 00:10:11,200
this out. 
Together Co wrote my first book 

192
00:10:11,200 --> 00:10:13,900
would tip house because we want 
to share what we learned so far.

193
00:10:13,900 --> 00:10:15,500
Because we knew there were other
challenges. 

194
00:10:15,500 --> 00:10:17,700
So it's been all about sharing 
experiences. 

195
00:10:17,700 --> 00:10:20,400
And I've been very fortunate to 
collaborate with Janet since 

196
00:10:20,400 --> 00:10:23,900
then not only on the books. 
But on conference time Since and

197
00:10:23,900 --> 00:10:26,800
read other articles and training
courses. 

198
00:10:26,900 --> 00:10:29,400
It's just been great to reach 
out to our software. 

199
00:10:29,400 --> 00:10:32,000
Can be maybe testing Community 
around the world, get everybody 

200
00:10:32,000 --> 00:10:34,200
to share experiences, and we 
kind of channel that out and 

201
00:10:34,200 --> 00:10:36,100
share it to everybody as much as
we can. 

202
00:10:37,100 --> 00:10:39,700
Thank you for sharing both of 
your stories if I can pick some 

203
00:10:39,700 --> 00:10:42,100
of the interesting things. 
So I didn't know you have stayed

204
00:10:42,100 --> 00:10:44,800
in Singapore and Indonesia China
lesion by the way. 

205
00:10:44,800 --> 00:10:47,300
So that's kind of like 
similarity from both of us. 

206
00:10:47,400 --> 00:10:50,600
So, both of you studied 
programming but then went into 

207
00:10:50,600 --> 00:10:52,900
testing. 
So tell us a little bit more 

208
00:10:52,900 --> 00:10:56,000
white Testing, apart from so 
many other areas in software, 

209
00:10:56,000 --> 00:10:59,300
engineering. 
Okay, my very first job in 

210
00:10:59,300 --> 00:11:01,600
programming. 
I work for the Vancouver Stock 

211
00:11:01,600 --> 00:11:04,600
Exchange and so things had to 
work. 

212
00:11:04,900 --> 00:11:09,200
I was very fortunate that my 
boss every day would come in and

213
00:11:09,200 --> 00:11:13,200
sit with me and say, so, Janet, 
what are you working on today? 

214
00:11:13,500 --> 00:11:15,900
There was no pairing. 
We all set with cubicles and 

215
00:11:15,900 --> 00:11:17,800
things. 
So I would tell him what I was 

216
00:11:17,800 --> 00:11:20,000
going to be working on. 
And then he'd say, so how are 

217
00:11:20,000 --> 00:11:23,000
you going to test that and in at
the end of the day, he would 

218
00:11:23,000 --> 00:11:25,100
come. 
And you'd say, so Janet, what 

219
00:11:25,100 --> 00:11:27,300
did you do today? 
How did you test that? 

220
00:11:27,500 --> 00:11:30,600
So I learned to think about 
testing right from the very 

221
00:11:30,600 --> 00:11:33,800
beginning for my first job. 
And I didn't know any 

222
00:11:33,800 --> 00:11:37,000
difference. 
So, when I went to my next job 

223
00:11:37,000 --> 00:11:41,500
which didn't do testing on their
things and was working out just 

224
00:11:41,500 --> 00:11:46,100
as critical application as Stock
Exchange, it bothered me. 

225
00:11:46,400 --> 00:11:49,500
I coded for a couple of years 
there and then my boss come to 

226
00:11:49,500 --> 00:11:52,900
me and said so Janet's. 
We think we need to have a test 

227
00:11:52,900 --> 00:11:53,400
team. 
Mmmmm. 

228
00:11:53,600 --> 00:11:57,700
You are the only one who was 
constantly complaining about our

229
00:11:57,700 --> 00:12:01,000
princess. 
I would like to be our QA 

230
00:12:01,000 --> 00:12:03,300
manager. 
I knew that I didn't want to 

231
00:12:03,308 --> 00:12:05,700
program forever. 
I knew that from the very 

232
00:12:05,700 --> 00:12:09,000
beginning and so I actually 
jumped at the chance and then I 

233
00:12:09,000 --> 00:12:13,400
took everything I possibly could
learn about testing and started 

234
00:12:13,400 --> 00:12:16,900
there, so it ended up not 
working out for me there, but 

235
00:12:17,200 --> 00:12:20,100
that's how I got into it. 
I've never looked back. 

236
00:12:21,000 --> 00:12:22,700
That's kind of like very 
interesting question. 

237
00:12:22,700 --> 00:12:25,200
How do you test that? 
Because I think many software 

238
00:12:25,200 --> 00:12:26,900
Engineers even don't think about
that. 

239
00:12:26,900 --> 00:12:30,300
So they just build the features 
and finish and maybe pass it to 

240
00:12:30,300 --> 00:12:32,000
Q&A. 
So, how about you Lisa? 

241
00:12:32,000 --> 00:12:34,900
Do you have any interesting 
insights, why you went into 

242
00:12:34,900 --> 00:12:38,000
testing world as well? 
That's a pretty good story. 

243
00:12:38,000 --> 00:12:41,200
So I was working in tech support
for a software vendor a pretty 

244
00:12:41,200 --> 00:12:42,800
big software. 
Vendor, this is back in the days

245
00:12:42,800 --> 00:12:45,000
when Microsoft and Apple were 
still small. 

246
00:12:45,100 --> 00:12:48,000
I think we were bigger, we're 
frustrated because that was 

247
00:12:48,000 --> 00:12:49,600
back. 
When you sent everything out on 

248
00:12:49,600 --> 00:12:53,200
paper to the customers, These 
are Big customers that was 

249
00:12:53,200 --> 00:12:57,200
mostly with database software. 
We do a release, the tapes would

250
00:12:57,200 --> 00:12:59,300
be sent out. 
We had nothing to do with it. 

251
00:12:59,500 --> 00:13:00,700
This was back. 
When people just called on the 

252
00:13:00,708 --> 00:13:03,000
phone, no faxes. 
No email. 

253
00:13:03,500 --> 00:13:05,200
Just had an angry customer on 
the phone. 

254
00:13:05,300 --> 00:13:08,000
Who said, I cannot believe you 
didn't see this giant bug. 

255
00:13:08,200 --> 00:13:10,600
We hadn't even seen it. 
We didn't have it yet either. 

256
00:13:10,900 --> 00:13:12,900
So it occurred to us to ask the 
developers. 

257
00:13:12,900 --> 00:13:15,200
We were in Colorado and they 
were back in Germany. 

258
00:13:15,200 --> 00:13:17,500
And we said, could it be 
possible to send us maybe a 

259
00:13:17,500 --> 00:13:20,200
preliminary version before you 
send it to the customers? 

260
00:13:20,200 --> 00:13:22,200
And we could just Start trying 
it out. 

261
00:13:22,200 --> 00:13:25,200
So they said, yeah, we could 
send it like a week ahead, so we

262
00:13:25,200 --> 00:13:27,900
could install it, we can start 
testing it, we could find bags 

263
00:13:27,900 --> 00:13:31,200
and we can start working on 
patches so that when they angry 

264
00:13:31,200 --> 00:13:32,800
customers called basic, don't 
worry. 

265
00:13:33,000 --> 00:13:34,900
We're working on a patch and 
we'll have that out to you next 

266
00:13:34,900 --> 00:13:38,300
week and we did it on our own 
and then our managers were like 

267
00:13:39,100 --> 00:13:44,900
Sting, what an interesting idea.
What if we have a department 

268
00:13:44,900 --> 00:13:48,900
devoted to people who do testy 
and coordinate their releases 

269
00:13:48,900 --> 00:13:51,300
and cut the released tapes and 
they will the customer Maybe we 

270
00:13:51,300 --> 00:13:54,200
should combine all these things 
and so I put my hand up like 

271
00:13:54,200 --> 00:13:55,800
that, sounds fun, and like 
jamming. 

272
00:13:55,800 --> 00:13:58,900
I've never looked back. 
Wow, so both of you took the 

273
00:13:58,900 --> 00:14:02,200
chance jam into the opportunity 
and went ahead an till now. 

274
00:14:02,400 --> 00:14:05,300
So which brought you into the 
concept of a gel testing. 

275
00:14:05,500 --> 00:14:08,600
The first book was written in 
2009 if I'm not wrong. 

276
00:14:08,800 --> 00:14:11,600
So what is agile testing? 
Maybe if you can give the 

277
00:14:11,600 --> 00:14:14,600
definition for all of us here 
who maybe have not heard about 

278
00:14:14,600 --> 00:14:17,900
it yet? 
Our official definition 

279
00:14:18,100 --> 00:14:22,100
collaborative, testing practices
that occur continuously from 

280
00:14:22,100 --> 00:14:26,200
inception to delivery and Beyond
supporting frequent delivery for

281
00:14:26,200 --> 00:14:29,900
our customers testing activities
focus on building quality into 

282
00:14:29,900 --> 00:14:33,400
the product using fast feedback 
loops to validate our 

283
00:14:33,400 --> 00:14:37,200
understanding in the practices, 
strengthen and support, the idea

284
00:14:37,200 --> 00:14:40,800
of whole team responsibility for
Quality, we spend a lot of time 

285
00:14:40,800 --> 00:14:44,300
trying to put that together but 
really what it means is play 

286
00:14:44,300 --> 00:14:47,800
nice in the sandbox. 
Work together, think about 

287
00:14:47,800 --> 00:14:51,300
testing from the very beginning 
when I think about this and we 

288
00:14:51,300 --> 00:14:54,800
might talk about a little bit 
later is that definition could 

289
00:14:54,800 --> 00:14:57,400
be applied to the term were 
using these days, which is 

290
00:14:57,400 --> 00:15:00,600
holistic testing. 
And by the way, that was a 

291
00:15:00,600 --> 00:15:02,600
community effort. 
We got a lot of inputs and 

292
00:15:02,600 --> 00:15:05,600
people in the community, we've 
written a couple blog posts on 

293
00:15:05,600 --> 00:15:08,600
as we were developing that and 
so you get more details on our 

294
00:15:08,600 --> 00:15:12,500
website at altestore.com slash 
blog and search a room for that.

295
00:15:12,600 --> 00:15:15,400
But yes, Janet that's a really 
good point that the definitions 

296
00:15:15,400 --> 00:15:16,500
don't. 
Works, even if we use a 

297
00:15:16,508 --> 00:15:19,700
different label. 
So if I read the descriptions, 

298
00:15:19,900 --> 00:15:22,500
definitely first, it's like, 
there are so many things mention

299
00:15:22,500 --> 00:15:24,000
there. 
But there are a few key things 

300
00:15:24,000 --> 00:15:25,700
that I observed. 
The first thing is to mention 

301
00:15:25,700 --> 00:15:27,900
about this concept called whole 
team. 

302
00:15:28,200 --> 00:15:31,100
What do you mean by whole team? 
What is the relation with 

303
00:15:31,100 --> 00:15:35,900
testing as a whole team and our 
experience being successful at 

304
00:15:35,900 --> 00:15:39,500
delivering valuable software 
products to customers is an 

305
00:15:39,500 --> 00:15:42,100
effort that requires everybody 
on the team to be committed to 

306
00:15:42,100 --> 00:15:45,600
it and work together. 
We've seen over the years that 

307
00:15:45,800 --> 00:15:48,900
Saying at the end only by tester
doesn't work, even when I work 

308
00:15:48,900 --> 00:15:52,100
in waterfall projects, everybody
still did testing the whole 

309
00:15:52,100 --> 00:15:54,000
time. 
Wonderful doesn't mean you have 

310
00:15:54,000 --> 00:15:58,600
to do a bad job and having that 
mindset where we've all talked 

311
00:15:58,600 --> 00:16:00,600
about that everybody on the 
team. 

312
00:16:00,600 --> 00:16:03,200
Regardless of our specialty, 
here's a global equality. 

313
00:16:03,200 --> 00:16:05,400
We want our software. 
This is what we want to get 

314
00:16:05,400 --> 00:16:07,100
through. 
This is our goal and we're 

315
00:16:07,100 --> 00:16:09,700
committed to getting there 
because we know, it'll be hard. 

316
00:16:09,800 --> 00:16:12,500
We know where run into a lot of 
obstacles, will have a lot of 

317
00:16:12,500 --> 00:16:15,300
hard problems to solve but 
together with all our different 

318
00:16:15,300 --> 00:16:18,700
skills, And experiences will be 
able to do it because we're 

319
00:16:18,700 --> 00:16:20,100
committed to it. 
We're not going to just throw 

320
00:16:20,100 --> 00:16:23,300
our hands up and give up. 
That's what it takes. 

321
00:16:23,500 --> 00:16:26,700
If it's just part of the team, 
even if some of the developers 

322
00:16:26,700 --> 00:16:28,400
like, oh yeah, I really care 
about quality. 

323
00:16:28,400 --> 00:16:31,000
Alright, some unit tests, but if
that's all they do, and they 

324
00:16:31,000 --> 00:16:33,600
don't involve. 
But testers or the testers go 

325
00:16:33,600 --> 00:16:35,400
off and try to do all the 
testing on their own. 

326
00:16:35,600 --> 00:16:39,300
We just seen it doesn't work. 
The state of devops survey has 

327
00:16:39,300 --> 00:16:41,600
really supported that with hard 
data that was damaged in my 

328
00:16:41,600 --> 00:16:44,900
experience over all the years. 
But then we saw data from that 

329
00:16:44,900 --> 00:16:48,700
survey supports It when 
developers own the testing, when

330
00:16:48,700 --> 00:16:52,500
they own the automated test, yet
they created them and maintain 

331
00:16:52,500 --> 00:16:55,100
them along with the testers. 
And the testers help them with 

332
00:16:55,100 --> 00:16:57,400
all these other activities, like
exploratory kissing. 

333
00:16:57,600 --> 00:16:59,600
That's what correlates with high
performing teams. 

334
00:16:59,900 --> 00:17:02,300
I know as humans we don't pay 
attention necessarily the hard 

335
00:17:02,300 --> 00:17:05,599
data but we have the hard data 
to back up our views and it 

336
00:17:05,608 --> 00:17:08,400
starts from the very beginning. 
That question. 

337
00:17:08,700 --> 00:17:12,200
How are we going to test this? 
If we start that at the very 

338
00:17:12,200 --> 00:17:15,200
beginning that drives the 
testing all the way through. 

339
00:17:15,900 --> 00:17:18,000
So I think I agree. 
It's a very important 

340
00:17:18,000 --> 00:17:20,500
distinction, having the whole 
team, really taking 

341
00:17:20,500 --> 00:17:24,000
responsibility of how to test 
the application or the system 

342
00:17:24,000 --> 00:17:26,099
that they're building. 
Because as you can tell in the 

343
00:17:26,099 --> 00:17:28,500
industry, I'm sure throughout 
your Consulting experience as 

344
00:17:28,500 --> 00:17:30,400
well. 
There are many teams still have 

345
00:17:30,400 --> 00:17:33,200
the silo responsibility 
developer just build the 

346
00:17:33,200 --> 00:17:34,900
software. 
While there's another team 

347
00:17:34,900 --> 00:17:37,400
called the testing team, QA 
team, quality engineering, 

348
00:17:37,400 --> 00:17:39,600
whatever you call it. 
That's the testing part. 

349
00:17:39,700 --> 00:17:43,600
So what in your opinion should 
change in order to make it more 

350
00:17:43,600 --> 00:17:47,200
like a whole team concept, all 
this Still kind of like a common

351
00:17:47,200 --> 00:17:49,800
Trend in the industry. 
I can tell at least from my part

352
00:17:49,800 --> 00:17:52,100
of the world. 
What should change in order to 

353
00:17:52,100 --> 00:17:54,200
go towards this whole team 
concept. 

354
00:17:54,700 --> 00:17:57,400
Well and I'm going to take this 
from Elizabeth Hendrick since 

355
00:17:57,400 --> 00:18:00,500
one of my favorite quotes 
testing is an activity that 

356
00:18:00,500 --> 00:18:03,900
happens throughout. 
It is not a phase that happens 

357
00:18:03,900 --> 00:18:06,900
at the end. 
So if we truly believe that 

358
00:18:06,900 --> 00:18:09,600
testing is an activity that 
happens from the very beginning.

359
00:18:10,100 --> 00:18:13,800
When we first see that feature 
that very first feature starting

360
00:18:13,800 --> 00:18:18,100
to think about what are the 
risks X to be able to do that. 

361
00:18:18,200 --> 00:18:20,600
That means that who's ever 
thinking about testers. 

362
00:18:20,600 --> 00:18:23,400
If you have testers on the team,
if you don't have testers on the

363
00:18:23,400 --> 00:18:27,800
team, start thinking about the 
risks at the very beginning and 

364
00:18:27,800 --> 00:18:30,100
start thinking about, how are we
going to mitigate that? 

365
00:18:30,100 --> 00:18:33,800
And a lot of that mitigation 
involves testing it starts 

366
00:18:33,800 --> 00:18:36,200
there. 
And so moving, both testing 

367
00:18:36,200 --> 00:18:40,700
activities, thinking about the 
early, I think that's how we're 

368
00:18:40,700 --> 00:18:44,600
going to change it, don't bring 
in testers when you've got code 

369
00:18:44,600 --> 00:18:47,300
to test. 
The man early to start thinking 

370
00:18:47,300 --> 00:18:50,400
about those risks and really 
talking about that level of 

371
00:18:50,400 --> 00:18:52,400
quality because that's how we 
start it. 

372
00:18:52,700 --> 00:18:56,500
And I think that's has to change
its the mindsets of fatigue and 

373
00:18:56,500 --> 00:19:00,000
individuals because it's how we 
think about testing. 

374
00:19:00,300 --> 00:19:05,000
So I don't use software tester. 
I don't use that term for myself

375
00:19:05,000 --> 00:19:07,000
because I think it's more than 
software. 

376
00:19:07,300 --> 00:19:10,300
We test ideas, we test 
assumptions. 

377
00:19:10,700 --> 00:19:13,700
We're testing many things. 
And so it's not only the 

378
00:19:13,700 --> 00:19:16,500
software. 
I think we test the Product and 

379
00:19:16,500 --> 00:19:20,900
the product is the whole. 
Yep, which you mention is, like 

380
00:19:20,900 --> 00:19:22,700
a mindset, right? 
So in the book, you mentioned 

381
00:19:22,700 --> 00:19:26,500
agile, testing mindset. 
So I can also see, observation 

382
00:19:26,500 --> 00:19:28,100
from at least my part of the 
world. 

383
00:19:28,100 --> 00:19:31,600
So testing is always the Miss. 
Like, maybe lower part of 

384
00:19:31,600 --> 00:19:34,500
engineering, right? 
Where people assume they just do

385
00:19:34,500 --> 00:19:37,000
manual testing during the 
reproductive jobs. 

386
00:19:37,400 --> 00:19:39,900
I know that some testers are 
like good programmers as well. 

387
00:19:39,900 --> 00:19:42,800
They're right great Automation 
and all that but still 

388
00:19:42,800 --> 00:19:46,000
unfortunately, there are still 
these misconceptions also This 

389
00:19:46,000 --> 00:19:49,200
term called quality engineering 
quality assurance, so people 

390
00:19:49,200 --> 00:19:52,400
think they added efficient that 
is responsible for Quality. 

391
00:19:52,700 --> 00:19:56,800
So I think all these, maybe need
your advice, how to change this 

392
00:19:56,800 --> 00:19:59,600
mindset, so that we can move 
towards this, a gel testing 

393
00:19:59,600 --> 00:20:02,100
mindset. 
Again, where everyone will 

394
00:20:02,100 --> 00:20:05,200
involve in the testing and 
really cares about quality since

395
00:20:05,200 --> 00:20:07,200
the beginning. 
Yeah, Ted. 

396
00:20:07,200 --> 00:20:09,900
And I were just chatting about 
that this morning, because it's 

397
00:20:09,900 --> 00:20:13,100
not easy, and I'm just trying to
think back that had in mind, 

398
00:20:13,100 --> 00:20:16,000
mindset changed. 
I remember the first first 

399
00:20:16,000 --> 00:20:18,700
two-week iteration on my first 
extreme programming team. 

400
00:20:18,900 --> 00:20:21,800
I was working for a start-up 
consulting company and most of 

401
00:20:21,800 --> 00:20:24,700
the first two weeks, I was 
helping another client of sight.

402
00:20:24,900 --> 00:20:28,300
So I came back just like the day
before we're going to demo to 

403
00:20:28,300 --> 00:20:30,400
the customers or a very first 
iteration. 

404
00:20:30,600 --> 00:20:32,700
One of the things I did was 
okay, I'm going to start up the 

405
00:20:32,700 --> 00:20:35,700
server and we got this little 
application going and I'm going 

406
00:20:35,700 --> 00:20:38,800
to log in as two different users
and the server crash. 

407
00:20:39,500 --> 00:20:41,700
And I was like, what? 
We can't show this to the 

408
00:20:41,700 --> 00:20:43,900
customers. 
It's terrible. 

409
00:20:43,900 --> 00:20:45,500
You can even have two people on 
it. 

410
00:20:46,100 --> 00:20:47,400
Fortunately, an extreme 
programming. 

411
00:20:47,400 --> 00:20:49,600
You have a coach part of your 
team. 

412
00:20:49,800 --> 00:20:50,800
Archive said. 
Now. 

413
00:20:50,800 --> 00:20:54,300
Lisa, we don't have a story in 
this iteration for supporting 

414
00:20:54,300 --> 00:20:56,900
more than one user. 
The reason for that is our 

415
00:20:56,900 --> 00:21:01,900
customer is developing this to 
show people the potential 

416
00:21:01,900 --> 00:21:04,400
futures of their product. 
They need funding and I'm going 

417
00:21:04,400 --> 00:21:06,200
to use it, they're just going to
demonstrate it. 

418
00:21:06,300 --> 00:21:08,100
They don't need to have two 
people logged in. 

419
00:21:08,200 --> 00:21:12,600
I was like what, what what oh 
that's mind-blowing to me. 

420
00:21:12,600 --> 00:21:15,500
Quality was the server says up 
its reliable. 

421
00:21:15,600 --> 00:21:18,000
Thing works. 
But now the quality is what the 

422
00:21:18,008 --> 00:21:20,800
customer needs. 
We really had to focus on that 

423
00:21:21,100 --> 00:21:24,200
so I really think it's an 
ongoing process, I think it 

424
00:21:24,200 --> 00:21:27,000
needs training. 
I think it needs daily coaching 

425
00:21:27,000 --> 00:21:30,200
from somebody who's done it 
before, who knows what they're 

426
00:21:30,200 --> 00:21:33,100
doing to help the team Janet. 
And I like to say, if you 

427
00:21:33,100 --> 00:21:36,300
haven't already been on a high 
performing, a dollar whole team 

428
00:21:36,300 --> 00:21:38,200
approach. 
Team, you can understand what 

429
00:21:38,200 --> 00:21:41,600
the unicorn magic of that is, 
you really have to experience 

430
00:21:41,600 --> 00:21:43,700
it. 
And so, you know, if people can 

431
00:21:43,700 --> 00:21:46,500
get somebody hire somebody at 
least temporarily, Like who does

432
00:21:46,500 --> 00:21:49,100
know, is who does understand it 
and can help the team get over 

433
00:21:49,100 --> 00:21:50,800
it, because mindset switches are
hard. 

434
00:21:50,800 --> 00:21:53,900
It's a cultural change. 
As Janet says, you know, we're 

435
00:21:53,900 --> 00:21:56,300
going to focus now on preventing
bugs not catching bugs at the 

436
00:21:56,300 --> 00:21:57,400
end. 
We're not going to be the 

437
00:21:57,400 --> 00:21:59,500
quality police because it's not 
our job to determine what 

438
00:21:59,500 --> 00:22:01,900
quality is. 
It's the customers now we've got

439
00:22:01,900 --> 00:22:03,300
to find out. 
What does the customer want? 

440
00:22:03,300 --> 00:22:06,300
That's big part of our job. 
As testers this, to help 

441
00:22:06,300 --> 00:22:08,700
everybody understand, get a 
chair and understanding of what 

442
00:22:08,700 --> 00:22:12,400
the customer really needs. 
That's a really good story Lisa.

443
00:22:12,400 --> 00:22:15,800
I wish I would have heard that 
20 years ago because That's 

444
00:22:15,800 --> 00:22:19,000
exactly it. 
From a testing perspective, the 

445
00:22:19,000 --> 00:22:23,600
hardest thing is to understand 
that we are testing this small 

446
00:22:23,600 --> 00:22:26,000
slice. 
We are making sure that this 

447
00:22:26,000 --> 00:22:29,200
small slice works right now, 
doesn't mean that we're going to

448
00:22:29,200 --> 00:22:32,700
give it to the customer right 
now if that's what they need, if

449
00:22:32,700 --> 00:22:33,800
they can use it. 
Yes. 

450
00:22:34,200 --> 00:22:38,600
But from a testing perspective 
that is the hardest thing is to 

451
00:22:38,600 --> 00:22:42,100
understand that their testing is
narrow for this particular 

452
00:22:42,100 --> 00:22:44,800
Story. 
Once testers get used to that 

453
00:22:44,900 --> 00:22:49,300
they realize Much easier it is 
to keep testing and adding 

454
00:22:49,300 --> 00:22:51,400
complexity. 
You have that first story and 

455
00:22:51,400 --> 00:22:54,400
then that works. 
You add complexity and you can 

456
00:22:54,400 --> 00:22:57,700
test that and that works. 
You just keep wrapping that 

457
00:22:57,700 --> 00:23:00,800
complexity around and you get a 
solid feature. 

458
00:23:01,200 --> 00:23:03,600
And I think that makes the world
of difference. 

459
00:23:03,900 --> 00:23:07,000
But it is a hard mindset to 
think that we have to do that 

460
00:23:07,000 --> 00:23:08,500
narrow. 
It's funny. 

461
00:23:08,500 --> 00:23:10,300
I have to laugh here. 
I just going to add a little 

462
00:23:10,300 --> 00:23:14,100
story because I can't talk 
without my hands. 

463
00:23:14,300 --> 00:23:19,100
So, and Gasps, you have to 
visualize that I'm holding my 

464
00:23:19,100 --> 00:23:21,300
hands up and I'm doing narrow 
Focus. 

465
00:23:23,300 --> 00:23:25,400
That's true. 
That's the thing in my early 

466
00:23:25,400 --> 00:23:29,900
days or even for the first two 
years that I was on agile teams,

467
00:23:30,200 --> 00:23:33,000
my teammates had to keep 
reminding me, Lisa, I know you 

468
00:23:33,000 --> 00:23:35,700
just think you found a bug could
you just make sure the happy 

469
00:23:35,700 --> 00:23:38,200
path works first so that we can 
be confident that we have at 

470
00:23:38,200 --> 00:23:41,000
least started in a good way 
because, you know, I was always 

471
00:23:41,000 --> 00:23:42,900
going off in the weeds. 
I bet there's a bug over here. 

472
00:23:43,100 --> 00:23:44,600
No. 
Lisa, we think it works. 

473
00:23:44,600 --> 00:23:47,800
We want you to help us. 
I know that it works driving 

474
00:23:47,800 --> 00:23:51,400
development with those tests, I 
was like, creating huge numbers.

475
00:23:51,400 --> 00:23:53,700
I was with the product owner, 
we're creating these huge 

476
00:23:53,700 --> 00:23:57,200
matrices of test cases and 
expected outputs and given that 

477
00:23:57,200 --> 00:23:58,900
the developers, okay, here's 
what you should do. 

478
00:23:58,900 --> 00:24:01,400
And they're like, oh my gosh, I 
can't see the forest for the 

479
00:24:01,400 --> 00:24:02,900
trees. 
So can you just give us the 

480
00:24:02,900 --> 00:24:04,900
happy path? 
Let's all make sure that works. 

481
00:24:05,100 --> 00:24:09,000
Then let's add on like one 
unusual scenario and just do the

482
00:24:09,000 --> 00:24:11,900
step at a time and that was 
something I had to learn because

483
00:24:11,900 --> 00:24:14,900
I had been more used to even 
though I was trying to always 

484
00:24:14,900 --> 00:24:17,800
collaborate with Developers and 
Tessa still, we were doing big, 

485
00:24:17,800 --> 00:24:19,500
bang things. 
So I was getting a whole bunch 

486
00:24:19,500 --> 00:24:21,700
of stuff to test that once and 
that's a quite a different 

487
00:24:21,700 --> 00:24:22,700
experience. 
Yeah. 

488
00:24:23,000 --> 00:24:26,100
And I think when I've taught a 
course hundreds of times and I 

489
00:24:26,108 --> 00:24:29,100
think that's the biggest 
takeaway most of the time from 

490
00:24:29,100 --> 00:24:32,200
the whole team because we teach 
it to the whole team is that 

491
00:24:32,200 --> 00:24:35,700
they can do that once. 
Well piece and add complexity as

492
00:24:35,700 --> 00:24:38,400
they go and test it. 
And I think that's a real 

493
00:24:38,400 --> 00:24:40,400
showstopper for a lot of people 
just going. 

494
00:24:40,400 --> 00:24:43,000
Whoa. 
And she said, I was laughing 

495
00:24:43,000 --> 00:24:46,100
when you shared that because I 
wasn't even Being about 

496
00:24:46,100 --> 00:24:48,300
application, that should just 
work for one people. 

497
00:24:48,500 --> 00:24:51,100
We always assume that it could 
work for multiple people. 

498
00:24:51,400 --> 00:24:54,200
So you brought a point where 
sometimes you have testing is 

499
00:24:54,200 --> 00:24:56,800
hard because you don't know what
kind of slice that you should be

500
00:24:56,800 --> 00:24:59,000
focusing on. 
And I think you mentioned the 

501
00:24:59,000 --> 00:25:02,100
key point where we should test 
within a smaller scope and 

502
00:25:02,100 --> 00:25:04,500
complexity afterwards. 
And it seems like this kind of 

503
00:25:04,508 --> 00:25:07,800
like a line to your concept of 
holistic testing where you have 

504
00:25:07,800 --> 00:25:10,900
this short cycle from build test
deploy and all that. 

505
00:25:11,100 --> 00:25:13,500
So maybe this is a good time to 
actually introduce. 

506
00:25:13,500 --> 00:25:16,300
What do you mean by a holistic? 
Testing that You are talking 

507
00:25:16,300 --> 00:25:19,300
pretty lately. 
I'll take this one, because 

508
00:25:19,400 --> 00:25:22,100
there's a lot of talk about 
shift left shift, right? 

509
00:25:22,400 --> 00:25:24,400
And when I hear shift, left 
shift, right? 

510
00:25:24,500 --> 00:25:28,200
I think of a lateral line, like 
a horizontal line, I'm thinking 

511
00:25:28,200 --> 00:25:30,000
that software development isn't 
like that. 

512
00:25:30,300 --> 00:25:34,600
So the devops loop started and I
think that's great for some I 

513
00:25:34,600 --> 00:25:38,300
had seen that was in discovered 
to deliver Island Goddess Diana 

514
00:25:38,300 --> 00:25:40,800
and Mary Gorman's book. 
They had this nice little loop, 

515
00:25:40,800 --> 00:25:44,800
the devops loop, and then Dan 
Ashby, and one of his blog post 

516
00:25:44,800 --> 00:25:48,300
talked about Newest testing, he 
said the step up so it was 

517
00:25:48,300 --> 00:25:50,600
great. 
But where does testing? 

518
00:25:50,700 --> 00:25:54,300
They always had this choke phase
for testing and it didn't work. 

519
00:25:54,500 --> 00:25:58,200
So he made this we test here and
he put all around the loop. 

520
00:25:58,200 --> 00:26:00,100
We test here and here and here 
and everywhere. 

521
00:26:00,300 --> 00:26:03,900
And I thought, perfect and I use
that for a long time but still 

522
00:26:03,900 --> 00:26:06,100
something bothered me about the 
devops loop. 

523
00:26:06,500 --> 00:26:09,800
So Lisa, come up with one 
iteration and then we just kept 

524
00:26:09,800 --> 00:26:11,400
playing with it. 
And finally, I kind of had this 

525
00:26:11,400 --> 00:26:15,100
little mini light bulb moment 
and took the the loop as it is 

526
00:26:15,100 --> 00:26:19,200
now, Now and wrote a blog post 
on it showing that testing 

527
00:26:19,200 --> 00:26:22,000
really does start from the 
discovery. 

528
00:26:22,300 --> 00:26:25,100
So when we start thinking about 
Discovery, that's a very 

529
00:26:25,100 --> 00:26:28,800
beginning thinking about the 
risks and planning for it going 

530
00:26:28,800 --> 00:26:32,300
through the building and those 
are all stages that we do, 

531
00:26:32,400 --> 00:26:34,600
whether we go really fast 
through them because we're doing

532
00:26:34,600 --> 00:26:39,100
a small story, we discover we 
understand and we deploy and 

533
00:26:39,100 --> 00:26:42,800
test, we might put it into 
production but one way or the 

534
00:26:42,800 --> 00:26:44,400
other, we have an internal 
release it. 

535
00:26:44,400 --> 00:26:47,300
And if we don't put it too, 
Action and if they gave out what

536
00:26:47,300 --> 00:26:50,700
we learn from it. 
So it really is looking at that 

537
00:26:50,700 --> 00:26:54,800
whole cycle, thinking about 
testing holistically, when I 

538
00:26:54,808 --> 00:26:57,400
read that definition, and I 
hadn't really thought about that

539
00:26:57,400 --> 00:27:01,300
definition agile, testing, and 
it's very applicable to holistic

540
00:27:01,300 --> 00:27:03,400
testing. 
But I'm going to let Lisa talk 

541
00:27:03,400 --> 00:27:06,000
about why we call. 
It holistic, testing versus 

542
00:27:06,000 --> 00:27:09,500
agile testing these days. 
I mean, obviously, it was Janet 

543
00:27:09,500 --> 00:27:12,500
that came up with this idea. 
But part of the problem was Dan 

544
00:27:12,500 --> 00:27:14,300
Ashe. 
We use continuous testing, which

545
00:27:14,300 --> 00:27:17,500
made a lot of sense to us. 
And then somehow people took the

546
00:27:17,500 --> 00:27:21,500
term continuous testing and 
co-opted it to only mean the 

547
00:27:21,500 --> 00:27:23,800
automated regression tests that 
run continuous integration. 

548
00:27:23,800 --> 00:27:27,600
And so when you s taste testing,
that's what people thought of it

549
00:27:27,600 --> 00:27:29,600
was like that's like a teeny 
tiny little part. 

550
00:27:29,600 --> 00:27:31,100
At that thing, it's an important
part. 

551
00:27:31,100 --> 00:27:34,600
What describes it better. 
So Jenna came up with the word 

552
00:27:34,600 --> 00:27:38,400
holistic, and, yeah, because 
whole team approach going 

553
00:27:38,400 --> 00:27:41,300
through the whole cycle. 
The testing activities are in 

554
00:27:41,300 --> 00:27:43,200
the whole cycle and it really 
sums it up. 

555
00:27:43,500 --> 00:27:45,500
What I found interesting, isn't 
it? 

556
00:27:45,600 --> 00:27:48,800
Just recently, I was still a 
hint on the tester on a future 

557
00:27:48,800 --> 00:27:52,300
team and we're going to company 
with several different teams. 

558
00:27:52,500 --> 00:27:55,800
I shared Janet's blog post on 
our slack. 

559
00:27:56,000 --> 00:27:59,400
One of the really experienced 
testers on another team was like

560
00:27:59,800 --> 00:28:02,400
and he had a light bulb moment 
to, he says, this model is 

561
00:28:02,400 --> 00:28:04,700
great. 
I can finally explain to people 

562
00:28:04,700 --> 00:28:07,600
what it is. 
We do because it really summed 

563
00:28:07,600 --> 00:28:10,000
up when there are well, 
performing teams that are doing 

564
00:28:10,000 --> 00:28:11,600
a good job of the holistic 
approach. 

565
00:28:11,600 --> 00:28:12,900
And there's a tester on that 
team. 

566
00:28:12,900 --> 00:28:14,600
Kind of acting is a testing 
consultant. 

567
00:28:14,600 --> 00:28:17,100
Guiding them. 
Leading that effort now, he can 

568
00:28:17,100 --> 00:28:18,800
take it explained. 
The other thing, this is what we

569
00:28:18,800 --> 00:28:21,100
do. 
I have a way to show you now, 

570
00:28:21,300 --> 00:28:23,800
other people have also come back
with that kind of feedback. 

571
00:28:24,000 --> 00:28:25,900
One of the ways that resonated 
with people who are already 

572
00:28:25,900 --> 00:28:28,900
doing it, recognize it. 
So, we hope it'll help other 

573
00:28:28,900 --> 00:28:32,900
people than be able to achieve 
that holistic approach in grow 

574
00:28:32,900 --> 00:28:36,600
that in their own teams. 
So maybe if you can walk us 

575
00:28:36,600 --> 00:28:38,800
through in terms of practical 
sense, right? 

576
00:28:38,800 --> 00:28:41,300
So you have this cycle, there 
are multiple stages in the 

577
00:28:41,300 --> 00:28:43,400
cycle. 
So I just pick a starting point 

578
00:28:43,400 --> 00:28:45,400
Discovery as you mentioned that 
you have planned. 

579
00:28:45,500 --> 00:28:49,000
Understanding building deploying
releasing observability and 

580
00:28:49,000 --> 00:28:50,800
learning throughout these 
stages. 

581
00:28:50,800 --> 00:28:53,800
There should be some level of 
testing involved, so maybe if 

582
00:28:53,800 --> 00:28:56,300
you can walk us through in 
Practical terms day to day, 

583
00:28:56,300 --> 00:28:58,500
maybe in the spring, how do you 
run this? 

584
00:28:58,700 --> 00:29:01,300
Do we create a story for each of
the stage, or do we have one 

585
00:29:01,300 --> 00:29:03,100
story where each has its own 
stage. 

586
00:29:03,700 --> 00:29:07,900
So if we think about it, say The
Discovery, I would say we have a

587
00:29:07,900 --> 00:29:10,200
story. 
Let's take the simplest story 

588
00:29:10,200 --> 00:29:15,300
that we possibly can one person 
can log in using leases. 

589
00:29:15,500 --> 00:29:18,700
I think so that story is the 
happy path. 

590
00:29:18,900 --> 00:29:22,400
Lisa can log into the system, 
what does that take? 

591
00:29:22,700 --> 00:29:25,600
And so we would take and we 
would think about what risks 

592
00:29:25,600 --> 00:29:28,600
there are. 
Maybe the risk is a servers and 

593
00:29:28,600 --> 00:29:31,000
up, she can't get validated, 
whatever it is. 

594
00:29:31,300 --> 00:29:34,600
We think about the risks. 
We might share some acceptance 

595
00:29:34,600 --> 00:29:36,300
tests. 
High-level acceptance test. 

596
00:29:36,300 --> 00:29:39,700
We might do example mapping if 
it needed to be, so that's part 

597
00:29:39,700 --> 00:29:42,800
of the understanding. 
So you're planning at a high 

598
00:29:42,800 --> 00:29:44,200
level. 
What is it take? 

599
00:29:44,200 --> 00:29:46,000
You understand that. 
All right. 

600
00:29:46,000 --> 00:29:48,600
And every story will be 
different depending on how well 

601
00:29:48,600 --> 00:29:50,600
loan it is. 
What are the risks? 

602
00:29:51,000 --> 00:29:53,900
So, then you have some 
high-level acceptance tests. 

603
00:29:53,900 --> 00:29:57,600
You were two examples and the 
whole team has an understanding 

604
00:29:57,600 --> 00:30:00,500
of what they are going to build 
and then they build it. 

605
00:30:00,600 --> 00:30:03,400
Of course, the building will 
include automating it. 

606
00:30:03,400 --> 00:30:07,400
Include exploratory, testing on 
that story and includes tdd for 

607
00:30:07,400 --> 00:30:10,200
the programmers. 
It includes whatever it takes to

608
00:30:10,200 --> 00:30:13,000
make that code into a Deployable
piece. 

609
00:30:13,400 --> 00:30:17,300
So you have that code. 
Deploy it your automated tests 

610
00:30:17,300 --> 00:30:19,800
are already running. 
You can run them on your system.

611
00:30:20,100 --> 00:30:23,700
You can do any other human 
Centric, kinds of testing, like 

612
00:30:23,700 --> 00:30:27,100
perhaps you have to do some 
performance testing or something

613
00:30:27,100 --> 00:30:30,800
like that on a deployed system. 
So all of these little testing 

614
00:30:30,800 --> 00:30:34,100
activities happen and then if 
you really sit with us 

615
00:30:34,200 --> 00:30:37,800
internally or externally to a 
customer, you're going to make 

616
00:30:37,800 --> 00:30:41,900
sure that happens and so testing
in production, every time I say,

617
00:30:41,900 --> 00:30:44,700
testing and production is that 
back 20 years ago? 

618
00:30:44,700 --> 00:30:47,000
No, no That's not what we mean 
anymore. 

619
00:30:47,500 --> 00:30:49,300
But you can do that. 
And then you can have your 

620
00:30:49,300 --> 00:30:52,600
observability and you're 
monitoring happening, but that's

621
00:30:52,600 --> 00:30:55,500
all about learning. 
How is your customer using it. 

622
00:30:55,500 --> 00:30:58,400
If you've actually put it out to
production if they're using it 

623
00:30:58,700 --> 00:31:01,800
and then you learn and you go 
into your next story. 

624
00:31:02,200 --> 00:31:06,300
So that cycle can be very fast. 
If you doing something very 

625
00:31:06,300 --> 00:31:10,200
simple like logging in with a 
valid user, we're not doing any 

626
00:31:10,200 --> 00:31:12,700
extra step, we're just doing 
that happy path. 

627
00:31:12,900 --> 00:31:15,300
The next story might be 
validating all. 

628
00:31:15,400 --> 00:31:20,000
The usernames adding complexity.
Maybe the third story is 

629
00:31:20,000 --> 00:31:23,600
something else, but you adding 
complexity all the way along and

630
00:31:23,600 --> 00:31:27,800
I think that is recognizing that
Loop is just a continuous cycle 

631
00:31:28,200 --> 00:31:31,900
constantly adding on. 
One thing I particularly like 

632
00:31:31,900 --> 00:31:35,600
about the loop is haven't this 
part where we're going to learn 

633
00:31:35,600 --> 00:31:38,600
observing production that's one 
of these past few years. 

634
00:31:38,600 --> 00:31:41,900
My mission has been to make sure
that testers get involved with 

635
00:31:41,900 --> 00:31:46,100
that because our products are so
complex these days were Is the 

636
00:31:46,100 --> 00:31:49,700
cloud we're using all these 
Services microservices we need 

637
00:31:49,700 --> 00:31:53,400
to start as we're thinking about
what code we're going to rate 

638
00:31:53,400 --> 00:31:56,300
for that little story that Jan 
has talking about, what 

639
00:31:56,300 --> 00:31:59,000
Telemetry do we need? 
How should we instrument that 

640
00:31:59,000 --> 00:32:00,400
code? 
What events do we need to 

641
00:32:00,400 --> 00:32:03,800
capture so that we can create 
our monitoring dashboard so that

642
00:32:03,800 --> 00:32:05,500
we can create our alerts, if 
something goes wrong in 

643
00:32:05,500 --> 00:32:07,500
production. 
So we have all the data we need 

644
00:32:07,500 --> 00:32:09,700
for the things, we didn't 
anticipate and didn't create 

645
00:32:09,700 --> 00:32:12,900
dashboards or alerts for, and we
can learn very quickly what went

646
00:32:12,900 --> 00:32:17,100
wrong and fixed it and also 
we're We're so lucky these days.

647
00:32:17,300 --> 00:32:20,200
The analytics tools that are out
there which we're done for 

648
00:32:20,200 --> 00:32:22,800
marketing and product so they 
know what customers are doing. 

649
00:32:22,900 --> 00:32:26,600
So valuable to us as testers, we
could go out and look at Sea, 

650
00:32:26,700 --> 00:32:29,500
power integers, using our 
product, you know, some of their

651
00:32:29,500 --> 00:32:31,600
tools and I'll show you a screen
cast that what somebody did, 

652
00:32:31,600 --> 00:32:34,300
which is kind of creepy. 
But it's very educational when 

653
00:32:34,300 --> 00:32:37,000
you see somebody clicking on 
something that's not clickable 

654
00:32:37,000 --> 00:32:39,200
or just moving the mouse around 
because they don't know what 

655
00:32:39,200 --> 00:32:41,700
they're doing this. 
So helpful, to improve our 

656
00:32:41,700 --> 00:32:45,200
product to know where to focus. 
Our testing to know how to help 

657
00:32:45,200 --> 00:32:47,600
you. 
Hers, we need the whole picture.

658
00:32:47,600 --> 00:32:49,700
We can't just be people say, 
shift left shift, right? 

659
00:32:49,700 --> 00:32:51,800
We can do one of the other. 
We've got to be holistic. 

660
00:32:51,800 --> 00:32:54,400
We've got to go through the 
whole cycle and be involved all 

661
00:32:54,400 --> 00:32:56,700
the time. 
Using our testing skills to 

662
00:32:56,700 --> 00:33:00,000
identify risks. 
It's a spot anomalies to spot 

663
00:33:00,000 --> 00:33:03,900
patterns to help the theme. 
See those, I was just actually 

664
00:33:03,900 --> 00:33:08,400
listening to your last podcast 
with Liz found Jones. 

665
00:33:08,500 --> 00:33:11,500
She was talking about 
observability, one of the things

666
00:33:11,500 --> 00:33:14,300
that she said and I went this is
the way to think about it 

667
00:33:14,400 --> 00:33:16,900
because it hit home. 
The way she said it was, can you

668
00:33:16,900 --> 00:33:21,500
understand what's happening in 
your system and why without 

669
00:33:21,500 --> 00:33:25,900
having to push additional code 
slicing and dicing data from the

670
00:33:25,900 --> 00:33:28,600
Telemetry signals. 
And I thought, what a wonderful 

671
00:33:28,600 --> 00:33:31,000
statement that is about 
observability. 

672
00:33:31,500 --> 00:33:34,700
It sometimes I well, because I 
remember somebody paid for years

673
00:33:34,700 --> 00:33:38,200
of oh God, we cannot figure out 
why it says 500 error happening 

674
00:33:38,200 --> 00:33:40,800
with our web server. 
Okay, well let's add this 

675
00:33:40,800 --> 00:33:43,600
Telemetry and let's redeploy. 
A little that didn't do it. 

676
00:33:43,600 --> 00:33:46,600
Let's have this very good boy. 
No, that lives, this one of my 

677
00:33:46,600 --> 00:33:51,000
sheroes, the people in that 
space and in observability and 

678
00:33:51,000 --> 00:33:54,000
site reliability, engineering. 
They understand testing, and 

679
00:33:54,000 --> 00:33:56,000
they understand the importance 
of having this data. 

680
00:33:56,300 --> 00:33:59,100
We all need to be aware of that.
And not think that some separate

681
00:33:59,100 --> 00:34:01,300
area of expertise that we don't 
need to know about. 

682
00:34:01,600 --> 00:34:04,400
Lisa said, that's one of the 
testing activities that happen 

683
00:34:04,500 --> 00:34:08,300
in that understanding is what do
we have to put into the coat? 

684
00:34:09,000 --> 00:34:11,900
Having heard what, you just 
described both of you in detail 

685
00:34:11,900 --> 00:34:14,900
in depth including an example 
how we can practice it? 

686
00:34:15,000 --> 00:34:18,500
I think It totally makes sense. 
I can see why in the industry, 

687
00:34:18,500 --> 00:34:21,900
sometimes we all work in silos, 
most of the times developer just

688
00:34:21,900 --> 00:34:23,699
read, okay? 
This is the story requirement. 

689
00:34:23,699 --> 00:34:25,800
That's it. 
I don't think about Telemetry, I

690
00:34:25,808 --> 00:34:28,400
don't think about how we can 
measure user success and all 

691
00:34:28,408 --> 00:34:29,800
that. 
I just built based on the 

692
00:34:29,800 --> 00:34:32,400
specification has this probably 
also follow. 

693
00:34:32,500 --> 00:34:35,300
So, I think the whole concept of
having this cycle for every 

694
00:34:35,300 --> 00:34:38,199
story, you can be bigger scope. 
It can be small scope. 

695
00:34:38,199 --> 00:34:40,699
But having all these cycle 
covered in a story actually, 

696
00:34:40,699 --> 00:34:43,300
really good, because you can 
have a good understanding in 

697
00:34:43,300 --> 00:34:45,199
depth from the start from the 
discovery up. 

698
00:34:45,400 --> 00:34:48,600
That use that themselves how we 
can monitor and make sure that 

699
00:34:48,600 --> 00:34:50,600
the story itself brings value to
the user. 

700
00:34:50,900 --> 00:34:53,500
Thank you so much for explaining
this concept. 

701
00:34:53,600 --> 00:34:56,900
One part that I think with the 
mention because you assume that 

702
00:34:56,900 --> 00:34:59,800
the story can be deployed to 
production where you have the 

703
00:34:59,900 --> 00:35:03,800
release deploy and observability
but so many people still haven't

704
00:35:03,800 --> 00:35:05,600
practiced this, which is 
continuous delivery. 

705
00:35:05,900 --> 00:35:07,800
How important is continuous 
delivery. 

706
00:35:07,800 --> 00:35:13,100
In this holistic testing, well I
see so many different contexts 

707
00:35:13,400 --> 00:35:17,300
to make continuous delivery is 
Is something to think about and 

708
00:35:17,300 --> 00:35:21,800
to strive for, even if you live 
or do it, a lot of things are on

709
00:35:21,900 --> 00:35:24,900
three months, release Cycles, 
delivering to the customer 

710
00:35:24,900 --> 00:35:29,000
between month, and it might be 
because their customer chooses, 

711
00:35:29,000 --> 00:35:31,600
not to take it. 
It's a requirement that a lot of

712
00:35:31,600 --> 00:35:34,700
customers do, they? 
Don't want almost little fixes 

713
00:35:34,700 --> 00:35:38,000
for many reasons, but if you 
have that goal and you're 

714
00:35:38,000 --> 00:35:41,400
thinking about it, then that 
means that your stories are 

715
00:35:41,400 --> 00:35:44,500
going to be testable. 
Your stories are going to be 

716
00:35:44,500 --> 00:35:47,100
small enough. 
You can test and you will get 

717
00:35:47,100 --> 00:35:50,500
those good habits. 
Even if you choose only to give 

718
00:35:50,500 --> 00:35:54,200
it to the customer who wants to.
Well I encourage teams to 

719
00:35:54,200 --> 00:35:57,800
practice continuous delivery to 
go for that even if it's on an 

720
00:35:57,800 --> 00:36:01,200
internal server and they're 
practicing it on their staging 

721
00:36:01,200 --> 00:36:05,100
environment, they can still get 
to that even if they don't push 

722
00:36:05,100 --> 00:36:09,800
it to the customer every time, 
it's a great way to get all of 

723
00:36:09,808 --> 00:36:13,600
those practices in place. 
So it's not going to happen 

724
00:36:13,600 --> 00:36:15,900
overnight because a lot of 
things Don't even have 

725
00:36:15,900 --> 00:36:18,900
automation yet. 
How did you get there? 

726
00:36:19,100 --> 00:36:22,800
That's just a series of small 
steps understanding. 

727
00:36:22,800 --> 00:36:25,000
That every team has a different 
context. 

728
00:36:25,300 --> 00:36:28,800
So we talked about it as 
everybody does but they don't 

729
00:36:29,200 --> 00:36:32,100
they don't there are still teams
out there that don't even have 

730
00:36:32,100 --> 00:36:35,300
continuous integration. 
I remember when I told Lisa that

731
00:36:35,300 --> 00:36:40,000
was probably 10 years ago or 15 
years ago and she goes what? 

732
00:36:40,200 --> 00:36:43,600
Because every team she worked on
problems and she couldn't see 

733
00:36:43,600 --> 00:36:46,900
working any other way. 
But they're still teams without 

734
00:36:46,900 --> 00:36:50,200
continuous integration, which, 
you know, it's kind of sad, but 

735
00:36:50,200 --> 00:36:53,500
I know they exist. 
It's puzzling to me because 

736
00:36:53,500 --> 00:36:54,900
like, it says everything I've 
been on. 

737
00:36:54,900 --> 00:36:57,100
I had it because it only took us
a day or two to get it going. 

738
00:36:57,100 --> 00:37:00,400
So, what is the problem? 
People, it's not hard, but I 

739
00:37:00,400 --> 00:37:02,800
like Janet says, before we even 
had the words continuous 

740
00:37:02,800 --> 00:37:05,300
delivery, which I first learned 
from Jazz humble, and they 

741
00:37:05,300 --> 00:37:08,300
Farley's excellent book of the 
same name, which is a book about

742
00:37:08,300 --> 00:37:11,100
testing. 
My team, took us a few years 

743
00:37:11,100 --> 00:37:13,800
because at first, we couldn't 
even get a passing build with an

744
00:37:13,800 --> 00:37:15,200
artifact. 
We could deploy to production. 

745
00:37:15,300 --> 00:37:17,500
In two weeks. 
And then we tried to get one 

746
00:37:17,500 --> 00:37:19,600
everything we set goals, okay? 
We're going to have a passing 

747
00:37:19,600 --> 00:37:21,700
build every day and for the days
we don't have that. 

748
00:37:21,700 --> 00:37:24,200
We're going to have a calendar 
where we put colors to remind 

749
00:37:24,200 --> 00:37:26,300
us. 
So everybody in the company can 

750
00:37:26,300 --> 00:37:29,200
see including the executives and
start to understand the 

751
00:37:29,200 --> 00:37:30,600
importance of having this 
passing. 

752
00:37:30,600 --> 00:37:32,900
Both everyday just build it a 
step by step. 

753
00:37:33,100 --> 00:37:35,200
It took us years to get there 
but then we always had a 

754
00:37:35,207 --> 00:37:38,000
Deployable artifacts. 
We could deploy to production in

755
00:37:38,000 --> 00:37:41,000
our business domain customer 
didn't want new changes every 

756
00:37:41,000 --> 00:37:44,300
day so we only did it every two 
weeks but we had the ability at,

757
00:37:44,300 --> 00:37:47,300
that's the goal. 
One thing I tell people that 

758
00:37:47,300 --> 00:37:50,400
don't have anything, they don't 
even have CI, you have a 

759
00:37:50,400 --> 00:37:53,200
deployment pipeline changes that
you make in your software 

760
00:37:53,200 --> 00:37:56,200
product gets production. 
Somehow there are steps that you

761
00:37:56,200 --> 00:37:59,400
take to get there even if their 
manual and it's really important

762
00:37:59,400 --> 00:38:01,800
that the whole team understands 
all those steps. 

763
00:38:02,100 --> 00:38:06,000
So we tell people sit down 
together, but use your virtual 

764
00:38:06,000 --> 00:38:10,100
whiteboard and virtual sticky 
notes and draw your pipeline, 

765
00:38:10,100 --> 00:38:12,800
even if it's manual, or even if 
it's some automation, some 

766
00:38:12,800 --> 00:38:16,800
manual, understand what you do. 
And then try to pick the play 

767
00:38:16,800 --> 00:38:18,600
sets. 
May be your biggest bottleneck 

768
00:38:18,600 --> 00:38:20,200
where it would help you to 
automate it. 

769
00:38:20,200 --> 00:38:24,500
Get that faster feedback, make 
your process more robust so that

770
00:38:24,500 --> 00:38:27,400
you do have something solid you 
could get to production but you 

771
00:38:27,400 --> 00:38:30,500
have to build a step-by-step. 
I learned from Janet years ago, 

772
00:38:30,500 --> 00:38:33,100
the first step in solving. 
A problem is make it visible. 

773
00:38:33,500 --> 00:38:36,200
So is your problem is you 
struggle to get anything to 

774
00:38:36,200 --> 00:38:38,800
production because you're still 
have good processes. 

775
00:38:39,000 --> 00:38:42,300
Start by visualizing what it is 
now and then decide how you want

776
00:38:42,300 --> 00:38:45,000
to improve it. 
I will be surprised if many 

777
00:38:45,000 --> 00:38:48,200
people Still, do not practice, 
continuous integration but few 

778
00:38:48,200 --> 00:38:51,500
years back I could even see some
teams do not even have Version 

779
00:38:51,500 --> 00:38:54,400
Control or maybe their Version 
Control even just locally. 

780
00:38:54,500 --> 00:38:56,100
So I think this is also a bad 
practice. 

781
00:38:56,400 --> 00:38:59,700
I think we should all avoid. 
But by now, I hope all the 

782
00:38:59,700 --> 00:39:02,000
listeners who have practiced at 
least Version Control and 

783
00:39:02,000 --> 00:39:05,000
continuous integration. 
So in continuous delivery, we 

784
00:39:05,000 --> 00:39:07,800
know that you have many stages 
in your deployment pipeline. 

785
00:39:08,000 --> 00:39:11,300
Obviously, many of those stages 
will be automation test. 

786
00:39:11,600 --> 00:39:14,400
So I know that you have this 
concept agile testing quadrants 

787
00:39:14,400 --> 00:39:16,500
where you have the garage. 
Has different types of tests. 

788
00:39:16,500 --> 00:39:18,500
Maybe, can you give us an 
overview? 

789
00:39:18,500 --> 00:39:21,100
What is agile? 
Testing quadrants and what are 

790
00:39:21,100 --> 00:39:25,200
the representations for each of 
the quadrants in actually wear 

791
00:39:25,200 --> 00:39:28,600
red petticord in myself and 
Brian Merrick and it was one of 

792
00:39:28,600 --> 00:39:31,900
those conference late night. 
Chats, we were talking about 

793
00:39:31,900 --> 00:39:35,100
this and Brian was explaining 
these quadrants so he had a 

794
00:39:35,107 --> 00:39:37,200
napkin it's one of those 
stories, right? 

795
00:39:37,200 --> 00:39:40,500
Say had a napkin he's drawing on
it and then Brett and I booked 

796
00:39:40,500 --> 00:39:43,800
him for months until he finally 
wrote it up in a blog post, so 

797
00:39:43,800 --> 00:39:46,800
then we could use it. 
We did ask his permission to use

798
00:39:46,800 --> 00:39:51,300
it but the quadrants is a way of
classifying, the tests four 

799
00:39:51,300 --> 00:39:56,800
quadrants so the left side is 
about guiding development tests 

800
00:39:56,800 --> 00:40:01,100
that you right before you start 
coding the right hand side is 

801
00:40:01,100 --> 00:40:05,600
about critiquing the product. 
So test that you do after coding

802
00:40:05,600 --> 00:40:10,000
is complete, the top bath is 
business facing tests, so those 

803
00:40:10,000 --> 00:40:13,600
are tests that your business 
would be able to look at would 

804
00:40:13,600 --> 00:40:17,100
be able to understand would be 
Able to do feedback on and the 

805
00:40:17,100 --> 00:40:20,200
bottom half is technical facing,
which just means that the 

806
00:40:20,200 --> 00:40:24,100
language that those tests are in
a something, the business 

807
00:40:24,100 --> 00:40:28,000
wouldn't look at, for example, 
Performance testing, they care 

808
00:40:28,000 --> 00:40:30,900
about the results, but they 
would never go look at the 

809
00:40:30,900 --> 00:40:33,700
actual tests. 
So the four quadrants, and 

810
00:40:33,700 --> 00:40:38,100
they're numbered for ease of use
only technology facing tests. 

811
00:40:38,100 --> 00:40:42,100
That guide development. 101 and 
those are kind of unit tests. 

812
00:40:42,400 --> 00:40:45,100
So the programmers are mostly 
responsible for guiding. 

813
00:40:45,300 --> 00:40:49,300
Development with the unit test 
quadrant 2 is business facing 

814
00:40:49,300 --> 00:40:52,600
tests that guy development. 
So this is where when we do 

815
00:40:52,600 --> 00:40:55,400
acceptance test-driven 
development or behavioral driven

816
00:40:55,400 --> 00:40:58,400
development thinking about how 
are we going to use those 

817
00:40:58,400 --> 00:41:02,800
examples to guide development? 
What tests can we write before? 

818
00:41:02,800 --> 00:41:06,300
So things like simulations or 
those sorts of things. 

819
00:41:06,300 --> 00:41:11,400
Those are all things that help 
us quadrant 3 is business facing

820
00:41:11,400 --> 00:41:14,500
that critique the product. 
So generally we put in 

821
00:41:14,500 --> 00:41:18,300
exploratory, Retesting in there 
or user acceptance, testing 

822
00:41:18,400 --> 00:41:21,800
business facing tests. 
We need to do them because 

823
00:41:21,800 --> 00:41:24,300
there's a notes, what didn't we 
think about? 

824
00:41:24,600 --> 00:41:28,600
And then Quadrant 4, which was 
the quadrant that made me really

825
00:41:28,600 --> 00:41:32,000
like this. 
Because at that time and that 

826
00:41:32,000 --> 00:41:35,800
was in 2003 that conference, 
every time I would talk about 

827
00:41:35,800 --> 00:41:38,400
testing in agile projects, every
tester. 

828
00:41:38,400 --> 00:41:42,500
I meant would say it won't work.
They talk about customer tests 

829
00:41:42,500 --> 00:41:45,100
and programmer tasks. 
But what about all those other? 

830
00:41:45,300 --> 00:41:49,000
Chests. 
Quadrant 4 gives us a way to 

831
00:41:49,008 --> 00:41:53,800
talk about all those other tests
performance reliability in all 

832
00:41:53,800 --> 00:41:57,200
the quality attributes that we 
know we have to work with. 

833
00:41:57,400 --> 00:42:01,700
So the quadrants are a way of 
visualizing your testing. 

834
00:42:01,700 --> 00:42:05,000
Again, we encourage teams to 
think about the test, they do 

835
00:42:05,000 --> 00:42:08,200
and then kind of put them in the
quadrants to think about why are

836
00:42:08,200 --> 00:42:10,100
they doing them? 
Are they doing it to support? 

837
00:42:10,100 --> 00:42:15,000
Because the left hand side 
supporting or guiding, that's 

838
00:42:15,000 --> 00:42:18,500
about, About preventing defects 
in code and we test our 

839
00:42:18,500 --> 00:42:21,500
assumptions. 
Can we test our knowledge before

840
00:42:21,500 --> 00:42:24,700
we ever write a line of code? 
The right hand side is about 

841
00:42:24,800 --> 00:42:28,400
finding defects. 
What didn't we think about? 

842
00:42:28,600 --> 00:42:31,900
So it's a way of classifying it.
So it's a way of visualizing the

843
00:42:31,908 --> 00:42:34,900
kinds of tests and where do we 
actually want to do this? 

844
00:42:35,600 --> 00:42:38,400
Thanks for walking us through 
the Adele testing quadrant. 

845
00:42:38,400 --> 00:42:41,300
So maybe for those of you who 
cannot visualize what Janet is 

846
00:42:41,300 --> 00:42:43,600
saying just now, so maybe I'll 
put it in the show notes. 

847
00:42:43,600 --> 00:42:46,400
So what the diagram looks like, 
But there are so many categories

848
00:42:46,400 --> 00:42:48,700
of tests. 
Depending on the left side, top 

849
00:42:48,700 --> 00:42:50,700
bottom, maybe you can have a 
look. 

850
00:42:50,800 --> 00:42:53,800
But importantly, there are so 
many types of tests for one 

851
00:42:53,800 --> 00:42:55,300
particular story should be 
built. 

852
00:42:55,300 --> 00:42:57,900
So many of those tests, what is 
the ratio here? 

853
00:42:58,000 --> 00:42:59,900
How should we build these test 
cases? 

854
00:43:00,100 --> 00:43:02,900
Because I also notice that you 
have this good practice which is

855
00:43:02,900 --> 00:43:05,100
called The Three Amigos or the 
power of three. 

856
00:43:05,500 --> 00:43:07,900
So maybe walk us through a 
little bit on how we should 

857
00:43:07,900 --> 00:43:11,800
write all those tests and how I 
wish it ran all the tests and 

858
00:43:11,800 --> 00:43:15,100
all the quadrants together with 
each feature. 

859
00:43:15,200 --> 00:43:17,700
Sure, that your team's going to 
work on and with each story that

860
00:43:17,700 --> 00:43:20,700
you work on, it depends for like
a web application. 

861
00:43:20,700 --> 00:43:23,000
Maybe you're doing something 
with a user interface and then 

862
00:43:23,000 --> 00:43:26,000
it has a back-end server and the
database have all those pieces, 

863
00:43:26,300 --> 00:43:29,600
we often will start in that 
quarter to that business facing 

864
00:43:29,600 --> 00:43:32,800
tested guide development, with 
prototypes and wireframes. 

865
00:43:32,800 --> 00:43:34,900
And what is rui going to look 
like? 

866
00:43:35,000 --> 00:43:38,200
We can test those, maybe get 
involved with designers. 

867
00:43:38,500 --> 00:43:41,900
When we say that the power of 
three these days, I think you're

868
00:43:41,900 --> 00:43:43,900
going to need four or five 
because you often need the 

869
00:43:43,900 --> 00:43:47,400
designers or We have experts or 
aberrations, experts, it could 

870
00:43:47,400 --> 00:43:50,000
be anybody. 
So we make sure that testing 

871
00:43:50,000 --> 00:43:52,200
those feature ideas is Janet 
mentioned earlier. 

872
00:43:52,500 --> 00:43:56,600
Once we've written sliced up 
this testable stories and start 

873
00:43:56,600 --> 00:43:59,600
working on them down. 
In that technology phasing 

874
00:43:59,900 --> 00:44:02,400
quadrant one that guides 
development. 

875
00:44:02,500 --> 00:44:05,500
The developers are the 
programmers are coding test. 

876
00:44:05,500 --> 00:44:08,400
First are doing test-driven 
development at the unit level, 

877
00:44:08,800 --> 00:44:12,300
that's going to be something 
that the programmer Zone because

878
00:44:12,300 --> 00:44:15,100
that's not a testing activity as
much as it is a code. 

879
00:44:15,200 --> 00:44:19,000
An activity, but it does help us
design, testable code, and 

880
00:44:19,000 --> 00:44:21,000
operable code, all those good 
things. 

881
00:44:21,300 --> 00:44:24,700
Then we're going to maybe go 
back into that quadrant two and 

882
00:44:24,700 --> 00:44:28,500
do our story testing. 
And again, it's great to have a 

883
00:44:28,508 --> 00:44:31,300
programmer in a tester pair up 
on that or attachment a product 

884
00:44:31,300 --> 00:44:33,900
owner or even better. 
For my point of view and 

885
00:44:33,900 --> 00:44:37,500
Ensemble with several people in 
different roles testing it 

886
00:44:37,500 --> 00:44:39,500
together. 
Well, I guess you can do that. 

887
00:44:39,500 --> 00:44:42,500
So we set the story Little know 
we were for exploring later on 

888
00:44:42,800 --> 00:44:46,600
but Beverly, you could do except
this testing at the Level a, 

889
00:44:46,600 --> 00:44:48,900
I've done that a lot doing the 
show me that jam. 

890
00:44:48,900 --> 00:44:51,900
It talks about the programmers, 
done thinks they're done with 

891
00:44:51,900 --> 00:44:55,700
the story and they asked, pester
hey, can I just walk you through

892
00:44:55,700 --> 00:44:57,500
what I've done and just a few 
minutes. 

893
00:44:57,500 --> 00:44:59,500
You may have had that. 
Aha moment of oh I didn't 

894
00:44:59,500 --> 00:45:01,800
understand that quite writer. 
Oops I forgot a piece. 

895
00:45:02,100 --> 00:45:04,500
You've prevented a bug. 
You haven't even checked the cut

896
00:45:04,500 --> 00:45:05,900
into the source code control 
yet. 

897
00:45:05,900 --> 00:45:07,900
So having that close 
collaboration. 

898
00:45:07,900 --> 00:45:10,400
It's like this iteration testing
coding, testing coding. 

899
00:45:10,800 --> 00:45:14,300
When we really think the stories
ready to go and we get enough 

900
00:45:14,300 --> 00:45:18,300
stories in a ER now we can start
into that quadrant three, the 

901
00:45:18,300 --> 00:45:21,700
critiquing the product but still
business facing exploratory, 

902
00:45:21,700 --> 00:45:23,500
testing usability, testing 
things. 

903
00:45:23,500 --> 00:45:26,500
We really need for code to have 
been written and deployed. 

904
00:45:26,800 --> 00:45:30,400
Again more collaboration 
involving more people. 

905
00:45:30,700 --> 00:45:32,900
One of my recent jobs, when 
somebody thought they had a 

906
00:45:32,908 --> 00:45:36,400
feature that was pretty ready to
go, we just opened it up to the 

907
00:45:36,400 --> 00:45:39,300
whole engineering organization. 
So hey, we're going to do 

908
00:45:39,300 --> 00:45:41,800
another sample. 
Testing session exploratory 

909
00:45:41,800 --> 00:45:44,400
testing on this feature for 30 
minutes at this time, please 

910
00:45:44,400 --> 00:45:46,400
join us. 
We have people rather tame join 

911
00:45:46,700 --> 00:45:49,400
fresh eyes. 
I've never seen the future 

912
00:45:49,400 --> 00:45:53,500
before that's so important. 
And just as such short time, we 

913
00:45:53,500 --> 00:45:56,500
would flush out anything. 
Any bugs that were there any 

914
00:45:56,600 --> 00:45:59,100
hidden assumptions that we 
hadn't thought about getting 

915
00:45:59,100 --> 00:46:02,100
that collaboration even from 
people with them teens and then 

916
00:46:02,100 --> 00:46:04,700
the technology Pages stuff you 
might do at the beginning. 

917
00:46:04,700 --> 00:46:08,000
Maybe we've got some new piece 
of a product that we want it 

918
00:46:08,000 --> 00:46:11,300
needs to support a large number 
of users at the same time, it 

919
00:46:11,300 --> 00:46:13,600
had really good performance. 
Well we have to think about the 

920
00:46:13,600 --> 00:46:15,000
architecture and how do we know 
that? 

921
00:46:15,100 --> 00:46:18,500
Architectural scale. 
So maybe then we Spike the code.

922
00:46:18,500 --> 00:46:22,100
We don't do the careful thing. 
We just do some throw a code 

923
00:46:22,100 --> 00:46:24,700
with that architecture and now 
we can do performance testing on

924
00:46:24,700 --> 00:46:27,900
it or load testing on it. 
See if it's scales and if it 

925
00:46:27,900 --> 00:46:30,800
does we throw away that code and
we start running the real code. 

926
00:46:30,800 --> 00:46:34,000
If it doesn't we Spike. 
Another architecture those 

927
00:46:34,000 --> 00:46:37,100
technology facing tests may be 
done at the very beginning if 

928
00:46:37,100 --> 00:46:39,200
that's the most important 
quality attributes who are 

929
00:46:39,200 --> 00:46:42,100
business and those become 
guiding development. 

930
00:46:42,100 --> 00:46:46,400
So the key thing that I observe 
when you Lisa is that is still 

931
00:46:46,400 --> 00:46:49,100
collaborative, right? 
It's not just one person or one 

932
00:46:49,100 --> 00:46:52,400
team who is responsible to come 
up with all these tests it could

933
00:46:52,400 --> 00:46:55,000
be from product manager, it 
could be from tested, it could 

934
00:46:55,000 --> 00:46:58,000
be from software developer 
designer and whatever the role 

935
00:46:58,000 --> 00:47:00,300
is in the team. 
And again this the feedback, 

936
00:47:00,300 --> 00:47:02,100
right? 
So the test sometimes drives 

937
00:47:02,100 --> 00:47:05,200
back the development and also by
zebras are so, I think I like 

938
00:47:05,200 --> 00:47:07,200
this concept of holistic 
starting already. 

939
00:47:07,400 --> 00:47:10,300
Thanks for sharing this concept.
So you mentioned a couple of 

940
00:47:10,300 --> 00:47:13,900
times about exploratory testing 
maybe this term, I think little 

941
00:47:13,900 --> 00:47:16,200
bit vague for some people. 
Well, some people think it's 

942
00:47:16,200 --> 00:47:18,400
like okay we just do any random 
test. 

943
00:47:18,600 --> 00:47:21,600
See whether the system crash 
maybe a little bit of Guiding 

944
00:47:21,600 --> 00:47:23,700
Light here. 
What do you mean by exploratory?

945
00:47:23,700 --> 00:47:26,100
Testing. 
Well first of all I would 

946
00:47:26,100 --> 00:47:28,900
recommend everybody read 
Elizabeth Hendrick since book, 

947
00:47:29,000 --> 00:47:31,400
explore it. 
I like the fact that she's 

948
00:47:31,400 --> 00:47:34,400
actually got a chapter in there 
for programmers, how to explore 

949
00:47:34,400 --> 00:47:37,800
when they're testing themselves.
She has so many good ideas. 

950
00:47:38,000 --> 00:47:42,500
One of the things that I really 
liked and I use to guide me is 

951
00:47:42,500 --> 00:47:47,200
the format to use Explorer. 
What am I going to explore with 

952
00:47:47,500 --> 00:47:50,100
chooses a word resources? 
I think variations. 

953
00:47:50,100 --> 00:47:53,100
But what is it that I'm 
exploring with and then to 

954
00:47:53,100 --> 00:47:56,300
discover. 
So when I put a exploratory test

955
00:47:56,300 --> 00:47:59,900
Charter together I use that 
format because it helps guide 

956
00:47:59,900 --> 00:48:02,400
me. 
It's a mission statement, it's 

957
00:48:02,400 --> 00:48:05,000
not necessarily easy to do 
because you don't want it too 

958
00:48:05,000 --> 00:48:07,300
small. 
You don't want it too big. 

959
00:48:07,500 --> 00:48:09,500
What is it? 
That I'm going to explore and it

960
00:48:09,508 --> 00:48:13,400
might be a specific risk. 
We think we have some security 

961
00:48:13,400 --> 00:48:16,700
issues in this area. 
So, maybe I'm going to be 

962
00:48:16,700 --> 00:48:20,300
exploring certain kinds of 
exploits or something, but 

963
00:48:20,300 --> 00:48:24,400
maybe, I want to see something. 
Say, we're adding this form and 

964
00:48:24,400 --> 00:48:27,300
we got people's names. 
So, if I use the agile testing 

965
00:48:27,300 --> 00:48:29,100
Fellowship, our website that we 
have. 

966
00:48:29,200 --> 00:48:32,200
One of the things we did was we 
wanted to make sure that we 

967
00:48:32,200 --> 00:48:35,600
could use any names. 
So, for example, your name 

968
00:48:35,600 --> 00:48:39,100
doesn't have any funny 
characters but if I put in 

969
00:48:39,100 --> 00:48:42,000
somebody from Thailand, they 
might have some different 

970
00:48:42,000 --> 00:48:45,000
characters or from Sweden and 
Norway, they have those 

971
00:48:45,100 --> 00:48:48,900
Different kinds of characters, 
will it supports every name? 

972
00:48:49,300 --> 00:48:51,900
And so we just put an 
exploratory test Charter to do 

973
00:48:51,900 --> 00:48:54,600
that. 
I also like to think about it, 

974
00:48:54,600 --> 00:48:59,500
as if I took that Charter and I 
tested, I would have my own 

975
00:48:59,500 --> 00:49:01,200
notes, I would have my own 
findings. 

976
00:49:01,700 --> 00:49:06,000
If I gave it to Lisa, she would 
explore differently. 

977
00:49:06,000 --> 00:49:08,000
She would use different 
examples. 

978
00:49:08,300 --> 00:49:11,900
She would have her own. 
So there's enough leeway that we

979
00:49:11,900 --> 00:49:15,700
would possibly test differently,
but we're still Exploring the 

980
00:49:15,700 --> 00:49:19,700
same thing and so it just takes 
practice, but there's a mission,

981
00:49:20,000 --> 00:49:24,000
not just randomly hitting keys. 
I want to look for this specific

982
00:49:24,000 --> 00:49:26,600
thing. 
Yeah, they're clearly be value 

983
00:49:26,600 --> 00:49:30,600
and ad hoc testing a lot of 
teams do bug bashes for they 

984
00:49:30,600 --> 00:49:33,000
just get a bunch of people and 
they start hammering on whatever

985
00:49:33,000 --> 00:49:36,500
they wanted release that's fine.
They'll find back to that way 

986
00:49:36,500 --> 00:49:41,900
too, but the exploratory testing
is designed more to try to find 

987
00:49:41,900 --> 00:49:44,800
those unknown unknowns before. 
They bite you in production and 

988
00:49:44,800 --> 00:49:48,000
be Organized. 
I actually got a great tip from 

989
00:49:48,000 --> 00:49:50,600
a programmer that I work with 
few years back. 

990
00:49:50,800 --> 00:49:54,300
When writing the charters, I 
like to start thinking of what 

991
00:49:54,300 --> 00:49:56,900
resources I'm at test with first
and that helps, you think of 

992
00:49:56,900 --> 00:49:59,600
that mission. 
As Janet says, what haven't got 

993
00:49:59,600 --> 00:50:02,400
my disposal that I can use to 
test this? 

994
00:50:02,700 --> 00:50:05,400
Maybe I've got some API 
endpoints I could use or maybe 

995
00:50:05,400 --> 00:50:08,600
I've got some production like 
test data, I could use or some 

996
00:50:08,600 --> 00:50:11,100
personas that the marketing 
department came up with that I 

997
00:50:11,100 --> 00:50:13,200
could use. 
So think about the resources, is

998
00:50:13,200 --> 00:50:14,900
one help because it's hard to 
learn, right? 

999
00:50:15,000 --> 00:50:18,800
Those things and I would also 
recommend pairing was already on

1000
00:50:18,800 --> 00:50:21,000
it. 
I would also put all those 

1001
00:50:21,000 --> 00:50:23,700
Charters as stories in your 
project, tracking tool in your 

1002
00:50:23,700 --> 00:50:25,800
backlog. 
Along with your feature stories,

1003
00:50:26,000 --> 00:50:28,700
they have to be visible. 
Everybody needs to know that 

1004
00:50:28,700 --> 00:50:32,100
those have to be done along with
all the coding and all the other

1005
00:50:32,100 --> 00:50:34,300
testing. 
We need to do that exploring as 

1006
00:50:34,300 --> 00:50:36,900
well. 
Thanks for clarifying this 

1007
00:50:36,900 --> 00:50:39,600
concept because many people 
still just assumed exploratory 

1008
00:50:39,600 --> 00:50:40,400
testing. 
Yeah, okay. 

1009
00:50:40,400 --> 00:50:43,100
You just go ahead and try to 
break the system or do random 

1010
00:50:43,100 --> 00:50:44,700
stuff. 
So I think there should be a 

1011
00:50:44,707 --> 00:50:46,700
path. 
Loud chatter Mission, you keep 

1012
00:50:46,700 --> 00:50:48,800
mentioning that. 
I think that's a very good 

1013
00:50:48,800 --> 00:50:51,000
explanation. 
One more misconception with 

1014
00:50:51,000 --> 00:50:53,600
Janet mentioned is about testing
in production. 

1015
00:50:53,800 --> 00:50:56,300
Again, it's got common 
misconception, people think oh 

1016
00:50:56,300 --> 00:50:59,400
maybe we should not write tests,
we just deployed, maybe let 

1017
00:50:59,400 --> 00:51:01,800
people figure it out. 
Maybe a little bit of Guiding 

1018
00:51:01,800 --> 00:51:06,400
Light here as well, but I 
remember listening to a keynote 

1019
00:51:06,400 --> 00:51:10,600
years ago at a conference and 
they said testing is dad, let 

1020
00:51:10,600 --> 00:51:13,600
the customers test. 
I wanted that person in the 

1021
00:51:13,600 --> 00:51:17,400
worst way and I kicked myself 
now for not putting my hand up 

1022
00:51:17,400 --> 00:51:21,300
and saying something, but I was 
hoping that person would say in 

1023
00:51:21,400 --> 00:51:26,600
my context because that is so 
important. 

1024
00:51:26,800 --> 00:51:30,300
So testing and production, if 
I'm doing a Google Maps, yeah, 

1025
00:51:30,300 --> 00:51:33,600
let the customers figured by the
bugs and report him, but if I'm 

1026
00:51:33,600 --> 00:51:38,500
doing code for a heart monitor, 
for example, I'm going to have 

1027
00:51:38,500 --> 00:51:41,600
way different risk profile. 
We're going to do all the 

1028
00:51:41,600 --> 00:51:44,900
testing we possibly can before 
we put it into production. 

1029
00:51:45,100 --> 00:51:48,800
And as such but testing and 
production and least I can talk 

1030
00:51:48,800 --> 00:51:51,100
about it much better than I can 
most of the time. 

1031
00:51:51,400 --> 00:51:55,200
But really if you have a safe 
way to do it, things like 

1032
00:51:55,200 --> 00:51:58,600
feature Flags, right? 
So the customer actually doesn't

1033
00:51:58,600 --> 00:52:00,800
see it. 
We're not touching customer data

1034
00:52:00,800 --> 00:52:03,200
where we're testing and we have 
it turned off. 

1035
00:52:03,500 --> 00:52:05,300
So there's different ways to do 
it. 

1036
00:52:05,500 --> 00:52:09,100
I think Katrina cookies book 
testing and devops is still the 

1037
00:52:09,100 --> 00:52:12,100
best one that I've come across 
for testing feature flags and 

1038
00:52:12,100 --> 00:52:14,800
working within that construct. 
Yeah, right. 

1039
00:52:15,000 --> 00:52:18,600
It'll guide for testing and 
devops that's available, van, 

1040
00:52:18,600 --> 00:52:20,000
leave Heaven. 
You Can Get It Free. 

1041
00:52:20,000 --> 00:52:22,200
She doesn't require you to pay 
although it would be nice to pay

1042
00:52:22,200 --> 00:52:23,900
because it's one of my go-to 
books for sure. 

1043
00:52:23,908 --> 00:52:27,100
But it's so important because as
good, a job. 

1044
00:52:27,107 --> 00:52:30,100
As we do have testing, we know 
our test environments, don't 

1045
00:52:30,100 --> 00:52:32,100
look like Productions. 
We're not going to test 

1046
00:52:32,100 --> 00:52:33,600
everything. 
We're going to miss things, and 

1047
00:52:33,600 --> 00:52:36,200
so we need to be able to put 
things in a production 

1048
00:52:36,200 --> 00:52:38,400
environment. 
You some kind of really strategy

1049
00:52:38,400 --> 00:52:39,500
that. 
Let's just do that safely. 

1050
00:52:39,500 --> 00:52:40,900
It takes build an 
infrastructure. 

1051
00:52:40,900 --> 00:52:43,900
And again, that's an effort that
your whole team needs to get 

1052
00:52:43,900 --> 00:52:46,000
involved in. 
And that's when you Need to have

1053
00:52:46,000 --> 00:52:48,500
operations specialist site, 
reliability, Engineers 

1054
00:52:48,500 --> 00:52:51,400
platforms, and ears in your team
are working with your team to 

1055
00:52:51,400 --> 00:52:54,600
help you with that. 
I think you just said something 

1056
00:52:54,600 --> 00:52:56,300
that really struck with me 
there. 

1057
00:52:56,300 --> 00:52:59,300
Lisa, you said, in the 
production environment, not 

1058
00:52:59,300 --> 00:53:04,400
necessarily with production data
and I think that's the key is 

1059
00:53:04,500 --> 00:53:08,900
the production environment, not 
the customers data, my last 

1060
00:53:08,900 --> 00:53:12,100
full-time job that I had. 
We could not test and production

1061
00:53:12,100 --> 00:53:16,000
in any way we just could not As 
if I as a little Services 

1062
00:53:16,000 --> 00:53:18,200
application, there was no way to
do it. 

1063
00:53:18,500 --> 00:53:20,600
If we had tried to do it with a 
got in trouble. 

1064
00:53:20,900 --> 00:53:24,600
And so then it became really 
important to be able to use our 

1065
00:53:24,600 --> 00:53:28,900
production data observability, 
what our customers doing power, 

1066
00:53:28,900 --> 00:53:32,500
its does think performing. 
And also the analytic tools, 

1067
00:53:32,700 --> 00:53:35,000
what are the individual users? 
We could actually see what they 

1068
00:53:35,000 --> 00:53:36,000
were doing. 
Like. 

1069
00:53:36,000 --> 00:53:38,900
Oh, people are complaining about
this and we can't reproduce it 

1070
00:53:38,900 --> 00:53:41,000
in our test environment. 
Let's go see what they're doing 

1071
00:53:41,000 --> 00:53:42,700
in production. 
Thank goodness. 

1072
00:53:42,700 --> 00:53:46,100
We have those tools now because 
if Can't test it in production, 

1073
00:53:46,100 --> 00:53:49,800
you need some way to understand 
what's happening there context. 

1074
00:53:49,900 --> 00:53:54,100
So important things will 
emphasizing that again context. 

1075
00:53:54,300 --> 00:53:57,300
So for people who learn about 
testing production, it doesn't 

1076
00:53:57,300 --> 00:54:00,000
mean that you just deploy your 
code without any kind of safe 

1077
00:54:00,000 --> 00:54:03,500
mechanism like feature Flags or 
maybe robust monitoring and 

1078
00:54:03,500 --> 00:54:05,800
observability. 
Because without knowing what 

1079
00:54:05,800 --> 00:54:08,300
your system does, its kind of 
like risky, especially if you're

1080
00:54:08,300 --> 00:54:10,900
dealing with Finance or health 
and all that. 

1081
00:54:11,100 --> 00:54:13,600
So unfortunately, due to time we
have to wrap up. 

1082
00:54:13,700 --> 00:54:16,600
It's been a really great 
Conversation about testing in 

1083
00:54:16,600 --> 00:54:20,100
overall but before I let both of
you go normally I would ask this

1084
00:54:20,100 --> 00:54:22,300
question called three technical 
leadership wisdom. 

1085
00:54:22,500 --> 00:54:25,000
So I will leave it whether you 
want a combined effort or if you

1086
00:54:25,000 --> 00:54:27,300
want to go individual. 
So what will be your tree 

1087
00:54:27,300 --> 00:54:29,300
technical leadership? 
Wisdom to share with us? 

1088
00:54:30,000 --> 00:54:32,500
Yeah, I thought that was an 
interesting question and I think

1089
00:54:32,500 --> 00:54:37,100
my answer is different now, but 
it would have been a few years 

1090
00:54:37,100 --> 00:54:38,800
ago. 
One of the things that we've 

1091
00:54:38,800 --> 00:54:41,800
learned in the last part, 10 
years now, but people don't pay 

1092
00:54:41,808 --> 00:54:44,500
enough attention to. 
It is a prerequisite for 

1093
00:54:44,600 --> 00:54:46,400
success. 
This fall software development 

1094
00:54:46,400 --> 00:54:49,500
is psychological safety. 
We had to feel safe to 

1095
00:54:49,500 --> 00:54:52,700
experiment and learn we have to 
feel safe to ask questions. 

1096
00:54:53,000 --> 00:54:55,500
When you don't feel safe, you're
going to burn out. 

1097
00:54:55,500 --> 00:54:58,000
My advice is, if you're in a 
toxic environment where you 

1098
00:54:58,000 --> 00:54:59,900
don't feel safe, where people 
are not safe. 

1099
00:55:00,100 --> 00:55:03,400
Try to be an agent for change. 
I've really found a good help 

1100
00:55:03,400 --> 00:55:06,100
from the books by Linda rising 
and Marilyn Mansfield has 

1101
00:55:06,100 --> 00:55:07,500
changed. 
And we're Fearless changed 

1102
00:55:07,600 --> 00:55:09,300
patterns to help influence 
people. 

1103
00:55:09,600 --> 00:55:13,600
You can maybe influence people 
and if it doesn't work, if you 

1104
00:55:13,600 --> 00:55:16,400
can get out, Out. 
Now I realize that's a 

1105
00:55:16,400 --> 00:55:18,200
privilege. 
Some people they need to pay 

1106
00:55:18,200 --> 00:55:21,500
their bills and they don't have 
any other options but also 

1107
00:55:21,500 --> 00:55:26,000
realize it isn't your problem. 
No amount of yoga and meditation

1108
00:55:26,000 --> 00:55:29,400
is going to fix it for you. 
Don't blame yourself. 

1109
00:55:30,000 --> 00:55:31,800
It's a problem with your 
organization. 

1110
00:55:32,100 --> 00:55:36,100
So pay attention to that before 
you burn out and really suffer 

1111
00:55:36,100 --> 00:55:38,300
the anger teams going to be 
suffering too, that's one of 

1112
00:55:38,308 --> 00:55:41,100
them. 
I'll let you do one Janet to, of

1113
00:55:41,100 --> 00:55:44,800
my go right in with that, one of
the things was small, f. 

1114
00:55:45,100 --> 00:55:49,300
Can bring about big changes, 
don't underestimate your scope 

1115
00:55:49,300 --> 00:55:52,300
of influence. 
The other one that kind of ties 

1116
00:55:52,300 --> 00:55:55,000
in with you two, were talking 
about psychological safety. 

1117
00:55:55,200 --> 00:55:59,400
Don't be afraid to try new 
things, keep learning many years

1118
00:55:59,400 --> 00:56:03,100
ago and this was in my pretty 
early days out of high school 

1119
00:56:03,200 --> 00:56:05,100
was my second job out of high 
school. 

1120
00:56:05,100 --> 00:56:08,500
I was in administer stencil 
administrative assistant and I 

1121
00:56:08,500 --> 00:56:13,100
had a really lazy boss. 
Really lazy people kept saying, 

1122
00:56:13,100 --> 00:56:16,400
why do you work for him? 
What I ended up doing was, I 

1123
00:56:16,400 --> 00:56:19,100
learned because he had me do 
most of the budget 

1124
00:56:19,100 --> 00:56:22,000
reconciliation and things 
because he didn't like to do it.

1125
00:56:22,000 --> 00:56:26,400
So I had to do it for him but 
then one day he was sick and it 

1126
00:56:26,408 --> 00:56:29,400
was a very important meeting of 
all the directors and a 

1127
00:56:29,400 --> 00:56:32,500
budgetary kind of meeting. 
And they were talking about it. 

1128
00:56:32,700 --> 00:56:36,800
I walked in with all the papers 
and they said, where's Brian? 

1129
00:56:36,900 --> 00:56:40,700
And I said he's sick and they 
said she would cancel a meeting 

1130
00:56:40,700 --> 00:56:45,500
and I said, no, let me explain 
because I had You can on that 

1131
00:56:45,500 --> 00:56:49,900
extra thing and done it. 
Learn that it just opened up a 

1132
00:56:49,900 --> 00:56:53,900
whole new world for me. 
I was viewed very differently 

1133
00:56:54,000 --> 00:56:57,300
than I had been the day before 
it paid off. 

1134
00:56:57,500 --> 00:57:01,700
So sometimes don't be afraid but
then take that jump take that 

1135
00:57:01,700 --> 00:57:04,200
opportunity that is in front of 
you. 

1136
00:57:04,300 --> 00:57:08,200
Don't be afraid to take it but 
that's only good if you have 

1137
00:57:08,200 --> 00:57:12,600
psychological safety. 
As Lisa said before my second 

1138
00:57:12,600 --> 00:57:15,400
big piece of advice, something I
have followed my elf for a 

1139
00:57:15,408 --> 00:57:16,800
change. 
I think my own advice. 

1140
00:57:17,100 --> 00:57:21,000
The last few jobs I have done, 
is build relationship. 

1141
00:57:21,000 --> 00:57:23,900
One of the things I got from 
Katrina cookies, book is about 

1142
00:57:23,900 --> 00:57:26,600
relationships within your team, 
but also outside of your team. 

1143
00:57:26,600 --> 00:57:31,500
I've gotten so much benefit from
asking people on the platform 

1144
00:57:31,500 --> 00:57:33,200
team. 
Could I have a one-on-one with 

1145
00:57:33,200 --> 00:57:34,800
you for 30 minutes every couple 
weeks? 

1146
00:57:35,300 --> 00:57:37,200
So helpful, I learned so much 
from them. 

1147
00:57:37,200 --> 00:57:40,200
I got so much support from them.
That help my team. 

1148
00:57:40,500 --> 00:57:42,300
I just felt like it was a secret
weapon. 

1149
00:57:42,600 --> 00:57:44,800
Do start with people who seem 
friendly and seem like they'll 

1150
00:57:45,000 --> 00:57:48,800
Your allies you got to pick them
out the just forms a foundation 

1151
00:57:48,800 --> 00:57:51,000
for so much. 
The kind of learning that Jim is

1152
00:57:51,000 --> 00:57:54,800
talking about and helping you 
get changed, because maybe 

1153
00:57:54,800 --> 00:57:56,500
there's something you want to 
do, and you can't interest 

1154
00:57:56,500 --> 00:57:59,300
anybody on your own team, maybe 
somebody on another team wants 

1155
00:57:59,300 --> 00:58:03,000
to help you really beautiful. 
So, everything aligned together,

1156
00:58:03,000 --> 00:58:05,900
I can see the Synergy between 
both of you, right, as you can 

1157
00:58:05,900 --> 00:58:08,000
tell from the books and all 
that. 

1158
00:58:08,000 --> 00:58:11,400
So for people who want to follow
up with you or maybe continue 

1159
00:58:11,400 --> 00:58:14,500
this conversation outside of 
this episode, is there a place 

1160
00:58:14,500 --> 00:58:15,800
where they I can find you 
online. 

1161
00:58:16,500 --> 00:58:18,900
Lots of places, we're both on 
Twitter. 

1162
00:58:19,300 --> 00:58:24,000
My Twitter ID is Janet Gregory 
chca for Canada because I'm 

1163
00:58:24,100 --> 00:58:26,300
Canada. 
People get that confused. 

1164
00:58:26,400 --> 00:58:27,700
Lisa. 
Your Twitter ID. 

1165
00:58:28,200 --> 00:58:31,200
It's just Lisa Chris and I'm 
Lisa Chrisman on all social 

1166
00:58:31,200 --> 00:58:34,700
media platforms that I'm on. 
We're on LinkedIn for any of our

1167
00:58:34,700 --> 00:58:37,200
websites. 
We all have a contact us. 

1168
00:58:37,600 --> 00:58:43,700
My website is Janet Gregory .c a
or agile tester, .c a or the 

1169
00:58:43,700 --> 00:58:47,700
agile testing fellow. 
And I'm Lisa Crispin.com are all

1170
00:58:47,700 --> 00:58:49,800
connected together. 
If you go below it's pretty easy

1171
00:58:49,800 --> 00:58:53,900
to find and Lisa and Janet also 
has a limp up book so it's the a

1172
00:58:53,900 --> 00:58:56,800
gel testing condensed. 
If you ever want to have this 

1173
00:58:56,800 --> 00:59:00,100
condensed Concept in one goal, I
think it's also a good place 

1174
00:59:00,100 --> 00:59:02,900
where you can get all these 
resources one last question. 

1175
00:59:02,900 --> 00:59:05,100
One fun fact. 
So when I read your book, I 

1176
00:59:05,100 --> 00:59:07,500
always see this dragon and 
Donkey. 

1177
00:59:08,400 --> 00:59:11,500
I have my hypothesis, but I'll 
let you explain what all this 

1178
00:59:11,500 --> 00:59:15,700
about dragon and Donkey. 
Well, the donkeys are kind of my

1179
00:59:15,700 --> 00:59:19,400
mascot because I have donkeys. 
So we have for donkeys here and 

1180
00:59:19,400 --> 00:59:22,700
our little Farm in Vermont. 
I've taught me a lot over the 

1181
00:59:22,700 --> 00:59:26,400
years about trust and Agility. 
I pitched them up to carts and 

1182
00:59:26,400 --> 00:59:28,900
wagons and drive them and take 
them places and they be work 

1183
00:59:28,900 --> 00:59:31,100
around the farm. 
There's sort of therapy donkeys.

1184
00:59:31,100 --> 00:59:34,200
I took him to senior centers and
schools and things like that. 

1185
00:59:34,400 --> 00:59:38,200
So yeah, it's just my passion. 
So three of those donkeys are 

1186
00:59:38,200 --> 00:59:39,800
miniature donkeys. 
Yeah. 

1187
00:59:39,900 --> 00:59:42,900
And then she's got one big one 
who protects the three little 

1188
00:59:42,900 --> 00:59:46,400
ones and for those kind of cool.
And the dragon is because I'm a 

1189
00:59:46,500 --> 00:59:49,300
fantasy but file of fantasy 
stories. 

1190
00:59:49,500 --> 00:59:53,000
My favorite series is a 
dragonriders of Pern, by Anne 

1191
00:59:53,000 --> 00:59:56,400
McCaffrey. 
So, I like friendly dragons, not

1192
00:59:56,400 --> 00:59:58,800
like the Game of Thrones 
dragons, the friendly ones. 

1193
00:59:59,300 --> 01:00:01,400
So it's just kind of something 
that appeals to me. 

1194
01:00:02,100 --> 01:00:04,600
Thanks for explaining that 
really fun fact, for those of 

1195
01:00:04,600 --> 01:00:07,500
you who also notice in the book 
that whatever you see dragon and

1196
01:00:07,500 --> 01:00:10,800
donkeys, this is what it means. 
Again really Pleasant to have 

1197
01:00:10,800 --> 01:00:12,600
this conversation. 
Thank you so much for spending 

1198
01:00:12,600 --> 01:00:14,700
your time Janet and Lisa goodbye
for now. 

1199
01:00:15,500 --> 01:00:18,500
Thank you so much for having us.
It was a fun conversation. 

1200
01:00:18,600 --> 01:00:25,300
It was Thank you for listening 
to this episode and for staying,

1201
01:00:25,300 --> 01:00:28,100
right until the end if you 
highly enjoyed it. 

1202
01:00:28,200 --> 01:00:30,600
I would appreciate if you share 
it with your friends and 

1203
01:00:30,600 --> 01:00:33,600
colleagues who you think would 
also benefit from listening to 

1204
01:00:33,600 --> 01:00:35,600
this episode. 
And if you are new to the 

1205
01:00:35,600 --> 01:00:38,600
podcast, make sure to subscribe 
and leave me your valuable 

1206
01:00:38,600 --> 01:00:41,200
review and feedback. 
It helps me a lot. 

1207
01:00:41,200 --> 01:00:43,200
In order to grow this podcast 
better. 

1208
01:00:43,600 --> 01:00:46,500
You can also find the full show 
notes of this conversation on 

1209
01:00:46,500 --> 01:00:50,200
the episode page at technology 
Arnold death website, including 

1210
01:00:50,200 --> 01:00:53,500
the full transcript enter, 
Testing courts and links to the 

1211
01:00:53,500 --> 01:00:55,800
resources mention from the 
conversation. 

1212
01:00:56,200 --> 01:00:59,300
And lastly, make sure to 
subscribe to the show's mailing 

1213
01:00:59,300 --> 01:01:02,700
list on pack leader. 
No dot f to get notified for any

1214
01:01:02,700 --> 01:01:05,300
future episodes. 
Stay tuned for the next 

1215
01:01:05,300 --> 01:01:06,700
technology. 
No episode. 

1216
01:01:07,000 --> 01:01:08,700
And until then goodbye.