1
00:00:00,200 --> 00:00:03,720
What if your job stopped being 
writing code and became building

2
00:00:03,720 --> 00:00:06,840
the factory that writes it? 
That's today's claim. 

3
00:00:07,080 --> 00:00:10,200
Software engineering is becoming
factory engineering. 

4
00:00:10,520 --> 00:00:13,840
So who designs the factory and 
who checks its work? 

5
00:00:18,560 --> 00:00:23,120
Welcome to Build Wiz AI Show, 
big AI ideas made clear for 

6
00:00:23,120 --> 00:00:26,080
builders. 
I'm Bob, and today I'm the one 

7
00:00:26,080 --> 00:00:28,720
asking the questions. 
And I'm Alice. 

8
00:00:29,000 --> 00:00:32,759
We're unpacking Ben Holmes's X 
article Becoming a Factory 

9
00:00:32,759 --> 00:00:34,680
Engineer. 
His big idea? 

10
00:00:34,920 --> 00:00:38,720
We went from writing code to 
writing prompts to engineering 

11
00:00:38,720 --> 00:00:41,160
the system of agents that runs 
those prompts. 

12
00:00:41,880 --> 00:00:45,040
So a factory, no forklifts I'm 
guessing. 

13
00:00:45,840 --> 00:00:49,000
No forklifts. 
He calls it a systems design 

14
00:00:49,000 --> 00:00:52,240
challenge, one that seems to 
have breathed new life into 

15
00:00:52,240 --> 00:00:56,240
software development. 3 guests 
and a few outside sources will 

16
00:00:56,240 --> 00:00:58,960
help us test it. 
Start me at zero. 

17
00:00:59,360 --> 00:01:03,200
What changed in the day job? 
Ben splits engineering into 

18
00:01:03,200 --> 00:01:07,160
three steps, planning, 
implementation, and code review.

19
00:01:07,640 --> 00:01:11,200
Before agents, implementation 
was the bulk of the deep work. 

20
00:01:11,560 --> 00:01:14,400
The time leftover went to 
architecture discussions and 

21
00:01:14,400 --> 00:01:17,520
code review. 
And now the agents do the 

22
00:01:17,520 --> 00:01:19,880
implementing. 
Exactly. 

23
00:01:20,000 --> 00:01:23,880
So the balance has flipped and 
his blunt line, if you stop 

24
00:01:23,880 --> 00:01:26,520
there, it feels like 
engineering, was replaced with 

25
00:01:26,520 --> 00:01:31,400
20% meetings and 80% drudgery. 
20% meetings. 

26
00:01:31,800 --> 00:01:33,200
That's my calendar on a good 
week. 

27
00:01:34,000 --> 00:01:37,640
Picture a chef who used to cook 
every plate and now only writes 

28
00:01:37,640 --> 00:01:40,880
menus and checks orders all day.
That's the trap. 

29
00:01:40,880 --> 00:01:44,560
He describes the cooking move to
the agents, so the fun part 

30
00:01:44,560 --> 00:01:47,240
looks gone. 
But he doesn't stop there, 

31
00:01:47,360 --> 00:01:49,760
right? 
No, he says. 

32
00:01:49,760 --> 00:01:52,000
There are interesting 
engineering challenges worth 

33
00:01:52,000 --> 00:01:54,400
solving at each step. 
Planning. 

34
00:01:54,640 --> 00:01:58,160
When should the agent involve a 
human to draft a plan and what 

35
00:01:58,160 --> 00:02:00,600
goes into a good product or 
technical spec? 

36
00:02:01,240 --> 00:02:04,080
And the other two steps. 
Implementation. 

37
00:02:04,360 --> 00:02:08,080
How do we optimize the cost and 
quality of agent code, and what 

38
00:02:08,080 --> 00:02:11,960
access do agents get across 
secrets, MCPS, and skills 

39
00:02:12,360 --> 00:02:15,080
review? 
How do agents verify their own 

40
00:02:15,080 --> 00:02:18,840
work end to end and assess the 
risk of a pull request to save 

41
00:02:18,840 --> 00:02:22,160
humans time? 
So the hard problems just moved 

42
00:02:22,160 --> 00:02:24,320
up a floor. 
That's his point. 

43
00:02:24,640 --> 00:02:27,800
These are the systems questions 
engineers used to answer in 

44
00:02:27,800 --> 00:02:30,920
syntax and abstractions, just at
a higher level. 

45
00:02:31,280 --> 00:02:34,920
Burgeter Buckler, writing on 
Martin Fowler's site, frames the

46
00:02:34,920 --> 00:02:37,920
same challenge. 
To let coding agents work with 

47
00:02:37,920 --> 00:02:41,440
less supervision, we need ways 
to increase our confidence in 

48
00:02:41,440 --> 00:02:44,520
their result. 
He turns that into two job 

49
00:02:44,520 --> 00:02:45,720
titles. 
What are they? 

50
00:02:46,440 --> 00:02:49,640
Product engineering is building 
the product itself, deciding 

51
00:02:49,640 --> 00:02:52,240
what needs to exist and how it 
should be architected. 

52
00:02:52,720 --> 00:02:55,880
Factory engineering is building 
the system of agents that builds

53
00:02:55,880 --> 00:02:58,960
the product. 
It optimizes the implementation 

54
00:02:58,960 --> 00:03:00,960
and review pieces through 
automation. 

55
00:03:01,680 --> 00:03:04,920
Wait, so factory engineers 
automate everything? 

56
00:03:05,640 --> 00:03:09,640
No, and he's firm on that. 
It means automating low risk 

57
00:03:09,640 --> 00:03:13,160
work and only requiring human 
attention when absolutely 

58
00:03:13,160 --> 00:03:15,760
necessary. 
Berkeler explains why people 

59
00:03:15,760 --> 00:03:18,200
stay in the loop. 
Human developers bring their 

60
00:03:18,200 --> 00:03:21,600
skills and experience as an 
implicit harness to every code 

61
00:03:21,600 --> 00:03:24,720
base, and an agent has no 
organizational memory. 

62
00:03:25,240 --> 00:03:28,320
Give me a picture. 
Think of a restaurant kitchen. 

63
00:03:28,560 --> 00:03:31,640
The head chef decides the menu 
and someone else designs the 

64
00:03:31,640 --> 00:03:34,640
stations, the order tickets, and
the tasting checks. 

65
00:03:34,840 --> 00:03:37,320
So most plates go out without 
the chef touching them. 

66
00:03:37,720 --> 00:03:41,320
The menu is product engineering.
That kitchen designer is the 

67
00:03:41,320 --> 00:03:44,600
factory engineer. 
What does a factory engineer 

68
00:03:44,600 --> 00:03:48,640
actually get asked? 
He gives 3 sample requests. 

69
00:03:49,000 --> 00:03:53,080
One, our agent keeps adding test
cases and code comments that it 

70
00:03:53,080 --> 00:03:56,680
really shouldn't. 
Two, someone on call keeps copy 

71
00:03:56,680 --> 00:04:00,160
pasting error monitor reports 
into an agent and asks could we 

72
00:04:00,160 --> 00:04:02,200
automate this? 
And #3. 

73
00:04:03,200 --> 00:04:06,480
The API bill skyrocketed after 
the latest model release. 

74
00:04:06,840 --> 00:04:09,320
Could some workflows move to 
cheaper models without 

75
00:04:09,320 --> 00:04:11,960
sacrificing code quality? 
And his rule? 

76
00:04:12,440 --> 00:04:16,000
The larger your product and the 
more agents you deploy, the more

77
00:04:16,000 --> 00:04:18,040
valuable factory engineering 
becomes. 

78
00:04:19,000 --> 00:04:21,680
Ah, the surprise invoice. 
Everyone's favorite e-mail. 

79
00:04:22,079 --> 00:04:25,560
OK, So what is a software 
factory concretely? 

80
00:04:26,240 --> 00:04:29,800
Ben's definition? 
An automation loop around agents

81
00:04:30,000 --> 00:04:32,960
designed to build better and 
better software over time. 

82
00:04:33,280 --> 00:04:36,160
Ideally you set the product 
requirements and it runs without

83
00:04:36,160 --> 00:04:39,960
human interaction, but agents 
aren't at that level yet, so 

84
00:04:39,960 --> 00:04:43,240
factory engineers build a 
factory around today's agents 

85
00:04:43,240 --> 00:04:45,120
that automates as much as 
possible. 

86
00:04:45,840 --> 00:04:47,960
Where does this factory actually
live? 

87
00:04:48,800 --> 00:04:52,280
In a shared setup that both the 
agents and the team can reach, 

88
00:04:52,640 --> 00:04:56,400
usually a centralized system in 
the cloud or on your own compute

89
00:04:56,600 --> 00:05:00,800
instead of local setups, humans 
and agents can configure it, the

90
00:05:00,800 --> 00:05:04,320
environment, the agents access 
levels, and the pipeline of 

91
00:05:04,320 --> 00:05:07,720
steps it follows. 
And where do the humans step in?

92
00:05:08,480 --> 00:05:12,280
Picture a car assembly line that
runs by itself, except at 2 

93
00:05:12,280 --> 00:05:15,200
stations where a person must 
sign off before anything moves 

94
00:05:15,200 --> 00:05:18,160
on. 
In Ben's example pipeline, those

95
00:05:18,160 --> 00:05:20,840
two stations are the spec and 
the pull request. 

96
00:05:21,560 --> 00:05:24,960
Isn't factory just a new word 
for what people call a harness? 

97
00:05:25,720 --> 00:05:29,720
Close cousins Buckler defines a 
harness as everything in an 

98
00:05:29,720 --> 00:05:33,240
agent except the model. 
She splits it into guides which 

99
00:05:33,240 --> 00:05:36,640
steer the agent before it acts, 
and sensors which observe 

100
00:05:36,640 --> 00:05:38,520
afterwards and help itself 
correct. 

101
00:05:39,320 --> 00:05:41,760
And how does that fit Ben's two 
human gates? 

102
00:05:42,480 --> 00:05:46,760
Neatly, she writes that a good 
harness should not necessarily 

103
00:05:46,760 --> 00:05:50,520
aim to fully eliminate human 
input, but to direct it to where

104
00:05:50,520 --> 00:05:54,640
our input is most important. 
Lena, you've been reading about 

105
00:05:54,640 --> 00:05:59,000
the ground under all of this. 
Right, Nathan Harvey and Derek 

106
00:05:59,000 --> 00:06:00,880
Debellis on the Google. 
Cloud Blog. 

107
00:06:01,200 --> 00:06:05,200
Announced the Dora report on AI 
assisted software development. 

108
00:06:05,520 --> 00:06:08,800
It's key insight. 
AI doesn't fix a team, it 

109
00:06:08,800 --> 00:06:12,560
amplifies what's already there. 
So a shaky setup just gets 

110
00:06:12,560 --> 00:06:15,280
louder. 
They call platform engineering 

111
00:06:15,280 --> 00:06:18,960
the foundation. 90% of 
organizations have adopted at 

112
00:06:18,960 --> 00:06:22,600
least one platform, and a high 
quality internal platform 

113
00:06:22,880 --> 00:06:25,640
correlates with being able to 
unlock the value of AI. 

114
00:06:26,000 --> 00:06:28,040
Think of. 
It as the concrete floor the 

115
00:06:28,040 --> 00:06:32,200
whole line is bolted to. 
Once the factory runs, how do 

116
00:06:32,200 --> 00:06:35,640
you know it's any good? 
Ben's first tool is spending 

117
00:06:35,640 --> 00:06:37,960
metrics. 
Track what agents cost you in 

118
00:06:37,960 --> 00:06:40,200
token usage and on device 
compute. 

119
00:06:40,640 --> 00:06:44,920
Then translate raw numbers into 
real value like cost per PR and 

120
00:06:44,920 --> 00:06:48,320
cycle time which he defines as 
initial prompt to release. 

121
00:06:49,000 --> 00:06:52,320
Cycle time? 
Isn't that a Dora thing? 

122
00:06:53,160 --> 00:06:56,320
A close relative, Nathan 
Harvey's guide on Dora's 

123
00:06:56,320 --> 00:06:59,720
delivery metrics defines change 
lead time as the time for a 

124
00:06:59,720 --> 00:07:03,040
change to go from committed to 
version control to deployed in 

125
00:07:03,040 --> 00:07:06,200
production. 
Ben's cycle time starts earlier 

126
00:07:06,280 --> 00:07:09,080
at the first prompt. 
And code quality. 

127
00:07:09,400 --> 00:07:11,080
You can't put taste in a 
spreadsheet. 

128
00:07:11,880 --> 00:07:16,080
That's his second tool, LLM as a
judge scoring for quality. 

129
00:07:16,080 --> 00:07:20,000
You can't check statically, 
architecture, quality, redundant

130
00:07:20,000 --> 00:07:23,360
or tautological tests how 
efficient the agent's tool calls

131
00:07:23,360 --> 00:07:26,080
are. 
Another agent reviews past agent

132
00:07:26,080 --> 00:07:29,720
conversations to score them. 
So how should I picture all 

133
00:07:29,720 --> 00:07:32,200
that? 
It's like a fitness tracker plus

134
00:07:32,200 --> 00:07:35,200
a coach. 
The tracker counts steps, cheap 

135
00:07:35,200 --> 00:07:38,000
and constant. 
The coach watches the replay and

136
00:07:38,000 --> 00:07:41,640
tells you your form is off. 
In Buckler's terms, the cheap 

137
00:07:41,640 --> 00:07:45,600
counters are computational 
sensors and the LLM judge is an 

138
00:07:45,600 --> 00:07:49,040
inferential 1. 
But can a judge be trusted every

139
00:07:49,040 --> 00:07:51,520
time? 
Buckler is careful here. 

140
00:07:51,920 --> 00:07:55,640
LLMS can partially address 
problems like redundant tests, 

141
00:07:55,920 --> 00:07:59,160
but expensively and 
probabilistically not on every 

142
00:07:59,160 --> 00:08:02,200
commit. 
Cheap deterministic checks like 

143
00:08:02,200 --> 00:08:06,120
tests and linters are fast 
enough to run on every change. 

144
00:08:07,000 --> 00:08:08,960
Any warning label on all these 
metrics? 

145
00:08:09,640 --> 00:08:13,440
Two from that door a guide. 
Don't set a metric as the goal 

146
00:08:13,600 --> 00:08:15,200
because teams will try to game 
it. 

147
00:08:15,520 --> 00:08:18,960
And don't pick one metric to 
rule them all, choose several 

148
00:08:18,960 --> 00:08:20,960
with a healthy amount of tension
between them. 

149
00:08:21,680 --> 00:08:24,600
Last piece, can the factory fix 
itself? 

150
00:08:25,280 --> 00:08:27,720
That's his third tool, self 
improvement. 

151
00:08:28,120 --> 00:08:31,440
Once you track these metrics, 
make them agent accessible. 

152
00:08:31,720 --> 00:08:35,480
His example is hypothetical, A 
redundant test scorer that does 

153
00:08:35,480 --> 00:08:40,320
poorly on 60% of conversations. 
A self improvement agent notices

154
00:08:40,320 --> 00:08:43,520
the trend and suggests changes 
to your skills and code 

155
00:08:43,520 --> 00:08:47,000
architecture. 60% pointless 
tests. 

156
00:08:47,200 --> 00:08:49,000
The robots are padding their 
homework. 

157
00:08:49,800 --> 00:08:53,880
In his made-up example, yes, 
Buckler describes the same loop 

158
00:08:53,880 --> 00:08:57,320
done by people whenever an issue
happens multiple times. 

159
00:08:57,560 --> 00:09:00,720
Improve the guides and sensors 
so it becomes less likely. 

160
00:09:01,120 --> 00:09:04,560
And she adds that agents can 
help build those controls, too. 

161
00:09:05,120 --> 00:09:08,840
And the fourth tool. 
Benchmarking Test theories on 

162
00:09:08,840 --> 00:09:12,080
how factory configurations 
change cost and quality. 

163
00:09:12,400 --> 00:09:15,360
The biggest lever is model 
selection, but you might also 

164
00:09:15,360 --> 00:09:18,200
cut stages or add and remove 
subagents. 

165
00:09:18,520 --> 00:09:22,520
Treat each as a hypothesis and 
change one variable at a time. 

166
00:09:23,040 --> 00:09:26,560
Why only one at a time? 
Imagine baking bread and 

167
00:09:26,560 --> 00:09:29,840
changing the flour, the oven, 
and the timer all at once. 

168
00:09:30,280 --> 00:09:32,560
If the loaf gets better, you 
can't say why. 

169
00:09:32,960 --> 00:09:36,080
Change one variable in the 
factory and watch the metrics 

170
00:09:36,280 --> 00:09:39,120
after setting a baseline for 
current performance, which is 

171
00:09:39,120 --> 00:09:43,200
step one in the door a guide. 
Time for push back Maya. 

172
00:09:43,400 --> 00:09:46,720
You looked at what happens when 
experienced developers actually 

173
00:09:46,720 --> 00:09:48,920
use these tools. 
I did. 

174
00:09:49,400 --> 00:09:54,240
Metra Nonprofit Research Group 
ran a randomized trial with 16 

175
00:09:54,240 --> 00:09:58,360
experienced open source 
developers on 246 real issues 

176
00:09:58,360 --> 00:10:00,000
from repositories they know 
well. 

177
00:10:00,480 --> 00:10:04,640
Each issue was randomly assigned
to allow or disallow AI and. 

178
00:10:05,200 --> 00:10:09,280
What happened? 
With AI allowed they took 19% 

179
00:10:09,280 --> 00:10:11,800
longer and the perception gap is
striking. 

180
00:10:12,080 --> 00:10:16,360
They expected a 24% speed up and
even afterwards believed AI had 

181
00:10:16,360 --> 00:10:20,440
speed them up by 20%. 
Slower, and they felt faster. 

182
00:10:20,920 --> 00:10:23,760
Ouch. 
To be fair, Metter says it 

183
00:10:23,760 --> 00:10:25,840
doesn't claim this holds for 
most developers. 

184
00:10:26,080 --> 00:10:28,760
It's setting is experienced 
people on projects they know 

185
00:10:28,760 --> 00:10:30,360
well. 
But it does show that 

186
00:10:30,360 --> 00:10:32,320
self-reports may be over 
optimistic. 

187
00:10:33,040 --> 00:10:36,920
Which is exactly why Ben wants 
measured numbers like cycle time

188
00:10:36,920 --> 00:10:40,000
from prompt to release, rather 
than a feeling of speed. 

189
00:10:40,720 --> 00:10:43,160
Sam, you think the whole factory
image is? 

190
00:10:43,160 --> 00:10:48,800
Off I do, Jeff Atwood on his 
blog Coding Horror asked is 

191
00:10:48,800 --> 00:10:51,720
software development like 
manufacturing, and he wasn't 

192
00:10:51,720 --> 00:10:54,680
convinced. 
His line variability is the 

193
00:10:54,720 --> 00:10:58,000
enemy in manufacturing. 
In software, it's the reason we 

194
00:10:58,000 --> 00:11:01,520
get up in the morning. 
So every project is a one off. 

195
00:11:02,240 --> 00:11:05,640
He says every worthwhile 
software development project is 

196
00:11:05,640 --> 00:11:08,320
a custom one off job for a 
unique problem. 

197
00:11:08,600 --> 00:11:10,800
And he quotes Mary and Tom 
Poppendieck. 

198
00:11:11,040 --> 00:11:14,240
Generating good software is not 
a production process, it is a 

199
00:11:14,240 --> 00:11:18,720
development process. 
Fair hit Ben's answer is partly 

200
00:11:18,720 --> 00:11:21,360
built in. 
He automates the low risk work 

201
00:11:21,360 --> 00:11:24,320
and keeps human gates at the 
spec and the pull request. 

202
00:11:25,080 --> 00:11:28,520
Lena, Does more output from the 
factory mean more breakage? 

203
00:11:29,240 --> 00:11:32,280
It can. 
The Dora team found AI adoption 

204
00:11:32,280 --> 00:11:34,680
still has a negative 
relationship with software 

205
00:11:34,680 --> 00:11:38,080
delivery stability. 
Without robust control systems 

206
00:11:38,080 --> 00:11:42,120
like strong automated testing 
and fast feedback loops, more 

207
00:11:42,120 --> 00:11:44,080
change volume leads to 
instability. 

208
00:11:44,880 --> 00:11:47,200
And one limit no factory 
removes. 

209
00:11:47,680 --> 00:11:51,200
Buckler notes that misdiagnosis 
over engineering and 

210
00:11:51,200 --> 00:11:54,800
misunderstood instructions 
aren't caught reliably In her 

211
00:11:54,800 --> 00:11:59,160
words, correctness is outside 
any sensors remit if the human 

212
00:11:59,160 --> 00:12:02,240
didn't clearly specify what they
wanted in the first place. 

213
00:12:03,000 --> 00:12:06,160
Time for Builders corner Alice. 
One thing to try this week. 

214
00:12:06,920 --> 00:12:10,240
Pull last week's agent pull 
requests and log 2 numbers for 

215
00:12:10,240 --> 00:12:14,040
each what it cost and it's cycle
time from first prompt to 

216
00:12:14,040 --> 00:12:17,080
release. 
That's Ben's cost per PR and 

217
00:12:17,080 --> 00:12:19,360
cycle time. 
Just two numbers. 

218
00:12:20,080 --> 00:12:23,280
Plus one judge score. 
Pick the behavior that bugs you 

219
00:12:23,280 --> 00:12:26,920
most, say redundant or 
tautological tests, and have 

220
00:12:26,920 --> 00:12:29,520
another agent score those past 
conversations. 

221
00:12:29,960 --> 00:12:32,920
Keep a healthy tension between 
the numbers, as the Dora guide 

222
00:12:32,920 --> 00:12:34,720
advises. 
And then? 

223
00:12:35,560 --> 00:12:39,480
Change one thing, say a cheaper 
model on one workflow and 

224
00:12:39,480 --> 00:12:42,280
compare next week 1 variable at 
a time. 

225
00:12:43,080 --> 00:12:46,920
Quick takeaways mine the work 
moved up a level from writing 

226
00:12:46,920 --> 00:12:48,920
code to designing the system 
that writes it. 

227
00:12:49,600 --> 00:12:53,840
Mine automate the low risk work 
and send human attention where 

228
00:12:53,840 --> 00:12:56,840
it matters most, at the spec and
the pull request. 

229
00:12:57,640 --> 00:13:00,960
And measure it cost, cycle time 
and a judge score. 

230
00:13:01,160 --> 00:13:05,080
Not a gut feeling. 
Because, as METR found, 

231
00:13:05,320 --> 00:13:08,280
developers can feel faster while
being slower. 

232
00:13:08,680 --> 00:13:12,480
What's still open? 
Ben's own first question when 

233
00:13:12,480 --> 00:13:15,720
should an agent involve a human 
to draft a plan instead of just 

234
00:13:15,720 --> 00:13:18,800
building the post, asks it and 
leaves it open. 

235
00:13:19,400 --> 00:13:22,640
And on the measuring side? 
Buckler asks a good one. 

236
00:13:23,000 --> 00:13:26,720
If sensors never fire, is that a
sign of high quality or 

237
00:13:26,720 --> 00:13:30,440
inadequate detection mechanisms?
A quiet factory might be a great

238
00:13:30,440 --> 00:13:33,920
one, or a blind one. 
A blind factory? 

239
00:13:34,440 --> 00:13:38,000
Now that's a scary thought. 
Thanks to Ben Holmes for his X 

240
00:13:38,000 --> 00:13:42,000
article, Becoming a Factory 
engineer and To begin to Buckler

241
00:13:42,000 --> 00:13:46,760
on Martin Fowler's site, Nathan 
Harvey's Dora Guide and Metr, 

242
00:13:47,200 --> 00:13:50,320
plus Nathan Harvey and Derek 
Debellis on the Google Cloud 

243
00:13:50,320 --> 00:13:52,720
blog, and Jeff Atwood at Coding 
Horror. 

244
00:13:53,480 --> 00:13:56,920
That was factory engineering 
from writing code to building 

245
00:13:56,920 --> 00:13:59,720
the system that writes it. 
That's all for today on Build 

246
00:13:59,720 --> 00:14:02,200
Wiz AI Show. 
Thanks for listening and see you

247
00:14:02,200 --> 00:14:02,720
next time.
