1
00:00:00,400 --> 00:00:03,840
Hey, hello and welcome back to 
Edge Talks. 

2
00:00:04,000 --> 00:00:07,520
So we had a bit of a summer 
risk, but no worry. 

3
00:00:07,520 --> 00:00:09,080
Fresh. 
I don't know about fresh. 

4
00:00:09,080 --> 00:00:13,120
Two hours first. 
Episode yeah, I know I mean we 

5
00:00:13,120 --> 00:00:16,480
just heard Mark a story so I 
don't know it's he's recovering 

6
00:00:16,480 --> 00:00:21,960
a bit but I think you will do 
fine for today. 

7
00:00:21,960 --> 00:00:28,480
So as always, we have Marcus and
Nisa here, and today's topic is 

8
00:00:28,480 --> 00:00:31,560
gonna be content scraping. 
No, no, no. 

9
00:00:31,840 --> 00:00:37,400
It's it's one of those amazing 
contents I. 

10
00:00:37,920 --> 00:00:40,000
Wonder if this content will be 
scraped. 

11
00:00:42,520 --> 00:00:48,880
I I know just the title makes 
90% of all podcast listeners 

12
00:00:48,880 --> 00:00:51,680
worldwide wanted want to just 
listen to this one right? 

13
00:00:54,120 --> 00:00:56,280
Yeah, I think so too. 
I think so too. 

14
00:00:56,280 --> 00:01:00,280
But what is it that? 
What content scraping is content

15
00:01:00,280 --> 00:01:05,519
scraping? 
In short, is someone getting 

16
00:01:06,000 --> 00:01:09,520
your information on your website
because that's what we talk 

17
00:01:09,520 --> 00:01:14,920
about right now, or your assets 
online at bulk or in a large 

18
00:01:14,920 --> 00:01:17,360
mass? 
Yeah, in an automated fashion. 

19
00:01:17,840 --> 00:01:23,360
Yeah, yeah, it could be manual. 
For sure click Farms. 

20
00:01:23,680 --> 00:01:25,320
Yeah, yeah, yeah. 
You could click farm it. 

21
00:01:25,320 --> 00:01:31,840
So that's why I But it's hard to
make some sort of economical 

22
00:01:32,480 --> 00:01:37,240
point if you would manually 
content scrape Amazon.com for. 

23
00:01:37,240 --> 00:01:38,520
Instance. 
Yeah, no truth. 

24
00:01:39,720 --> 00:01:41,240
I would assume that takes a 
little bit. 

25
00:01:41,240 --> 00:01:42,800
Of time. 
I guess that that would be what 

26
00:01:42,800 --> 00:01:45,960
you call a Mechanical Turk as 
well, so go and check the 

27
00:01:45,960 --> 00:01:49,360
Wikipedia page. 
Or just pure madness and. 

28
00:01:50,800 --> 00:01:53,880
That's a big. 
So I assume then that we're in 

29
00:01:54,520 --> 00:01:56,520
the AI? 
We'll touch on it definitely. 

30
00:01:57,440 --> 00:02:01,640
Yeah, 100%. 
But it is sort of like content 

31
00:02:01,640 --> 00:02:07,200
scraping in a way you could. 
It's a pretty wide definition if

32
00:02:07,200 --> 00:02:09,520
you look at it. 
And you know, we what we said 

33
00:02:09,520 --> 00:02:12,920
was like getting you getting 
your content in bulk, most 

34
00:02:12,920 --> 00:02:18,120
likely in an automated fashion. 
So is a crawler bot like 

35
00:02:18,120 --> 00:02:20,680
Googlebot a content scraper? 
Yeah. 

36
00:02:21,000 --> 00:02:22,000
It is. 
It is. 

37
00:02:22,520 --> 00:02:25,120
Is it AI? 
Maybe not. 

38
00:02:25,240 --> 00:02:28,160
Not necessarily, or at least 
historically it wasn't. 

39
00:02:28,600 --> 00:02:34,680
But so there is sort of like the
I think that the the definition 

40
00:02:34,680 --> 00:02:38,360
is the important one that hey, 
it's something that automat in 

41
00:02:38,360 --> 00:02:41,760
an automated fashion gets your 
content in bulk. 

42
00:02:42,360 --> 00:02:49,280
And I also think that two years 
ago or more or less a year ago, 

43
00:02:49,840 --> 00:02:53,680
people wanted to be scraped by 
Google, for instance, and all of

44
00:02:53,680 --> 00:02:55,560
the other search engine 
optimiser. 

45
00:02:55,560 --> 00:03:01,480
And you were fine with scraping 
and you paid people to make sure

46
00:03:01,480 --> 00:03:04,400
they scraped you well and 
correctly. 

47
00:03:04,400 --> 00:03:10,800
And, and yeah, often and search 
like the whole search engine 

48
00:03:10,800 --> 00:03:15,320
optimization market that that 
was something you wanted to 

49
00:03:15,320 --> 00:03:19,240
happen and you wanted to be best
in class because that would give

50
00:03:19,240 --> 00:03:21,120
you the most clicks and the most
revenue. 

51
00:03:21,600 --> 00:03:23,640
Yeah, it, it was like the SEO. 
Yeah. 

52
00:03:23,760 --> 00:03:27,640
And. 
And now content scraping has 

53
00:03:27,640 --> 00:03:31,320
become something that's more 
negative. 

54
00:03:32,000 --> 00:03:36,360
In my line of work, I talk to 
customers who have a problem 

55
00:03:36,480 --> 00:03:40,200
with being scraped. 
A year ago, no one had a problem

56
00:03:40,200 --> 00:03:43,040
being indexed and scraped by 
Google, for instance. 

57
00:03:45,160 --> 00:03:49,120
It's not only that people block 
Yandex because of a war in 

58
00:03:49,120 --> 00:03:54,720
Ukraine, but the scraping is 
scraping has become a problem. 

59
00:03:56,080 --> 00:03:58,120
Me and Lisa spoke with a 
customer. 

60
00:03:58,160 --> 00:04:00,920
Was it this week? 
Yes. 

61
00:04:01,520 --> 00:04:03,600
Yes. 
And last week, it feels like we 

62
00:04:03,600 --> 00:04:07,440
have these conversations. 
Where like the the the bandwidth

63
00:04:07,440 --> 00:04:12,800
bill is twice because of 1 
scraper being greedy. 

64
00:04:13,360 --> 00:04:16,480
Yeah. 
So why is it like this? 

65
00:04:16,480 --> 00:04:19,440
Why are there? 
So many 'cause people take a 

66
00:04:19,440 --> 00:04:23,040
photograph of their fridge and 
say I want to make pancakes, 

67
00:04:23,040 --> 00:04:28,880
what am I missing and send that 
to shat GPT so. 

68
00:04:29,760 --> 00:04:34,560
So the scraping is actually 
based on us having we're too 

69
00:04:34,560 --> 00:04:36,200
lazy. 
So we're not thinking anymore. 

70
00:04:36,600 --> 00:04:40,840
So I'm, I'm I'm pro AI because 
of the technologies cool and 

71
00:04:40,840 --> 00:04:42,840
machine learning. 
It's not AI if you ask me. 

72
00:04:42,840 --> 00:04:46,480
It's machine learning in LLMS. 
But we are being becoming more 

73
00:04:46,480 --> 00:04:50,720
and more lazy and the usage is 
dirt cheap. 

74
00:04:51,200 --> 00:04:56,000
But, but I think that there is 
just to, to back back a little 

75
00:04:56,000 --> 00:04:59,120
bit, I think that there is sort 
of like all these things are 

76
00:04:59,160 --> 00:05:03,240
definitely in waves. 
So when we, if we talked about 

77
00:05:03,240 --> 00:05:07,160
the, the Google bot running 
around your website, everyone 

78
00:05:07,160 --> 00:05:10,000
was fine with it. 
Then we found out all about all 

79
00:05:10,000 --> 00:05:13,160
the Frontier labs sucking down 
all the data to sort of like 

80
00:05:13,160 --> 00:05:17,600
build a huge amount of data to, 
to teach their, you know, train 

81
00:05:17,600 --> 00:05:21,200
their LLMS on. 
That was sort of like when 

82
00:05:21,200 --> 00:05:23,800
everyone was, hey, they're 
stealing our data. 

83
00:05:24,520 --> 00:05:26,640
Yeah. 
And now we're sort of like in 

84
00:05:26,640 --> 00:05:31,000
the scenario where that's still 
going on to a lesser extent 

85
00:05:31,000 --> 00:05:34,320
because I guess that they sucked
out most of the data on the 

86
00:05:34,320 --> 00:05:37,080
Internet already. 
But then it's sort of like when 

87
00:05:37,080 --> 00:05:40,960
you start talking about agents 
doing checking out, you know, 

88
00:05:40,960 --> 00:05:45,840
that pancake recipe on your 
behalf, it becomes even more 

89
00:05:45,840 --> 00:05:49,480
difficult to start to think 
about how you know, how should, 

90
00:05:49,720 --> 00:05:51,760
how should I manage content 
scraping? 

91
00:05:52,320 --> 00:05:58,400
Yeah, and that's the that's the 
the end discussion when when me 

92
00:05:58,400 --> 00:06:02,680
and Lisa talks with with 
prospect prospects and and and 

93
00:06:02,680 --> 00:06:07,880
existing customers and going 
back to scraping like scraping 

94
00:06:08,040 --> 00:06:13,760
like sure Byte Dance and Google 
and and Open AI, they've they've

95
00:06:13,760 --> 00:06:15,840
scraped the Internet. 
The Internet is scraped and 

96
00:06:15,840 --> 00:06:19,520
copied into multiple instances 
by these operators and stored 

97
00:06:19,520 --> 00:06:24,120
and God what not that with the 
data, but they continue to 

98
00:06:24,120 --> 00:06:28,120
verify that the date is accurate
and there's always new data and 

99
00:06:28,640 --> 00:06:30,920
you scrape the lowest hanging 
fruit. 

100
00:06:30,920 --> 00:06:34,200
So you start with Wikipedia. 
That's the that's the simple 1 

101
00:06:34,200 --> 00:06:38,720
to scrape. 
Now you're scraping like niche 

102
00:06:39,000 --> 00:06:44,560
car manufacturers second hand 
stuff. 

103
00:06:44,560 --> 00:06:50,800
You're scraping eBay. 
You're scarce scraping, umm, 

104
00:06:51,520 --> 00:06:57,880
weird pages, uh, and, and, and 
like, you've done the lowest 

105
00:06:57,880 --> 00:07:01,400
hanging fruit from an scraper 
perspective, so now you're going

106
00:07:01,400 --> 00:07:04,560
into smaller ones and those get 
hurt the most. 

107
00:07:05,040 --> 00:07:08,840
If you look at Bonya news, short
scrapers steal their news 

108
00:07:08,840 --> 00:07:14,320
articles, but they aren't hurt 
by having uh, uh, another 

109
00:07:14,320 --> 00:07:17,560
million requests a month because
they already have a few 100 

110
00:07:17,560 --> 00:07:21,800
million requests. 
But if you're, if you're used to

111
00:07:21,800 --> 00:07:25,280
five to maybe, yeah, if you used
to have 5 million requests per 

112
00:07:25,280 --> 00:07:29,880
month and then additional 
million, uh, and that is also 

113
00:07:29,880 --> 00:07:34,400
using an AI scraper. 
So it will see that you're on 

114
00:07:34,400 --> 00:07:38,600
the WordPress or Magenta site. 
So it will start using your 

115
00:07:38,920 --> 00:07:44,320
category tags greedy and it will
just go, just blow through the, 

116
00:07:44,640 --> 00:07:46,480
the cache and go straight to 
origin. 

117
00:07:46,920 --> 00:07:52,240
And your server that's sized for
a load that you normally have 

118
00:07:52,240 --> 00:07:54,720
won't, won't cope with it. 
So you will get a bandwidth 

119
00:07:54,720 --> 00:07:58,280
build. 
So the the the whole thing is 

120
00:07:58,280 --> 00:08:00,200
like, well do you want to be 
script? 

121
00:08:00,480 --> 00:08:03,400
Yeah, I think that's the that's 
the crux of the issue, right? 

122
00:08:03,680 --> 00:08:05,680
Yeah. 
And also. 

123
00:08:06,280 --> 00:08:08,520
Who scrapes? 
Yeah, yeah, yeah, no. 

124
00:08:08,520 --> 00:08:11,760
And I I can definitely from my 
personal perspective, it's sort 

125
00:08:11,760 --> 00:08:16,880
of like it is a hard no. 
I'm since in a way at Fastly 

126
00:08:16,880 --> 00:08:18,840
we're sitting on a little bit on
the both sides. 

127
00:08:18,840 --> 00:08:22,240
We want to make sure our cust 
our customers can deliver the 

128
00:08:22,240 --> 00:08:27,560
best experience to their users. 
But personally, like I was doing

129
00:08:27,560 --> 00:08:32,400
research on on a pretty big 
purchase this summer and 

130
00:08:32,400 --> 00:08:38,559
obviously I pointed my my claw 
to go and search the web for dot

131
00:08:38,559 --> 00:08:42,880
dot dot dot dot because I want 
the best information and I want 

132
00:08:42,880 --> 00:08:46,080
the LLM to summarize that. 
So it's sort of like the LLM 

133
00:08:46,080 --> 00:08:51,000
plus some kind of a new 
information reviews, forums 

134
00:08:51,000 --> 00:08:55,720
about product X, etcetera. 
So I can make the best decision.

135
00:08:55,960 --> 00:08:59,800
So I can totally understand. 
It's sort of like that scraping,

136
00:09:00,080 --> 00:09:03,400
slash searching, slash 
downloading of that content 

137
00:09:04,200 --> 00:09:09,200
becomes a question for for the 
company is how should we handle 

138
00:09:09,200 --> 00:09:10,800
it? 
Do we want to be part of that 

139
00:09:10,800 --> 00:09:14,280
result? 
Or do we want to block bots or 

140
00:09:14,280 --> 00:09:17,560
scrapers because they cost us 
money? 

141
00:09:18,240 --> 00:09:19,840
Could they actually make us 
money? 

142
00:09:20,240 --> 00:09:23,760
Yeah, and, and and here's what. 
Sorry, Lisa, go ahead. 

143
00:09:24,200 --> 00:09:27,840
No, but it's not the only 
concern and issues. 

144
00:09:27,840 --> 00:09:30,760
You would say the balance here, 
it's the cost you want it, but 

145
00:09:30,760 --> 00:09:34,200
you have to be able to take the 
cost or you don't want it, but 

146
00:09:34,200 --> 00:09:35,680
then you have to figure out what
don't. 

147
00:09:35,680 --> 00:09:37,440
Yeah, no, it's not a black and 
white. 

148
00:09:37,440 --> 00:09:39,960
It's not binary. 
It's definitely like, you know, 

149
00:09:39,960 --> 00:09:49,000
yeah, 5050 Shades of Grey. 
Quantum, I think especially if 

150
00:09:49,000 --> 00:09:53,960
if you used the e-commerce as an
example, which is a returning 

151
00:09:53,960 --> 00:09:56,520
example for us because it's 
something that all of us know, 

152
00:09:56,520 --> 00:10:02,400
including the audience. 
But e-commerce is, is a vertical

153
00:10:02,400 --> 00:10:06,720
under constant pressure for 
pricing and shipping fees and, 

154
00:10:06,720 --> 00:10:08,920
and all of that. 
And it's highly, highly 

155
00:10:08,920 --> 00:10:11,640
competitive, especially with the
globalisation. 

156
00:10:12,000 --> 00:10:20,920
So you want to be scraped so 
that if someone asks their AI 

157
00:10:21,520 --> 00:10:26,600
tool for assistance or help, you
want them to use like you, you 

158
00:10:26,600 --> 00:10:31,040
want Claude to go to, let's say 
Claus Olson instead of going to 

159
00:10:31,040 --> 00:10:33,480
do lab, for instance, That, 
that, that would be the 

160
00:10:33,480 --> 00:10:37,800
marketing department of each 
company to make sure that the, 

161
00:10:38,720 --> 00:10:41,920
the weapon of choice for an AI 
tool would be to use their 

162
00:10:41,920 --> 00:10:45,680
website and their brand. 
So you want to be indexed, You 

163
00:10:45,680 --> 00:10:48,080
want to be crawled. 
You want to, you want to make it

164
00:10:48,080 --> 00:10:53,280
simple, but you also don't want 
your competitors to crawl you 

165
00:10:55,200 --> 00:10:58,560
and do you? 
How much money are you willing 

166
00:10:58,560 --> 00:11:01,400
to spend? 
What's the return on investment?

167
00:11:01,400 --> 00:11:05,240
So there's, there is the 
financial player. 

168
00:11:06,480 --> 00:11:11,080
And and we have like I think 
it's, it's worth mentioning some

169
00:11:11,080 --> 00:11:18,360
facts, 49% of all the traffic 
currently is bot traffic or AI 

170
00:11:18,360 --> 00:11:27,720
traffic and it's increasing 6.5 
faster growth than the normal 

171
00:11:27,720 --> 00:11:29,560
human traffic than us. 
People, we can only. 

172
00:11:29,560 --> 00:11:31,800
Click. 
Just so fast. 

173
00:11:32,760 --> 00:11:36,560
Yeah. 
And, and, and I mean Lisa, we 

174
00:11:36,560 --> 00:11:42,200
spoke, this was a few months ago
with, with one customer, they 

175
00:11:42,200 --> 00:11:50,200
had a, an increase of 5K in U.S.
dollars in traffic fees because 

176
00:11:50,200 --> 00:11:56,160
of bike dance AI scraper and 
they're a niche car part 

177
00:11:56,880 --> 00:12:03,640
e-commerce site, which is a 3X 
increase in their in their IT 

178
00:12:04,480 --> 00:12:07,400
spend. 
And that hurts. 

179
00:12:08,800 --> 00:12:12,160
That hurts, definitely. 
Yeah, yeah, that month will will

180
00:12:12,160 --> 00:12:13,960
not be. 
And I mean that's what we see on

181
00:12:13,960 --> 00:12:16,480
all our tests. 
Yeah, all our, I mean we see 

182
00:12:16,480 --> 00:12:18,240
that on all our e-commerce 
customers. 

183
00:12:18,320 --> 00:12:21,360
This is especially e-commerce, I
would say, isn't it? 

184
00:12:21,640 --> 00:12:24,880
Yeah, it's, but what can we do 
about them? 

185
00:12:25,800 --> 00:12:27,280
What's our option? 
Give up. 

186
00:12:27,560 --> 00:12:30,920
Yeah, I'd say, you know, just. 
Go for it. 

187
00:12:31,320 --> 00:12:33,840
Go to bed. 
No, I I think the the the only 

188
00:12:33,840 --> 00:12:37,400
solution here is to go back to 
send out magazines to everybody 

189
00:12:37,760 --> 00:12:41,160
and then just connect the fax 
machine again. 

190
00:12:41,280 --> 00:12:46,680
Printed catalogues. 
More exactly, more analogue. 

191
00:12:47,760 --> 00:12:50,160
You can. 
You can only order when there's 

192
00:12:50,160 --> 00:12:52,000
statue tuning. 
Yeah, exactly. 

193
00:12:52,000 --> 00:12:54,680
No. 
Yeah, and post the weather. 

194
00:12:55,320 --> 00:12:56,720
Maybe water content, Yeah, I 
know. 

195
00:12:56,960 --> 00:13:00,880
But I think it's sort of like we
generally talk about tech, but I

196
00:13:00,880 --> 00:13:04,360
think that that in the end that 
becomes a business decision on 

197
00:13:05,160 --> 00:13:07,440
how you want to handle that kind
of traffic. 

198
00:13:09,120 --> 00:13:16,280
I guess it's hard, probably hard
to to associate, you know, sales

199
00:13:16,280 --> 00:13:21,680
to scraping, but I think that 
somewhere there is it lies this 

200
00:13:21,680 --> 00:13:26,080
sort of like the truth in terms 
of figuring out what what you 

201
00:13:26,080 --> 00:13:28,400
should actually be doing. 
Yeah. 

202
00:13:29,320 --> 00:13:33,280
Or if you're willing to sort of 
like, yeah, do a calculated risk

203
00:13:33,280 --> 00:13:35,600
and say that, hey, scrapers have
have added. 

204
00:13:36,600 --> 00:13:40,840
Yeah. 
I think that normally when I 

205
00:13:40,840 --> 00:13:45,560
come into the discussion, the 
business have realized that the 

206
00:13:45,560 --> 00:13:48,760
bandwidth bill and the cost of 
serving the website have gone 

207
00:13:48,760 --> 00:13:52,000
up. 
So they go to the IT side of 

208
00:13:52,000 --> 00:13:56,720
their business and say, hey, 
there isn't there's a problem of

209
00:13:56,720 --> 00:14:01,040
the bill being too high, let's 
fix that. 

210
00:14:02,160 --> 00:14:05,960
And that that's, that's the 
thing like to apply a technical 

211
00:14:05,960 --> 00:14:09,360
solution to this problem of a, 
of a large bandwidth bill. 

212
00:14:10,080 --> 00:14:15,160
Whereas we, our response is that
sure, we have the tools and 

213
00:14:15,160 --> 00:14:19,120
measures and we, we have things 
that will help you and we can 

214
00:14:19,120 --> 00:14:20,600
help you and we can work with 
you. 

215
00:14:21,240 --> 00:14:26,040
But the first step is to 
visualize what the reason behind

216
00:14:26,040 --> 00:14:26,800
this. 
Who is it? 

217
00:14:26,800 --> 00:14:30,440
What are they doing? 
Is it malicious crawler or, or, 

218
00:14:30,560 --> 00:14:34,680
or like, is there a business 
case to be made from your point 

219
00:14:34,680 --> 00:14:37,600
of view? 
So the first step is that we 

220
00:14:37,600 --> 00:14:41,800
just visualize the traffic for 
the users using our bot 

221
00:14:42,800 --> 00:14:45,680
features. 
And then we actually asked the 

222
00:14:45,720 --> 00:14:49,080
IT part to go back to the 
business and say we need to 

223
00:14:49,080 --> 00:14:52,520
decide from a business point of 
view, what should we keep as 

224
00:14:52,520 --> 00:14:54,240
traffic and what should we 
mitigate. 

225
00:14:54,520 --> 00:14:55,960
Yeah. 
Yeah. 

226
00:14:56,480 --> 00:14:58,760
I think that's a great point 
because I think people forget 

227
00:14:58,760 --> 00:15:01,480
about that and I think this is a
very common that they see, OK, 

228
00:15:01,840 --> 00:15:05,000
our traffic has just gone up and
then people are guessing, oh, we

229
00:15:05,000 --> 00:15:09,680
believe this is a scrapers and I
think a lot of organizations 

230
00:15:09,680 --> 00:15:12,960
forget that. 
This is a decision for the 

231
00:15:12,960 --> 00:15:16,360
business and for the leadership 
to decide how should we handle 

232
00:15:16,360 --> 00:15:19,400
and what's our strategy for 
scraping? 

233
00:15:19,520 --> 00:15:23,200
No, definitely. 
But I think I mean, still this 

234
00:15:23,200 --> 00:15:29,640
is quite, I mean, new that it 
looks like this or this is a new

235
00:15:29,640 --> 00:15:32,400
environment, right? 
But I think that's for sure it's

236
00:15:32,480 --> 00:15:35,240
some like where the 
organizations need to start. 

237
00:15:36,200 --> 00:15:39,240
And as you said, they said, 
there's a bunch of whistles and 

238
00:15:39,240 --> 00:15:41,680
bells how you can handle it. 
But first you have to make a 

239
00:15:41,680 --> 00:15:43,600
decision as an organization. 
What's your strategy? 

240
00:15:43,600 --> 00:15:45,840
No, I think the. 
Key key there is is that you 

241
00:15:45,840 --> 00:15:49,560
need to first of all visualize 
and see sort of attribute that 

242
00:15:49,560 --> 00:15:53,840
increase in billing, you know, 
in the money to what where it's 

243
00:15:53,840 --> 00:15:59,360
actually going. 
Of course you can be hit by a 

244
00:15:59,360 --> 00:16:03,400
tonne of bad bots, you know, 
malicious bots, and that's 

245
00:16:03,400 --> 00:16:05,480
obviously traffic you don't want
to pay for. 

246
00:16:05,800 --> 00:16:07,560
So obviously you should block 
those. 

247
00:16:07,560 --> 00:16:13,280
But then it's the grey zone of 
of scrapers, good, good behaving

248
00:16:13,280 --> 00:16:15,880
bots, so to say. 
And that is where you need to 

249
00:16:15,880 --> 00:16:20,960
sort of like figure out, do we 
want, is this a potential 

250
00:16:21,000 --> 00:16:24,680
potential revenue stream or is 
it just something that we don't 

251
00:16:24,880 --> 00:16:29,680
want to avoid or is there an, or
is there an upside in paying 

252
00:16:29,680 --> 00:16:31,640
that extra for, for that 
traffic? 

253
00:16:32,080 --> 00:16:33,880
Yeah. 
But how do you know? 

254
00:16:33,920 --> 00:16:36,720
How do you know if it's a good 
scape or a good? 

255
00:16:36,760 --> 00:16:38,720
Funny you should. 
Funny you should ask. 

256
00:16:40,160 --> 00:16:43,760
You should. 
Call me no but. 

257
00:16:44,040 --> 00:16:48,680
There are bot management tools 
obviously that can categorize 

258
00:16:48,680 --> 00:16:51,600
the different types of bots. 
And as Nisa said, you know, one 

259
00:16:51,600 --> 00:16:55,440
part of this is basically 
visualizing what is the traffic 

260
00:16:55,440 --> 00:16:59,080
coming from and, and 
categorizing that traffic into 

261
00:16:59,080 --> 00:17:04,839
different buckets like malicious
bots, good bots, scrapers, AI 

262
00:17:05,720 --> 00:17:08,880
and so forth and so forth. 
Because that's sort of like when

263
00:17:08,880 --> 00:17:13,200
you can get the relationship 
between good, bad, acceptable, 

264
00:17:13,200 --> 00:17:16,680
non acceptable traffic and start
making decision better 

265
00:17:16,680 --> 00:17:18,040
decisions. 
Yeah, yeah. 

266
00:17:18,480 --> 00:17:22,599
So I think what what we do 
because we see a lot of the the 

267
00:17:22,599 --> 00:17:26,400
global Internet traffic of 
course and and we speak 

268
00:17:26,680 --> 00:17:31,440
constantly with the big IT 
providers and Google cloud 

269
00:17:31,440 --> 00:17:36,080
providers and bot operators. 
So we do a lot of verifications,

270
00:17:36,080 --> 00:17:41,120
we do a lot of collections and 
intelligence on these bot 

271
00:17:41,160 --> 00:17:46,440
operators. 
So we know if it's sat GPT in 

272
00:17:46,440 --> 00:17:53,000
the majority of cases because 
there's been, I have one 

273
00:17:53,000 --> 00:17:56,440
customer that are really good at
identifying bots on their own 

274
00:17:56,920 --> 00:17:59,680
and they've done their own tests
with honeypots. 

275
00:18:00,240 --> 00:18:04,800
They can actually see that even 
ChatGPT, they aren't really 

276
00:18:04,800 --> 00:18:10,120
honest about the user agent 
string sometimes because the end

277
00:18:10,120 --> 00:18:13,960
user experience is more 
important than being honest with

278
00:18:13,960 --> 00:18:16,720
who you are on the website 
you're trying to visit. 

279
00:18:17,200 --> 00:18:21,920
So there, there's, there's that 
and they're, they're currently 

280
00:18:21,920 --> 00:18:24,560
trying to write the blog and 
seeing, but there there's some 

281
00:18:25,080 --> 00:18:27,680
probably a legal perspective as 
well, but they're, they're 

282
00:18:27,680 --> 00:18:29,160
trying to figure out these 
things. 

283
00:18:29,920 --> 00:18:31,680
But but. 
Coming. 

284
00:18:32,400 --> 00:18:34,440
Yeah. 
Now coming back to that with 

285
00:18:34,520 --> 00:18:39,960
sort of like the question is if,
if I ask an AI to search the web

286
00:18:39,960 --> 00:18:43,600
for me and pull that content, is
that, yeah, synonymous with a 

287
00:18:43,600 --> 00:18:49,040
browser or a bot. 
So it's not that it's not easy. 

288
00:18:49,760 --> 00:18:52,520
Yeah, that's the $10,000 
question. 

289
00:18:54,000 --> 00:19:01,560
And should should my should if I
ask a bot, should the bot VPN 

290
00:19:01,560 --> 00:19:03,800
through my computer send in the 
query? 

291
00:19:04,680 --> 00:19:07,040
Is that a solution? 
Would it? 

292
00:19:07,400 --> 00:19:12,760
Would having technology where 
the bot can use my user agent 

293
00:19:12,760 --> 00:19:17,800
and send things like that, would
that constitute as a request 

294
00:19:17,800 --> 00:19:21,160
from a scraper? 
Is it scraping or is it just 

295
00:19:22,000 --> 00:19:24,360
automated data from a user? 
Yeah. 

296
00:19:25,280 --> 00:19:26,920
No. 
For example, the currently there

297
00:19:26,920 --> 00:19:32,760
are MCP extensions to browsers. 
Yeah. 100% and what is what? 

298
00:19:32,760 --> 00:19:35,280
Is that then, is that a bot or 
is that a browser? 

299
00:19:35,840 --> 00:19:38,440
Yeah. 
And it's it's not going to be 

300
00:19:38,440 --> 00:19:44,040
long until we're going to see 
specific endpoints where E COM 

301
00:19:44,040 --> 00:19:49,640
providers, for instance, will 
point AI agents to where it's 

302
00:19:49,640 --> 00:19:53,400
going to be more efficient. 
So an API, an MCP instead of 

303
00:19:53,720 --> 00:19:57,040
loading a sheet, ton of graphics
and static content that would 

304
00:19:57,040 --> 00:20:02,480
just cost extra. 
So I I think that that's the 

305
00:20:02,480 --> 00:20:07,440
path, yeah. 
So so if we. 

306
00:20:07,440 --> 00:20:12,280
Think about then we, well, we 
haven't, but let's, let's just 

307
00:20:12,280 --> 00:20:15,760
pretend that we have come up 
with a, with a solution or a 

308
00:20:15,760 --> 00:20:18,360
strategy for how we want to 
handle bots. 

309
00:20:18,880 --> 00:20:24,880
Yeah, what, what are the means? 
You know, we talked about bot 

310
00:20:24,880 --> 00:20:28,120
management, but that's sort of 
like identifying things. 

311
00:20:28,640 --> 00:20:31,240
Yeah. 
How can we couple that with 

312
00:20:31,240 --> 00:20:35,920
actual actions? 
I think the, the, the, the 

313
00:20:35,920 --> 00:20:42,080
normal technology has been to 
use a capture which is horrible 

314
00:20:42,080 --> 00:20:48,880
user experience and like select 
all the boxes that contain a bus

315
00:20:48,880 --> 00:20:53,120
and you have 4 pixels on the 
wrong side and all of those 

316
00:20:53,200 --> 00:20:57,480
different things. 
At fast they were trying to do 

317
00:20:57,480 --> 00:21:02,080
something we call proof of work 
or silent challenges where we 

318
00:21:02,080 --> 00:21:08,480
and I know we mentioned this 
before, but we we send like a 

319
00:21:08,480 --> 00:21:13,400
mathematical exercise to the 
client and they will use the the

320
00:21:13,400 --> 00:21:17,400
Java engine in the browser, the 
JavaScript engine in the browser

321
00:21:17,400 --> 00:21:23,560
to to solve the the algorithm 
and and return with a value and 

322
00:21:23,680 --> 00:21:26,080
proof. 
Of work, yeah, basically. 

323
00:21:26,720 --> 00:21:30,920
And then we said a cookie, and 
that cookie is also set with a 

324
00:21:30,920 --> 00:21:33,480
lot of value. 
So even if you take that cookie 

325
00:21:33,480 --> 00:21:37,200
and move to another machine, 
that cookie value wouldn't 

326
00:21:37,520 --> 00:21:42,320
calculate and be verifiable 
because it might connect with 

327
00:21:42,320 --> 00:21:49,360
other things, TLS signatures and
and some secret sauce that we 

328
00:21:49,360 --> 00:21:52,440
would want to have secret. 
And then of course there's a lot

329
00:21:52,440 --> 00:21:55,880
of other stuff to identify 
headless browsers. 

330
00:21:57,400 --> 00:22:01,240
So the The thing is that you 
would need to verify it's a 

331
00:22:01,240 --> 00:22:07,000
human. 
Yeah, that's hard it. 

332
00:22:07,000 --> 00:22:09,640
Is hard definitely. 
And and sort of like the 

333
00:22:09,640 --> 00:22:13,960
impersonation is, you know, 
there's a there's a tug of war 

334
00:22:14,120 --> 00:22:18,160
going on that as soon as you 
make a detection better, you 

335
00:22:18,160 --> 00:22:22,520
know, the, the getting, you 
know, masking as something else 

336
00:22:22,520 --> 00:22:25,240
gets better. 
And it's sort of like goes on 

337
00:22:25,240 --> 00:22:28,160
and on and on. 
But yeah, but I think it's sort 

338
00:22:28,160 --> 00:22:31,800
of like the I think that's the 
one of the key, key things is 

339
00:22:31,800 --> 00:22:37,480
actually starting to to try and 
bucket good and bad traffic or 

340
00:22:37,600 --> 00:22:42,720
not normal human and automated 
traffic. 

341
00:22:44,360 --> 00:22:48,640
So you so you can do something 
And then then after that, I 

342
00:22:48,640 --> 00:22:53,480
think that the there are ideas 
that could come in mind is sort 

343
00:22:53,480 --> 00:22:58,000
of like thinking about, you 
know, giving things a scraping 

344
00:22:58,000 --> 00:23:00,960
budget. 
How much traffic are we willing 

345
00:23:00,960 --> 00:23:06,000
to accept? 
You know, traditional means that

346
00:23:06,000 --> 00:23:10,000
has been used for a long time 
are still applicable, you know. 

347
00:23:10,800 --> 00:23:12,280
Yeah. 
Well, rate, rate. 

348
00:23:12,280 --> 00:23:17,160
Limit that, Bob, if you want to 
have it, if you want to let it 

349
00:23:17,160 --> 00:23:22,800
through, but but not like go 
bananas on your website rate 

350
00:23:22,800 --> 00:23:25,160
limited. 
Yeah, maybe there are still 

351
00:23:25,160 --> 00:23:29,040
areas, areas where you don't 
want to let let the bot come in 

352
00:23:30,240 --> 00:23:34,360
like old archives because those 
are most likely going to hit 

353
00:23:34,360 --> 00:23:36,480
your cash it ratio and things 
like that. 

354
00:23:38,920 --> 00:23:42,560
So there are, there are many 
things to think about, yeah. 

355
00:23:42,560 --> 00:23:47,520
And I think it's if you worked 
10 years with online 

356
00:23:47,520 --> 00:23:55,160
experiences, you would have to 
rethink a lot of those more or 

357
00:23:55,160 --> 00:23:57,600
less truths, old time truth we 
have. 

358
00:23:57,600 --> 00:24:03,400
We have to revisit a lot of 
given decisions and re evaluate 

359
00:24:04,120 --> 00:24:10,200
a lot of assumptions such As, 
for instance, as Marcus said, 

360
00:24:11,480 --> 00:24:20,280
budget, budget visitors, would 
my page be fully available? 

361
00:24:21,000 --> 00:24:24,600
And then will the people 
visiting us be honest? 

362
00:24:24,880 --> 00:24:28,240
Well, they are honest because 
it's it's a news site and the 

363
00:24:28,240 --> 00:24:32,320
only people normally visiting 
visiting our site is humans. 

364
00:24:32,960 --> 00:24:37,800
But now scrapers are greedier 
for news than humans, yeah? 

365
00:24:38,400 --> 00:24:41,080
I think that means that. 
You would have to yeah, you 

366
00:24:41,080 --> 00:24:44,800
would have to guard your content
if, if that's the value of the 

367
00:24:45,640 --> 00:24:47,080
Yeah, I. 
Think that that's sort of like 

368
00:24:47,320 --> 00:24:49,800
five years ago. 
The strategy for your web page 

369
00:24:49,800 --> 00:24:52,680
or API, your application or 
whatever was very easy. 

370
00:24:53,040 --> 00:24:56,680
Make it as fast as possible. 
And if you know when you when 

371
00:24:56,680 --> 00:24:59,760
you're done adding feature, make
try to make that faster. 

372
00:24:59,920 --> 00:25:02,880
And if you don't can't come up 
with anything new, make try to 

373
00:25:02,880 --> 00:25:05,600
make it even faster. 
Yeah, but you know. 

374
00:25:05,600 --> 00:25:08,600
Just that was. 
Always the solution, yeah. 

375
00:25:08,600 --> 00:25:12,040
Yeah, exactly. 
But it's like that that doesn't 

376
00:25:12,400 --> 00:25:15,280
apply because you need to 
categorize traffic, yeah. 

377
00:25:15,880 --> 00:25:20,840
And shatke PT or any other of 
the scrapers, they will always 

378
00:25:20,840 --> 00:25:27,240
have more bandwidth than you 
will to be honest, which means 

379
00:25:27,240 --> 00:25:31,760
that they will outperform any of
the websites more or less. 

380
00:25:32,120 --> 00:25:36,000
But I also think that the 
technology is the same now as it

381
00:25:36,000 --> 00:25:40,680
was in the 1990s, more or less. 
You still own the website, you 

382
00:25:40,680 --> 00:25:43,320
still own the decision to send 
the data. 

383
00:25:43,800 --> 00:25:47,720
You can't control who asks, but 
you can control who to give. 

384
00:25:47,720 --> 00:25:50,520
And that's the key, key part 
there. 

385
00:25:50,760 --> 00:25:56,320
It's just that we haven't had 
the the requirement or the need 

386
00:25:56,600 --> 00:26:00,760
to control when responding. 
We've always just as Mark 

387
00:26:00,760 --> 00:26:04,760
Cassette responded as fast as 
humanly possible or machine 

388
00:26:04,760 --> 00:26:09,320
possible, more or less. 
But we always try to, to to 

389
00:26:09,320 --> 00:26:12,120
respond. 
And now we have to think like, 

390
00:26:12,120 --> 00:26:16,840
is this a question I should I 
should respond to? 

391
00:26:17,600 --> 00:26:20,480
Should I give it all or should I
give just enough? 

392
00:26:21,800 --> 00:26:27,280
And if the question is XY 
respond with X + Y, maybe X is 

393
00:26:27,280 --> 00:26:30,840
enough. 
Uh, so there's, there's a lot of

394
00:26:32,160 --> 00:26:36,200
decisions to be made and I work 
with fastly. 

395
00:26:36,200 --> 00:26:43,160
So those decisions should be 
done in the edge, of course, 

396
00:26:43,160 --> 00:26:47,320
because we've had the, the YCDN 
before. 

397
00:26:48,040 --> 00:26:50,320
Of course, yeah. 
Totally. 

398
00:26:50,560 --> 00:26:54,720
No, I, I think that that's umm, 
umm, I'm not sure if we will get

399
00:26:54,720 --> 00:26:59,440
any further in terms of, you 
know, I actual, you know, there 

400
00:26:59,440 --> 00:27:03,680
are tools, but it's up to you 
how to use those tools in the 

401
00:27:03,680 --> 00:27:07,040
most efficient way. 
And that requires a fair bit of 

402
00:27:07,040 --> 00:27:09,200
thinking. 
I think that's sort of like the 

403
00:27:09,200 --> 00:27:13,680
conclusion that it all boils 
down to that there is no silver 

404
00:27:13,680 --> 00:27:18,160
bullet because everyone's 
different and it it so much 

405
00:27:18,160 --> 00:27:22,400
depends on who you are, what 
kind, what kind of business 

406
00:27:22,400 --> 00:27:27,200
you're in, what kind of value 
can you attribute to actual 

407
00:27:27,280 --> 00:27:30,120
scraping. 
It might be totally valuable for

408
00:27:30,120 --> 00:27:35,640
you how? 
Do you think, do you think 

409
00:27:35,640 --> 00:27:41,760
there's added values to be made 
by being as simple as possible 

410
00:27:41,760 --> 00:27:47,240
for a scraper or having an MCP? 
If if you're an e-commerce, do 

411
00:27:47,240 --> 00:27:51,200
you think there's an added value
of investing in in let's say an 

412
00:27:51,200 --> 00:27:55,120
MCP server that can place orders
for you? 

413
00:27:56,200 --> 00:27:58,880
That is a good question. 
I I think that there has been a 

414
00:27:58,880 --> 00:28:04,080
lot of talk about AI shopping, 
letting the AI shop for you. 

415
00:28:04,480 --> 00:28:10,160
The question is how much has 
that taken off and will it take 

416
00:28:10,160 --> 00:28:13,760
off? 
Would you would you be 

417
00:28:13,760 --> 00:28:19,840
comfortable asking Claude? 
Hey, I saw there was there was a

418
00:28:20,560 --> 00:28:26,680
discount for the the Samsung 
frame version like 2026 model 

419
00:28:26,840 --> 00:28:30,400
A32 inch. 
Can you get me the cheapest 

420
00:28:30,400 --> 00:28:33,120
possible and get it delivered to
my home? 

421
00:28:33,520 --> 00:28:35,640
I I guess. 
That I would get something like 

422
00:28:35,640 --> 00:28:38,200
a a postcard sized one from 
Timu, yeah. 

423
00:28:42,000 --> 00:28:44,480
Probably. 
But yeah, I. 

424
00:28:44,480 --> 00:28:45,840
Think that there are some things
that don't. 

425
00:28:45,960 --> 00:28:50,400
Get 10,000 of those. 
I think that there is still some

426
00:28:50,400 --> 00:28:54,040
things that are held true and I 
many years ago I worked with a 

427
00:28:54,040 --> 00:28:59,400
customer that actually did some.
They did airplane tickets, 

428
00:29:00,080 --> 00:29:02,040
right? 
They wanted bots. 

429
00:29:02,200 --> 00:29:06,880
That was half their business, 
but they had an interesting stat

430
00:29:06,880 --> 00:29:10,680
as well that generally all the 
search that happened came from 

431
00:29:10,680 --> 00:29:13,320
cell phones. 
Purchase is always from desktop.

432
00:29:14,200 --> 00:29:18,760
So I think that probably for 
bigger items, probably we are 

433
00:29:18,760 --> 00:29:22,960
conservative enough not to let 
the AI bought go bananas and 

434
00:29:22,960 --> 00:29:25,240
just buy buy all the stuff for 
us. 

435
00:29:25,760 --> 00:29:27,240
I. 
Think that's a generational 

436
00:29:27,240 --> 00:29:29,320
thing because probably, like 
probably. 

437
00:29:29,320 --> 00:29:31,560
Like. 
I'm born in the 80s, so there 

438
00:29:31,560 --> 00:29:35,520
are activities online that is a 
big screen activity, like buying

439
00:29:35,560 --> 00:29:39,760
tickets, travelling, bigger 
purchases. 

440
00:29:39,960 --> 00:29:44,880
Yeah, I could buy socks and 
yeah, order a cable with my 

441
00:29:44,880 --> 00:29:48,560
phone. 
Is there spend limits that 

442
00:29:48,680 --> 00:29:50,920
connected to you if you buy it 
from your phone I think? 

443
00:29:50,920 --> 00:29:55,240
That's the cost of the regret of
buying the wrong thing that that

444
00:29:55,240 --> 00:29:57,560
makes the decision. 
There is no set limit. 

445
00:30:00,120 --> 00:30:03,440
And will there be a similar 
limit that only for letting your

446
00:30:03,720 --> 00:30:05,600
AI do your shopping? 
No, but I. 

447
00:30:05,600 --> 00:30:09,600
Don't want sort of like 500 
kilos of sugar when I need 2 

448
00:30:09,600 --> 00:30:17,720
it's but I don't know maybe, 
maybe maybe some some time 

449
00:30:17,760 --> 00:30:20,680
there. 
I still haven't found a really 

450
00:30:20,680 --> 00:30:23,280
good case. 
And as I said this summer I did 

451
00:30:23,280 --> 00:30:29,040
some research with with Claude, 
but and OK, they they don't have

452
00:30:29,040 --> 00:30:34,280
a shopping feature yet, but in 
it definitely gave me a price 

453
00:30:34,280 --> 00:30:39,400
range, but I was able to get 
that price a lot lower for sure.

454
00:30:40,520 --> 00:30:44,720
Yeah, by with the Google normal 
conditional. 

455
00:30:46,000 --> 00:30:50,760
Yeah, I would never give my AI 
assistant my credit card. 

456
00:30:51,240 --> 00:30:56,080
Like here's my Amex card. 
No, I'm thinking there's a lot 

457
00:30:56,080 --> 00:30:59,320
of people asking for other 
people's credit card numbers. 

458
00:31:00,240 --> 00:31:02,360
It's probably something you 
would be able to get. 

459
00:31:02,720 --> 00:31:06,760
So I think there's I think 
there's ways to go because as we

460
00:31:06,760 --> 00:31:08,960
said, there's big screen 
activities. 

461
00:31:09,720 --> 00:31:12,840
So there will be AI activities 
that are safe to do. 

462
00:31:12,840 --> 00:31:17,840
And one of them is probably the 
the research part will be more 

463
00:31:17,840 --> 00:31:22,600
automated, which is where like 
operators like please Yacht for 

464
00:31:22,600 --> 00:31:27,760
instance, shameless plug here, 
but and others like price 

465
00:31:27,760 --> 00:31:32,200
comparison features, they will 
play a bigger role totally. 

466
00:31:33,000 --> 00:31:36,000
Yeah, great. 
Can you guys, we're running out 

467
00:31:36,000 --> 00:31:37,440
all the time today. 
This was the least. 

468
00:31:37,440 --> 00:31:39,080
Technical session ever? 
No. 

469
00:31:39,080 --> 00:31:40,240
Yeah, yeah. 
No, I said. 

470
00:31:40,240 --> 00:31:44,520
It's it's a lot of fluff, Yeah. 
I think it's, yeah. 

471
00:31:44,640 --> 00:31:46,920
I think it's also, you know, the
summer holiday. 

472
00:31:47,000 --> 00:31:48,520
We're a little bit more relaxed 
maybe. 

473
00:31:48,640 --> 00:31:53,000
I forgot about everything. 
But. 

474
00:31:53,280 --> 00:31:57,280
That's why I think. 
No, but I think the topic at 

475
00:31:57,280 --> 00:32:00,720
hand is still, you know, it's 
everyone needs to make their own

476
00:32:00,720 --> 00:32:05,680
decisions on on how to do 
things, IE as I said, there is 

477
00:32:05,680 --> 00:32:08,200
no silver bullet. 
So you definitely sort of like 

478
00:32:08,200 --> 00:32:12,000
need to come up with with your 
own strategy and then, you know 

479
00:32:12,000 --> 00:32:15,880
attach, you know, attach tooling
to that accordingly. 

480
00:32:17,240 --> 00:32:21,200
So it's it is not an easy thing,
but it's still a thing worth 

481
00:32:21,200 --> 00:32:25,560
talking about because it is it 
is a category of traffic that is

482
00:32:25,560 --> 00:32:31,120
growing so much and has an 
effect on all the customers that

483
00:32:31,120 --> 00:32:33,120
are on the web basically. 
Yeah. 

484
00:32:33,680 --> 00:32:37,680
And the tools are the same 
because the technology and the 

485
00:32:37,680 --> 00:32:41,760
protocols are still the same. 
So you you still have those 

486
00:32:42,040 --> 00:32:45,640
technology, that type of 
technology we need to work with 

487
00:32:45,960 --> 00:32:54,280
good old HTTP, Yes, yes. 
Let's leave it at that. 

488
00:32:54,760 --> 00:32:56,400
That's. 
Marcus's new tattoo. 

489
00:32:59,280 --> 00:33:00,640
Yes. 
Good. 

490
00:33:00,640 --> 00:33:04,600
Old HTTP. 
Thank you all. 

491
00:33:04,600 --> 00:33:07,160
Thank you. 
And welcome back after summer. 

492
00:33:07,720 --> 00:33:07,800
Bye.