1
00:00:00,040 --> 00:00:02,240
This episode is brought to you 
by Beam OPS. 

2
00:00:02,400 --> 00:00:05,520
If your Elixir deployments are 
painful, or your infrastructure 

3
00:00:05,520 --> 00:00:07,960
is a mess, or you just need 
experience hands to help you 

4
00:00:07,960 --> 00:00:11,880
scale, these are your people. 
Beam OPS specializes in scalable

5
00:00:11,880 --> 00:00:14,840
Elixir development, automated 
deployment pipelines, and 

6
00:00:14,840 --> 00:00:18,040
infrastructure migrations. 
They will help you work smarter 

7
00:00:18,080 --> 00:00:21,520
and ship faster. 
As authors of the Pragprog book 

8
00:00:21,560 --> 00:00:24,520
Engineering Elixir Applications,
they know their stuff. 

9
00:00:24,760 --> 00:00:30,480
Check them out at beamops.co.uk.
Coding Elixir with AI or 

10
00:00:30,480 --> 00:00:32,479
building AI into your Elixir 
apps? 

11
00:00:32,720 --> 00:00:36,240
The Elixir AI Collective Discord
is your community for both. 

12
00:00:36,360 --> 00:00:45,880
Link in the description below. 
Testing what's up? 

13
00:00:45,880 --> 00:00:48,720
I'm Jacob Litzow, your Elixir 
mentor and welcome to another 

14
00:00:48,720 --> 00:00:51,080
exciting episode of the Elixir 
Mentor podcast. 

15
00:00:51,080 --> 00:00:53,440
This is where we discuss 
everything related to Elixir, 

16
00:00:53,440 --> 00:00:55,480
from interviews with 
enthusiasts, pioneers in the 

17
00:00:55,480 --> 00:00:58,760
community to innovative projects
and library shaping Elixir's 

18
00:00:58,760 --> 00:01:01,280
future. 
Today I'm joined by Mark Kotner,

19
00:01:01,960 --> 00:01:05,120
telecom engineer, open source 
contributor, and the person 

20
00:01:05,120 --> 00:01:09,040
behind something big in Elixir, 
observability. 

21
00:01:09,560 --> 00:01:10,920
Thanks for joining me today, 
Mark. 

22
00:01:10,920 --> 00:01:14,480
I'm excited to to chat. 
It's good to be here. 

23
00:01:16,120 --> 00:01:18,600
To kick things off, do you want 
to just tell us a little bit 

24
00:01:18,600 --> 00:01:20,920
about yourself, like where 
you're best, what's your 

25
00:01:20,920 --> 00:01:23,680
background and what your 
day-to-day looks like? 

26
00:01:24,600 --> 00:01:28,760
Sure, I'm in Atlanta, GA, my 
day-to-day is pretty busy. 

27
00:01:28,760 --> 00:01:30,880
I have basically 2 full time 
jobs. 

28
00:01:31,560 --> 00:01:37,080
I'm ADBA for Mannheim Auto 
Auctions, which is largest 

29
00:01:37,080 --> 00:01:42,640
auction provider for used cars 
in the United States and at 

30
00:01:42,640 --> 00:01:44,760
night and I have about 1000 
databases or so. 

31
00:01:45,200 --> 00:01:49,040
And at night and weekends I am a
CTO of a startup. 

32
00:01:49,440 --> 00:01:55,080
So the startup is we do small 
cable plants, we do mostly 

33
00:01:55,080 --> 00:02:00,280
hospitality networks and you 
know, 2 to 300 modem small 

34
00:02:01,560 --> 00:02:04,480
networks with cable modems. 
That's awesome. 

35
00:02:04,480 --> 00:02:07,240
Very cool. 
What what ended up getting you 

36
00:02:07,480 --> 00:02:11,200
into that into that space? 
That's where I started. 

37
00:02:11,720 --> 00:02:16,080
So I my background is cable. 
So I started, yeah, my first 

38
00:02:16,080 --> 00:02:21,320
cable modem was a one way 
download only ISA card that you,

39
00:02:21,840 --> 00:02:24,320
you used your dial up for 
uploads. 

40
00:02:24,840 --> 00:02:28,000
So yeah, I've been in the cable 
industry for a very long time. 

41
00:02:28,880 --> 00:02:34,040
I remember dial up. 
I also remember when like ISDN 

42
00:02:34,040 --> 00:02:36,720
came out and it like being 
permanently connected to 

43
00:02:36,720 --> 00:02:39,080
something became a thing that 
was that was cool. 

44
00:02:41,160 --> 00:02:44,320
Yeah, actually. 
So I, I started very early in 

45
00:02:44,320 --> 00:02:51,200
that and I sat for the the DOXIS
steering committee for Cox 

46
00:02:51,600 --> 00:02:56,040
Communications back in the day. 
So the DOXIS is the cable modem 

47
00:02:56,520 --> 00:02:59,440
protocol that actually operates 
the cable modems. 

48
00:02:59,840 --> 00:03:02,920
So yeah, I've been around it for
a very long time. 

49
00:03:03,280 --> 00:03:05,960
That's awesome for a minute. 
Like in high school, it's been a

50
00:03:05,960 --> 00:03:08,040
while. 
I did some Cisco networking 

51
00:03:08,040 --> 00:03:14,760
stuff and I don't remember any 
of it, but it was, I learned a 

52
00:03:14,760 --> 00:03:16,800
lot. 
I remember that and it was good.

53
00:03:17,640 --> 00:03:21,960
I'm curious what what, what was 
your background like? 

54
00:03:21,960 --> 00:03:26,960
What got you into like the cable
modem and then into databases? 

55
00:03:27,000 --> 00:03:30,080
And now you're writing Elixir, 
I'm guessing with the 

56
00:03:30,080 --> 00:03:32,600
hospitality stuff, right? 
I am. 

57
00:03:32,880 --> 00:03:37,360
So originally we, you know, I 
started as sort of a sys admin 

58
00:03:37,360 --> 00:03:39,880
network engineer for the cable 
and home side. 

59
00:03:40,800 --> 00:03:44,200
And then I became a specialist 
and started doing the steering 

60
00:03:44,200 --> 00:03:48,040
committee. 
And as part of that I decided we

61
00:03:48,040 --> 00:03:50,760
needed better observability in 
our cable networks. 

62
00:03:51,600 --> 00:03:54,880
And so I built an application 
that helped with that. 

63
00:03:55,160 --> 00:03:57,600
And so it was used for every 
single call, like if you were a 

64
00:03:57,600 --> 00:04:01,920
Cox cable customer and you 
called in our app was what 

65
00:04:02,160 --> 00:04:04,920
basically helped them 
troubleshoot your cable modem 

66
00:04:04,920 --> 00:04:07,120
and see how it was feeling today
and all that stuff. 

67
00:04:07,600 --> 00:04:12,320
So that's of course required a 
very large database. 

68
00:04:12,320 --> 00:04:14,720
And so I got a lot of attention 
in the my sequel space. 

69
00:04:14,720 --> 00:04:17,800
This was a my sequel database 
back in the day, got a lot of 

70
00:04:17,800 --> 00:04:20,920
attention back in the day for my
sequel. 

71
00:04:21,880 --> 00:04:24,840
And so I helped them get their 
second round of funding, made 

72
00:04:24,840 --> 00:04:31,080
them a lot of money. 
And and so, yeah, just after 

73
00:04:31,080 --> 00:04:36,120
that, I was asked by Manheim was
having some performance issues 

74
00:04:36,120 --> 00:04:38,640
on their databases. 
They asked me to come over. 

75
00:04:39,400 --> 00:04:42,720
Manheim is also part of the same
company that Cox Communications 

76
00:04:42,720 --> 00:04:45,240
is. 
And so I came over from Cox 

77
00:04:45,240 --> 00:04:48,640
Communications to Manheim to 
help them with their databases. 

78
00:04:48,640 --> 00:04:50,960
And that was, you know, 6-8 
databases at the time. 

79
00:04:50,960 --> 00:04:55,520
And now it's, you know, 1000. 
But, but that's how I, I became 

80
00:04:55,520 --> 00:04:59,800
ADBA during, you know, my day 
job is I built an app that 

81
00:04:59,920 --> 00:05:02,360
basically gave us observability 
for cable modems. 

82
00:05:02,720 --> 00:05:06,120
So my, my background is 
definitely observability and 

83
00:05:06,120 --> 00:05:10,040
that's why I broke timeless the 
observability databases, right? 

84
00:05:10,320 --> 00:05:13,680
So there's a timeless metrics, 
which is a metrics database. 

85
00:05:14,000 --> 00:05:17,800
There is timeless logs, which is
a log compression. 

86
00:05:17,800 --> 00:05:22,160
You know, log ingestion allows 
you to search database. 

87
00:05:22,160 --> 00:05:24,920
And then there's the traces, 
right? 

88
00:05:25,240 --> 00:05:29,520
So I, I ingest all the traces 
and then you can go and search 

89
00:05:29,920 --> 00:05:32,000
for traces and look them up and 
stuff like that. 

90
00:05:32,720 --> 00:05:37,960
Very high compression, very, 
very low CPO overhead, which is 

91
00:05:37,960 --> 00:05:41,000
why I sort of pursued the 
embedded side, which you guys 

92
00:05:41,000 --> 00:05:44,200
will see later. 
And so you can just embed your 

93
00:05:44,200 --> 00:05:47,480
observability right in your 
Phoenix app is the goal. 

94
00:05:48,280 --> 00:05:51,040
Yeah, that's really cool. 
I, yeah, you sent me a demo 

95
00:05:51,440 --> 00:05:54,360
right before this and I was 
trying to watch it, but, but to 

96
00:05:54,360 --> 00:05:57,960
be honest, I was like watching 
it and also multitasking. 

97
00:05:57,960 --> 00:06:04,280
So I probably absorbed like 10%.
But it seems like it seems 

98
00:06:04,280 --> 00:06:05,200
really cool. 
What? 

99
00:06:06,000 --> 00:06:10,160
So like, I guess we can, I'm 
going to, we'll shift to the 

100
00:06:10,360 --> 00:06:13,720
your, your observability engine 
a little bit here first. 

101
00:06:16,520 --> 00:06:19,640
Why that and what makes it so 
complex? 

102
00:06:21,200 --> 00:06:25,320
So I was actually putting 
together Ansible scripts to 

103
00:06:25,320 --> 00:06:30,920
deploy some of our apps in the, 
on the, the ISP side, the the 

104
00:06:31,360 --> 00:06:33,880
CTO slash, you know, my, my 
night job. 

105
00:06:35,080 --> 00:06:39,480
And it was a bit more difficult 
than I was expecting to get the 

106
00:06:39,480 --> 00:06:42,200
Ansible just right with the 
stack that was, you know, highly

107
00:06:42,200 --> 00:06:44,680
available. 
It had metrics, you know, I was 

108
00:06:44,680 --> 00:06:48,680
using Victoria metrics at the 
time and which is fantastic. 

109
00:06:48,800 --> 00:06:51,640
You know, I, I don't necessarily
want to compete with them 

110
00:06:51,640 --> 00:06:53,040
because they're quite good at 
what they do. 

111
00:06:53,400 --> 00:06:57,200
But but yeah, so it was just 
really difficult to sort of get 

112
00:06:57,200 --> 00:07:00,680
all the Ansible right to get it 
in high availability, everything

113
00:07:00,680 --> 00:07:02,400
that I wanted. 
And I was like, you know what, 

114
00:07:02,400 --> 00:07:08,360
this should be easier because 99
Percent, 95% of apps don't 

115
00:07:08,360 --> 00:07:10,920
really need external 
observability stacks. 

116
00:07:10,920 --> 00:07:13,240
You know where you're pushing 
your logs to a server, you're 

117
00:07:13,240 --> 00:07:16,760
pushing your traces to a server,
you're pushing your metrics to a

118
00:07:16,760 --> 00:07:20,440
server. 
So, you know, I, I needed a high

119
00:07:20,440 --> 00:07:22,720
performance. 
It has to be high performance if

120
00:07:22,720 --> 00:07:25,320
you embed it because you don't 
want it to take away from the 

121
00:07:25,320 --> 00:07:30,520
app's ability to serve, right? 
So it needs to be like Rust or 

122
00:07:30,520 --> 00:07:35,360
something like that Wrestler. 
And so, you know, basically just

123
00:07:35,360 --> 00:07:37,760
a really lightweight 
observability stack that 

124
00:07:37,760 --> 00:07:40,240
compresses really well that you 
can just run inside your 

125
00:07:40,240 --> 00:07:42,800
application. 
I even added them in an in 

126
00:07:42,800 --> 00:07:47,640
memory mode so that you could do
it with say, a nerves app. 

127
00:07:48,720 --> 00:07:52,280
You don't want to necessarily 
write metrics to your SD card, 

128
00:07:52,280 --> 00:07:53,800
right? 
Because those things don't 

129
00:07:53,800 --> 00:07:55,520
necessarily take a lot of 
rights. 

130
00:07:56,160 --> 00:07:57,560
And so there's an in memory 
mode. 

131
00:07:57,560 --> 00:08:00,400
So you can actually just take 
the metrics, compress them and 

132
00:08:00,400 --> 00:08:02,720
just store them in memory, which
is pretty cool as well. 

133
00:08:03,040 --> 00:08:05,160
That is cool. 
I don't remember who I was just 

134
00:08:05,160 --> 00:08:10,720
talking to because all my 
podcast guests kind of mesh, but

135
00:08:10,720 --> 00:08:17,040
that we were talking about. 
Like if you want to like up your

136
00:08:17,160 --> 00:08:20,240
software engineering game to 
work on embedded systems because

137
00:08:20,240 --> 00:08:22,800
you have limited resources, 
which is something we're not 

138
00:08:22,800 --> 00:08:26,960
very used to anymore in the 
software world 'cause we just 

139
00:08:26,960 --> 00:08:30,280
throw more and more RAM at it, 
or more GPU, whatever we might 

140
00:08:30,280 --> 00:08:33,000
need. 
Yeah, there's a, there's a 

141
00:08:33,000 --> 00:08:37,200
number of things that I've 
written that are to run on 

142
00:08:37,400 --> 00:08:39,880
routers, which is the same, 
right? 

143
00:08:39,880 --> 00:08:42,240
You just need it to be as small 
as possible and take up this 

144
00:08:42,320 --> 00:08:44,360
lease. 
The least amount of memory is 

145
00:08:44,360 --> 00:08:45,840
possible, so. 
Yeah, that makes sense. 

146
00:08:45,840 --> 00:08:47,480
It's the same same concept. 
Yep. 

147
00:08:48,560 --> 00:08:52,080
And like a router has to always 
work too. 

148
00:08:52,080 --> 00:08:55,720
Like you know, you plug it in 
and it needs to just start doing

149
00:08:55,720 --> 00:08:57,680
its thing. 
Yeah, you can't. 

150
00:08:57,680 --> 00:08:59,880
You can't interfere with it 
passing packets, right? 

151
00:09:00,120 --> 00:09:02,040
That's the most important thing 
that it does. 

152
00:09:02,480 --> 00:09:07,840
Yeah, yeah, that's cool. 
I feel like, like I really 

153
00:09:07,840 --> 00:09:13,440
appreciate like people that do 
the observability stuff and 

154
00:09:13,440 --> 00:09:16,200
everything because I feel like 
it's like it's something I don't

155
00:09:16,200 --> 00:09:18,360
want to think about, but it's 
really nice to have those 

156
00:09:18,360 --> 00:09:20,000
metrics and logs when you need 
them. 

157
00:09:22,320 --> 00:09:23,720
Yeah, well, the nice. 
Go ahead. 

158
00:09:24,160 --> 00:09:26,200
Oh, no, I was just. 
Yeah, continue. 

159
00:09:26,800 --> 00:09:29,200
Well, the nice thing about this 
is it's, you know, it's Igniter,

160
00:09:29,280 --> 00:09:30,800
right? 
And so you don't have to think 

161
00:09:30,800 --> 00:09:33,880
about it like it's, I mean, you 
probably saw it from the, from 

162
00:09:33,880 --> 00:09:36,720
the part of the demo that you, 
you were able to watch. 

163
00:09:37,360 --> 00:09:40,840
It takes less than 5 minutes. 
It's one of the first things you

164
00:09:40,840 --> 00:09:44,520
do when you start up an app. 
You just say, hey, you run an 

165
00:09:44,520 --> 00:09:47,280
igniter script that does all the
wiring for you. 

166
00:09:47,280 --> 00:09:50,480
It starts ingesting your logs, 
it starts ingesting your traces 

167
00:09:51,200 --> 00:09:53,920
and it starts ingesting your 
metrics and it does it all for 

168
00:09:53,920 --> 00:09:55,000
you. 
It's, it's pretty great. 

169
00:09:55,240 --> 00:09:58,040
Igniter is just an amazing, 
amazing addition to the. 

170
00:09:58,080 --> 00:10:02,320
Ecosystem igniter is so cool. 
It's so cool that and usage 

171
00:10:02,320 --> 00:10:06,880
rules like those are like, I 
think my two favorite, like just

172
00:10:07,840 --> 00:10:13,720
they're like such simple 
libraries that are so powerful. 

173
00:10:15,680 --> 00:10:18,600
And the nice thing about this 
too also is that, you know, 

174
00:10:19,080 --> 00:10:21,640
because I wanted it to be 
embedded, I've actually made the

175
00:10:21,640 --> 00:10:25,720
performance so good that it 
actually runs really well as a 

176
00:10:25,720 --> 00:10:28,320
stand alone. 
You could actually replace your 

177
00:10:28,320 --> 00:10:33,080
entire observability stack from 
say, Victoria metrics, logs, 

178
00:10:33,080 --> 00:10:35,840
traces, right? 
You could just replace that with

179
00:10:35,840 --> 00:10:38,680
a single server and it would 
ingest, you know, millions of 

180
00:10:38,680 --> 00:10:41,720
entries per second. 
I was watching the the Tiger, 

181
00:10:41,800 --> 00:10:46,240
Tiger Beetle demo this week that
that's been making its rounds 

182
00:10:46,240 --> 00:10:48,840
around the Internet and they, 
they talk about a trillion 

183
00:10:48,840 --> 00:10:51,440
transactions and all this stuff.
And you look at it and it's only

184
00:10:51,440 --> 00:10:53,680
doing a half a million 
transactions a second, which 

185
00:10:53,960 --> 00:10:57,560
sounds like a lot, but it's, 
it's really not compared to what

186
00:10:57,560 --> 00:11:01,240
metrics ingestion is at today, 
right? 

187
00:11:02,400 --> 00:11:05,120
Yeah. 
But we get around 3,000,000 

188
00:11:05,120 --> 00:11:08,320
metric ingestion like metrics 
per second. 

189
00:11:09,400 --> 00:11:13,520
And we can do a million logs per
second, obviously not at the 

190
00:11:13,520 --> 00:11:17,000
same time, but each individual 
server is capable of these, 

191
00:11:17,000 --> 00:11:22,520
these and I think traces were up
to like 800,000 per second or 

192
00:11:22,520 --> 00:11:24,120
something like that. 
It's it's a big number. 

193
00:11:24,400 --> 00:11:26,040
That is cool. 
Yeah. 

194
00:11:26,480 --> 00:11:29,680
We're like, I'm guessing like 
the compression side of things 

195
00:11:29,680 --> 00:11:32,240
is like the huge bottleneck, 
right? 

196
00:11:32,240 --> 00:11:34,640
Or no. 
It's actually not. 

197
00:11:35,840 --> 00:11:41,080
The compression has advanced in 
so many ways the last few years.

198
00:11:41,800 --> 00:11:47,400
I've obviously been following 
telemetry for a long time, 30 

199
00:11:47,400 --> 00:11:52,840
years now, but but really during
COVID, I think every smart 

200
00:11:52,840 --> 00:11:56,120
person smarter than me, 
certainly every smart person 

201
00:11:56,120 --> 00:11:58,800
just sat at home and just 
figured out great ways to 

202
00:11:58,800 --> 00:12:02,160
compress things and use CPU 
offloading. 

203
00:12:02,880 --> 00:12:07,320
You know, there's, there's kind 
of like MMX extensions, right, 

204
00:12:07,320 --> 00:12:14,200
For CPUs, Neon for, for ARM 64. 
There's, there's basically CPUs 

205
00:12:14,200 --> 00:12:17,400
do some acceleration there. 
And so you can actually do it 

206
00:12:17,400 --> 00:12:21,000
even down to the pie, you know, 
even Raspberry Pi's will do some

207
00:12:21,000 --> 00:12:25,160
hardware acceleration for that. 
So it's, it's become, it's 

208
00:12:25,160 --> 00:12:31,160
become, you know, really nice. 
We'll just say, and it, there 

209
00:12:31,160 --> 00:12:34,200
are lots of options, which is 
why this has taken me two years 

210
00:12:34,560 --> 00:12:37,640
is because there's so many 
compression options for logs and

211
00:12:37,640 --> 00:12:40,080
traces. 
And, and I actually split, 

212
00:12:40,560 --> 00:12:43,920
there's something called PCO, 
which is the compression that we

213
00:12:43,920 --> 00:12:45,720
use for metrics, which is 
fantastic. 

214
00:12:46,240 --> 00:12:49,280
Usually get over 12:50. 
It's, it's, I think around 14 to

215
00:12:49,280 --> 00:12:52,560
one right now on my, my test 
bed. 

216
00:12:53,480 --> 00:12:57,000
And then, you know, getting 
12:50 nine to one, something 

217
00:12:57,000 --> 00:12:59,000
like that for logs and traces is
really good. 

218
00:12:59,000 --> 00:13:03,160
And we use Open ZL for that, 
which is a columnar compression 

219
00:13:03,160 --> 00:13:05,880
that's actually format aware, 
which is really cool. 

220
00:13:08,000 --> 00:13:10,880
That is, that's cool. 
So what's the, what's the 

221
00:13:10,880 --> 00:13:14,560
biggest bottleneck then for like
your ingestion rate? 

222
00:13:14,600 --> 00:13:17,920
Like what are you competing 
against to get that number up? 

223
00:13:18,920 --> 00:13:21,920
So it's, again, it's been a 
journey. 

224
00:13:21,920 --> 00:13:24,560
So I've been pulling off pieces 
and doing neffs right? 

225
00:13:24,560 --> 00:13:28,760
And I would do some C Neffs. 
I actually built a little web 

226
00:13:28,760 --> 00:13:31,240
server. 
It's called Rocket. 

227
00:13:32,040 --> 00:13:34,360
That was a little faster, 
actually quite a bit faster, 30 

228
00:13:34,360 --> 00:13:39,280
times faster from a latency 
perspective than Bandit. 

229
00:13:39,800 --> 00:13:43,640
And so I've just been pulling 
pieces apart. 

230
00:13:44,040 --> 00:13:46,720
You know, I find a bottleneck 
and I'll, I'll write a nif for 

231
00:13:46,720 --> 00:13:50,080
it. 
And the problem that I ran into 

232
00:13:50,080 --> 00:13:53,560
was I kept pulling pieces, 
excuse me, pulling pieces off 

233
00:13:54,000 --> 00:13:56,600
and writing Nifs for them. 
But then when the Nifs would 

234
00:13:56,600 --> 00:13:59,000
have to talk to each other, they
would have to go through the 

235
00:13:59,000 --> 00:14:01,080
Elixir layer and there's a 
translation layer there. 

236
00:14:01,760 --> 00:14:05,440
And so that became slow. 
And So what I did was I, I 

237
00:14:05,440 --> 00:14:08,280
pulled enough pieces off. 
Now I can just meld the pieces 

238
00:14:08,280 --> 00:14:10,120
together and just make one NIF, 
right? 

239
00:14:10,120 --> 00:14:13,400
That does most of the work. 
And then there's just one that 

240
00:14:13,400 --> 00:14:16,480
that one translation layer that 
goes basically from Elixir down 

241
00:14:16,480 --> 00:14:23,280
to to Rust on the metric side. 
And so, yeah, the bottleneck is,

242
00:14:23,280 --> 00:14:25,960
is sometimes the translation 
layer between the Elixir and the

243
00:14:25,960 --> 00:14:29,080
NIF. 
And that's really just to tell 

244
00:14:29,080 --> 00:14:31,800
you how, how much we're, we're 
slicing the time here. 

245
00:14:32,360 --> 00:14:34,320
That's where our our bottleneck 
is. 

246
00:14:34,680 --> 00:14:37,840
OK. 
Like I'm curious then would the 

247
00:14:37,840 --> 00:14:41,320
next step be it? 
Like is Elixir your bottleneck? 

248
00:14:42,920 --> 00:14:44,480
There's no. 
No. 

249
00:14:44,680 --> 00:14:47,440
No, orchestration in Elixir is 
just so good. 

250
00:14:47,440 --> 00:14:51,760
I would never consider, no, 
yeah, the ability to do 

251
00:14:51,760 --> 00:14:58,320
clustering, you know, just the 
the niceness of I mean, we're 

252
00:14:58,320 --> 00:15:00,840
all here because Elixir is 
fantastic for orchestration, 

253
00:15:00,840 --> 00:15:05,000
right? 
And so no, I, I think, I think 

254
00:15:05,120 --> 00:15:08,440
it's this, this is definitely an
Elixir application from from 

255
00:15:08,440 --> 00:15:11,000
beginning to end. 
I've gotten the performance that

256
00:15:11,000 --> 00:15:14,720
I feel like I need out of it. 
I feel like I could do Facebook 

257
00:15:14,720 --> 00:15:16,480
scale. 
I could do all the things that 

258
00:15:16,520 --> 00:15:19,800
that you need to do with the 
number of nodes that they have 

259
00:15:19,800 --> 00:15:22,680
doing ingestion anyway in AWS 
scale. 

260
00:15:22,680 --> 00:15:27,400
So I think, I think I think it's
fine and it works great, you 

261
00:15:27,400 --> 00:15:30,080
know, and it's it's it's really 
nice to have it in an Elixir 

262
00:15:30,080 --> 00:15:33,160
because you can, you can do 
things like this embedded side, 

263
00:15:33,160 --> 00:15:36,800
right where you just drop it 
into your app and it ingests and

264
00:15:36,920 --> 00:15:39,080
gives you all of the 
intelligence that you need for 

265
00:15:39,080 --> 00:15:43,320
your app traces and everything 
for nothing almost right. 

266
00:15:43,360 --> 00:15:44,880
I mean, it doesn't really cost 
you anything. 

267
00:15:44,880 --> 00:15:47,760
It's less than 5% probably load 
on top of your app so. 

268
00:15:48,360 --> 00:15:49,360
That's awesome. 
Cool. 

269
00:15:50,120 --> 00:15:55,200
Now how does Timeless work its 
way into your hospitality 

270
00:15:55,200 --> 00:15:56,200
business? 
Does it? 

271
00:15:56,640 --> 00:15:58,280
Did you build your reason? 
Yeah. 

272
00:15:58,640 --> 00:16:01,080
We, we have about 10,000 cable 
modems. 

273
00:16:01,720 --> 00:16:06,240
We have about 1200 or we'll have
1200 lines very shortly of 

274
00:16:06,240 --> 00:16:09,280
telephony. 
And so we have metrics that we 

275
00:16:09,280 --> 00:16:12,080
collect on all those things and 
we, we want to observe ability 

276
00:16:12,080 --> 00:16:14,600
for those. 
And that's what it was, that was

277
00:16:14,600 --> 00:16:16,160
what it was originally written 
for. 

278
00:16:17,240 --> 00:16:20,400
And so I thought I would share 
it with the world and see if, 

279
00:16:20,600 --> 00:16:22,600
you know, see if other people 
find value. 

280
00:16:22,880 --> 00:16:24,640
But we'll be ingesting quite a 
few metrics. 

281
00:16:24,640 --> 00:16:28,000
We'll have about 200,000 series,
maybe 300,000 series that we'll 

282
00:16:28,000 --> 00:16:31,000
put into it. 
And series is just a a metric 

283
00:16:31,000 --> 00:16:32,760
over time, right? 
Yeah, yeah. 

284
00:16:33,560 --> 00:16:41,720
Cool to go into the hospitality 
network business like paint, 

285
00:16:41,720 --> 00:16:44,880
paint me a picture. 
What what does it look like? 

286
00:16:44,880 --> 00:16:46,920
Like What does the actual 
network look like? 

287
00:16:46,920 --> 00:16:50,320
Like does, I know, I don't know 
if I made this up or if we 

288
00:16:50,320 --> 00:16:53,560
talked about it, but like it 
like some of your clients would 

289
00:16:53,560 --> 00:16:59,760
like include like RV parks, 
nursing home, nursing homes, 

290
00:16:59,760 --> 00:17:02,560
hotels, like all of that. 
OK. 

291
00:17:02,560 --> 00:17:05,520
We call them multi dwelling 
units, Mdus. 

292
00:17:06,119 --> 00:17:09,520
And so it's just a number of 
whatever, right, could be 

293
00:17:09,520 --> 00:17:12,280
apartments, it could be nursing 
home, it could be anything. 

294
00:17:12,920 --> 00:17:16,079
And it's like it just a small 
cable company, you know they 

295
00:17:16,079 --> 00:17:19,640
have a router, a cable router 
that we directly import from 

296
00:17:19,640 --> 00:17:21,240
China. 
We brand it, we have our own 

297
00:17:21,240 --> 00:17:25,960
brand and we do all the 
provisioning for that and the 

298
00:17:26,000 --> 00:17:30,560
observability, right. 
So we help integrators who will 

299
00:17:30,560 --> 00:17:33,840
go out and do the installs. 
We help them with all of the 

300
00:17:33,840 --> 00:17:35,880
things that are hard for them to
do right. 

301
00:17:36,440 --> 00:17:39,280
It's easy for them to travel and
do and understand networks well 

302
00:17:39,280 --> 00:17:41,880
enough to set up the networks. 
But we take over and, and make 

303
00:17:41,880 --> 00:17:45,080
sure that it's healthy and stays
healthy and, and stuff like that

304
00:17:45,120 --> 00:17:50,240
and sell them the hardware. 
So it, it is, it's, it's, it's 

305
00:17:50,240 --> 00:17:52,960
basically just tiny little cable
companies that we build, you 

306
00:17:52,960 --> 00:17:56,200
know, that are, you know, 2 or 
300 modems or something like 

307
00:17:56,200 --> 00:17:57,760
that. 
OK, that's cool. 

308
00:17:58,000 --> 00:18:00,680
So like an apartment complex, 
for instance, you would have 

309
00:18:00,680 --> 00:18:05,160
like do you have like a main 
entry point closet and then kind

310
00:18:05,160 --> 00:18:09,520
of splits off to every building 
and then every apartment and. 

311
00:18:10,520 --> 00:18:11,840
Yeah. 
And it will use like a lot of 

312
00:18:11,840 --> 00:18:14,760
times those apartment complexes 
already have cable run and so 

313
00:18:14,760 --> 00:18:17,600
it'll use their existing. 
They won't have to run fiber, 

314
00:18:17,600 --> 00:18:19,400
they won't have to run the 
Ethernet or any of that stuff. 

315
00:18:19,400 --> 00:18:21,040
It just operates over cable 
modems. 

316
00:18:21,480 --> 00:18:23,120
OK. 
And then it's your own hardware 

317
00:18:23,120 --> 00:18:24,880
or your branded hardware at 
least? 

318
00:18:24,960 --> 00:18:29,560
Yeah, that's cool. 
And then are you embedding 

319
00:18:31,960 --> 00:18:35,640
Timeless into your hardware or 
how's that working? 

320
00:18:36,360 --> 00:18:39,880
Now we pull centrally and so 
it's going to be a, there's a, 

321
00:18:39,880 --> 00:18:42,720
there's a version of Timeless 
that is that embeds into your 

322
00:18:42,720 --> 00:18:45,520
Phoenix application. 
And it's designed to be really 

323
00:18:45,520 --> 00:18:49,120
small, take very, very little 
memory, very little CPU. 

324
00:18:49,360 --> 00:18:53,280
There's also the server version,
right, which you know, will 

325
00:18:53,320 --> 00:18:56,120
consume as much memory as you 
give it, just like any other 

326
00:18:56,120 --> 00:18:59,600
database. 
And we'll return results quite a

327
00:18:59,600 --> 00:19:01,400
bit faster. 
And that's what we're we have a 

328
00:19:01,400 --> 00:19:02,480
central. 
Yeah. 

329
00:19:02,480 --> 00:19:04,000
OK, OK. 
I see. 

330
00:19:04,320 --> 00:19:09,520
That's cool. 
Now mentioning like you've had 

331
00:19:09,520 --> 00:19:14,200
100% annual growth. 
Oh yeah, that's wild. 

332
00:19:15,360 --> 00:19:17,280
That's, that's not necessarily 
me. 

333
00:19:17,280 --> 00:19:20,240
I'm, I'm late to the game. 
I just joined the company 

334
00:19:20,240 --> 00:19:24,240
recently. 
I'm the CTO, but the CEO has 

335
00:19:24,240 --> 00:19:27,560
been growing the company for 
about 7 years and he's, he's 

336
00:19:27,560 --> 00:19:30,800
gotten to a point now where he's
about 100% year over year. 

337
00:19:31,120 --> 00:19:35,440
And you know, people were it 
it's really advantageous for 

338
00:19:35,440 --> 00:19:38,400
them because again, it uses 
their existing cable plant, 

339
00:19:38,400 --> 00:19:40,640
right. 
So if you have coax run to 

340
00:19:40,760 --> 00:19:44,040
whatever it is you're trying to 
get to in a nursing home or in 

341
00:19:44,040 --> 00:19:48,760
in any multi dwelling unit, you 
know, you don't have to run 

342
00:19:48,760 --> 00:19:52,120
fiber Ethernet or so it's it's 
become a popular solution, 

343
00:19:52,320 --> 00:19:54,360
especially in the hospitality 
side, right? 

344
00:19:54,360 --> 00:20:00,080
RV parks, there's 17 thousand 
19,000 in the United States and 

345
00:20:00,360 --> 00:20:03,120
and they a lot of times have 
cable infrastructure as well. 

346
00:20:03,160 --> 00:20:06,520
And so they'll they'll take 
advantage of that and and put up

347
00:20:06,520 --> 00:20:09,520
a little Wi-Fi towers. 
We have a Wi-Fi solution where, 

348
00:20:10,200 --> 00:20:12,840
you know, you can walk around 
with a computer like you walk to

349
00:20:12,840 --> 00:20:15,840
the pool and your computer 
continues to stay on the same 

350
00:20:15,840 --> 00:20:19,120
Wi-Fi connection. 
It's it's pretty cool. 

351
00:20:19,120 --> 00:20:20,880
It's kind of like the Wi-Fi 6 
stuff, but it's. 

352
00:20:21,160 --> 00:20:23,200
OK, now it's like the mesh 
system stuff. 

353
00:20:23,920 --> 00:20:26,160
Exactly. 
It's, it's very much a just, it 

354
00:20:26,160 --> 00:20:32,240
works just like a mesh, but but 
yeah, so you can, you can, you 

355
00:20:32,280 --> 00:20:35,240
know, take your computer around 
and it continues to work and it,

356
00:20:35,240 --> 00:20:39,120
it works through pedestals. 
So they have little Wi-Fi 

357
00:20:39,120 --> 00:20:42,760
pedestals and basically 
everybody on the park is on the 

358
00:20:42,760 --> 00:20:44,240
same. 
Nice. 

359
00:20:44,680 --> 00:20:48,720
There is a period in my life 
where I wanted to buy an RV park

360
00:20:48,720 --> 00:20:52,120
as an investment property. 
I don't remember what happened 

361
00:20:52,120 --> 00:20:55,600
but it it it disappeared from my
my dream. 

362
00:20:55,600 --> 00:21:00,080
But it's not a bad route to go. 
Oh, it's very popular, yeah. 

363
00:21:00,320 --> 00:21:02,760
That's why I think that's one of
the reasons why we have 17 to 

364
00:21:02,760 --> 00:21:05,400
19,000. 
It's pretty easy to run you, you

365
00:21:05,400 --> 00:21:07,400
can get people to run it pretty 
easily for you. 

366
00:21:07,400 --> 00:21:09,880
And it's, it's a great 
investment for sure. 

367
00:21:10,240 --> 00:21:15,400
Yeah, that makes sense. 
I lived in an RV for 2-2 or 

368
00:21:15,400 --> 00:21:19,120
three years, but I mostly 
boondocked because I don't. 

369
00:21:19,160 --> 00:21:23,600
I didn't want to actually pay to
stay at parks and do that. 

370
00:21:23,640 --> 00:21:27,560
I was too cheap. 
I was boondocking before remote 

371
00:21:27,560 --> 00:21:32,360
work was cool after COVID. 
So if you did stay apart at a 

372
00:21:32,360 --> 00:21:34,720
park, how was your Internet? 
It probably wasn't very good, 

373
00:21:34,720 --> 00:21:38,520
right? 
I usually just with hotspot off 

374
00:21:38,520 --> 00:21:40,080
my mobile. 
Would you? 

375
00:21:40,480 --> 00:21:43,360
Yeah, yeah, we're trying to 
improve that situation. 

376
00:21:43,360 --> 00:21:47,000
So imagine if you had, you know,
a homestyle networking while you

377
00:21:47,000 --> 00:21:49,680
were out RV. 
You know, in an RV you could 

378
00:21:49,680 --> 00:21:51,320
stream video and do whatever, 
right? 

379
00:21:51,400 --> 00:21:54,080
I wish Starlink was a thing when
I was doing it. 

380
00:21:54,080 --> 00:21:56,280
That would have been a game 
changer for me because I had 

381
00:21:56,280 --> 00:22:00,160
like the crappy, I forget what 
they were called, like the the 

382
00:22:00,160 --> 00:22:04,400
cell phone boosters that kind of
work but they don't really work.

383
00:22:06,040 --> 00:22:08,840
I would always like have because
I'd be in places where I'd have 

384
00:22:08,840 --> 00:22:13,800
like 1 bar and then sometimes I 
can get my booster to work but I

385
00:22:13,800 --> 00:22:19,720
feel like it it hallucinated and
just made my phone think I had a

386
00:22:19,720 --> 00:22:21,280
better signal than it actually 
did. 

387
00:22:22,000 --> 00:22:24,200
Yeah, you don't actually pass 
any traffic at all. 

388
00:22:24,400 --> 00:22:27,640
Yeah, yeah, exactly. 
So that was the struggle. 

389
00:22:28,440 --> 00:22:32,240
But for the most part, I worked 
as a remote developer without 

390
00:22:32,240 --> 00:22:33,120
issue. 
Ever. 

391
00:22:33,160 --> 00:22:36,760
From time to time I would get on
public Wi-Fi if I really needed 

392
00:22:36,760 --> 00:22:42,120
to move a lot of data, but it 
worked for being I was an iOS 

393
00:22:42,120 --> 00:22:43,680
developer at the time and it 
worked. 

394
00:22:43,760 --> 00:22:48,440
So that was cool. 
How, how do you how do you think

395
00:22:48,440 --> 00:22:51,760
like Starlink and like satellite
Internet will affect your 

396
00:22:51,760 --> 00:22:55,280
business? 
I think we may use Starlink for 

397
00:22:55,280 --> 00:22:57,840
backhaul a lot of times because,
you know, the parks have to have

398
00:22:57,840 --> 00:22:59,960
a backhaul connection of some 
kind. 

399
00:23:00,000 --> 00:23:02,360
A lot of times we do fiber for 
that. 

400
00:23:02,360 --> 00:23:06,680
But but yeah, I mean, I think 
Starlink for backhaul and, and, 

401
00:23:06,720 --> 00:23:10,080
you know, there may come a time 
where everyone has Starlink and 

402
00:23:10,080 --> 00:23:13,720
it, you know, our, our services 
aren't as valuable. 

403
00:23:13,720 --> 00:23:16,600
But I don't see that in the near
future. 

404
00:23:16,720 --> 00:23:19,520
It'll, it'll be a while before 
it sort of takes over. 

405
00:23:19,560 --> 00:23:21,320
Yeah. 
I think we got some runway 

406
00:23:21,320 --> 00:23:23,880
there. 
I think like we'll probably see 

407
00:23:23,880 --> 00:23:27,000
it transforming like our 
cellular networks and cell 

408
00:23:27,000 --> 00:23:32,240
phones 1st and then when those 
stop being a thing and it's all 

409
00:23:32,240 --> 00:23:34,480
satellites, things will start 
shifting. 

410
00:23:35,440 --> 00:23:38,040
Yeah, I think probably when, 
when you can, if you have 

411
00:23:38,040 --> 00:23:41,920
Starlink in your cell phone and 
you can tether, that's going to 

412
00:23:41,920 --> 00:23:45,200
change the game for us. 
I think probably our services 

413
00:23:45,200 --> 00:23:46,840
will be less needed at that 
point. 

414
00:23:48,160 --> 00:23:50,440
Yeah. 
Well, you, you still might, you 

415
00:23:50,440 --> 00:23:54,760
know, I feel like people will 
still need like network health 

416
00:23:54,840 --> 00:23:58,680
and other services. 
You just will have to like shift

417
00:23:58,680 --> 00:24:00,040
a little. 
Yeah. 

418
00:24:00,440 --> 00:24:03,840
Yeah, exactly. 
Like I think like that's like 

419
00:24:03,880 --> 00:24:07,320
the whole fear mongering but 
behind LLMS right now, right? 

420
00:24:07,320 --> 00:24:09,560
Everyone thinks our jobs are 
going to go to the wayside. 

421
00:24:09,840 --> 00:24:12,400
And I just like we were talking 
a little bit before we started, 

422
00:24:12,400 --> 00:24:15,880
I think we're just going to not 
be writing code anymore. 

423
00:24:15,880 --> 00:24:20,360
We're going to be orchestrating 
code workers and agents and 

424
00:24:20,800 --> 00:24:23,280
reading it all. 
Yeah. 

425
00:24:23,280 --> 00:24:24,760
And I've mentioned this to you 
before. 

426
00:24:24,760 --> 00:24:27,120
You know, one of the nice things
about being into physical 

427
00:24:27,120 --> 00:24:29,840
networking is you don't have to 
worry about AI taking that 

428
00:24:29,840 --> 00:24:31,640
right? 
Yeah, that's true. 

429
00:24:31,760 --> 00:24:34,360
Yeah, until there's robots that 
can do it all. 

430
00:24:35,240 --> 00:24:37,320
Robots in Starlink, right? 
They're going to, they're going 

431
00:24:37,320 --> 00:24:41,720
to take all of our jobs. 
I mean, I would love, well, I 

432
00:24:41,720 --> 00:24:44,480
actually I don't know if I would
love this because I don't like 

433
00:24:44,480 --> 00:24:48,840
all the. 
What do you call the the Amazon 

434
00:24:48,840 --> 00:24:51,880
boxes that you the Siri boxes 
that you talk to? 

435
00:24:51,880 --> 00:24:53,560
I forget. 
I don't know why I'm blanking on

436
00:24:53,560 --> 00:24:55,960
it right now. 
Oh, yeah, Like the home. 

437
00:24:56,560 --> 00:24:57,960
I don't. 
I don't know what they're 

438
00:24:57,960 --> 00:25:00,120
called. 
I don't know what you're talking

439
00:25:00,120 --> 00:25:00,760
about. 
Yeah, but I. 

440
00:25:00,760 --> 00:25:03,160
Refuse to have those in my house
and I was just about to say I 

441
00:25:03,160 --> 00:25:06,400
can't wait to have like a robot 
servant that will clean my house

442
00:25:06,400 --> 00:25:08,680
and cook for me and then I'm 
like wait what? 

443
00:25:08,680 --> 00:25:11,280
I actually want one in my house.
Would you? 

444
00:25:11,280 --> 00:25:12,400
Would you talk to it? 
Right. 

445
00:25:12,400 --> 00:25:13,880
Would you trust it? 
Yeah. 

446
00:25:14,080 --> 00:25:15,160
I get it. 
I get it. 

447
00:25:16,760 --> 00:25:21,320
Yeah, so I don't know if I'd use
it, but part of me thinks it 

448
00:25:21,320 --> 00:25:26,840
would be nice to have a little 
robot servant until AGI and IT 

449
00:25:26,840 --> 00:25:29,880
Terminator it turns into a 
Terminator situation. 

450
00:25:30,440 --> 00:25:33,240
Yeah, yeah. 
And I think you know. 

451
00:25:33,360 --> 00:25:36,720
Jim Clark said Echo dot that 
Yeah, but what it what are those

452
00:25:36,760 --> 00:25:38,960
systems called? 
Like they're, they got to be 

453
00:25:38,960 --> 00:25:40,840
called something, right? 
Because doesn't Apple have one 

454
00:25:40,840 --> 00:25:42,400
too? 
Like a home assistant? 

455
00:25:42,520 --> 00:25:44,920
Home assistant maybe? 
Yeah, maybe. 

456
00:25:44,920 --> 00:25:46,880
I don't know. 
Freaks me out though. 

457
00:25:47,240 --> 00:25:51,040
Won't do it. 
It's always listening, yeah. 

458
00:25:51,240 --> 00:25:53,800
Listens to everything you say. 
I mean, I guess so. 

459
00:25:53,840 --> 00:25:58,080
So does so does this but. 
It really, it really does, yeah.

460
00:25:59,840 --> 00:26:03,040
But sometimes I take it for my 
advantage, like I'll talk about 

461
00:26:03,040 --> 00:26:06,160
something because I want the ad 
to show up on like, Instagram 

462
00:26:06,160 --> 00:26:08,960
because I can't find it again. 
And then it will. 

463
00:26:08,960 --> 00:26:10,120
It shows up. 
Oh yeah. 

464
00:26:10,880 --> 00:26:13,160
Yeah, it's a little scary. 
Actually, it's a lot scary. 

465
00:26:13,480 --> 00:26:15,200
It's, it's actually really 
scary. 

466
00:26:16,520 --> 00:26:20,000
I, I tend to be like a little 
more paranoid, I think with my 

467
00:26:20,000 --> 00:26:24,320
security and like privacy, But 
then there's also I don't put in

468
00:26:24,320 --> 00:26:29,400
the effort, like I know the 
effort required to actually have

469
00:26:29,400 --> 00:26:32,520
privacy and I just don't have 
the patience. 

470
00:26:34,240 --> 00:26:35,560
Well, it's everything now, 
right? 

471
00:26:35,560 --> 00:26:37,400
I mean, everything wants to 
listen so. 

472
00:26:37,960 --> 00:26:40,960
It's it's hard. 
Every everything wants our data,

473
00:26:42,640 --> 00:26:49,960
our data. 
Yeah, I Oh yeah. 

474
00:26:50,120 --> 00:26:53,320
I wanted to ask. 
So you you mentioned that you 

475
00:26:53,320 --> 00:26:56,960
are probably one of the like the
biggest noobs in the Elixir 

476
00:26:56,960 --> 00:26:59,520
space, and you about two. 
Years. 

477
00:26:59,720 --> 00:27:01,520
You've been writing an Elixir 
for two years. 

478
00:27:02,720 --> 00:27:04,600
What brought Elixir on your 
radar? 

479
00:27:06,240 --> 00:27:10,760
So originally, so I used to be 
the host of the Atlanta Ruby 

480
00:27:10,760 --> 00:27:14,440
Users group before Rails, right?
So I know I'm I'm one of those 

481
00:27:14,440 --> 00:27:20,200
Rails people who came to Elixir 
just like most, but but yeah, so

482
00:27:20,200 --> 00:27:23,200
I was the the host of the 
Atlanta Ruby Users group before 

483
00:27:23,200 --> 00:27:28,560
Rails and Obi Obi Fernandez, I 
don't know how to say his name, 

484
00:27:28,560 --> 00:27:31,680
but the Rails way guy came up 
from Jacksonville and and 

485
00:27:31,680 --> 00:27:34,680
presented Rails one point O so 
I've been around the Ruby 

486
00:27:34,680 --> 00:27:40,400
community for a long time and 
and I I used J Ruby to do a lot 

487
00:27:40,400 --> 00:27:44,000
of concurrency stuff. 
And I actually wrote basically 

488
00:27:44,000 --> 00:27:48,440
Gen. stage for J Ruby back in 
the day and and you know, came 

489
00:27:48,440 --> 00:27:52,160
up with the idea of message 
passing instead of simafores and

490
00:27:52,160 --> 00:27:56,520
all that stuff, right? 
Which seems like lots of people 

491
00:27:56,520 --> 00:27:58,640
have independently come to that 
conclusion, right? 

492
00:27:58,640 --> 00:28:00,800
Message passing is so much 
better than all the the. 

493
00:28:00,880 --> 00:28:03,600
Other as of recently too. 
They're just trying to reinvent 

494
00:28:03,960 --> 00:28:04,960
reinvent the. 
This was. 

495
00:28:05,800 --> 00:28:09,640
This was 18 years ago, so but 
yeah, so I wrote Gen. stage 18 

496
00:28:09,640 --> 00:28:15,240
years ago and relied completely 
on message passing with J Ruby 

497
00:28:15,240 --> 00:28:18,240
and and when I saw Elixir, I was
like, you know what? 

498
00:28:18,600 --> 00:28:21,640
They solved all the problems 
that I felt like I solved with. 

499
00:28:21,920 --> 00:28:23,680
It's a library called Common 
Thread. 

500
00:28:24,640 --> 00:28:27,160
And they solved all the problems
that I had back then. 

501
00:28:28,160 --> 00:28:31,640
You know, before I just didn't 
know about it, right? 

502
00:28:31,640 --> 00:28:35,600
Erlang was, Erlang was around, 
Elixir wasn't around, but Erlang

503
00:28:35,600 --> 00:28:40,120
was around and it's just a, it's
a, it's just a treasure. 

504
00:28:40,120 --> 00:28:45,840
I think I really, I really value
the Elixir community and all of 

505
00:28:45,840 --> 00:28:48,560
the technical decisions that 
were made to get us to this 

506
00:28:48,560 --> 00:28:51,520
point. 
It's really a a strong piece of 

507
00:28:51,520 --> 00:28:53,160
software. 
It is. 

508
00:28:53,280 --> 00:28:56,520
I'm thankful for really smart 
people like, you know, the 

509
00:28:56,520 --> 00:29:01,080
Josie's and the Christmas chords
and all that. 

510
00:29:02,080 --> 00:29:05,280
I'm, I'm only a user of great 
technology. 

511
00:29:05,280 --> 00:29:08,600
I don't think I'm, I'll ever be 
on the leading edge of something

512
00:29:08,600 --> 00:29:10,960
great like that. 
But I appreciate it. 

513
00:29:11,000 --> 00:29:13,920
And I get to talk to him because
I decided to start a podcast. 

514
00:29:15,080 --> 00:29:16,080
Which is great. 
Yeah. 

515
00:29:17,160 --> 00:29:19,920
Yeah, It's extremely helpful. 
I've been watching the podcast, 

516
00:29:19,920 --> 00:29:22,120
you know it again, just two 
years, right? 

517
00:29:22,120 --> 00:29:24,800
And so I've just really immersed
myself in the community, watched

518
00:29:24,800 --> 00:29:27,640
all the podcasts and really 
appreciated them. 

519
00:29:27,640 --> 00:29:31,880
So yeah, I mean, I, I really 
appreciate the work that you put

520
00:29:31,880 --> 00:29:35,080
in to to mentor everyone. 
I always love to hear that 

521
00:29:35,800 --> 00:29:39,960
because yeah, I don't know, It's
I talk about this all the time, 

522
00:29:39,960 --> 00:29:42,480
like that imposter syndrome, it 
never goes away. 

523
00:29:42,480 --> 00:29:46,000
You just like, keep faking it, 
keep faking it, and then 

524
00:29:46,000 --> 00:29:52,760
eventually it'll work out right?
Yeah, it's so I, I value Elixir 

525
00:29:52,760 --> 00:29:55,720
so much that I tell people, you 
know, I've learned about 17 

526
00:29:55,720 --> 00:29:58,320
languages over my lifetime. 
I've been doing this since I was

527
00:29:58,320 --> 00:30:01,120
8. 
Elixir is my 17th and final 

528
00:30:01,120 --> 00:30:03,280
language. 
I think I'm, I think I'm done. 

529
00:30:03,520 --> 00:30:05,920
It's really great. 
I wonder how many I've learned. 

530
00:30:06,680 --> 00:30:13,160
Let's see, like Java, C#, a 
little bit of C++, JavaScript, 

531
00:30:13,320 --> 00:30:16,760
Does TypeScript count or are 
JavaScript and TypeScript the 

532
00:30:16,760 --> 00:30:24,040
same thing? 
And then Kotlin, Swift, 

533
00:30:24,240 --> 00:30:31,000
Objective C, which I hated 
Objective C, it's about like my 

534
00:30:31,000 --> 00:30:33,240
ninth language. 
And I also want it to be my 

535
00:30:33,240 --> 00:30:37,520
last. 
And then I haven't really take 

536
00:30:37,520 --> 00:30:40,360
taken the time I kind of want 
to, but I want to learn like 

537
00:30:40,360 --> 00:30:44,440
more Erlang just because like 
there's so much more under the 

538
00:30:44,440 --> 00:30:48,840
hood that we have access to and 
we don't really take advantage 

539
00:30:48,840 --> 00:30:52,400
of it. 
Do you find yourself using it 

540
00:30:52,400 --> 00:30:57,680
more with like the telemetry and
all that not? 

541
00:30:58,160 --> 00:31:01,280
So I have, I have an interesting
Erlang story. 

542
00:31:01,520 --> 00:31:05,880
So I went, one of the libraries 
that I've published is SNMP Kit,

543
00:31:06,280 --> 00:31:12,960
which is, is a SNMP library, 
which simple network management 

544
00:31:12,960 --> 00:31:15,520
protocol. 
If, if you're not familiar, it's

545
00:31:16,160 --> 00:31:20,640
basically ancient at this point.
But I, I felt like it was, you 

546
00:31:20,640 --> 00:31:24,120
know, sort of a miss in the 
community and I, I had some 

547
00:31:24,120 --> 00:31:27,600
ideas around what I, I wanted to
publish for a SNMP library. 

548
00:31:28,120 --> 00:31:31,920
And so I went to the Erlang code
and went through, you know, the,

549
00:31:32,440 --> 00:31:35,040
there's some SNMP built into 
Erlang. 

550
00:31:35,560 --> 00:31:38,760
And I was like, you know what, 
all it's really doing is just 

551
00:31:38,760 --> 00:31:43,720
compiling down to, you know, 
Erlang beam machine code, right?

552
00:31:44,120 --> 00:31:46,480
And so if I just write this in 
Elixir, it'll be easier to 

553
00:31:46,480 --> 00:31:49,280
maintain. 
And so I, I basically just 

554
00:31:49,280 --> 00:31:54,000
abandoned the idea of writing 
wrappers around the Erlang and 

555
00:31:54,000 --> 00:31:57,600
just wrote it in the elixir. 
And I felt like I got at equal 

556
00:31:57,600 --> 00:32:01,480
or better performance. 
And so it was, it was an 

557
00:32:01,480 --> 00:32:05,440
interesting experiment, you 
know, reading that much Erlang 

558
00:32:05,440 --> 00:32:08,920
code. 
But, but yeah, I even have a mid

559
00:32:08,920 --> 00:32:12,600
compiler and, and all of the, 
all of the interesting side 

560
00:32:12,600 --> 00:32:15,960
effects of, of, of having to do 
SNMP. 

561
00:32:15,960 --> 00:32:20,040
But, but yeah, it was, it was 
interesting reading Erlang. 

562
00:32:20,280 --> 00:32:23,240
I'm, I have always been a syntax
light guy. 

563
00:32:24,160 --> 00:32:28,920
I've learned Rex in college. 
And hold on one second, I'm 

564
00:32:28,960 --> 00:32:33,520
sorry. 
I learned Rex in college and I, 

565
00:32:33,640 --> 00:32:37,680
I really loved it. 
And then, you know, I, I started

566
00:32:37,800 --> 00:32:42,520
working in networking labs and I
did TCL, which was very Syntex 

567
00:32:42,520 --> 00:32:45,640
light as well, which is why I 
gravitated toward Ruby and 

568
00:32:45,640 --> 00:32:48,760
eventually Elixir. 
So I like Erlang. 

569
00:32:48,760 --> 00:32:50,680
I think it's fine. 
I think it's great for those 

570
00:32:50,680 --> 00:32:53,960
people who enjoy it, but I 
really like syntax light 

571
00:32:53,960 --> 00:32:58,840
languages like like Elixir. 
Yeah, I feel like that's like 

572
00:32:59,120 --> 00:33:02,720
probably why Erlang didn't pick 
up in popularity too A. 

573
00:33:03,560 --> 00:33:06,800
Lot of roller plate. 
Yeah, there's a lot of structure

574
00:33:06,960 --> 00:33:08,440
to your code, right? 
Right. 

575
00:33:08,880 --> 00:33:13,320
And then I don't know, Elixir is
just nice. 

576
00:33:13,560 --> 00:33:16,440
It just flows. 
Yeah. 

577
00:33:16,840 --> 00:33:18,200
All right. 
I'm going to put this on hold 

578
00:33:18,200 --> 00:33:20,800
for a second and we'll be right 
back. 

579
00:33:21,560 --> 00:33:25,080
Hey, I'm Kimberly with Elixir 
mentor Jacob recently added 

580
00:33:25,080 --> 00:33:28,440
recruitment services to help 
companies hire pre bedded Elixir

581
00:33:28,440 --> 00:33:31,400
devs. 
I help with company outreach, so

582
00:33:31,400 --> 00:33:34,720
if you're hiring, I may be 
reaching out, or you can connect

583
00:33:34,720 --> 00:33:36,880
with us anytime at 
elixirmentor.com. 

584
00:33:39,960 --> 00:33:43,200
All right, and we're back. 
So what? 

585
00:33:43,200 --> 00:33:46,680
What is the what? 
What's the next thing? 

586
00:33:46,680 --> 00:33:52,800
What are you most excited about 
with like your telecom business 

587
00:33:53,280 --> 00:33:55,240
and using Elixir? 
Like where? 

588
00:33:55,240 --> 00:33:56,840
What do you, what are you 
building next? 

589
00:33:56,840 --> 00:34:00,680
Do you have any plans? 
Well, for me personally, I get 

590
00:34:00,680 --> 00:34:04,320
to, you know, maybe, maybe get 
down to one job, which would be 

591
00:34:04,320 --> 00:34:11,320
awesome instead of two. 
And I think I would like to 

592
00:34:11,320 --> 00:34:15,800
probably get, do additional work
on the server side of Timeless. 

593
00:34:17,040 --> 00:34:20,040
You know, the embedded was what 
I focused on for this because I 

594
00:34:20,040 --> 00:34:22,480
thought it would be the biggest 
benefit to the community. 

595
00:34:23,480 --> 00:34:27,400
And so I think that, you know, 
80% of apps could just drop this

596
00:34:27,400 --> 00:34:30,560
in and have full observability 
without any overhead. 

597
00:34:31,560 --> 00:34:35,320
And I, I feel like that, you 
know, most apps tend to be 

598
00:34:35,320 --> 00:34:39,280
relatively small, right? 
They don't get 200,000 hits a 

599
00:34:39,280 --> 00:34:43,239
second or anything like that. 
And so if I can help those 

600
00:34:43,239 --> 00:34:46,159
people first, that's my 
intention. 

601
00:34:46,560 --> 00:34:49,600
And then, you know, you can also
sample like traces and stuff 

602
00:34:49,600 --> 00:34:52,120
like that when you have the 
bigger apps and send them up to 

603
00:34:52,719 --> 00:34:55,360
a central server. 
And so I'm going to spend some 

604
00:34:55,360 --> 00:34:58,960
more time on the server making 
it, making sure it, it continues

605
00:34:58,960 --> 00:35:00,960
to be performance. 
So there's, there's going to be,

606
00:35:01,800 --> 00:35:03,200
there's a number of components, 
right? 

607
00:35:03,200 --> 00:35:05,680
There's three different 
databases and then there's 

608
00:35:05,680 --> 00:35:08,440
Timeless Phoenix, which is the 
embedded and then there's going 

609
00:35:08,440 --> 00:35:12,360
to be a timeless stack, which is
basically all three databases 

610
00:35:12,360 --> 00:35:15,600
and one as a server component 
that you can push your metrics 

611
00:35:15,600 --> 00:35:19,080
to centrally if you have 
multiple app servers, for 

612
00:35:19,080 --> 00:35:22,120
instance. 
And so that's going to be next 

613
00:35:22,680 --> 00:35:24,240
going to spend a lot more time 
on that. 

614
00:35:24,760 --> 00:35:28,960
And as far as libraries that I'm
interested in writing, I don't 

615
00:35:28,960 --> 00:35:33,640
know, you know, that the telecom
business is taking a lot of 

616
00:35:33,640 --> 00:35:35,760
time. 
And so I don't know if open 

617
00:35:35,760 --> 00:35:38,800
source contributions are going 
to be a thing that I've I 

618
00:35:38,800 --> 00:35:43,760
everything that I've done so far
as far as evaluating compression

619
00:35:43,760 --> 00:35:48,200
libraries, making a faster web 
server, all of those things I've

620
00:35:48,200 --> 00:35:50,080
sort of peeled off and made open
source. 

621
00:35:51,520 --> 00:35:55,440
And so I'll continue to do that.
If I find another compression 

622
00:35:55,440 --> 00:35:59,920
technology that works really 
well for say, logs, then I'll 

623
00:35:59,920 --> 00:36:02,400
wrap that and then make it 
available to the community. 

624
00:36:03,400 --> 00:36:06,360
But but I don't have anything 
right now. 

625
00:36:06,360 --> 00:36:08,640
That's just off the top of my 
head that I would like to do, 

626
00:36:09,960 --> 00:36:12,440
except just maybe continue with 
the timeless thing, right? 

627
00:36:12,440 --> 00:36:16,640
And just make it make it better.
Yeah, When you're when you're 

628
00:36:16,640 --> 00:36:21,040
using timeless like server side,
is there, is it at all possible 

629
00:36:21,040 --> 00:36:25,040
to make like a hosted service 
for it or would it always have 

630
00:36:25,040 --> 00:36:27,120
to be kind of directly tied in 
to? 

631
00:36:27,600 --> 00:36:30,680
It could be, and the nice thing 
about this, remember you're 

632
00:36:30,680 --> 00:36:33,880
compressing at the edge, right? 
And so when you're sending those

633
00:36:33,880 --> 00:36:37,360
logs up, if you're like just as 
a perfect example of nerves, 

634
00:36:37,360 --> 00:36:40,600
right? 
A lot of those nerves links are 

635
00:36:40,600 --> 00:36:44,360
limited in bandwidth, right? 
They're using cellular or 

636
00:36:44,360 --> 00:36:48,040
something like that. 
And so if we can compress at the

637
00:36:48,040 --> 00:36:51,480
edge and then send the packets 
up, it would be great to have a 

638
00:36:51,480 --> 00:36:53,800
cloud service provider that 
understands those already 

639
00:36:53,800 --> 00:36:56,800
compressed packets, right? 
And then can give you a single 

640
00:36:56,800 --> 00:37:00,800
pane of glass for observability 
in a dashboard, Grafana or 

641
00:37:00,840 --> 00:37:02,960
whatever. 
And I actually have an 

642
00:37:02,960 --> 00:37:07,360
interesting dashboard that I've 
built that's based on Scotty 

643
00:37:07,360 --> 00:37:12,520
Toolkit, which is 30 years old. 
But, but it's, it's fun. 

644
00:37:12,760 --> 00:37:14,600
Like it's got a, a timeline at 
the bottom. 

645
00:37:14,600 --> 00:37:18,120
You can put your logs and 
everything on your screen and 

646
00:37:18,120 --> 00:37:20,960
then you can track it over time.
But you can grab, there's a 

647
00:37:20,960 --> 00:37:23,320
timeline at the bottom. 
You can grab it and go back in 

648
00:37:23,320 --> 00:37:25,480
time and it'll show you the logs
from that time. 

649
00:37:25,480 --> 00:37:27,760
It'll show you the traces from 
that time and it'll show you the

650
00:37:27,760 --> 00:37:29,960
graphs from that time. 
So when you're troubleshooting a

651
00:37:29,960 --> 00:37:32,840
problem, like if you had a 
problem say at 11:00 PM last 

652
00:37:32,840 --> 00:37:37,520
night, you could actually just 
go back to that timeline and you

653
00:37:37,520 --> 00:37:39,520
can see all the traces from that
time, the logs. 

654
00:37:39,560 --> 00:37:43,520
From that. 
Yeah, it's an interesting idea. 

655
00:37:43,520 --> 00:37:45,200
I don't think anybody's ever 
tried it before. 

656
00:37:45,720 --> 00:37:48,400
The Polish is not there. 
It doesn't look as good as your 

657
00:37:48,480 --> 00:37:52,480
your typical dashboards do today
because it is based on 30 year 

658
00:37:52,480 --> 00:37:54,560
old technology. 
But now you've got flawed to 

659
00:37:54,560 --> 00:37:57,400
help you clean it up. 
I do, I do. 

660
00:37:57,680 --> 00:38:01,280
Well, and yeah, there's, 
there's, there's some UI cleanup

661
00:38:01,280 --> 00:38:04,080
that definitely needs to happen,
but that'll be on the server 

662
00:38:04,080 --> 00:38:06,440
side and it'll be optional 
because everything has an 

663
00:38:06,440 --> 00:38:11,520
Http://endpoint just like 
Victoria Metrics or Prometheus 

664
00:38:11,520 --> 00:38:13,720
or any of those things. 
So you can actually just point 

665
00:38:13,720 --> 00:38:16,040
Grafana at it. 
If you want to just use Grafana 

666
00:38:16,520 --> 00:38:18,840
and you can, you can create all 
your dashboards, you can look at

667
00:38:18,840 --> 00:38:21,600
your traces, you can, you can 
look at your logs. 

668
00:38:22,000 --> 00:38:25,480
And so you'd, you'd be able to 
get a farm of web servers. 

669
00:38:25,480 --> 00:38:29,480
You could just push all your 
metrics up to that, but I also 

670
00:38:29,480 --> 00:38:31,760
really like the idea of 
compressing at the edge, right? 

671
00:38:31,760 --> 00:38:34,560
So you just embed this in all 
your apps, it pushes the metrics

672
00:38:34,560 --> 00:38:36,160
up, and then you get a single 
pane of glass. 

673
00:38:36,280 --> 00:38:38,080
Yeah, I like that. 
That would be cool. 

674
00:38:39,040 --> 00:38:40,880
I don't know which way I'm going
yet. 

675
00:38:41,360 --> 00:38:44,600
I think you ought to build a 
service, a host, a service. 

676
00:38:44,800 --> 00:38:45,800
I think that would be really 
cool. 

677
00:38:45,840 --> 00:38:50,760
Could be it could be a thing. 
We'll, we'll see how people like

678
00:38:50,760 --> 00:38:53,760
it on this podcast and, and I 
might pursue that. 

679
00:38:54,200 --> 00:38:57,440
Yeah, I'll make sure to link. 
I'll link your GitHub and 

680
00:38:57,440 --> 00:39:00,240
whatever open source links, make
sure you send them to me and 

681
00:39:00,240 --> 00:39:04,000
I'll make sure they're up there.
Also, and I, I wanted to 

682
00:39:04,000 --> 00:39:07,240
announce this on this podcast, 
so nothing has been published 

683
00:39:07,520 --> 00:39:12,240
yet except GitHub, if you follow
my GitHub, yeah. 

684
00:39:12,240 --> 00:39:15,600
So this timeless is going to be 
available to everyone. 

685
00:39:15,600 --> 00:39:20,160
I'm going to put it on Hex this 
afternoon and you can try it in 

686
00:39:20,160 --> 00:39:25,600
your app and I'll have a little,
Jacob's already seen this, but a

687
00:39:25,720 --> 00:39:29,240
little snippet, a little 6 
minute video of how to install 

688
00:39:29,240 --> 00:39:31,960
it. 
You know what it gives you and 

689
00:39:31,960 --> 00:39:34,840
it's it's very short. 
It's very simple to use this. 

690
00:39:34,960 --> 00:39:37,440
It was made to be simple. 
That's awesome. 

691
00:39:37,480 --> 00:39:40,360
A lot simpler than trying to 
wire up your own observability. 

692
00:39:40,360 --> 00:39:44,640
I, I assure you. 
But but yeah, and I'll, I'll 

693
00:39:44,720 --> 00:39:46,600
make all that stuff public 
today. 

694
00:39:47,040 --> 00:39:49,200
That's exciting. 
Oh, I like that. 

695
00:39:49,600 --> 00:39:52,600
So make sure you get me the hex 
link and I'll put it in the show

696
00:39:52,600 --> 00:39:56,240
notes. 
OK, cuz yeah, that'll be that'll

697
00:39:56,240 --> 00:39:58,640
be awesome. 
I'm excited to check it out. 

698
00:39:58,640 --> 00:40:02,200
I might have to give it a little
test spin for kill switch. 

699
00:40:03,920 --> 00:40:07,040
Now there's. 
So one thing that you probably 

700
00:40:07,040 --> 00:40:10,880
should keep in mind is normally 
the dashboard isn't included in 

701
00:40:10,880 --> 00:40:13,440
prod. 
It's only sort of a dev thing. 

702
00:40:13,880 --> 00:40:18,040
So you can actually authenticate
or create a auth for your 

703
00:40:18,040 --> 00:40:21,200
dashboard. 
So you can, you can wire it up 

704
00:40:21,200 --> 00:40:24,160
so that your dashboard does get 
to, you know, push the prod 

705
00:40:24,320 --> 00:40:26,240
right? 
And then you can put auth in 

706
00:40:26,240 --> 00:40:28,280
front of it. 
And there's instructions in the 

707
00:40:28,920 --> 00:40:32,440
in the repository for that so 
that you can actually sort of 

708
00:40:32,760 --> 00:40:36,840
you can in prod, you can get all
the metrics that you would get 

709
00:40:36,840 --> 00:40:40,080
in dev and help you troubleshoot
problems, which would be pretty 

710
00:40:40,080 --> 00:40:41,800
amazing. 
That is nice. 

711
00:40:42,080 --> 00:40:46,800
Yeah, that's how I have my, I 
use air tracker on kill switch 

712
00:40:46,800 --> 00:40:52,120
and then my Oban, my Oban 
dashboard always put that stuff 

713
00:40:52,120 --> 00:40:58,360
behind auth guys. 
But yeah, yeah, that's cool. 

714
00:40:58,360 --> 00:41:02,280
I'm excited. 
I think I'm excited to hear 

715
00:41:02,280 --> 00:41:05,280
about your new business. 
You said you wanted to get down 

716
00:41:05,280 --> 00:41:07,240
to one business, but now it 
sounds like you're gonna be 

717
00:41:07,240 --> 00:41:11,800
starting another one anyways. 
Yeah, I'll eventually stop being

718
00:41:11,800 --> 00:41:15,360
a DBA and then I'll go do the 
CTO thing full time. 

719
00:41:15,760 --> 00:41:19,600
I'm hopefully buy out the CEO 
and actually become a CEO. 

720
00:41:19,600 --> 00:41:23,320
But then you're gonna start this
timeless service and that's 

721
00:41:23,320 --> 00:41:25,040
gonna be another product and 
business. 

722
00:41:25,640 --> 00:41:28,840
If, if there, if people find 
value in it, I will be happy to 

723
00:41:28,840 --> 00:41:31,840
provide it at a a very 
reasonable cost for sure. 

724
00:41:32,320 --> 00:41:34,840
Because remember, it's all, it's
the compression's really, 

725
00:41:35,000 --> 00:41:37,320
really, really good. 
And so I don't have to store 

726
00:41:37,320 --> 00:41:39,360
much data. 
I just have to present it right?

727
00:41:39,440 --> 00:41:43,720
Yeah, that's because I feel like
there is there is something nice

728
00:41:43,720 --> 00:41:45,120
when you don't want to self 
host. 

729
00:41:45,360 --> 00:41:49,920
Like there's a time frame in 
every startup where you just 

730
00:41:49,920 --> 00:41:53,320
don't have time or the resources
to self host everything, and 

731
00:41:53,320 --> 00:41:55,840
it's kind of nice when you can 
just reach out to services that 

732
00:41:55,840 --> 00:41:59,840
are super easy to integrate. 
Yeah, I agree with that. 

733
00:42:00,560 --> 00:42:04,000
Yeah, but then there's a point 
to where, like, I feel like 

734
00:42:04,000 --> 00:42:06,400
there's growing pains, right? 
You get to a certain point and 

735
00:42:06,400 --> 00:42:08,480
then you want to self host and 
control everything. 

736
00:42:09,840 --> 00:42:14,920
And I think, you know, where I 
see this landing for most people

737
00:42:14,920 --> 00:42:19,040
is that 80% of people who 
develop a small app, they want 

738
00:42:19,040 --> 00:42:21,800
observability. 
It's hard to wire up. 

739
00:42:21,800 --> 00:42:25,040
You don't necessarily have a 
provider you know and, and you 

740
00:42:25,040 --> 00:42:27,240
just. 
You run one command, one in one 

741
00:42:27,240 --> 00:42:29,960
igniter, and you get full 
observability for your. 

742
00:42:30,040 --> 00:42:32,280
App. 
And, and it doesn't take up much

743
00:42:32,280 --> 00:42:34,160
storage because we compress 
really well. 

744
00:42:34,160 --> 00:42:37,920
And we roll the data off for you
so that, you know, you get your 

745
00:42:37,920 --> 00:42:41,400
7 days of time window or 
whatever, whatever you dictate, 

746
00:42:41,400 --> 00:42:43,120
right? 
30 days, seven days, whatever 

747
00:42:43,120 --> 00:42:45,480
you want. 
But we roll the data off for 

748
00:42:45,480 --> 00:42:47,680
you. 
And so it's basically self 

749
00:42:47,680 --> 00:42:48,960
managing. 
You just turn it on. 

750
00:42:49,720 --> 00:42:51,960
That's awesome. 
Does it just? 

751
00:42:52,440 --> 00:42:56,160
Is it Postgres or how are you? 
What? 

752
00:42:56,160 --> 00:42:58,520
Are you No. 
So let's talk architecture. 

753
00:42:58,520 --> 00:43:00,840
Sorry, I haven't even really 
covered that well. 

754
00:43:00,840 --> 00:43:03,920
That's that's. 
We have plenty of time, so this 

755
00:43:03,920 --> 00:43:05,480
is good. 
OK. 

756
00:43:06,240 --> 00:43:09,800
So the architecture for we'll 
start with metrics, that's the 

757
00:43:09,800 --> 00:43:13,760
easy one, are the more complex 
one depending on how you look at

758
00:43:13,760 --> 00:43:16,680
it. 
But metrics, the metrics come in

759
00:43:16,920 --> 00:43:20,800
via either HTTP or you can push 
them directly from Elixir. 

760
00:43:22,080 --> 00:43:24,920
And so you don't necessarily 
need an endpoint because it's 

761
00:43:25,040 --> 00:43:29,640
integrated into your app. 
But the metrics come in and then

762
00:43:29,640 --> 00:43:32,200
we push them to Rust, which 
indexes them. 

763
00:43:32,440 --> 00:43:37,080
So it keeps track of of, you 
know, the, the tags and then the

764
00:43:37,080 --> 00:43:39,520
data itself. 
All, it all gets compressed. 

765
00:43:39,800 --> 00:43:43,320
We, we compress, there's 16 
bytes of data per metric. 

766
00:43:43,720 --> 00:43:47,400
We compress it down to just over
1 byte, which is fantastic. 

767
00:43:49,120 --> 00:43:51,800
And it's all searchable. 
There's a nice index. 

768
00:43:52,240 --> 00:43:54,080
We do use Sequel Lite for some 
things. 

769
00:43:54,080 --> 00:43:55,680
It's because it's embedded, 
right? 

770
00:43:56,240 --> 00:44:04,320
We, we have the ability to alert
you and that those alerts have 

771
00:44:04,320 --> 00:44:07,200
to be stored somewhere when you 
set them up. 

772
00:44:07,280 --> 00:44:11,040
And so those alerts are stored 
in Sequel Lite. 

773
00:44:13,320 --> 00:44:15,760
There's a couple other things 
I'm trying to remember exactly, 

774
00:44:16,480 --> 00:44:19,280
but but yeah, we do have a 
Sequel Lite database. 

775
00:44:19,360 --> 00:44:23,240
OK, do you use ETS tables at all
or anything like that? 

776
00:44:24,760 --> 00:44:28,560
Not as much that was in every 
possible like I've been working 

777
00:44:28,560 --> 00:44:30,800
on this for two years. 
So I've re architected it about 

778
00:44:31,000 --> 00:44:34,840
15 * 14 times. 
Yes, lots of bets. 

779
00:44:35,520 --> 00:44:39,520
The 15th time I went to Rust. 
And so it, it basically just 

780
00:44:39,520 --> 00:44:42,400
hands the data off to Rust and 
it it keeps track of the 

781
00:44:42,400 --> 00:44:45,080
indexing and does all the things
I see. 

782
00:44:45,080 --> 00:44:47,800
But yeah, and then logs and 
traces. 

783
00:44:48,520 --> 00:44:52,560
There is a sequel I database for
those and but we just compress 

784
00:44:52,560 --> 00:44:55,040
the, the values. 
We keep an index so that you can

785
00:44:55,040 --> 00:44:57,280
search, right. 
The search is very fast too, by 

786
00:44:57,280 --> 00:45:00,000
the way. 
And so if you wanted to go find,

787
00:45:00,000 --> 00:45:04,640
you know, all of the traces for 
the guy who or forget, you know,

788
00:45:04,640 --> 00:45:09,320
slash this, this URL, you could 
just go and pull those traces 

789
00:45:09,320 --> 00:45:11,560
and then you can click on it and
you can look at the trace. 

790
00:45:12,280 --> 00:45:14,880
Just like any other 
observability application. 

791
00:45:14,880 --> 00:45:17,960
It's, it's very simple. 
We kept it simple for a reason, 

792
00:45:18,880 --> 00:45:24,960
but but it's it's, you know, 
it's quite searchable and it's. 

793
00:45:24,960 --> 00:45:29,880
I feel like it's a good, it's a 
good example for Sequel Lite as 

794
00:45:29,880 --> 00:45:33,440
well, because you just start 
passing or you just have a Jason

795
00:45:33,440 --> 00:45:36,160
file living in the project at 
that point and you don't have to

796
00:45:36,160 --> 00:45:40,360
worry about Postgres database 
being there or an instance of 

797
00:45:40,360 --> 00:45:43,680
something larger, right? 
It just kind of works, yeah. 

798
00:45:44,560 --> 00:45:47,920
Yeah, we even. 
I even played with storing the 

799
00:45:48,680 --> 00:45:51,240
you know traces and everything 
you know, compressing them down,

800
00:45:51,240 --> 00:45:54,920
store them as blobs inside 
Sequel Lite and that worked OK, 

801
00:45:55,000 --> 00:45:58,920
it was fine. 
But block store is is much. 

802
00:45:58,920 --> 00:46:04,640
Faster, OK, interesting. 
I feel like it's I, I'm pushing 

803
00:46:04,640 --> 00:46:08,240
my skill level, my technical 
skill level, especially like 

804
00:46:08,240 --> 00:46:12,120
when you come down like, like, 
you know, just performance in 

805
00:46:12,120 --> 00:46:17,680
general is a very hard thing to 
like really dig into, right? 

806
00:46:18,200 --> 00:46:23,160
And like when you're picking in 
between like Sequel Lights or 

807
00:46:23,160 --> 00:46:28,920
maybe a My Sequel database or an
ETS table, Like what made you 

808
00:46:28,920 --> 00:46:32,280
lean towards Sequel Light over 
using like an ETS table? 

809
00:46:33,440 --> 00:46:36,840
Well I had this idea from the 
very beginning that I wanted to 

810
00:46:36,880 --> 00:46:38,680
have an embedded version of 
this. 

811
00:46:38,960 --> 00:46:42,600
It may not be the the final end 
all be all, but there is going 

812
00:46:42,600 --> 00:46:44,280
to be an embedded version, 
right? 

813
00:46:44,280 --> 00:46:47,120
Yeah. 
And so Sqlite's just much easier

814
00:46:47,120 --> 00:46:49,560
is. 
There a limitation with ETS or 

815
00:46:49,560 --> 00:46:54,080
ETS on an embedded hardware? 
No, no, it's. 

816
00:46:54,080 --> 00:46:56,320
Just the persistence isn't 
there, right? 

817
00:46:56,800 --> 00:46:59,840
So right, like if you lose your 
instance, everything's gone. 

818
00:47:00,800 --> 00:47:04,320
Yeah, we now you can have it 
obviously you can take your ADS 

819
00:47:04,320 --> 00:47:07,880
tables and dump them to debts 
and you can do or dump them to 

820
00:47:07,880 --> 00:47:09,600
Sequel Lite for persistence, 
right. 

821
00:47:10,160 --> 00:47:12,560
You can do like a like a slow 
path for that. 

822
00:47:13,080 --> 00:47:17,120
You can push it to ADS very 
quickly and then have a process 

823
00:47:17,120 --> 00:47:20,320
that kicks off and and sends it 
over to Sequel Lite for 

824
00:47:20,320 --> 00:47:22,880
persistence. 
I've done all of those things. 

825
00:47:23,120 --> 00:47:26,880
I've tried 15 different versions
of every one of these three 

826
00:47:26,880 --> 00:47:30,960
servers, but but yeah, I've 
settled on what I think is 

827
00:47:30,960 --> 00:47:34,880
probably the least the least 
amount of overhead for an 

828
00:47:34,880 --> 00:47:37,160
embedded app. 
So I've always had that that 

829
00:47:38,400 --> 00:47:40,920
embedded in my head from the 
very beginning. 

830
00:47:41,200 --> 00:47:44,040
And that's those are the 
important decisions that like 

831
00:47:44,040 --> 00:47:47,200
those architectural decisions 
are not easy ones. 

832
00:47:47,200 --> 00:47:50,520
And see like that guys is where 
we're not getting replaced by 

833
00:47:50,520 --> 00:47:55,480
LLMS, you know what I mean? 
Like you would not get your your

834
00:47:55,480 --> 00:47:59,800
architecture choices. 
Like if we put this into Claude 

835
00:47:59,800 --> 00:48:01,320
right now, we'd probably get 
some. 

836
00:48:01,560 --> 00:48:04,760
I'm sure we'd get like a working
plan that would probably be 

837
00:48:04,760 --> 00:48:07,520
functional, but it probably 
wouldn't perform the same way. 

838
00:48:08,800 --> 00:48:11,120
I will say though, I've, you 
know, I've worked in tech for a 

839
00:48:11,120 --> 00:48:14,240
long time and I've worked in and
worked with a lot of really 

840
00:48:14,240 --> 00:48:17,160
intelligent people. 
And I learned a long time ago to

841
00:48:17,160 --> 00:48:20,400
ask open-ended questions, you 
know, just go and say, I'm 

842
00:48:20,400 --> 00:48:24,960
looking to do this and then have
it come back with some possible 

843
00:48:24,960 --> 00:48:27,120
solutions. 
And, you know, maybe they're 

844
00:48:27,120 --> 00:48:31,520
good, maybe they're not, but, 
and then maybe they apply and 

845
00:48:31,520 --> 00:48:34,320
maybe they don't. 
But if you ask the open-ended 

846
00:48:34,320 --> 00:48:37,040
question, you, they may think of
something that you have not 

847
00:48:37,040 --> 00:48:40,000
thought of, which is, and, and 
the LLMS will do that too. 

848
00:48:40,000 --> 00:48:43,640
So if you ask open-ended 
questions of LLMS, they can help

849
00:48:43,640 --> 00:48:46,400
guide you. 
I think it's a great way to like

850
00:48:46,400 --> 00:48:49,280
rubber duck, like especially for
a lot of developers that are 

851
00:48:49,280 --> 00:48:51,240
remote. 
You like we get on our own 

852
00:48:51,440 --> 00:48:58,360
little eco or echo chamber, and 
it's nice to talk to a robot 

853
00:48:58,360 --> 00:49:01,600
from time to time. 
About your ideas, I mean. 

854
00:49:01,680 --> 00:49:04,400
Well, I learned they're 
encyclopedias. 

855
00:49:04,400 --> 00:49:06,960
It's why wouldn't you ask an 
encyclopedia what all the 

856
00:49:06,960 --> 00:49:09,000
options are? 
You know, why wouldn't you go 

857
00:49:09,000 --> 00:49:11,600
there and do that? 
They may not have the best 

858
00:49:11,600 --> 00:49:14,720
option for you and they may not.
It may not bubble up. 

859
00:49:14,760 --> 00:49:17,680
But as long as they know all of 
the options, it's more than you,

860
00:49:18,160 --> 00:49:20,320
right? 
And so they can help you. 

861
00:49:21,080 --> 00:49:24,760
And like when I did Killswitch, 
which you know, it's like a zero

862
00:49:24,760 --> 00:49:27,200
knowledge encryption document 
store, right? 

863
00:49:27,200 --> 00:49:30,800
I didn't really know how to 
implement a zero knowledge 

864
00:49:30,800 --> 00:49:32,440
encryption system. 
Well, right. 

865
00:49:32,760 --> 00:49:36,600
Like I've read my encryption 
algorithm books and things like 

866
00:49:36,600 --> 00:49:40,720
that but they don't actually 
like I know how they work at the

867
00:49:40,720 --> 00:49:43,760
encryption level but like how do
you properly implement it? 

868
00:49:44,360 --> 00:49:47,640
And dude, I'd like open-ended 
questions and rubber ducking 

869
00:49:47,640 --> 00:49:51,320
with Claude back and forth how I
wanted the system to work client

870
00:49:51,320 --> 00:49:55,520
side freaking help me iron 
things out so much faster than I

871
00:49:55,520 --> 00:49:58,280
would have by myself. 
I think I would have still got 

872
00:49:58,280 --> 00:50:01,320
to the same solution, but it 
would have taken me a lot 

873
00:50:01,320 --> 00:50:04,440
longer. 
Yeah, I agree with that. 

874
00:50:04,840 --> 00:50:08,880
It's, you know, it's, it's just 
nice. 

875
00:50:08,880 --> 00:50:12,360
I mean, just having something 
that knows more than no matter 

876
00:50:12,360 --> 00:50:14,240
what, they know more than you 
do, right? 

877
00:50:14,240 --> 00:50:17,400
They have more scenarios in 
their heads than than you do. 

878
00:50:17,720 --> 00:50:20,760
You've you've been down this 
path and so you have your 

879
00:50:20,760 --> 00:50:23,200
predefined things that you would
like to accomplish. 

880
00:50:23,560 --> 00:50:26,120
So you have to obviously measure
it against that. 

881
00:50:26,440 --> 00:50:29,360
But yeah, it's great. 
It really is great. 

882
00:50:29,640 --> 00:50:31,920
It's helped quite a bit on time 
was for sure. 

883
00:50:32,080 --> 00:50:34,640
Yeah, I'm sure that's cool. 
What? 

884
00:50:34,800 --> 00:50:38,520
What does your like day-to-day 
workflow look like using and 

885
00:50:38,520 --> 00:50:40,600
like? 
Do you use Claude or what LLM do

886
00:50:40,600 --> 00:50:42,640
you prefer? 
Oh, sure. 

887
00:50:43,320 --> 00:50:48,200
So, you know, in the car. 
In the car, I use Grok. 

888
00:50:48,920 --> 00:50:51,680
I'll just start it up and I'll 
start having a conversation 

889
00:50:51,680 --> 00:50:54,400
about how I would like to 
compress things or how I would 

890
00:50:54,400 --> 00:50:59,440
like to index things, right. 
And Claude, sometimes I'll, I'll

891
00:50:59,440 --> 00:51:03,120
take those ideas and bounce them
off of him once they're sort of 

892
00:51:03,120 --> 00:51:07,720
formed and then, you know, still
trying to answer, you know, ask 

893
00:51:07,720 --> 00:51:10,040
open-ended questions if it, if 
at all possible. 

894
00:51:10,400 --> 00:51:14,640
And then for clean up, I found 
GPT to be really good. 

895
00:51:15,200 --> 00:51:17,240
So, you know, Claude, you can 
have it. 

896
00:51:17,680 --> 00:51:21,440
It's very innovative when it 
writes code, but it doesn't 

897
00:51:21,440 --> 00:51:25,520
necessarily get it right. 
So I like the innovation 

898
00:51:25,600 --> 00:51:28,160
sometimes. 
And so when I'm trying to 

899
00:51:28,160 --> 00:51:31,920
innovate, I'll use Claude and 
then I'll come back with GPT and

900
00:51:31,920 --> 00:51:34,040
sort of clean it up and make 
sure that it's doing the right 

901
00:51:34,040 --> 00:51:36,600
things. 
But that's my, that's my work. 

902
00:51:37,040 --> 00:51:39,960
Excuse me, that's my workflow. 
OK, nice. 

903
00:51:41,240 --> 00:51:44,240
When you're like thinking 
through a feature, what does 

904
00:51:44,240 --> 00:51:45,480
that? 
What does that look like? 

905
00:51:45,520 --> 00:51:48,680
Are you chatting back and forth 
with Claude and creating like a 

906
00:51:48,680 --> 00:51:52,400
markdown plan or what's it? 
What are you doing? 

907
00:51:53,520 --> 00:51:58,200
So, yeah, so previous versions 
of of timeless metrics used 

908
00:51:58,200 --> 00:52:03,600
that's quite heavily. 
And you know, I, I was hitting 

909
00:52:03,600 --> 00:52:06,320
performance bottlenecks, I was 
trying to hit numbers that 

910
00:52:06,720 --> 00:52:10,960
approached, you know, the ghost 
space, the the Victoria metric 

911
00:52:10,960 --> 00:52:15,200
space and I was unable to do so 
I got really close and I 

912
00:52:15,200 --> 00:52:18,480
eventually got faster, but it, I
had to get away from ETS 

913
00:52:18,480 --> 00:52:22,640
unfortunately. 
And so I used a, what's it 

914
00:52:22,640 --> 00:52:27,440
called, persistent terms, and 
that made it a good bit faster. 

915
00:52:29,080 --> 00:52:33,800
And then, yeah. 
And then, you know, eventually I

916
00:52:33,800 --> 00:52:38,080
just kept breaking off pieces 
and, and making them Nifs. 

917
00:52:38,400 --> 00:52:41,120
And then of course, those Nifs 
would have to talk through 

918
00:52:41,120 --> 00:52:44,240
Elixir, right? 
And so that became a bottleneck 

919
00:52:44,240 --> 00:52:46,440
too. 
And then I just basically 

920
00:52:46,480 --> 00:52:49,920
bundled it all together. 
But but yeah, it's. 

921
00:52:50,120 --> 00:52:54,000
Interesting, like having more, 
having too many nifs was a 

922
00:52:54,000 --> 00:52:57,600
bottleneck, but having like 1? 
I was passing the data through 

923
00:52:57,600 --> 00:52:59,560
Elixir. 
Yeah, from one NIF to the other.

924
00:53:00,680 --> 00:53:03,200
So I see. 
You just take the 2 Nifs and 

925
00:53:03,200 --> 00:53:04,440
just put them into one. 
Right. 

926
00:53:04,440 --> 00:53:07,560
Yeah, yeah. 
And so it's, it's easier for it 

927
00:53:07,560 --> 00:53:10,120
to, you know, Rust to talk to 
Rust, you know, right. 

928
00:53:10,120 --> 00:53:12,600
That makes sense. 
You had just a middle man that 

929
00:53:12,600 --> 00:53:16,760
was slowing things down. 
And it's still again, Elixir 

930
00:53:16,760 --> 00:53:20,280
fantastic, best best there is as
far as orchestration layers is 

931
00:53:20,280 --> 00:53:22,800
concerned. 
I wouldn't trade it for anything

932
00:53:22,800 --> 00:53:26,520
for that layer of it, even if it
does cost a little bit in 

933
00:53:26,520 --> 00:53:29,800
performance. 
And you know, 3,000,000 inserts 

934
00:53:29,800 --> 00:53:32,880
per second is that's a lot. 
You know, back in the day, you 

935
00:53:32,880 --> 00:53:36,800
know, I told you I, I wrote this
application for monitoring cable

936
00:53:36,800 --> 00:53:39,120
modems for, for Cox back in the 
day. 

937
00:53:39,800 --> 00:53:43,240
And you know, I was just 
completely ecstatic to get 

938
00:53:43,240 --> 00:53:46,000
10,000 inserts per second back, 
you know, with those little 

939
00:53:46,000 --> 00:53:49,200
pizza boxes, the single core 
pizza boxes. 

940
00:53:50,320 --> 00:53:53,320
And now we're doing, you know, 
3,000,000 inserts per second, 

941
00:53:53,360 --> 00:53:56,160
which is crazy and compressing 
it, right? 

942
00:53:56,720 --> 00:53:59,480
Yeah, no, that's wild. 
Like it's actually hard to kind 

943
00:53:59,480 --> 00:54:01,840
of wrap your head around how 
much data that is. 

944
00:54:02,480 --> 00:54:04,960
Yeah, yeah. 
Well, it's more than most people

945
00:54:04,960 --> 00:54:07,920
will need, but it's, it's also 
necessary, right, to keep that 

946
00:54:07,920 --> 00:54:10,920
overhead very low, right. 
So if you want to put this in 

947
00:54:10,920 --> 00:54:13,720
your running application, you 
want it to to not affect the 

948
00:54:13,720 --> 00:54:15,960
running application. 
You want to be able to throw as 

949
00:54:15,960 --> 00:54:18,520
many logs as you want to at it 
without affecting the running 

950
00:54:18,520 --> 00:54:22,200
application. 
And so, you know, I, I felt like

951
00:54:22,200 --> 00:54:26,240
performance was, was absolutely 
paramount and, and I was 

952
00:54:26,240 --> 00:54:29,160
achieve, I was able to achieve 
everything that I was set out, I

953
00:54:29,200 --> 00:54:31,320
basically set out to do. 
Yeah, that's cool. 

954
00:54:31,720 --> 00:54:35,200
I'm not like very familiar with 
Rust or using NIFS. 

955
00:54:35,360 --> 00:54:37,160
What was your learning curve 
there? 

956
00:54:37,600 --> 00:54:38,640
How did you? 
Huge. 

957
00:54:38,920 --> 00:54:41,760
Yeah, huge. 
I'd I'd not done it before 

958
00:54:41,800 --> 00:54:45,640
either. 
And you know, it was it was 

959
00:54:45,640 --> 00:54:48,720
required, I think, for the 
performance that I was looking 

960
00:54:48,720 --> 00:54:50,920
for. 
And so I just, you know, head 

961
00:54:50,920 --> 00:54:54,160
down, you know, figured it out, 
asked a bunch of questions of 

962
00:54:54,160 --> 00:54:59,320
the LMS right, to help me 
understand the the trans in your

963
00:54:59,320 --> 00:55:01,080
wrestlers. 
Just fantastic. 

964
00:55:01,360 --> 00:55:04,440
It does take care of and I use 
fine too. 

965
00:55:04,920 --> 00:55:09,480
Fine is is used, I think on the 
rocket side. 

966
00:55:10,840 --> 00:55:15,600
I do have a couple of different 
interfaces, but but yeah, it's, 

967
00:55:15,800 --> 00:55:18,480
it's not as hard as you think. 
But it's it's a learning curve 

968
00:55:18,480 --> 00:55:23,120
for sure. 
Yeah, I heard like handling like

969
00:55:23,160 --> 00:55:27,600
errors and like hard crashes is 
difficult to like get back out 

970
00:55:27,600 --> 00:55:31,560
to the Elixir layer to actually 
be like helpful. 

971
00:55:32,320 --> 00:55:36,680
That was a big part of it. 
You have to actually, you sort 

972
00:55:36,680 --> 00:55:38,960
of write REST the way you would 
Elixir, right? 

973
00:55:38,960 --> 00:55:43,240
You send errors back. 
So yeah, if anything breaks, you

974
00:55:43,240 --> 00:55:45,080
send an error back. 
And so that Elixir can 

975
00:55:45,080 --> 00:55:47,720
understand what broke, right. 
Yeah. 

976
00:55:47,720 --> 00:55:52,440
You try to behave like an Elixir
application at that layer. 

977
00:55:53,000 --> 00:55:56,560
Because it's really easy to just
swallow, swallow the airs right 

978
00:55:56,560 --> 00:55:58,760
when you're like. 
Creating or crash, yeah, you 

979
00:55:58,760 --> 00:56:00,000
really don't want to crash 
either. 

980
00:56:00,720 --> 00:56:03,440
So yeah, that's going to be one 
of the interesting things about 

981
00:56:03,440 --> 00:56:05,360
making this public is, you know,
people are going to use it in 

982
00:56:05,360 --> 00:56:08,440
ways that I have not. 
And so there there may be some 

983
00:56:08,440 --> 00:56:10,600
crashes out there. 
We'll get it all worked out. 

984
00:56:10,720 --> 00:56:13,520
Try to treat it as at least beta
level software. 

985
00:56:13,880 --> 00:56:16,920
Probably, probably not alpha, 
probably beta level software, 

986
00:56:17,320 --> 00:56:19,240
but. 
But yeah, be careful. 

987
00:56:20,120 --> 00:56:21,440
Because. 
We are dealing with a lot of 

988
00:56:21,440 --> 00:56:23,720
Nifs. 
Is this your first like open 

989
00:56:23,720 --> 00:56:25,960
source project that you're 
taking public? 

990
00:56:26,640 --> 00:56:31,000
No, I've so I've maintained 
something called SNMP 4J. 

991
00:56:31,080 --> 00:56:32,440
Oh yeah, you've said that. 
Yeah. 

992
00:56:32,600 --> 00:56:35,320
Yeah. 
So I had an SNMP library that I 

993
00:56:35,400 --> 00:56:39,160
I open sourced. 
It was kind of a wrapper around 

994
00:56:39,160 --> 00:56:45,400
SNMP 4 J and it's for J Ruby 18 
years now I guess. 

995
00:56:45,680 --> 00:56:47,880
But I haven't had a lot of open 
source projects. 

996
00:56:48,080 --> 00:56:51,840
I've really embraced the Elixir 
community and I have, I want to 

997
00:56:51,840 --> 00:56:57,080
say 10 now, but yeah, quite a 
few things that I've released on

998
00:56:57,080 --> 00:57:00,640
the Elixir side. 
I have a SOAP library because we

999
00:57:00,640 --> 00:57:03,280
needed SOAP. 
We're we're a telephony provider

1000
00:57:03,280 --> 00:57:05,120
and telephony providers love 
SOAP. 

1001
00:57:05,640 --> 00:57:09,560
So I have a SOAP library that is
free for all and it's actually 

1002
00:57:09,560 --> 00:57:11,960
the most, I think it's the most 
popular library that I've 

1003
00:57:11,960 --> 00:57:17,200
released on Elixir ecosystem. 
The SNMP kit is very popular as 

1004
00:57:17,200 --> 00:57:19,800
well. 
I wouldn't say very popular, but

1005
00:57:19,800 --> 00:57:22,200
it's, it's more popular than I 
was expecting. 

1006
00:57:23,520 --> 00:57:25,760
And then, you know, all these 
compression libraries that I've 

1007
00:57:25,760 --> 00:57:31,040
been wrapping all the, the 
little NIF accesses, you know, I

1008
00:57:31,040 --> 00:57:33,720
may not be using them anymore, 
but I have released those as 

1009
00:57:33,720 --> 00:57:35,200
open source. 
So you can use them if you 

1010
00:57:35,200 --> 00:57:37,840
wanted to. 
Things like Z not Z standard 

1011
00:57:38,600 --> 00:57:43,920
open ZL I'm trying to think. 
You're making me look bad. 

1012
00:57:43,920 --> 00:57:47,680
I have no open source projects. 
Really, yeah, no, you should 

1013
00:57:47,680 --> 00:57:50,720
think, just peel off little 
pieces like as you're working, 

1014
00:57:50,800 --> 00:57:54,080
just think, hey, you know, this 
this little piece right here I 

1015
00:57:54,080 --> 00:57:55,960
think could probably be useful 
to someone else. 

1016
00:57:56,320 --> 00:57:59,320
And then just peel it off, you 
know, and there's some packaging

1017
00:57:59,320 --> 00:58:01,800
that's required, right? 
It takes a little bit of work, 

1018
00:58:02,080 --> 00:58:05,120
but with the LLMS, right, the 
packaging becomes a lot easier. 

1019
00:58:06,120 --> 00:58:09,320
And so I've just peeled them off
and and just made them available

1020
00:58:09,320 --> 00:58:11,680
to the community. 
So it's it's relatively easy to 

1021
00:58:11,680 --> 00:58:13,960
do. 
Yeah, no, I think, I think it's 

1022
00:58:13,960 --> 00:58:16,280
awesome too. 
Like I wish I was a better 

1023
00:58:16,280 --> 00:58:22,400
person. 
You know there's some sort of 

1024
00:58:22,400 --> 00:58:24,960
encryption you're using, right? 
Is there a way to peel off the 

1025
00:58:24,960 --> 00:58:26,520
encryption and maybe release it?
Yeah. 

1026
00:58:26,800 --> 00:58:30,800
Not to think about that the 
client, the problem is like all 

1027
00:58:30,800 --> 00:58:34,760
the client side stuff is done in
JavaScript and so. 

1028
00:58:35,280 --> 00:58:39,040
But there's probably, I don't 
know, by the way, I could do it.

1029
00:58:41,960 --> 00:58:44,440
I'll have to think about it 
because for any back end 

1030
00:58:44,440 --> 00:58:47,440
encryption, I'm just using ash 
cloak for that. 

1031
00:58:47,440 --> 00:58:53,320
And dude, I I love using ash for
my production stuff now because 

1032
00:58:53,320 --> 00:58:56,080
it's just so fast. 
I don't know if you've used ash 

1033
00:58:56,080 --> 00:58:58,200
at all, but wow. 
So. 

1034
00:58:58,400 --> 00:59:00,200
That's a part of the 
conversation I haven't really 

1035
00:59:00,200 --> 00:59:02,640
gotten into. 
SO you know, the provisioning 

1036
00:59:02,640 --> 00:59:05,400
side of DOCSIS, the cable modem 
stuff. 

1037
00:59:05,880 --> 00:59:10,680
We that entire stack is Elixir. 
Now we've got a DHCP server 

1038
00:59:10,680 --> 00:59:12,560
that's commercial quality, 
right? 

1039
00:59:12,680 --> 00:59:15,360
TFTP server that is commercial 
quality. 

1040
00:59:16,200 --> 00:59:19,400
We have an NTP server that is 
stratum 2 compliant, which is 

1041
00:59:19,400 --> 00:59:22,120
pretty cool. 
That's that's, you know, sort of

1042
00:59:22,120 --> 00:59:28,880
telephony level NTP we have. 
I'm trying to think of what else

1043
00:59:29,680 --> 00:59:32,840
DOCSIS Boot File Editor, which 
doesn't mean anything to anybody

1044
00:59:32,840 --> 00:59:36,840
on this call, but the number of 
things that help us do the 

1045
00:59:36,840 --> 00:59:38,920
provisioning for cable modems. 
That's all. 

1046
00:59:38,920 --> 00:59:40,760
Elixir. 
That's a full Elixir stack. 

1047
00:59:40,960 --> 00:59:46,000
OK, cool. 
Yeah, yeah, but for So I started

1048
00:59:46,000 --> 00:59:48,800
using Ash and it was a hard 
learning curve for me because 

1049
00:59:48,960 --> 00:59:53,080
it's like another language. 
It's not Elixir, it's, you know,

1050
00:59:53,080 --> 00:59:56,440
they're their own DSL thing. 
But what's nice is like for 

1051
00:59:56,440 --> 01:00:00,560
production stuff like you know, 
what your schemas look like and 

1052
01:00:00,760 --> 01:00:06,120
all of that. 
And it takes away the overhead 

1053
01:00:06,120 --> 01:00:08,680
of. 
Writing an active schema and 

1054
01:00:08,680 --> 01:00:14,120
then fine tuning your database 
migrations like it does that all

1055
01:00:14,120 --> 01:00:16,560
just like the way you set up 
your relationships and 

1056
01:00:16,560 --> 01:00:18,560
resources. 
It's amazing. 

1057
01:00:18,560 --> 01:00:20,520
I don't know. 
I suggest it reminds me. 

1058
01:00:21,600 --> 01:00:24,000
Yeah, it reminds me a little bit
of Rails. 

1059
01:00:24,880 --> 01:00:27,800
It helps you sort of translate 
your business logic very easily 

1060
01:00:27,800 --> 01:00:29,000
into your. 
Scheme, yeah. 

1061
01:00:29,200 --> 01:00:30,560
Yes. 
And that's what we've done on 

1062
01:00:30,560 --> 01:00:34,120
the the you know, DOXA services 
side is we've we've used ash. 

1063
01:00:34,400 --> 01:00:36,880
I forgot to mention that the 
reason why I told you all those 

1064
01:00:36,880 --> 01:00:41,000
things is because we did use Ash
for that and and yes, it's, it's

1065
01:00:41,000 --> 01:00:44,840
fantastic to just apply your 
business logic to to what you're

1066
01:00:44,840 --> 01:00:47,680
doing on the database side. 
Just write your resources, your 

1067
01:00:47,680 --> 01:00:52,000
relationships and actions 
essentially, and then you'd have

1068
01:00:52,000 --> 01:00:56,120
your schema and context files 
basically all in one small 

1069
01:00:56,120 --> 01:00:57,800
module. 
It's really nice, yeah. 

1070
01:00:59,200 --> 01:01:02,520
We really like it. 
Yeah, and I struggled with it 

1071
01:01:02,520 --> 01:01:06,240
though, until the Ash book got 
released because I just needed, 

1072
01:01:06,240 --> 01:01:09,520
I don't know, I needed to read 
it and see how it was being 

1073
01:01:09,520 --> 01:01:11,480
built. 
Like I looked at tons of 

1074
01:01:11,480 --> 01:01:15,280
examples in the documentation 
and I was just like, I always 

1075
01:01:15,480 --> 01:01:19,240
struggled with like, OK, I know 
how to do this in Elixir in 

1076
01:01:19,240 --> 01:01:26,280
Phoenix, and I know the ash way 
is like a one liner in a config 

1077
01:01:26,280 --> 01:01:27,760
somewhere, but I don't know how 
to do it. 

1078
01:01:27,760 --> 01:01:29,960
So it's just like it turned into
just being frustrating. 

1079
01:01:30,920 --> 01:01:35,880
But then once you unlock that 
the syntax and how to use their 

1080
01:01:35,880 --> 01:01:43,160
DSL, it's just you move quick. 
We had difficulty when we we 

1081
01:01:43,160 --> 01:01:47,280
when we first started the apps 
with Ash, the LMS didn't know it

1082
01:01:47,280 --> 01:01:49,640
very well. 
I had trouble with Claude and 

1083
01:01:49,640 --> 01:01:53,480
some of and that has improved 
dramatically in the last couple 

1084
01:01:53,480 --> 01:01:56,520
years. 
When they released usage rules 

1085
01:01:56,520 --> 01:01:59,440
that helped a lot because you 
didn't have to write your own 

1086
01:01:59,440 --> 01:02:03,000
ash like you didn't have to add 
to your own like MD file you 

1087
01:02:03,000 --> 01:02:04,320
know? 
Right. 

1088
01:02:04,840 --> 01:02:10,640
Yeah, yeah, that's it was it was
painful, unfortunately, using 

1089
01:02:10,640 --> 01:02:12,800
ash back in the day. 
This was two years ago. 

1090
01:02:13,640 --> 01:02:16,800
But but yeah, I've done a couple
of things recently and it's been

1091
01:02:16,800 --> 01:02:18,840
so much better. 
It's much, much, much better 

1092
01:02:18,840 --> 01:02:20,160
now. 
Mm hmm. 

1093
01:02:20,240 --> 01:02:25,080
I agree, but yeah, game changer.
I don't even know if I I can 

1094
01:02:25,080 --> 01:02:26,960
write normal Elixir code. 
I was going to say I don't know 

1095
01:02:26,960 --> 01:02:30,040
if I can anymore, but I write it
every day. 

1096
01:02:32,680 --> 01:02:36,480
The day job keeps me current 
with, you know, Acto and Elixir 

1097
01:02:36,480 --> 01:02:38,880
in Phoenix. 
And then all my side projects 

1098
01:02:38,880 --> 01:02:43,360
are are now Ash. 
I will say it's much. 

1099
01:02:43,480 --> 01:02:46,480
It's much nicer not to have to 
shovel, you know, shove the bits

1100
01:02:46,480 --> 01:02:49,360
around like I can. 
I can really envision what I'm 

1101
01:02:49,360 --> 01:02:54,000
trying to do and really, really 
get down to exactly how I want 

1102
01:02:54,000 --> 01:02:56,600
it to execute. 
And I don't have to write, you 

1103
01:02:56,600 --> 01:03:00,000
know, all the little bits and 
pieces and, you know, 

1104
01:03:00,040 --> 01:03:03,280
accidentally type something 
wrong like, you know, it's just 

1105
01:03:03,360 --> 01:03:08,080
I, I find it is very liberating.
It accelerates me for sure. 

1106
01:03:09,120 --> 01:03:12,240
Yeah, I agree. 
Well, like one thing that I 

1107
01:03:12,240 --> 01:03:16,880
think is the superpowers, just 
like I always look at my 

1108
01:03:16,920 --> 01:03:21,560
database migration files when 
they're generated, but I I've 

1109
01:03:21,560 --> 01:03:26,880
never had to alter them or add 
an index or anything with ash 

1110
01:03:26,880 --> 01:03:29,280
like it just it always is, 
right? 

1111
01:03:30,360 --> 01:03:34,360
Very intelligent. 
Yeah, so that's nice because I 

1112
01:03:34,360 --> 01:03:35,600
feel like that's the missing 
link. 

1113
01:03:35,600 --> 01:03:41,560
A lot of people like late lately
I've been reviewing so much I 

1114
01:03:41,560 --> 01:03:47,280
shouldn't be mean on my podcast.
A lot of AI driven projects that

1115
01:03:47,280 --> 01:03:50,880
are just kind of like over 
engineered slop and you can tell

1116
01:03:51,440 --> 01:03:55,200
that an AI generated it. 
It's like like sure, it's good 

1117
01:03:55,200 --> 01:03:58,800
code, it works, but it's like 
it's too much. 

1118
01:03:58,800 --> 01:04:01,320
Like I know you didn't put any 
thought into this. 

1119
01:04:01,320 --> 01:04:05,960
You let Claude engineer the 
whole solution and that's been 

1120
01:04:05,960 --> 01:04:09,080
getting old. 
But one thing Claude sucks at is

1121
01:04:09,080 --> 01:04:11,800
database performance. 
Yeah. 

1122
01:04:12,800 --> 01:04:16,400
So it'll miss indexes in the 
migration files. 

1123
01:04:16,400 --> 01:04:19,720
It'll do all that. 
And like that's a great, like if

1124
01:04:19,720 --> 01:04:22,040
you're getting interviewed, 
guys, a great got you. 

1125
01:04:22,440 --> 01:04:28,240
Like, OK, what happens if I, I 
need to query half a million 

1126
01:04:28,240 --> 01:04:31,360
records off this? 
Like is it going to perform or 

1127
01:04:31,360 --> 01:04:35,800
is it going to time out? 
Yeah, as a DBAI, fully support 

1128
01:04:35,800 --> 01:04:37,160
this message. 
Yeah. 

1129
01:04:37,440 --> 01:04:40,640
Check, check, check your LMS and
make sure that they're doing the

1130
01:04:40,640 --> 01:04:41,520
right thing. 
So. 

1131
01:04:42,280 --> 01:04:46,560
And honestly, I took for granted
because I've, like we were 

1132
01:04:46,560 --> 01:04:51,120
talking about before, like the 
average software business, you 

1133
01:04:51,120 --> 01:04:54,600
have a couple 1000 users, right?
Like you're never going to hit 

1134
01:04:54,600 --> 01:04:59,160
performance hurdles or cliffs 
just because you can kind of 

1135
01:04:59,160 --> 01:05:01,920
just write sloppy code and get 
away with it because it's just 

1136
01:05:01,920 --> 01:05:03,400
going to work. 
You don't have enough data. 

1137
01:05:04,560 --> 01:05:08,320
But then when I started working 
at collects, I had to Start 

1138
01:05:08,320 --> 01:05:13,080
learning about, I had to be 
mindful about, you know, where 

1139
01:05:13,080 --> 01:05:16,760
you place indexes, how you're 
using them and how you're doing 

1140
01:05:17,240 --> 01:05:21,520
table joins and all these things
because I like the very first 

1141
01:05:21,520 --> 01:05:25,240
feature I released, I like 
crashed production and it was 

1142
01:05:25,240 --> 01:05:27,960
very stressful, but I had to go 
in and fix it. 

1143
01:05:27,960 --> 01:05:33,680
And I learned all about taking a
deeper look at what was my, my 

1144
01:05:33,680 --> 01:05:36,280
sequel query, What did it look 
like? 

1145
01:05:36,280 --> 01:05:41,360
And was it what I expected? 
And it wasn't because we use, we

1146
01:05:41,360 --> 01:05:44,840
use App Synth, which is like 
draft QL, you know, and a lot 

1147
01:05:45,160 --> 01:05:48,000
like. 
For the most part it generates 

1148
01:05:48,000 --> 01:05:50,720
your queries properly, but 
sometimes it doesn't and you 

1149
01:05:50,720 --> 01:05:55,040
have to kind of force it to 
build the query you want. 

1150
01:05:56,520 --> 01:05:59,960
Yeah, I work with my day job, I 
work with 60 different dev 

1151
01:05:59,960 --> 01:06:03,000
groups and they have all 
different flavors of Orams and 

1152
01:06:03,000 --> 01:06:07,000
we have we have issues like the 
graph QL issues all the time. 

1153
01:06:07,040 --> 01:06:09,720
Yeah, actually after I finish 
this call, I have a call with 

1154
01:06:09,720 --> 01:06:13,960
someone who has a oops, I have a
600 million row table. 

1155
01:06:13,960 --> 01:06:17,560
How can you help me clean it up?
So that's, that's, you know, 

1156
01:06:17,720 --> 01:06:24,680
it's the life of ADBA. 
Do you, I'm like, how do you 

1157
01:06:24,680 --> 01:06:26,120
handle like, how do you handle 
that? 

1158
01:06:26,120 --> 01:06:29,320
Do you back up the table before 
you start messing with things 

1159
01:06:29,320 --> 01:06:31,560
and then? 
We figure out what they want to 

1160
01:06:31,560 --> 01:06:35,880
keep and then we we have some 
online schema change utilities 

1161
01:06:36,600 --> 01:06:40,160
that Facebook published a long 
time ago that we've modified 

1162
01:06:40,840 --> 01:06:43,360
Facebook, you know, they do a 
couple of passes. 

1163
01:06:43,360 --> 01:06:45,880
We do 3. 
So we'll take this very large 

1164
01:06:45,880 --> 01:06:49,320
table, we'll copy the data out 
that we want to keep and then 

1165
01:06:49,360 --> 01:06:52,320
we'll, and that can take a 
while, right? 

1166
01:06:52,320 --> 01:06:54,200
Because you sometimes you want 
to keep quite a bit. 

1167
01:06:54,920 --> 01:06:57,840
And then we do a delta and then 
we copy the delta over and then 

1168
01:06:57,840 --> 01:07:00,120
we lock the table. 
We copy the last little bit and 

1169
01:07:00,120 --> 01:07:02,480
then we swap the tables. 
OK. 

1170
01:07:02,520 --> 01:07:05,600
All right. 
So you never officially drop the

1171
01:07:05,880 --> 01:07:09,840
the table you're fixing until 
you confirm that everything's 

1172
01:07:09,840 --> 01:07:12,600
good. 
It depends, you know, sometimes 

1173
01:07:12,600 --> 01:07:14,400
it's just a queue, right? 
And they don't care. 

1174
01:07:14,520 --> 01:07:17,120
That's sometimes Wild West 
shifting, honestly. 

1175
01:07:18,040 --> 01:07:22,640
There's, there's a you'd be. 
So it's, it's a lot of different

1176
01:07:22,640 --> 01:07:25,720
dev groups and some of them 
have, you know, are focused on 

1177
01:07:25,720 --> 01:07:27,440
the, the size of their data and 
some are not. 

1178
01:07:27,920 --> 01:07:31,320
Yeah. 
Yeah, yeah. 

1179
01:07:31,320 --> 01:07:33,960
Touching a prod database always 
makes me nervous. 

1180
01:07:34,400 --> 01:07:37,760
I always aim for the replica 
first guys. 

1181
01:07:38,960 --> 01:07:40,640
You should you should try it as 
a full time job. 

1182
01:07:40,920 --> 01:07:43,880
It's exciting. 
I don't know. 

1183
01:07:43,960 --> 01:07:46,440
I mean, I'm sure, honestly, I'm 
sure it is it. 

1184
01:07:46,720 --> 01:07:50,240
It's probably a lot of fun, but 
it seems like a lot of 

1185
01:07:50,240 --> 01:07:53,160
responsibility. 
It is, you just have to be 

1186
01:07:53,160 --> 01:07:54,360
careful. 
Yeah. 

1187
01:07:54,520 --> 01:07:56,360
And you get, yeah, it comes with
time. 

1188
01:07:57,120 --> 01:08:00,360
Yeah, what? 
I don't know if you actually 

1189
01:08:00,360 --> 01:08:04,640
answered this at the beginning, 
but what got you into the 

1190
01:08:05,160 --> 01:08:07,480
database space? 
Like how did you become a 

1191
01:08:07,480 --> 01:08:11,520
consultant for that? 
So I built this really large 

1192
01:08:11,520 --> 01:08:14,240
data warehouse on my sequel of 
all things. 

1193
01:08:14,920 --> 01:08:16,680
Back in the day. 
It was a Snowflake database 

1194
01:08:16,680 --> 01:08:19,960
before a Snowflake was a thing. 
Snowflake's an actual design of 

1195
01:08:19,960 --> 01:08:24,000
a database, not a vendor. 
We use a snowflake IDs, I know 

1196
01:08:24,000 --> 01:08:29,560
that, but OK, basically because 
it was a bad architecture 

1197
01:08:29,560 --> 01:08:32,640
decision and we were running out
of primary IDs. 

1198
01:08:33,960 --> 01:08:37,399
Snowflake is basically you take 
your data, you have a set of raw

1199
01:08:37,399 --> 01:08:39,439
data, right? 
And then you have derivative 

1200
01:08:39,439 --> 01:08:41,600
tables that are around the 
Snowflake, right? 

1201
01:08:41,600 --> 01:08:45,160
That are their derivative 
intelligence that you've derived

1202
01:08:45,160 --> 01:08:47,880
from the data. 
Then you take those derivative 

1203
01:08:47,880 --> 01:08:51,000
tables and you sort of join them
together and you make Snowflake 

1204
01:08:51,000 --> 01:08:55,760
tables out at the edge, right? 
And so we did that with my 

1205
01:08:55,760 --> 01:08:59,359
sequel back in the day with the 
what was at the time a very 

1206
01:08:59,359 --> 01:09:02,600
large database, 3 terabytes. 
Which is one of the reasons why 

1207
01:09:02,600 --> 01:09:07,120
I love observability so much is 
because I built that database 

1208
01:09:07,520 --> 01:09:11,240
and we put so much data in it 
that it would take three weeks 

1209
01:09:11,240 --> 01:09:15,760
to do like a column add, right. 
With observability with the 

1210
01:09:15,920 --> 01:09:19,600
these new time series databases 
that we've we've come up with 

1211
01:09:20,120 --> 01:09:23,600
over time, unfortunately didn't 
have them at the time, but you 

1212
01:09:23,600 --> 01:09:25,240
know, it becomes a columnar 
format. 

1213
01:09:25,240 --> 01:09:28,040
You can very easily change the 
data that's flowing in. 

1214
01:09:28,720 --> 01:09:30,399
You could have new 
infrastructure that just 

1215
01:09:30,399 --> 01:09:33,240
automatically starts getting 
scraped or pulled or whatever. 

1216
01:09:33,240 --> 01:09:36,960
And it, it, it's really cool. 
I, I really find the, the, the 

1217
01:09:36,960 --> 01:09:39,680
concept of a time series 
database just completely 

1218
01:09:39,680 --> 01:09:43,640
fascinating because I've been 
completely, you know, scared to 

1219
01:09:43,640 --> 01:09:47,680
touch any part of my database 
because it was so fixed. 

1220
01:09:48,240 --> 01:09:50,160
Yeah, it would take three weeks 
to to modify. 

1221
01:09:50,160 --> 01:09:53,720
Well that's wild. 
Like doing a column change or 

1222
01:09:53,920 --> 01:09:55,800
anything. 
If it takes 3 weeks and then it 

1223
01:09:55,800 --> 01:09:59,400
doesn't work the way you expect 
it to, then you got another 

1224
01:09:59,400 --> 01:10:04,560
three weeks to test the fix. 
Yeah, it's a very, it's not 

1225
01:10:04,560 --> 01:10:07,120
exactly what you'd call a tight 
iteration cycle. 

1226
01:10:07,560 --> 01:10:11,520
Yeah, yeah, yeah, yeah. 1 oops 
cost you 3 weeks. 

1227
01:10:11,840 --> 01:10:13,600
That's crazy. 
Yeah. 

1228
01:10:14,080 --> 01:10:18,720
It's an expensive 3 weeks. 
It is, yeah. 

1229
01:10:19,520 --> 01:10:21,080
Yeah. 
And that's, again, if you 

1230
01:10:21,080 --> 01:10:24,120
compress it down, right, your 
data gets smaller and you make 

1231
01:10:24,120 --> 01:10:27,640
it columnar so that you can 
easily change what data goes in,

1232
01:10:27,640 --> 01:10:31,160
what data doesn't, and it 
compresses it so nicely. 

1233
01:10:32,040 --> 01:10:35,240
I'm just very fond of it. 
And I've always kind of wanted 

1234
01:10:35,240 --> 01:10:38,080
to write 1. 
And so it's been a really good 

1235
01:10:38,080 --> 01:10:39,880
experience for me to finally get
to do that. 

1236
01:10:39,880 --> 01:10:43,640
I read the the the papers that 
came out for Gorilla when they 

1237
01:10:43,640 --> 01:10:46,160
first came out. 
You know, Facebook published a 

1238
01:10:46,160 --> 01:10:50,600
paper on Gorilla probably 15 
years ago and I found it all 

1239
01:10:50,840 --> 01:10:53,400
very fascinating. 
And actually writing one finally

1240
01:10:53,400 --> 01:10:56,440
is is been quite. 
It's very rewarding. 

1241
01:10:56,680 --> 01:10:59,040
Yeah, that's cool. 
Well, I mean, I feel like that's

1242
01:10:59,040 --> 01:11:01,800
the only way to learn is to 
build it, you know? 

1243
01:11:01,920 --> 01:11:04,080
That's the primary reason I 
didn't necessarily want to 

1244
01:11:04,080 --> 01:11:08,520
release it, but the performance 
was so good, like I had pretty 

1245
01:11:08,520 --> 01:11:12,640
high expectations for myself. 
I ended up with the something 

1246
01:11:12,640 --> 01:11:15,560
that performs relatively well. 
We're at least, you know, as 

1247
01:11:15,560 --> 01:11:17,480
fast as Victoria metrics or 
probably. 

1248
01:11:17,480 --> 01:11:19,320
Now you got the bragging rights.
You have to. 

1249
01:11:19,360 --> 01:11:22,120
The second fastest, Yeah, we're 
probably the second fastest 

1250
01:11:22,120 --> 01:11:24,640
database as far as ingestion and
and so that's awesome. 

1251
01:11:24,680 --> 01:11:27,360
That's very cool. 
So yeah, and not bad. 

1252
01:11:28,840 --> 01:11:32,480
Guys, he's only been in the 
elixir world for two years. 

1253
01:11:32,880 --> 01:11:33,880
Two years? 
Yeah. 

1254
01:11:33,880 --> 01:11:36,320
He's blowing our pants off. 
He's making us look bad. 

1255
01:11:38,160 --> 01:11:41,320
That's awesome. 
That's so cool, and now it's 

1256
01:11:41,360 --> 01:11:44,360
going to be officially an A hex 
package this afternoon. 

1257
01:11:45,200 --> 01:11:48,560
Yes, this afternoon I should be 
able to publish the The Timeless

1258
01:11:48,560 --> 01:11:49,200
Phoenix. 
Go ahead. 

1259
01:11:49,440 --> 01:11:51,480
How many years have you been 
writing this? 

1260
01:11:52,880 --> 01:11:56,680
2 Yeah, it was I started. 
Oh, so you like started it as 

1261
01:11:56,680 --> 01:11:58,880
you learned Elixir? 
Yeah, OK. 

1262
01:11:58,880 --> 01:12:02,320
Yeah, pieces, you know, again, I
read that that gorilla paper, 

1263
01:12:02,520 --> 01:12:04,120
you know, back in the day. 
And so the first thing that I 

1264
01:12:04,120 --> 01:12:06,480
did was gorilla. 
And of course along the way I 

1265
01:12:06,480 --> 01:12:09,760
decided gorilla wasn't a good 
enough compression algorithm so 

1266
01:12:09,760 --> 01:12:12,240
I threw away that code. 
But it, it did start two years 

1267
01:12:12,240 --> 01:12:15,200
ago when I started writing the 
Gorilla compression algorithm 

1268
01:12:16,320 --> 01:12:19,680
and, and you know, we use much 
better compression now. 

1269
01:12:20,080 --> 01:12:23,560
It's faster. 
It's it's, you know, better in 

1270
01:12:23,560 --> 01:12:27,480
every way than Gorilla, But but 
get started with Gorilla. 

1271
01:12:27,920 --> 01:12:39,080
OK, that's cool. 
All right, I'm what? 

1272
01:12:41,080 --> 01:12:45,040
What does your like supervision 
tree and things look like? 

1273
01:12:45,040 --> 01:12:47,520
Like how are you handling? 
How are you handling that side 

1274
01:12:47,520 --> 01:12:50,520
of things? 
Cuz we talk about, we talk about

1275
01:12:50,560 --> 01:12:52,880
how awesome Elixir is at 
orchestrating. 

1276
01:12:52,880 --> 01:12:57,080
But what does that mean, right? 
That's a really good question 

1277
01:12:57,080 --> 01:12:59,000
and something I kind of wanted 
to cover on this. 

1278
01:12:59,600 --> 01:13:02,800
The nice thing about having 
supervision trees, it means that

1279
01:13:02,800 --> 01:13:06,720
you have layers of isolation for
your application and you can do 

1280
01:13:06,720 --> 01:13:10,560
things right that you wouldn't 
necessarily do with C or go, 

1281
01:13:10,920 --> 01:13:12,360
right? 
So if you're, if you're 

1282
01:13:12,360 --> 01:13:15,640
developing a, a time series 
database and, and go, you're 

1283
01:13:15,640 --> 01:13:19,240
going to do that in isolation. 
If you are developing a log 

1284
01:13:19,240 --> 01:13:21,000
database and go, you're going to
do that in isolation. 

1285
01:13:21,000 --> 01:13:23,120
If you're going to do that in 
isolation, if you're going to do

1286
01:13:23,120 --> 01:13:24,280
that in isolation, you're going 
to do that. 

1287
01:13:24,280 --> 01:13:25,960
You know, for traces, you're 
going to do it in isolation. 

1288
01:13:25,960 --> 01:13:28,240
You're going to end up with 
three different services because

1289
01:13:28,480 --> 01:13:31,080
you don't want one service 
crashing and taking down the 

1290
01:13:31,080 --> 01:13:35,080
entire thing, right? 
But with Elixir, doesn't cost 

1291
01:13:35,080 --> 01:13:37,200
you anything, right? 
So you can just throw all those 

1292
01:13:37,200 --> 01:13:41,480
services in one, excuse me, with
the supervision tree. 

1293
01:13:41,880 --> 01:13:45,200
And then, you know, if if one of
them goes down, it's just the 

1294
01:13:45,200 --> 01:13:47,360
one that goes down and it comes 
right back up, right? 

1295
01:13:47,440 --> 01:13:49,200
Yeah. 
Yeah. 

1296
01:13:49,200 --> 01:13:52,200
We do that on the DHCP server. 
We do that on all of our our 

1297
01:13:52,200 --> 01:13:54,680
services. 
We, we piled every one of the 

1298
01:13:54,680 --> 01:13:59,480
DOCSIS services into a single 
Elixir app and we don't worry 

1299
01:13:59,480 --> 01:14:02,520
about 1 going down and, and 
taking down the entire app 

1300
01:14:02,520 --> 01:14:05,800
because it won't, right? 
Supervision trees are amazing. 

1301
01:14:06,200 --> 01:14:09,120
We even have sort of sort of 
advanced supervision trees on 

1302
01:14:09,120 --> 01:14:13,440
the, the DOCSIS app side where 
you can stop and start services 

1303
01:14:13,600 --> 01:14:14,800
and it's all green threads, 
right? 

1304
01:14:14,800 --> 01:14:17,120
So the services immediately 
stopped and they immediately 

1305
01:14:17,120 --> 01:14:19,680
start right back up, which is 
pretty cool to see. 

1306
01:14:20,040 --> 01:14:22,400
It's not like running a, a 
binary, right? 

1307
01:14:23,560 --> 01:14:26,400
So. 
You're kind of you're, you're 

1308
01:14:26,400 --> 01:14:28,720
running like a dynamic 
supervision tree and then 

1309
01:14:28,720 --> 01:14:31,160
dynamic processes off of that as
well. 

1310
01:14:31,160 --> 01:14:32,840
Yep. 
Yeah, that's exactly how it's 

1311
01:14:32,880 --> 01:14:38,520
done. 
Yeah, yeah, the yeah, the the 

1312
01:14:38,520 --> 01:14:43,080
actor model and like just the 
fault tolerance of Elixir, I 

1313
01:14:43,080 --> 01:14:45,640
guess. 
I can't say of Elixir of of 

1314
01:14:45,840 --> 01:14:49,520
Beam. 
VM is freaking magical honestly.

1315
01:14:51,160 --> 01:14:54,320
I used to have so much respect. 
So I, I've worked in telco for a

1316
01:14:54,320 --> 01:14:57,040
long time, right? 
So we had five E switches and we

1317
01:14:57,040 --> 01:14:59,640
didn't have Ericsson switches. 
We did have 5E switches back in 

1318
01:14:59,640 --> 01:15:02,320
the day. 
And we had a ton of respect for 

1319
01:15:02,320 --> 01:15:04,520
those guys. 
You know, the, the, the five 

1320
01:15:04,520 --> 01:15:08,800
nines, the 10, you know, 6 nines
or whatever of up time that they

1321
01:15:08,800 --> 01:15:11,440
were able to obtain. 
And you know, they were always 

1322
01:15:11,440 --> 01:15:13,760
in the data center and they were
sitting there by themselves, you

1323
01:15:13,760 --> 01:15:15,440
know, just just doing their 
thing. 

1324
01:15:15,440 --> 01:15:16,880
But I always had tons of 
respect. 

1325
01:15:17,200 --> 01:15:19,600
I was a data engineer and they 
were the telephony guys, right? 

1326
01:15:19,760 --> 01:15:23,400
OK. 
And you know, it's, it's nice 

1327
01:15:23,400 --> 01:15:25,960
actually working with the 
software that they were working 

1328
01:15:25,960 --> 01:15:28,040
with at the time, even though 
it's not the same, right. 

1329
01:15:28,040 --> 01:15:30,080
It's a 5 E switch versus an 
Ericsson switch. 

1330
01:15:30,520 --> 01:15:35,360
But but still switch technology 
is is phenomenal and and I think

1331
01:15:35,360 --> 01:15:37,400
we should not take it for 
granted. 

1332
01:15:37,400 --> 01:15:40,680
It's really good stuff. 
I mean, we take it for granted 

1333
01:15:40,680 --> 01:15:42,360
for sure. 
Yeah. 

1334
01:15:42,640 --> 01:15:46,800
But like we were able to do that
because of how well it just, it 

1335
01:15:46,800 --> 01:15:48,560
just works. 
Just works. 

1336
01:15:49,120 --> 01:15:50,360
Yeah. 
It's fantastic. 

1337
01:15:51,160 --> 01:15:56,400
Did you like when you were 
saying you said 6 to 10 nines? 

1338
01:15:57,520 --> 01:15:59,520
Oh yeah, they've up time like 
that's. 

1339
01:15:59,880 --> 01:16:02,080
I think there's an Ericsson 
switch that ran for decades, 

1340
01:16:02,080 --> 01:16:04,760
right? 
There's like probably multiple 

1341
01:16:04,760 --> 01:16:07,200
decades of up time. 
Yeah, it's pretty crazy. 

1342
01:16:07,720 --> 01:16:09,920
That's wild. 
What's crazy is like there's no 

1343
01:16:09,920 --> 01:16:12,440
hardware failure. 
That's even more. 

1344
01:16:12,440 --> 01:16:14,040
Well, they have redundant 
hardware. 

1345
01:16:14,400 --> 01:16:17,480
OK, it's, yeah, it's the way 
it's designed. 

1346
01:16:17,520 --> 01:16:20,600
It has redundant hardware, they 
can pull pull boards out if they

1347
01:16:20,600 --> 01:16:23,560
go bad and put boards back in 
and it's hot swappable. 

1348
01:16:23,560 --> 01:16:26,760
OK, yeah, that makes. 
Sense it was, it was really 

1349
01:16:26,760 --> 01:16:30,080
advanced stuff for the time, you
know, and I, you know, that was 

1350
01:16:30,240 --> 01:16:32,320
again, that was the beginning of
my career. 

1351
01:16:32,320 --> 01:16:35,520
So that was 30 years ago. 
Yeah, yeah, that's wild. 

1352
01:16:35,640 --> 01:16:38,600
They solved everything in the 
mid 80s, really. 

1353
01:16:38,640 --> 01:16:41,680
And then we just keep getting 
new problems that have the same 

1354
01:16:41,680 --> 01:16:45,440
pattern, which is cool. 
Exactly. 

1355
01:16:46,400 --> 01:16:48,080
Why? 
It's funny watching all the 

1356
01:16:48,080 --> 01:16:51,840
banter on like X with all the 
LLM companies of like, what's 

1357
01:16:51,840 --> 01:16:56,200
the best tech stack now for 
agents and all this and I don't 

1358
01:16:56,200 --> 01:16:57,480
know. 
Are you familiar with Mike 

1359
01:16:57,640 --> 01:17:00,600
Hostetler? 
Oh yeah, yeah, I. 

1360
01:17:02,080 --> 01:17:05,160
He he likes to, he likes to get 
in there and get all 

1361
01:17:05,160 --> 01:17:07,040
controversial. 
It's fun to watch. 

1362
01:17:07,360 --> 01:17:10,880
I sit back, you know, I just 
like to be, I'll be the nice 

1363
01:17:10,880 --> 01:17:15,920
face of Elixir and but it's fun 
to watch him poke, poke the 

1364
01:17:15,920 --> 01:17:19,000
bear. 
Yeah, and that's why I, I agree 

1365
01:17:19,000 --> 01:17:20,880
with everything. 
And he says I think Elixir is a 

1366
01:17:20,880 --> 01:17:23,840
fantastic environment, 
especially for agents and stuff 

1367
01:17:23,840 --> 01:17:25,760
like that. 
But unfortunately I'm on the 

1368
01:17:25,760 --> 01:17:28,320
performance side, right? 
So I, I had to do some Nifs, 

1369
01:17:28,320 --> 01:17:33,360
but, but yeah, I think the 
performance is, you know, 1/4 of

1370
01:17:33,360 --> 01:17:36,760
what you get like in Go, that's 
still very, very good. 

1371
01:17:36,760 --> 01:17:41,160
That's 10 times faster than any 
interpreted language, right? 

1372
01:17:41,160 --> 01:17:44,080
Yeah. 
So the performance is good 

1373
01:17:44,080 --> 01:17:47,120
enough for just about anything 
you wanna do, except for maybe 

1374
01:17:47,120 --> 01:17:48,120
build a database. 
Right. 

1375
01:17:48,560 --> 01:17:50,000
Right. 
Well, I feel like it's 

1376
01:17:50,000 --> 01:17:52,360
fantastic. 
I feel like he would agree with 

1377
01:17:52,360 --> 01:17:54,840
you there, right? 
Like you're using Elixir for the

1378
01:17:54,840 --> 01:17:58,200
orchestration and like, the 
process capability, like 

1379
01:17:58,440 --> 01:18:00,960
spinning up thousands of 
processes like nothing. 

1380
01:18:00,960 --> 01:18:02,960
Right. 
But you still mind things. 

1381
01:18:03,280 --> 01:18:05,800
Yeah. 
Depending on what kind of data 

1382
01:18:05,800 --> 01:18:09,320
you're pounding through, you 
might need an. 

1383
01:18:09,320 --> 01:18:11,920
If you might need to use Rust, 
you might need to use something 

1384
01:18:11,920 --> 01:18:16,480
that's just more efficient at, 
you know, heavy calculations in 

1385
01:18:16,480 --> 01:18:20,200
math. 
And Elixir is a polyglot, right?

1386
01:18:20,240 --> 01:18:24,280
You could write Lua, you could 
write Python, you could write 

1387
01:18:24,320 --> 01:18:28,200
Rust. 
You can write what's Zig, right?

1388
01:18:28,600 --> 01:18:33,480
Ziglar CC, It's it's whatever 
you want to do. 

1389
01:18:33,800 --> 01:18:36,120
You can pretty much do it in 
Elixir and just peel off the 

1390
01:18:36,120 --> 01:18:39,240
parts that need just a tiny bit 
more performance. 

1391
01:18:39,240 --> 01:18:41,080
You just peel that off and do a 
nif, right? 

1392
01:18:41,960 --> 01:18:44,240
Yeah. 
How do you, how do you go, how 

1393
01:18:44,240 --> 01:18:47,360
did you go about like breaking 
down like what needed the 

1394
01:18:47,360 --> 01:18:50,520
performance boosts like how do 
you how do you find those 

1395
01:18:50,520 --> 01:18:53,840
problem spots and how do you 
tackle those problems? 

1396
01:18:54,800 --> 01:18:59,240
Most of them were obvious, like 
you know when I started trying 

1397
01:18:59,240 --> 01:19:01,080
to. 
So you, what you do when you 

1398
01:19:01,080 --> 01:19:05,000
ingest metric data is you have a
set of tags for this particular 

1399
01:19:05,000 --> 01:19:08,960
piece of data and then you have 
a value and those tags that look

1400
01:19:08,960 --> 01:19:12,880
up takes time, right? 
That's, that's the, that's the 

1401
01:19:13,360 --> 01:19:15,320
sort of the bulk of what takes 
time. 

1402
01:19:16,160 --> 01:19:19,720
And so, you know, I first did it
in Ed's tables, then I, you 

1403
01:19:19,880 --> 01:19:24,280
know, and just, you know, a lot 
of different, I tried 

1404
01:19:24,280 --> 01:19:27,040
everything. 
But but yeah, we first did it in

1405
01:19:27,040 --> 01:19:28,480
Ed's. 
And you know, that look up would

1406
01:19:28,480 --> 01:19:31,480
take time because it is such a 
kind of a long string, right? 

1407
01:19:31,960 --> 01:19:37,080
And and so it was, it became 
obvious, you know, it was, I 

1408
01:19:37,080 --> 01:19:41,880
don't want to say that I did 
some grand deep dive into the 

1409
01:19:41,880 --> 01:19:44,680
beam and figure out exactly what
the performance issues were. 

1410
01:19:44,680 --> 01:19:47,480
I kind of knew. 
You just saw your bottlenecks 

1411
01:19:47,480 --> 01:19:51,360
just with time like the query 
length or whatever. 

1412
01:19:51,360 --> 01:19:55,160
I guess it would be a query. 
And I also have the ability so 

1413
01:19:56,480 --> 01:19:59,840
and someday, and this day is not
today, but someday, I'm going to

1414
01:19:59,840 --> 01:20:03,720
have the ability to already know
what your metric ID is going to 

1415
01:20:03,720 --> 01:20:05,680
be, right? 
So you have to derive that long 

1416
01:20:05,680 --> 01:20:07,760
string into an ID and then store
the metric. 

1417
01:20:08,160 --> 01:20:11,120
And so if you just store the ID 
and the metric, I've gotten up 

1418
01:20:11,120 --> 01:20:13,560
to like 13 million inserts per 
second, right? 

1419
01:20:13,960 --> 01:20:18,840
So, you know, if you just do 
that and you have something that

1420
01:20:18,840 --> 01:20:20,840
collects the data, that's 
intelligent enough. 

1421
01:20:21,560 --> 01:20:25,920
Sorry, my dog's about to bark. 
You have something that's 

1422
01:20:26,080 --> 01:20:28,960
intelligent enough to collect 
the data in such a way that it 

1423
01:20:28,960 --> 01:20:31,480
knows that that ID exists. 
You could actually insert it, 

1424
01:20:31,600 --> 01:20:34,120
you know, 1313 million inserts 
per second. 

1425
01:20:34,680 --> 01:20:37,600
But but yeah, so it makes it 
obvious, right? 

1426
01:20:37,600 --> 01:20:40,240
If you can insert that at 13 
million per second and you're 

1427
01:20:40,240 --> 01:20:42,480
only at 3,000,000 per second. 
If you have to do that, look up 

1428
01:20:42,840 --> 01:20:44,360
your bottlenecks. 
The look up right? 

1429
01:20:44,800 --> 01:20:47,400
It's, it's pretty easy to 
determine what your, your 

1430
01:20:47,400 --> 01:20:51,400
performance penalty is. 
And so, yeah. 

1431
01:20:51,440 --> 01:20:53,400
And so unfortunately it was 
easy. 

1432
01:20:53,440 --> 01:20:56,240
I, I wish I could give you some 
insight on, you know, how to 

1433
01:20:57,080 --> 01:21:00,280
find performance issues in the 
beam, but it was for us, it was,

1434
01:21:00,480 --> 01:21:01,760
it was obvious. 
Yeah. 

1435
01:21:02,120 --> 01:21:05,400
Well, if there you always want 
to know, if you want to know 

1436
01:21:05,400 --> 01:21:07,920
everyone's tricks, right? 
Like, that's where you learn, 

1437
01:21:07,960 --> 01:21:10,000
that's where you learn the juicy
stuff, you know? 

1438
01:21:11,360 --> 01:21:12,920
Don't have any tricks 
unfortunately. 

1439
01:21:14,360 --> 01:21:20,440
How do you, how are you like 
pushing your boundaries and how 

1440
01:21:20,440 --> 01:21:23,240
are you keeping those metrics? 
Like what kind of test system do

1441
01:21:23,240 --> 01:21:25,960
you have to like know what your 
limits are? 

1442
01:21:27,080 --> 01:21:31,840
So you know I was doing, I did 
most of the development on a air

1443
01:21:31,960 --> 01:21:36,520
right AM 4 air. 
And so it was OK at sort of 

1444
01:21:36,520 --> 01:21:41,400
judging performance. 
I did upgrade to the M5 Pro with

1445
01:21:41,400 --> 01:21:45,240
the maximum number of CPU's, but
you know, lower, lower number of

1446
01:21:45,440 --> 01:21:47,600
GPU's because I don't, I don't 
need GPU cores. 

1447
01:21:48,440 --> 01:21:52,160
So I could sort of test, you 
know, with, you know, best of 

1448
01:21:52,160 --> 01:21:55,480
breed processors. 
I also have an AWS account. 

1449
01:21:55,480 --> 01:22:01,240
And so I, I did spin up 192 core
ARM instance and try this 

1450
01:22:01,240 --> 01:22:07,200
against Victoria metrics. 
And then I also did a 92 core 

1451
01:22:07,200 --> 01:22:12,240
A-96 core with against Victoria 
metrics and we did well. 

1452
01:22:12,640 --> 01:22:14,880
We didn't actually beat them, 
but we did well. 

1453
01:22:15,360 --> 01:22:17,800
And I really feel strongly that 
we're probably the second 

1454
01:22:17,800 --> 01:22:20,920
fastest time series database. 
The time will come to be #1 I 

1455
01:22:20,920 --> 01:22:24,400
believe in you. 
Yeah, eventually maybe we'll get

1456
01:22:24,400 --> 01:22:27,800
there. 
But yeah, it's, it's the 

1457
01:22:27,800 --> 01:22:30,560
performance is pretty phenomenal
and you'll see it when you start

1458
01:22:30,560 --> 01:22:33,760
to use your app, especially if 
it's, you know, you're not doing

1459
01:22:33,760 --> 01:22:36,960
millions of inserts per second. 
It the the query performance is 

1460
01:22:36,960 --> 01:22:40,240
phenomenal. 
It's it's way sub millisecond. 

1461
01:22:40,240 --> 01:22:43,240
It's like we're in like 10s of 
microseconds. 

1462
01:22:43,400 --> 01:22:46,280
That's why. 
I feel like like running your 

1463
01:22:46,280 --> 01:22:49,360
benchmarks is like the hardest 
part because. 

1464
01:22:49,560 --> 01:22:51,520
It is when you have to do that, 
right? 

1465
01:22:51,680 --> 01:22:53,640
Every little thing that you 
change, you have to run all the 

1466
01:22:53,640 --> 01:22:56,040
benchmarks to make sure that you
didn't affect it too much, 

1467
01:22:56,080 --> 01:22:58,760
right? 
And so, yeah, benchmarks become 

1468
01:22:58,760 --> 01:23:01,160
a big part of your life when 
you're doing database stuff. 

1469
01:23:01,840 --> 01:23:05,640
Do you set up tests for like 
time benchmark, like speed 

1470
01:23:05,840 --> 01:23:09,520
benchmarks and things like that?
Like how do you know if like 

1471
01:23:09,800 --> 01:23:13,360
something you just tweaked 
actually after everything up? 

1472
01:23:14,320 --> 01:23:17,960
I probably should I, they 
probably should I, but it's, 

1473
01:23:18,080 --> 01:23:20,640
it's all manual. 
You know, I'll, I'll finish the 

1474
01:23:20,640 --> 01:23:23,480
session and I'll, I'll run a 
benchmark and make sure that I 

1475
01:23:23,480 --> 01:23:26,200
didn't regress. 
I usually do it on a branch 

1476
01:23:26,200 --> 01:23:28,800
because I worry so much about 
performance regression. 

1477
01:23:29,520 --> 01:23:32,240
I'll do it on a branch. 
I'll, I'll make sure and test 

1478
01:23:32,240 --> 01:23:35,440
before I merge. 
So I know exactly. 

1479
01:23:35,880 --> 01:23:38,240
And then sometimes you just take
performance penalties. 

1480
01:23:38,680 --> 01:23:41,560
Like sometimes it's just, well, 
you know, this was valuable 

1481
01:23:41,560 --> 01:23:44,240
enough that I'm gonna take a 10%
hit on my performance. 

1482
01:23:44,240 --> 01:23:46,360
But you know, it makes it a more
robust. 

1483
01:23:46,360 --> 01:23:47,920
Service. 
How often has that happened 

1484
01:23:47,920 --> 01:23:51,040
where you've had to take the 
performance hit for something 

1485
01:23:51,040 --> 01:23:52,920
like a few times? 
Yeah, a few times. 

1486
01:23:53,240 --> 01:23:55,400
Yeah, cuz I feel like especially
on the rest side. 

1487
01:23:55,600 --> 01:23:59,400
Your whole, your whole thing 
though, like performance is so 

1488
01:23:59,400 --> 01:24:01,760
important. 
So it's, it's a hard decision, 

1489
01:24:01,760 --> 01:24:03,680
right? 
It is. 

1490
01:24:03,720 --> 01:24:08,080
It's a hard decision and you 
know, again, my, it's, it's made

1491
01:24:08,080 --> 01:24:12,520
me, it's made it easier on me 
just knowing that embedded is my

1492
01:24:12,600 --> 01:24:14,960
my goal, right, at least 
initially. 

1493
01:24:15,200 --> 01:24:18,720
So if embedded is your goal, 
then performance, there's no way

1494
01:24:18,760 --> 01:24:21,400
that any one of these apps, I 
mean, you could have a server 

1495
01:24:21,400 --> 01:24:30,680
that's 192 cores taken 200,000 
HTTP hits a second and it still 

1496
01:24:30,680 --> 01:24:34,320
couldn't generate enough logs 
for you to cough right. 

1497
01:24:34,720 --> 01:24:40,240
And so having that as a, your 
baseline and, and, and you know,

1498
01:24:40,240 --> 01:24:44,560
shooting for performance numbers
that are so high, it makes it 

1499
01:24:44,560 --> 01:24:46,960
easier, right? 
So you can say, well, I can take

1500
01:24:46,960 --> 01:24:51,880
a 10% to 10% performance hit on 
this and it still will be good 

1501
01:24:51,880 --> 01:24:54,560
enough for 99% of the users. 
Right. 

1502
01:24:54,880 --> 01:25:00,960
So yeah, yeah, that's cool. 
I feel like yeah, like your 

1503
01:25:00,960 --> 01:25:04,600
benchmarking was probably the 
hardest process to set up too. 

1504
01:25:05,800 --> 01:25:08,640
And I did lots of different 
kinds of benchmarks. 

1505
01:25:09,000 --> 01:25:12,280
I, I would do like a sustained 
load on queries and a sustained 

1506
01:25:12,280 --> 01:25:15,680
load on inserts and see how they
affected each other, right? 

1507
01:25:16,240 --> 01:25:19,400
Because usually with metrics 
databases, they're different, 

1508
01:25:19,400 --> 01:25:21,120
right? 
So usually with a normal 

1509
01:25:21,120 --> 01:25:26,040
database, which is what I deal 
with in my day job, you're 9995%

1510
01:25:26,040 --> 01:25:35,720
reads and and 5% writes, whereas
this is 9995% writes, 5% reads. 

1511
01:25:36,160 --> 01:25:38,440
And so it's a very different way
to look at databases. 

1512
01:25:38,440 --> 01:25:40,600
But I wanted the read 
performance to be very good. 

1513
01:25:40,600 --> 01:25:44,640
And so I, I really did focus on 
that and, you know, kept the 

1514
01:25:44,640 --> 01:25:46,680
read performance really good 
too, so. 

1515
01:25:47,000 --> 01:25:48,160
That's good. 
Yeah. 

1516
01:25:48,160 --> 01:25:52,560
I could see that kind of being 
ignored because you'd be more 

1517
01:25:52,560 --> 01:25:57,480
willing to wait on getting data 
back to investigate issues than 

1518
01:25:57,480 --> 01:26:00,360
obviously writing data because 
you need all your logs, you 

1519
01:26:00,360 --> 01:26:04,040
can't miss any. 
But I also still had that in the

1520
01:26:04,040 --> 01:26:06,720
back of my head, right? 
I wanted that canvas where you 

1521
01:26:06,720 --> 01:26:08,360
could drop your graphs on, 
right? 

1522
01:26:08,640 --> 01:26:11,120
All of all of the graphs that 
are, they're relevant to 

1523
01:26:11,120 --> 01:26:14,880
whatever you're looking at and, 
and grab that timeline and go 

1524
01:26:14,880 --> 01:26:17,640
back and, and have every one of 
those graphs repopulate as 

1525
01:26:17,640 --> 01:26:19,200
quickly as possible, right? 
Yeah, that's cool. 

1526
01:26:19,200 --> 01:26:23,000
So I did have some, some 
performance goals in mind for 

1527
01:26:23,000 --> 01:26:25,400
that. 
And I think everyone's, you 

1528
01:26:25,400 --> 01:26:30,160
know, I love the the canvas, but
I don't know that it's going to 

1529
01:26:30,160 --> 01:26:32,360
be something that everyone 
embraces. 

1530
01:26:32,360 --> 01:26:34,960
But you know, you always have 
Grafana. 

1531
01:26:34,960 --> 01:26:37,480
You can always go back to 
Grafana and do your normal, you 

1532
01:26:37,480 --> 01:26:41,160
know, trace, look up or your, 
your graphing or your dashboards

1533
01:26:41,160 --> 01:26:45,280
or whatever. 
But but I wanted it to perform, 

1534
01:26:45,400 --> 01:26:46,600
you know, from a read 
perspective. 

1535
01:26:46,600 --> 01:26:49,640
I was, I was, I was really 
hoping for really good 

1536
01:26:49,640 --> 01:26:51,840
performance. 
Yeah, that's cool. 

1537
01:26:51,920 --> 01:26:56,280
We just started using Grafana at
the day job, so I'm like kind of

1538
01:26:56,280 --> 01:26:58,160
learning that. 
It's not too complicated, but 

1539
01:26:58,560 --> 01:27:01,000
it's pretty cool with the amount
of data you can have. 

1540
01:27:03,040 --> 01:27:05,760
Yeah, it tends to be convoluted 
like it. 

1541
01:27:05,840 --> 01:27:09,280
It has a lot of little knobs and
buttons and things, and I think 

1542
01:27:09,280 --> 01:27:12,040
that that that doesn't work well
for a lot of people. 

1543
01:27:12,760 --> 01:27:14,880
I tried for something different,
yeah. 

1544
01:27:15,160 --> 01:27:18,640
And I I've achieved it. 
Hopefully I'll release the the 

1545
01:27:18,640 --> 01:27:22,040
server component with the canvas
engine later. 

1546
01:27:22,800 --> 01:27:25,160
And it will, it'll be hopefully 
people will like it. 

1547
01:27:25,160 --> 01:27:27,080
We'll see. 
I think people will love it. 

1548
01:27:28,880 --> 01:27:30,560
I I'm excited to get my hands 
on. 

1549
01:27:30,560 --> 01:27:34,280
I'm going to put Timeless into 
kill switch today, this 

1550
01:27:34,280 --> 01:27:35,160
afternoon. 
Sweet. 

1551
01:27:36,000 --> 01:27:37,320
And I'll give it around. 
That's awesome. 

1552
01:27:37,400 --> 01:27:39,520
I'll play around with it. 
Our first user. 

1553
01:27:41,240 --> 01:27:44,440
I don't know we we got we got a 
couple people here just waiting 

1554
01:27:44,440 --> 01:27:49,920
for it. 
Let's see, Daniel said. 

1555
01:27:49,920 --> 01:27:54,960
Elixir plus C equals conquer the
world, man Yeah CC is a good 

1556
01:27:56,160 --> 01:27:59,480
that's a good happy. 
Lifter Yeah, there's a there's a

1557
01:28:00,160 --> 01:28:03,640
web server called Rocket that I 
developed specifically for this 

1558
01:28:03,640 --> 01:28:06,320
project so that I could get my 
latency down for 

1559
01:28:06,320 --> 01:28:11,200
Http://requests, and it's all C.
And it's nice and it's. 

1560
01:28:11,200 --> 01:28:12,960
Fast. 
Yeah, it's very fast. 

1561
01:28:13,440 --> 01:28:15,520
It's 30 times faster than 
Bandit. 

1562
01:28:15,960 --> 01:28:18,320
Wow, that's. 
From a latency perspective, 

1563
01:28:18,440 --> 01:28:20,520
which is latency is what what I 
cared about, yeah. 

1564
01:28:20,640 --> 01:28:23,360
Right. 
But 30 times, that's a lot. 

1565
01:28:23,400 --> 01:28:26,840
That's a big number. 
Well, and it, you know, it's 

1566
01:28:26,840 --> 01:28:29,320
just, it's just the latency, 
right? 

1567
01:28:29,320 --> 01:28:35,280
So it's three times faster in 
requests per second or something

1568
01:28:35,280 --> 01:28:37,120
like that. 
And depending on your pull size.

1569
01:28:37,600 --> 01:28:40,360
But, but yeah, it's faster. 
Let's just say it's faster. 

1570
01:28:40,640 --> 01:28:42,960
And the latency was what I was 
concerned about because I wanted

1571
01:28:42,960 --> 01:28:45,120
to be able to pull the data out 
as quickly as possible. 

1572
01:28:45,120 --> 01:28:50,680
And so I just bypassed, you 
know, basically Bandit, any of 

1573
01:28:50,680 --> 01:28:55,240
the HTTP, the plug, right? 
All that stuff is bypassed. 

1574
01:28:55,240 --> 01:28:57,920
It just basically interprets 
your request, figures out what 

1575
01:28:57,920 --> 01:29:00,080
you wanted to do and sends the 
data back as quickly as 

1576
01:29:00,080 --> 01:29:01,840
possible. 
It's called rocket. 

1577
01:29:02,000 --> 01:29:04,800
It's, it's released separately 
as an open source project as 

1578
01:29:04,800 --> 01:29:07,520
well. 
But it's, you'd have to know you

1579
01:29:07,520 --> 01:29:10,920
need it, right? 
Because bandit's extremely fast,

1580
01:29:11,040 --> 01:29:13,600
right? 
There's not a lot of people that

1581
01:29:13,600 --> 01:29:16,600
need a a, you know, lower 
latency than Bandit. 

1582
01:29:17,440 --> 01:29:21,880
Yeah, well, it's like you, you 
have to run into these limits 

1583
01:29:21,880 --> 01:29:23,760
before you know you need to fix 
them, right? 

1584
01:29:24,880 --> 01:29:28,320
Yeah, I was doing benchmarking 
and a lot of half of my 

1585
01:29:28,320 --> 01:29:30,680
benchmarking is internal 
metrics, like I'll just push the

1586
01:29:30,680 --> 01:29:32,240
metrics directly into the 
database. 

1587
01:29:32,520 --> 01:29:37,400
The other half is, hey, how am I
going to do against things like 

1588
01:29:37,760 --> 01:29:40,680
Victoria metrics and, and 
Prometheus and those guys, And 

1589
01:29:40,680 --> 01:29:43,760
they all have Http://interfaces.
And so I wanted it to be 

1590
01:29:44,120 --> 01:29:47,200
relatively fast. 
And so I, I, I I thought it was 

1591
01:29:47,200 --> 01:29:48,520
an important part of the 
project. 

1592
01:29:50,000 --> 01:29:54,880
So, so all these systems you're 
using basically like just a REST

1593
01:29:54,880 --> 01:29:59,040
API for taking in data or not 
necessarily? 

1594
01:29:59,520 --> 01:30:03,560
They're well established 
interfaces that I had to conform

1595
01:30:03,560 --> 01:30:05,840
to in order for things like 
Grafada to work. 

1596
01:30:06,200 --> 01:30:09,160
And so I just followed those. 
I there's nothing. 

1597
01:30:09,240 --> 01:30:11,480
Yeah. 
And it's, it's not rest it's 

1598
01:30:11,600 --> 01:30:17,400
similar, but it's, it's there's 
prom QL, I think is the one that

1599
01:30:17,400 --> 01:30:22,960
I, I conform to the most. 
And so you have a certain way 

1600
01:30:22,960 --> 01:30:25,800
that you query and a way that 
you push data in. 

1601
01:30:26,240 --> 01:30:28,880
I see and and I just. 
I've I had to follow those 

1602
01:30:28,880 --> 01:30:32,200
standards in order to work with 
Grafana and and other. 

1603
01:30:32,600 --> 01:30:35,600
That makes sense. 
Open Telemetry pushes metrics, 

1604
01:30:35,640 --> 01:30:39,400
has a certain format that it 
wants to send and so you don't 

1605
01:30:39,400 --> 01:30:41,920
have a lot of. 
Would sockets ever be an option 

1606
01:30:41,920 --> 01:30:44,440
or not really? 
Yeah, I mean it. 

1607
01:30:44,520 --> 01:30:47,600
It'll hold the, it'll hold the 
HTTP connection open and just 

1608
01:30:47,600 --> 01:30:48,480
keep. 
OK. 

1609
01:30:48,520 --> 01:30:49,840
Yeah, yeah. 
Gotcha. 

1610
01:30:50,040 --> 01:30:52,520
It's like web sockets, but it's 
not technically, it's just an 

1611
01:30:52,520 --> 01:30:54,680
HTTP connection. 
I see, got it. 

1612
01:30:54,960 --> 01:31:00,880
OK, excuse my slowness. 
I have to sometimes ask ask the 

1613
01:31:00,880 --> 01:31:04,280
dumb questions. 
It's just like web sockets 

1614
01:31:04,280 --> 01:31:06,400
though it it holds the 
connection and just continues to

1615
01:31:06,400 --> 01:31:09,640
send data or query right? 
OK, cool. 

1616
01:31:10,280 --> 01:31:15,240
Yeah, that's awesome. 
Yeah, I, I think we can wrap 

1617
01:31:15,240 --> 01:31:17,600
this up. 
This has been a lot of fun time.

1618
01:31:17,720 --> 01:31:20,400
The time goes past so fast on 
these things. 

1619
01:31:20,760 --> 01:31:23,280
I don't know how it was for you,
but yeah, me too. 

1620
01:31:23,280 --> 01:31:28,400
And it was great meeting you. 
Do you have any advice for for 

1621
01:31:28,400 --> 01:31:31,800
developers that are just getting
into Elixir, have been learning 

1622
01:31:31,800 --> 01:31:33,840
it a while? 
Just anything. 

1623
01:31:33,960 --> 01:31:38,360
Any. 
Know your strengths, know know 

1624
01:31:38,360 --> 01:31:40,880
what Elixir is good at. 
Really take advantage of that. 

1625
01:31:41,440 --> 01:31:45,600
Don't necessarily pull off nifs 
because you, you feel like you 

1626
01:31:45,600 --> 01:31:48,520
need to. 
I had very specific performance 

1627
01:31:48,520 --> 01:31:53,200
targets and I, I felt like that 
that was necessary for me. 

1628
01:31:53,360 --> 01:31:55,840
But Elixir is a very fast 
language. 

1629
01:31:55,880 --> 01:31:59,880
I think you're going to be fine.
And, you know, oh, ask 

1630
01:31:59,880 --> 01:32:02,160
open-ended questions to your 
LMII. 

1631
01:32:02,160 --> 01:32:05,560
Think that really helped me a 
lot, understand what is out 

1632
01:32:05,560 --> 01:32:09,520
there as far as compression 
technologies and then zero in on

1633
01:32:09,520 --> 01:32:11,400
the ones that are most important
for me. 

1634
01:32:12,400 --> 01:32:14,760
And, you know, they have a lot 
of data. 

1635
01:32:14,880 --> 01:32:17,440
They don't necessarily 
understand it as well as you do,

1636
01:32:18,080 --> 01:32:19,960
but they can get you to the 
right places. 

1637
01:32:20,760 --> 01:32:24,640
Yeah, I love that. 
All right, well, this has been a

1638
01:32:24,640 --> 01:32:26,800
lot of fun. 
Thanks for joining us today, 

1639
01:32:26,800 --> 01:32:30,800
Mark and everyone else. 
Thanks for tuning in and have a 

1640
01:32:30,800 --> 01:32:31,520
good weekend.