This episode is brought to you by Beam OPS.
If your Elixir deployments are painful, or your infrastructure
is a mess, or you just need experience hands to help you
scale, these are your people. Beam OPS specializes in scalable
Elixir development, automated deployment pipelines, and
infrastructure migrations. They will help you work smarter
and ship faster. As authors of the Pragprog book
Engineering Elixir Applications, they know their stuff.
Check them out at beamops.co.uk. Coding Elixir with AI or
building AI into your Elixir apps?
The Elixir AI Collective Discord is your community for both.
Link in the description below. Testing what's up?
I'm Jacob Litzow, your Elixir mentor and welcome to another
exciting episode of the Elixir Mentor podcast.
This is where we discuss everything related to Elixir,
from interviews with enthusiasts, pioneers in the
community to innovative projects and library shaping Elixir's
future. Today I'm joined by Mark Kotner,
telecom engineer, open source contributor, and the person
behind something big in Elixir, observability.
Thanks for joining me today, Mark.
I'm excited to to chat. It's good to be here.
To kick things off, do you want to just tell us a little bit
about yourself, like where you're best, what's your
background and what your day-to-day looks like?
Sure, I'm in Atlanta, GA, my day-to-day is pretty busy.
I have basically 2 full time jobs.
I'm ADBA for Mannheim Auto Auctions, which is largest
auction provider for used cars in the United States and at
night and I have about 1000 databases or so.
And at night and weekends I am a CTO of a startup.
So the startup is we do small cable plants, we do mostly
hospitality networks and you know, 2 to 300 modem small
networks with cable modems. That's awesome.
Very cool. What what ended up getting you
into that into that space? That's where I started.
So I my background is cable. So I started, yeah, my first
cable modem was a one way download only ISA card that you,
you used your dial up for uploads.
So yeah, I've been in the cable industry for a very long time.
I remember dial up. I also remember when like ISDN
came out and it like being permanently connected to
something became a thing that was that was cool.
Yeah, actually. So I, I started very early in
that and I sat for the the DOXIS steering committee for Cox
Communications back in the day. So the DOXIS is the cable modem
protocol that actually operates the cable modems.
So yeah, I've been around it for a very long time.
That's awesome for a minute. Like in high school, it's been a
while. I did some Cisco networking
stuff and I don't remember any of it, but it was, I learned a
lot. I remember that and it was good.
I'm curious what what, what was your background like?
What got you into like the cable modem and then into databases?
And now you're writing Elixir, I'm guessing with the
hospitality stuff, right? I am.
So originally we, you know, I started as sort of a sys admin
network engineer for the cable and home side.
And then I became a specialist and started doing the steering
committee. And as part of that I decided we
needed better observability in our cable networks.
And so I built an application that helped with that.
And so it was used for every single call, like if you were a
Cox cable customer and you called in our app was what
basically helped them troubleshoot your cable modem
and see how it was feeling today and all that stuff.
So that's of course required a very large database.
And so I got a lot of attention in the my sequel space.
This was a my sequel database back in the day, got a lot of
attention back in the day for my sequel.
And so I helped them get their second round of funding, made
them a lot of money. And and so, yeah, just after
that, I was asked by Manheim was having some performance issues
on their databases. They asked me to come over.
Manheim is also part of the same company that Cox Communications
is. And so I came over from Cox
Communications to Manheim to help them with their databases.
And that was, you know, 6-8 databases at the time.
And now it's, you know, 1000. But, but that's how I, I became
ADBA during, you know, my day job is I built an app that
basically gave us observability for cable modems.
So my, my background is definitely observability and
that's why I broke timeless the observability databases, right?
So there's a timeless metrics, which is a metrics database.
There is timeless logs, which is a log compression.
You know, log ingestion allows you to search database.
And then there's the traces, right?
So I, I ingest all the traces and then you can go and search
for traces and look them up and stuff like that.
Very high compression, very, very low CPO overhead, which is
why I sort of pursued the embedded side, which you guys
will see later. And so you can just embed your
observability right in your Phoenix app is the goal.
Yeah, that's really cool. I, yeah, you sent me a demo
right before this and I was trying to watch it, but, but to
be honest, I was like watching it and also multitasking.
So I probably absorbed like 10%. But it seems like it seems
really cool. What?
So like, I guess we can, I'm going to, we'll shift to the
your, your observability engine a little bit here first.
Why that and what makes it so complex?
So I was actually putting together Ansible scripts to
deploy some of our apps in the, on the, the ISP side, the the
CTO slash, you know, my, my night job.
And it was a bit more difficult than I was expecting to get the
Ansible just right with the stack that was, you know, highly
available. It had metrics, you know, I was
using Victoria metrics at the time and which is fantastic.
You know, I, I don't necessarily want to compete with them
because they're quite good at what they do.
But but yeah, so it was just really difficult to sort of get
all the Ansible right to get it in high availability, everything
that I wanted. And I was like, you know what,
this should be easier because 99 Percent, 95% of apps don't
really need external observability stacks.
You know where you're pushing your logs to a server, you're
pushing your traces to a server, you're pushing your metrics to a
server. So, you know, I, I needed a high
performance. It has to be high performance if
you embed it because you don't want it to take away from the
app's ability to serve, right? So it needs to be like Rust or
something like that Wrestler. And so, you know, basically just
a really lightweight observability stack that
compresses really well that you can just run inside your
application. I even added them in an in
memory mode so that you could do it with say, a nerves app.
You don't want to necessarily write metrics to your SD card,
right? Because those things don't
necessarily take a lot of rights.
And so there's an in memory mode.
So you can actually just take the metrics, compress them and
just store them in memory, which is pretty cool as well.
That is cool. I don't remember who I was just
talking to because all my podcast guests kind of mesh, but
that we were talking about. Like if you want to like up your
software engineering game to work on embedded systems because
you have limited resources, which is something we're not
very used to anymore in the software world 'cause we just
throw more and more RAM at it, or more GPU, whatever we might
need. Yeah, there's a, there's a
number of things that I've written that are to run on
routers, which is the same, right?
You just need it to be as small as possible and take up this
lease. The least amount of memory is
possible, so. Yeah, that makes sense.
It's the same same concept. Yep.
And like a router has to always work too.
Like you know, you plug it in and it needs to just start doing
its thing. Yeah, you can't.
You can't interfere with it passing packets, right?
That's the most important thing that it does.
Yeah, yeah, that's cool. I feel like, like I really
appreciate like people that do the observability stuff and
everything because I feel like it's like it's something I don't
want to think about, but it's really nice to have those
metrics and logs when you need them.
Yeah, well, the nice. Go ahead.
Oh, no, I was just. Yeah, continue.
Well, the nice thing about this is it's, you know, it's Igniter,
right? And so you don't have to think
about it like it's, I mean, you probably saw it from the, from
the part of the demo that you, you were able to watch.
It takes less than 5 minutes. It's one of the first things you
do when you start up an app. You just say, hey, you run an
igniter script that does all the wiring for you.
It starts ingesting your logs, it starts ingesting your traces
and it starts ingesting your metrics and it does it all for
you. It's, it's pretty great.
Igniter is just an amazing, amazing addition to the.
Ecosystem igniter is so cool. It's so cool that and usage
rules like those are like, I think my two favorite, like just
they're like such simple libraries that are so powerful.
And the nice thing about this too also is that, you know,
because I wanted it to be embedded, I've actually made the
performance so good that it actually runs really well as a
stand alone. You could actually replace your
entire observability stack from say, Victoria metrics, logs,
traces, right? You could just replace that with
a single server and it would ingest, you know, millions of
entries per second. I was watching the the Tiger,
Tiger Beetle demo this week that that's been making its rounds
around the Internet and they, they talk about a trillion
transactions and all this stuff. And you look at it and it's only
doing a half a million transactions a second, which
sounds like a lot, but it's, it's really not compared to what
metrics ingestion is at today, right?
Yeah. But we get around 3,000,000
metric ingestion like metrics per second.
And we can do a million logs per second, obviously not at the
same time, but each individual server is capable of these,
these and I think traces were up to like 800,000 per second or
something like that. It's it's a big number.
That is cool. Yeah.
We're like, I'm guessing like the compression side of things
is like the huge bottleneck, right?
Or no. It's actually not.
The compression has advanced in so many ways the last few years.
I've obviously been following telemetry for a long time, 30
years now, but but really during COVID, I think every smart
person smarter than me, certainly every smart person
just sat at home and just figured out great ways to
compress things and use CPU offloading.
You know, there's, there's kind of like MMX extensions, right,
For CPUs, Neon for, for ARM 64. There's, there's basically CPUs
do some acceleration there. And so you can actually do it
even down to the pie, you know, even Raspberry Pi's will do some
hardware acceleration for that. So it's, it's become, it's
become, you know, really nice. We'll just say, and it, there
are lots of options, which is why this has taken me two years
is because there's so many compression options for logs and
traces. And, and I actually split,
there's something called PCO, which is the compression that we
use for metrics, which is fantastic.
Usually get over 12:50. It's, it's, I think around 14 to
one right now on my, my test bed.
And then, you know, getting 12:50 nine to one, something
like that for logs and traces is really good.
And we use Open ZL for that, which is a columnar compression
that's actually format aware, which is really cool.
That is, that's cool. So what's the, what's the
biggest bottleneck then for like your ingestion rate?
Like what are you competing against to get that number up?
So it's, again, it's been a journey.
So I've been pulling off pieces and doing neffs right?
And I would do some C Neffs. I actually built a little web
server. It's called Rocket.
That was a little faster, actually quite a bit faster, 30
times faster from a latency perspective than Bandit.
And so I've just been pulling pieces apart.
You know, I find a bottleneck and I'll, I'll write a nif for
it. And the problem that I ran into
was I kept pulling pieces, excuse me, pulling pieces off
and writing Nifs for them. But then when the Nifs would
have to talk to each other, they would have to go through the
Elixir layer and there's a translation layer there.
And so that became slow. And So what I did was I, I
pulled enough pieces off. Now I can just meld the pieces
together and just make one NIF, right?
That does most of the work. And then there's just one that
that one translation layer that goes basically from Elixir down
to to Rust on the metric side. And so, yeah, the bottleneck is,
is sometimes the translation layer between the Elixir and the
NIF. And that's really just to tell
you how, how much we're, we're slicing the time here.
That's where our our bottleneck is.
OK. Like I'm curious then would the
next step be it? Like is Elixir your bottleneck?
There's no. No.
No, orchestration in Elixir is just so good.
I would never consider, no, yeah, the ability to do
clustering, you know, just the the niceness of I mean, we're
all here because Elixir is fantastic for orchestration,
right? And so no, I, I think, I think
it's this, this is definitely an Elixir application from from
beginning to end. I've gotten the performance that
I feel like I need out of it. I feel like I could do Facebook
scale. I could do all the things that
that you need to do with the number of nodes that they have
doing ingestion anyway in AWS scale.
So I think, I think I think it's fine and it works great, you
know, and it's it's it's really nice to have it in an Elixir
because you can, you can do things like this embedded side,
right where you just drop it into your app and it ingests and
gives you all of the intelligence that you need for
your app traces and everything for nothing almost right.
I mean, it doesn't really cost you anything.
It's less than 5% probably load on top of your app so.
That's awesome. Cool.
Now how does Timeless work its way into your hospitality
business? Does it?
Did you build your reason? Yeah.
We, we have about 10,000 cable modems.
We have about 1200 or we'll have 1200 lines very shortly of
telephony. And so we have metrics that we
collect on all those things and we, we want to observe ability
for those. And that's what it was, that was
what it was originally written for.
And so I thought I would share it with the world and see if,
you know, see if other people find value.
But we'll be ingesting quite a few metrics.
We'll have about 200,000 series, maybe 300,000 series that we'll
put into it. And series is just a a metric
over time, right? Yeah, yeah.
Cool to go into the hospitality network business like paint,
paint me a picture. What what does it look like?
Like What does the actual network look like?
Like does, I know, I don't know if I made this up or if we
talked about it, but like it like some of your clients would
like include like RV parks, nursing home, nursing homes,
hotels, like all of that. OK.
We call them multi dwelling units, Mdus.
And so it's just a number of whatever, right, could be
apartments, it could be nursing home, it could be anything.
And it's like it just a small cable company, you know they
have a router, a cable router that we directly import from
China. We brand it, we have our own
brand and we do all the provisioning for that and the
observability, right. So we help integrators who will
go out and do the installs. We help them with all of the
things that are hard for them to do right.
It's easy for them to travel and do and understand networks well
enough to set up the networks. But we take over and, and make
sure that it's healthy and stays healthy and, and stuff like that
and sell them the hardware. So it, it is, it's, it's, it's
basically just tiny little cable companies that we build, you
know, that are, you know, 2 or 300 modems or something like
that. OK, that's cool.
So like an apartment complex, for instance, you would have
like do you have like a main entry point closet and then kind
of splits off to every building and then every apartment and.
Yeah. And it will use like a lot of
times those apartment complexes already have cable run and so
it'll use their existing. They won't have to run fiber,
they won't have to run the Ethernet or any of that stuff.
It just operates over cable modems.
OK. And then it's your own hardware
or your branded hardware at least?
Yeah, that's cool. And then are you embedding
Timeless into your hardware or how's that working?
Now we pull centrally and so it's going to be a, there's a,
there's a version of Timeless that is that embeds into your
Phoenix application. And it's designed to be really
small, take very, very little memory, very little CPU.
There's also the server version, right, which you know, will
consume as much memory as you give it, just like any other
database. And we'll return results quite a
bit faster. And that's what we're we have a
central. Yeah.
OK, OK. I see.
That's cool. Now mentioning like you've had
100% annual growth. Oh yeah, that's wild.
That's, that's not necessarily me.
I'm, I'm late to the game. I just joined the company
recently. I'm the CTO, but the CEO has
been growing the company for about 7 years and he's, he's
gotten to a point now where he's about 100% year over year.
And you know, people were it it's really advantageous for
them because again, it uses their existing cable plant,
right. So if you have coax run to
whatever it is you're trying to get to in a nursing home or in
in any multi dwelling unit, you know, you don't have to run
fiber Ethernet or so it's it's become a popular solution,
especially in the hospitality side, right?
RV parks, there's 17 thousand 19,000 in the United States and
and they a lot of times have cable infrastructure as well.
And so they'll they'll take advantage of that and and put up
a little Wi-Fi towers. We have a Wi-Fi solution where,
you know, you can walk around with a computer like you walk to
the pool and your computer continues to stay on the same
Wi-Fi connection. It's it's pretty cool.
It's kind of like the Wi-Fi 6 stuff, but it's.
OK, now it's like the mesh system stuff.
Exactly. It's, it's very much a just, it
works just like a mesh, but but yeah, so you can, you can, you
know, take your computer around and it continues to work and it,
it works through pedestals. So they have little Wi-Fi
pedestals and basically everybody on the park is on the
same. Nice.
There is a period in my life where I wanted to buy an RV park
as an investment property. I don't remember what happened
but it it it disappeared from my my dream.
But it's not a bad route to go. Oh, it's very popular, yeah.
That's why I think that's one of the reasons why we have 17 to
19,000. It's pretty easy to run you, you
can get people to run it pretty easily for you.
And it's, it's a great investment for sure.
Yeah, that makes sense. I lived in an RV for 2-2 or
three years, but I mostly boondocked because I don't.
I didn't want to actually pay to stay at parks and do that.
I was too cheap. I was boondocking before remote
work was cool after COVID. So if you did stay apart at a
park, how was your Internet? It probably wasn't very good,
right? I usually just with hotspot off
my mobile. Would you?
Yeah, yeah, we're trying to improve that situation.
So imagine if you had, you know, a homestyle networking while you
were out RV. You know, in an RV you could
stream video and do whatever, right?
I wish Starlink was a thing when I was doing it.
That would have been a game changer for me because I had
like the crappy, I forget what they were called, like the the
cell phone boosters that kind of work but they don't really work.
I would always like have because I'd be in places where I'd have
like 1 bar and then sometimes I can get my booster to work but I
feel like it it hallucinated and just made my phone think I had a
better signal than it actually did.
Yeah, you don't actually pass any traffic at all.
Yeah, yeah, exactly. So that was the struggle.
But for the most part, I worked as a remote developer without
issue. Ever.
From time to time I would get on public Wi-Fi if I really needed
to move a lot of data, but it worked for being I was an iOS
developer at the time and it worked.
So that was cool. How, how do you how do you think
like Starlink and like satellite Internet will affect your
business? I think we may use Starlink for
backhaul a lot of times because, you know, the parks have to have
a backhaul connection of some kind.
A lot of times we do fiber for that.
But but yeah, I mean, I think Starlink for backhaul and, and,
you know, there may come a time where everyone has Starlink and
it, you know, our, our services aren't as valuable.
But I don't see that in the near future.
It'll, it'll be a while before it sort of takes over.
Yeah. I think we got some runway
there. I think like we'll probably see
it transforming like our cellular networks and cell
phones 1st and then when those stop being a thing and it's all
satellites, things will start shifting.
Yeah, I think probably when, when you can, if you have
Starlink in your cell phone and you can tether, that's going to
change the game for us. I think probably our services
will be less needed at that point.
Yeah. Well, you, you still might, you
know, I feel like people will still need like network health
and other services. You just will have to like shift
a little. Yeah.
Yeah, exactly. Like I think like that's like
the whole fear mongering but behind LLMS right now, right?
Everyone thinks our jobs are going to go to the wayside.
And I just like we were talking a little bit before we started,
I think we're just going to not be writing code anymore.
We're going to be orchestrating code workers and agents and
reading it all. Yeah.
And I've mentioned this to you before.
You know, one of the nice things about being into physical
networking is you don't have to worry about AI taking that
right? Yeah, that's true.
Yeah, until there's robots that can do it all.
Robots in Starlink, right? They're going to, they're going
to take all of our jobs. I mean, I would love, well, I
actually I don't know if I would love this because I don't like
all the. What do you call the the Amazon
boxes that you the Siri boxes that you talk to?
I forget. I don't know why I'm blanking on
it right now. Oh, yeah, Like the home.
I don't. I don't know what they're
called. I don't know what you're talking
about. Yeah, but I.
Refuse to have those in my house and I was just about to say I
can't wait to have like a robot servant that will clean my house
and cook for me and then I'm like wait what?
I actually want one in my house. Would you?
Would you talk to it? Right.
Would you trust it? Yeah.
I get it. I get it.
Yeah, so I don't know if I'd use it, but part of me thinks it
would be nice to have a little robot servant until AGI and IT
Terminator it turns into a Terminator situation.
Yeah, yeah. And I think you know.
Jim Clark said Echo dot that Yeah, but what it what are those
systems called? Like they're, they got to be
called something, right? Because doesn't Apple have one
too? Like a home assistant?
Home assistant maybe? Yeah, maybe.
I don't know. Freaks me out though.
Won't do it. It's always listening, yeah.
Listens to everything you say. I mean, I guess so.
So does so does this but. It really, it really does, yeah.
But sometimes I take it for my advantage, like I'll talk about
something because I want the ad to show up on like, Instagram
because I can't find it again. And then it will.
It shows up. Oh yeah.
Yeah, it's a little scary. Actually, it's a lot scary.
It's, it's actually really scary.
I, I tend to be like a little more paranoid, I think with my
security and like privacy, But then there's also I don't put in
the effort, like I know the effort required to actually have
privacy and I just don't have the patience.
Well, it's everything now, right?
I mean, everything wants to listen so.
It's it's hard. Every everything wants our data,
our data. Yeah, I Oh yeah.
I wanted to ask. So you you mentioned that you
are probably one of the like the biggest noobs in the Elixir
space, and you about two. Years.
You've been writing an Elixir for two years.
What brought Elixir on your radar?
So originally, so I used to be the host of the Atlanta Ruby
Users group before Rails, right? So I know I'm I'm one of those
Rails people who came to Elixir just like most, but but yeah, so
I was the the host of the Atlanta Ruby Users group before
Rails and Obi Obi Fernandez, I don't know how to say his name,
but the Rails way guy came up from Jacksonville and and
presented Rails one point O so I've been around the Ruby
community for a long time and and I I used J Ruby to do a lot
of concurrency stuff. And I actually wrote basically
Gen. stage for J Ruby back in the day and and you know, came
up with the idea of message passing instead of simafores and
all that stuff, right? Which seems like lots of people
have independently come to that conclusion, right?
Message passing is so much better than all the the.
Other as of recently too. They're just trying to reinvent
reinvent the. This was.
This was 18 years ago, so but yeah, so I wrote Gen. stage 18
years ago and relied completely on message passing with J Ruby
and and when I saw Elixir, I was like, you know what?
They solved all the problems that I felt like I solved with.
It's a library called Common Thread.
And they solved all the problems that I had back then.
You know, before I just didn't know about it, right?
Erlang was, Erlang was around, Elixir wasn't around, but Erlang
was around and it's just a, it's a, it's just a treasure.
I think I really, I really value the Elixir community and all of
the technical decisions that were made to get us to this
point. It's really a a strong piece of
software. It is.
I'm thankful for really smart people like, you know, the
Josie's and the Christmas chords and all that.
I'm, I'm only a user of great technology.
I don't think I'm, I'll ever be on the leading edge of something
great like that. But I appreciate it.
And I get to talk to him because I decided to start a podcast.
Which is great. Yeah.
Yeah, It's extremely helpful. I've been watching the podcast,
you know it again, just two years, right?
And so I've just really immersed myself in the community, watched
all the podcasts and really appreciated them.
So yeah, I mean, I, I really appreciate the work that you put
in to to mentor everyone. I always love to hear that
because yeah, I don't know, It's I talk about this all the time,
like that imposter syndrome, it never goes away.
You just like, keep faking it, keep faking it, and then
eventually it'll work out right? Yeah, it's so I, I value Elixir
so much that I tell people, you know, I've learned about 17
languages over my lifetime. I've been doing this since I was
8. Elixir is my 17th and final
language. I think I'm, I think I'm done.
It's really great. I wonder how many I've learned.
Let's see, like Java, C#, a little bit of C++, JavaScript,
Does TypeScript count or are JavaScript and TypeScript the
same thing? And then Kotlin, Swift,
Objective C, which I hated Objective C, it's about like my
ninth language. And I also want it to be my
last. And then I haven't really take
taken the time I kind of want to, but I want to learn like
more Erlang just because like there's so much more under the
hood that we have access to and we don't really take advantage
of it. Do you find yourself using it
more with like the telemetry and all that not?
So I have, I have an interesting Erlang story.
So I went, one of the libraries that I've published is SNMP Kit,
which is, is a SNMP library, which simple network management
protocol. If, if you're not familiar, it's
basically ancient at this point. But I, I felt like it was, you
know, sort of a miss in the community and I, I had some
ideas around what I, I wanted to publish for a SNMP library.
And so I went to the Erlang code and went through, you know, the,
there's some SNMP built into Erlang.
And I was like, you know what, all it's really doing is just
compiling down to, you know, Erlang beam machine code, right?
And so if I just write this in Elixir, it'll be easier to
maintain. And so I, I basically just
abandoned the idea of writing wrappers around the Erlang and
just wrote it in the elixir. And I felt like I got at equal
or better performance. And so it was, it was an
interesting experiment, you know, reading that much Erlang
code. But, but yeah, I even have a mid
compiler and, and all of the, all of the interesting side
effects of, of, of having to do SNMP.
But, but yeah, it was, it was interesting reading Erlang.
I'm, I have always been a syntax light guy.
I've learned Rex in college. And hold on one second, I'm
sorry. I learned Rex in college and I,
I really loved it. And then, you know, I, I started
working in networking labs and I did TCL, which was very Syntex
light as well, which is why I gravitated toward Ruby and
eventually Elixir. So I like Erlang.
I think it's fine. I think it's great for those
people who enjoy it, but I really like syntax light
languages like like Elixir. Yeah, I feel like that's like
probably why Erlang didn't pick up in popularity too A.
Lot of roller plate. Yeah, there's a lot of structure
to your code, right? Right.
And then I don't know, Elixir is just nice.
It just flows. Yeah.
All right. I'm going to put this on hold
for a second and we'll be right back.
Hey, I'm Kimberly with Elixir mentor Jacob recently added
recruitment services to help companies hire pre bedded Elixir
devs. I help with company outreach, so
if you're hiring, I may be reaching out, or you can connect
with us anytime at elixirmentor.com.
All right, and we're back. So what?
What is the what? What's the next thing?
What are you most excited about with like your telecom business
and using Elixir? Like where?
What do you, what are you building next?
Do you have any plans? Well, for me personally, I get
to, you know, maybe, maybe get down to one job, which would be
awesome instead of two. And I think I would like to
probably get, do additional work on the server side of Timeless.
You know, the embedded was what I focused on for this because I
thought it would be the biggest benefit to the community.
And so I think that, you know, 80% of apps could just drop this
in and have full observability without any overhead.
And I, I feel like that, you know, most apps tend to be
relatively small, right? They don't get 200,000 hits a
second or anything like that. And so if I can help those
people first, that's my intention.
And then, you know, you can also sample like traces and stuff
like that when you have the bigger apps and send them up to
a central server. And so I'm going to spend some
more time on the server making it, making sure it, it continues
to be performance. So there's, there's going to be,
there's a number of components, right?
There's three different databases and then there's
Timeless Phoenix, which is the embedded and then there's going
to be a timeless stack, which is basically all three databases
and one as a server component that you can push your metrics
to centrally if you have multiple app servers, for
instance. And so that's going to be next
going to spend a lot more time on that.
And as far as libraries that I'm interested in writing, I don't
know, you know, that the telecom business is taking a lot of
time. And so I don't know if open
source contributions are going to be a thing that I've I
everything that I've done so far as far as evaluating compression
libraries, making a faster web server, all of those things I've
sort of peeled off and made open source.
And so I'll continue to do that. If I find another compression
technology that works really well for say, logs, then I'll
wrap that and then make it available to the community.
But but I don't have anything right now.
That's just off the top of my head that I would like to do,
except just maybe continue with the timeless thing, right?
And just make it make it better. Yeah, When you're when you're
using timeless like server side, is there, is it at all possible
to make like a hosted service for it or would it always have
to be kind of directly tied in to?
It could be, and the nice thing about this, remember you're
compressing at the edge, right? And so when you're sending those
logs up, if you're like just as a perfect example of nerves,
right? A lot of those nerves links are
limited in bandwidth, right? They're using cellular or
something like that. And so if we can compress at the
edge and then send the packets up, it would be great to have a
cloud service provider that understands those already
compressed packets, right? And then can give you a single
pane of glass for observability in a dashboard, Grafana or
whatever. And I actually have an
interesting dashboard that I've built that's based on Scotty
Toolkit, which is 30 years old. But, but it's, it's fun.
Like it's got a, a timeline at the bottom.
You can put your logs and everything on your screen and
then you can track it over time. But you can grab, there's a
timeline at the bottom. You can grab it and go back in
time and it'll show you the logs from that time.
It'll show you the traces from that time and it'll show you the
graphs from that time. So when you're troubleshooting a
problem, like if you had a problem say at 11:00 PM last
night, you could actually just go back to that timeline and you
can see all the traces from that time, the logs.
From that. Yeah, it's an interesting idea.
I don't think anybody's ever tried it before.
The Polish is not there. It doesn't look as good as your
your typical dashboards do today because it is based on 30 year
old technology. But now you've got flawed to
help you clean it up. I do, I do.
Well, and yeah, there's, there's, there's some UI cleanup
that definitely needs to happen, but that'll be on the server
side and it'll be optional because everything has an
Http://endpoint just like Victoria Metrics or Prometheus
or any of those things. So you can actually just point
Grafana at it. If you want to just use Grafana
and you can, you can create all your dashboards, you can look at
your traces, you can, you can look at your logs.
And so you'd, you'd be able to get a farm of web servers.
You could just push all your metrics up to that, but I also
really like the idea of compressing at the edge, right?
So you just embed this in all your apps, it pushes the metrics
up, and then you get a single pane of glass.
Yeah, I like that. That would be cool.
I don't know which way I'm going yet.
I think you ought to build a service, a host, a service.
I think that would be really cool.
Could be it could be a thing. We'll, we'll see how people like
it on this podcast and, and I might pursue that.
Yeah, I'll make sure to link. I'll link your GitHub and
whatever open source links, make sure you send them to me and
I'll make sure they're up there. Also, and I, I wanted to
announce this on this podcast, so nothing has been published
yet except GitHub, if you follow my GitHub, yeah.
So this timeless is going to be available to everyone.
I'm going to put it on Hex this afternoon and you can try it in
your app and I'll have a little, Jacob's already seen this, but a
little snippet, a little 6 minute video of how to install
it. You know what it gives you and
it's it's very short. It's very simple to use this.
It was made to be simple. That's awesome.
A lot simpler than trying to wire up your own observability.
I, I assure you. But but yeah, and I'll, I'll
make all that stuff public today.
That's exciting. Oh, I like that.
So make sure you get me the hex link and I'll put it in the show
notes. OK, cuz yeah, that'll be that'll
be awesome. I'm excited to check it out.
I might have to give it a little test spin for kill switch.
Now there's. So one thing that you probably
should keep in mind is normally the dashboard isn't included in
prod. It's only sort of a dev thing.
So you can actually authenticate or create a auth for your
dashboard. So you can, you can wire it up
so that your dashboard does get to, you know, push the prod
right? And then you can put auth in
front of it. And there's instructions in the
in the repository for that so that you can actually sort of
you can in prod, you can get all the metrics that you would get
in dev and help you troubleshoot problems, which would be pretty
amazing. That is nice.
Yeah, that's how I have my, I use air tracker on kill switch
and then my Oban, my Oban dashboard always put that stuff
behind auth guys. But yeah, yeah, that's cool.
I'm excited. I think I'm excited to hear
about your new business. You said you wanted to get down
to one business, but now it sounds like you're gonna be
starting another one anyways. Yeah, I'll eventually stop being
a DBA and then I'll go do the CTO thing full time.
I'm hopefully buy out the CEO and actually become a CEO.
But then you're gonna start this timeless service and that's
gonna be another product and business.
If, if there, if people find value in it, I will be happy to
provide it at a a very reasonable cost for sure.
Because remember, it's all, it's the compression's really,
really, really good. And so I don't have to store
much data. I just have to present it right?
Yeah, that's because I feel like there is there is something nice
when you don't want to self host.
Like there's a time frame in every startup where you just
don't have time or the resources to self host everything, and
it's kind of nice when you can just reach out to services that
are super easy to integrate. Yeah, I agree with that.
Yeah, but then there's a point to where, like, I feel like
there's growing pains, right? You get to a certain point and
then you want to self host and control everything.
And I think, you know, where I see this landing for most people
is that 80% of people who develop a small app, they want
observability. It's hard to wire up.
You don't necessarily have a provider you know and, and you
just. You run one command, one in one
igniter, and you get full observability for your.
App. And, and it doesn't take up much
storage because we compress really well.
And we roll the data off for you so that, you know, you get your
7 days of time window or whatever, whatever you dictate,
right? 30 days, seven days, whatever
you want. But we roll the data off for
you. And so it's basically self
managing. You just turn it on.
That's awesome. Does it just?
Is it Postgres or how are you? What?
Are you No. So let's talk architecture.
Sorry, I haven't even really covered that well.
That's that's. We have plenty of time, so this
is good. OK.
So the architecture for we'll start with metrics, that's the
easy one, are the more complex one depending on how you look at
it. But metrics, the metrics come in
via either HTTP or you can push them directly from Elixir.
And so you don't necessarily need an endpoint because it's
integrated into your app. But the metrics come in and then
we push them to Rust, which indexes them.
So it keeps track of of, you know, the, the tags and then the
data itself. All, it all gets compressed.
We, we compress, there's 16 bytes of data per metric.
We compress it down to just over 1 byte, which is fantastic.
And it's all searchable. There's a nice index.
We do use Sequel Lite for some things.
It's because it's embedded, right?
We, we have the ability to alert you and that those alerts have
to be stored somewhere when you set them up.
And so those alerts are stored in Sequel Lite.
There's a couple other things I'm trying to remember exactly,
but but yeah, we do have a Sequel Lite database.
OK, do you use ETS tables at all or anything like that?
Not as much that was in every possible like I've been working
on this for two years. So I've re architected it about
15 * 14 times. Yes, lots of bets.
The 15th time I went to Rust. And so it, it basically just
hands the data off to Rust and it it keeps track of the
indexing and does all the things I see.
But yeah, and then logs and traces.
There is a sequel I database for those and but we just compress
the, the values. We keep an index so that you can
search, right. The search is very fast too, by
the way. And so if you wanted to go find,
you know, all of the traces for the guy who or forget, you know,
slash this, this URL, you could just go and pull those traces
and then you can click on it and you can look at the trace.
Just like any other observability application.
It's, it's very simple. We kept it simple for a reason,
but but it's it's, you know, it's quite searchable and it's.
I feel like it's a good, it's a good example for Sequel Lite as
well, because you just start passing or you just have a Jason
file living in the project at that point and you don't have to
worry about Postgres database being there or an instance of
something larger, right? It just kind of works, yeah.
Yeah, we even. I even played with storing the
you know traces and everything you know, compressing them down,
store them as blobs inside Sequel Lite and that worked OK,
it was fine. But block store is is much.
Faster, OK, interesting. I feel like it's I, I'm pushing
my skill level, my technical skill level, especially like
when you come down like, like, you know, just performance in
general is a very hard thing to like really dig into, right?
And like when you're picking in between like Sequel Lights or
maybe a My Sequel database or an ETS table, Like what made you
lean towards Sequel Light over using like an ETS table?
Well I had this idea from the very beginning that I wanted to
have an embedded version of this.
It may not be the the final end all be all, but there is going
to be an embedded version, right?
Yeah. And so Sqlite's just much easier
is. There a limitation with ETS or
ETS on an embedded hardware? No, no, it's.
Just the persistence isn't there, right?
So right, like if you lose your instance, everything's gone.
Yeah, we now you can have it obviously you can take your ADS
tables and dump them to debts and you can do or dump them to
Sequel Lite for persistence, right.
You can do like a like a slow path for that.
You can push it to ADS very quickly and then have a process
that kicks off and and sends it over to Sequel Lite for
persistence. I've done all of those things.
I've tried 15 different versions of every one of these three
servers, but but yeah, I've settled on what I think is
probably the least the least amount of overhead for an
embedded app. So I've always had that that
embedded in my head from the very beginning.
And that's those are the important decisions that like
those architectural decisions are not easy ones.
And see like that guys is where we're not getting replaced by
LLMS, you know what I mean? Like you would not get your your
architecture choices. Like if we put this into Claude
right now, we'd probably get some.
I'm sure we'd get like a working plan that would probably be
functional, but it probably wouldn't perform the same way.
I will say though, I've, you know, I've worked in tech for a
long time and I've worked in and worked with a lot of really
intelligent people. And I learned a long time ago to
ask open-ended questions, you know, just go and say, I'm
looking to do this and then have it come back with some possible
solutions. And, you know, maybe they're
good, maybe they're not, but, and then maybe they apply and
maybe they don't. But if you ask the open-ended
question, you, they may think of something that you have not
thought of, which is, and, and the LLMS will do that too.
So if you ask open-ended questions of LLMS, they can help
guide you. I think it's a great way to like
rubber duck, like especially for a lot of developers that are
remote. You like we get on our own
little eco or echo chamber, and it's nice to talk to a robot
from time to time. About your ideas, I mean.
Well, I learned they're encyclopedias.
It's why wouldn't you ask an encyclopedia what all the
options are? You know, why wouldn't you go
there and do that? They may not have the best
option for you and they may not. It may not bubble up.
But as long as they know all of the options, it's more than you,
right? And so they can help you.
And like when I did Killswitch, which you know, it's like a zero
knowledge encryption document store, right?
I didn't really know how to implement a zero knowledge
encryption system. Well, right.
Like I've read my encryption algorithm books and things like
that but they don't actually like I know how they work at the
encryption level but like how do you properly implement it?
And dude, I'd like open-ended questions and rubber ducking
with Claude back and forth how I wanted the system to work client
side freaking help me iron things out so much faster than I
would have by myself. I think I would have still got
to the same solution, but it would have taken me a lot
longer. Yeah, I agree with that.
It's, you know, it's, it's just nice.
I mean, just having something that knows more than no matter
what, they know more than you do, right?
They have more scenarios in their heads than than you do.
You've you've been down this path and so you have your
predefined things that you would like to accomplish.
So you have to obviously measure it against that.
But yeah, it's great. It really is great.
It's helped quite a bit on time was for sure.
Yeah, I'm sure that's cool. What?
What does your like day-to-day workflow look like using and
like? Do you use Claude or what LLM do
you prefer? Oh, sure.
So, you know, in the car. In the car, I use Grok.
I'll just start it up and I'll start having a conversation
about how I would like to compress things or how I would
like to index things, right. And Claude, sometimes I'll, I'll
take those ideas and bounce them off of him once they're sort of
formed and then, you know, still trying to answer, you know, ask
open-ended questions if it, if at all possible.
And then for clean up, I found GPT to be really good.
So, you know, Claude, you can have it.
It's very innovative when it writes code, but it doesn't
necessarily get it right. So I like the innovation
sometimes. And so when I'm trying to
innovate, I'll use Claude and then I'll come back with GPT and
sort of clean it up and make sure that it's doing the right
things. But that's my, that's my work.
Excuse me, that's my workflow. OK, nice.
When you're like thinking through a feature, what does
that? What does that look like?
Are you chatting back and forth with Claude and creating like a
markdown plan or what's it? What are you doing?
So, yeah, so previous versions of of timeless metrics used
that's quite heavily. And you know, I, I was hitting
performance bottlenecks, I was trying to hit numbers that
approached, you know, the ghost space, the the Victoria metric
space and I was unable to do so I got really close and I
eventually got faster, but it, I had to get away from ETS
unfortunately. And so I used a, what's it
called, persistent terms, and that made it a good bit faster.
And then, yeah. And then, you know, eventually I
just kept breaking off pieces and, and making them Nifs.
And then of course, those Nifs would have to talk through
Elixir, right? And so that became a bottleneck
too. And then I just basically
bundled it all together. But but yeah, it's.
Interesting, like having more, having too many nifs was a
bottleneck, but having like 1? I was passing the data through
Elixir. Yeah, from one NIF to the other.
So I see. You just take the 2 Nifs and
just put them into one. Right.
Yeah, yeah. And so it's, it's easier for it
to, you know, Rust to talk to Rust, you know, right.
That makes sense. You had just a middle man that
was slowing things down. And it's still again, Elixir
fantastic, best best there is as far as orchestration layers is
concerned. I wouldn't trade it for anything
for that layer of it, even if it does cost a little bit in
performance. And you know, 3,000,000 inserts
per second is that's a lot. You know, back in the day, you
know, I told you I, I wrote this application for monitoring cable
modems for, for Cox back in the day.
And you know, I was just completely ecstatic to get
10,000 inserts per second back, you know, with those little
pizza boxes, the single core pizza boxes.
And now we're doing, you know, 3,000,000 inserts per second,
which is crazy and compressing it, right?
Yeah, no, that's wild. Like it's actually hard to kind
of wrap your head around how much data that is.
Yeah, yeah. Well, it's more than most people
will need, but it's, it's also necessary, right, to keep that
overhead very low, right. So if you want to put this in
your running application, you want it to to not affect the
running application. You want to be able to throw as
many logs as you want to at it without affecting the running
application. And so, you know, I, I felt like
performance was, was absolutely paramount and, and I was
achieve, I was able to achieve everything that I was set out, I
basically set out to do. Yeah, that's cool.
I'm not like very familiar with Rust or using NIFS.
What was your learning curve there?
How did you? Huge.
Yeah, huge. I'd I'd not done it before
either. And you know, it was it was
required, I think, for the performance that I was looking
for. And so I just, you know, head
down, you know, figured it out, asked a bunch of questions of
the LMS right, to help me understand the the trans in your
wrestlers. Just fantastic.
It does take care of and I use fine too.
Fine is is used, I think on the rocket side.
I do have a couple of different interfaces, but but yeah, it's,
it's not as hard as you think. But it's it's a learning curve
for sure. Yeah, I heard like handling like
errors and like hard crashes is difficult to like get back out
to the Elixir layer to actually be like helpful.
That was a big part of it. You have to actually, you sort
of write REST the way you would Elixir, right?
You send errors back. So yeah, if anything breaks, you
send an error back. And so that Elixir can
understand what broke, right. Yeah.
You try to behave like an Elixir application at that layer.
Because it's really easy to just swallow, swallow the airs right
when you're like. Creating or crash, yeah, you
really don't want to crash either.
So yeah, that's going to be one of the interesting things about
making this public is, you know, people are going to use it in
ways that I have not. And so there there may be some
crashes out there. We'll get it all worked out.
Try to treat it as at least beta level software.
Probably, probably not alpha, probably beta level software,
but. But yeah, be careful.
Because. We are dealing with a lot of
Nifs. Is this your first like open
source project that you're taking public?
No, I've so I've maintained something called SNMP 4J.
Oh yeah, you've said that. Yeah.
So I had an SNMP library that I
I open sourced. It was kind of a wrapper around
SNMP 4 J and it's for J Ruby 18 years now I guess.
But I haven't had a lot of open source projects.
I've really embraced the Elixir community and I have, I want to
say 10 now, but yeah, quite a few things that I've released on
the Elixir side. I have a SOAP library because we
needed SOAP. We're we're a telephony provider
and telephony providers love SOAP.
So I have a SOAP library that is free for all and it's actually
the most, I think it's the most popular library that I've
released on Elixir ecosystem. The SNMP kit is very popular as
well. I wouldn't say very popular, but
it's, it's more popular than I was expecting.
And then, you know, all these compression libraries that I've
been wrapping all the, the little NIF accesses, you know, I
may not be using them anymore, but I have released those as
open source. So you can use them if you
wanted to. Things like Z not Z standard
open ZL I'm trying to think. You're making me look bad.
I have no open source projects. Really, yeah, no, you should
think, just peel off little pieces like as you're working,
just think, hey, you know, this this little piece right here I
think could probably be useful to someone else.
And then just peel it off, you know, and there's some packaging
that's required, right? It takes a little bit of work,
but with the LLMS, right, the packaging becomes a lot easier.
And so I've just peeled them off and and just made them available
to the community. So it's it's relatively easy to
do. Yeah, no, I think, I think it's
awesome too. Like I wish I was a better
person. You know there's some sort of
encryption you're using, right? Is there a way to peel off the
encryption and maybe release it? Yeah.
Not to think about that the client, the problem is like all
the client side stuff is done in JavaScript and so.
But there's probably, I don't know, by the way, I could do it.
I'll have to think about it because for any back end
encryption, I'm just using ash cloak for that.
And dude, I I love using ash for my production stuff now because
it's just so fast. I don't know if you've used ash
at all, but wow. So.
That's a part of the conversation I haven't really
gotten into. SO you know, the provisioning
side of DOCSIS, the cable modem stuff.
We that entire stack is Elixir. Now we've got a DHCP server
that's commercial quality, right?
TFTP server that is commercial quality.
We have an NTP server that is stratum 2 compliant, which is
pretty cool. That's that's, you know, sort of
telephony level NTP we have. I'm trying to think of what else
DOCSIS Boot File Editor, which doesn't mean anything to anybody
on this call, but the number of things that help us do the
provisioning for cable modems. That's all.
Elixir. That's a full Elixir stack.
OK, cool. Yeah, yeah, but for So I started
using Ash and it was a hard learning curve for me because
it's like another language. It's not Elixir, it's, you know,
they're their own DSL thing. But what's nice is like for
production stuff like you know, what your schemas look like and
all of that. And it takes away the overhead
of. Writing an active schema and
then fine tuning your database migrations like it does that all
just like the way you set up your relationships and
resources. It's amazing.
I don't know. I suggest it reminds me.
Yeah, it reminds me a little bit of Rails.
It helps you sort of translate your business logic very easily
into your. Scheme, yeah.
Yes. And that's what we've done on
the the you know, DOXA services side is we've we've used ash.
I forgot to mention that the reason why I told you all those
things is because we did use Ash for that and and yes, it's, it's
fantastic to just apply your business logic to to what you're
doing on the database side. Just write your resources, your
relationships and actions essentially, and then you'd have
your schema and context files basically all in one small
module. It's really nice, yeah.
We really like it. Yeah, and I struggled with it
though, until the Ash book got released because I just needed,
I don't know, I needed to read it and see how it was being
built. Like I looked at tons of
examples in the documentation and I was just like, I always
struggled with like, OK, I know how to do this in Elixir in
Phoenix, and I know the ash way is like a one liner in a config
somewhere, but I don't know how to do it.
So it's just like it turned into just being frustrating.
But then once you unlock that the syntax and how to use their
DSL, it's just you move quick. We had difficulty when we we
when we first started the apps with Ash, the LMS didn't know it
very well. I had trouble with Claude and
some of and that has improved dramatically in the last couple
years. When they released usage rules
that helped a lot because you didn't have to write your own
ash like you didn't have to add to your own like MD file you
know? Right.
Yeah, yeah, that's it was it was painful, unfortunately, using
ash back in the day. This was two years ago.
But but yeah, I've done a couple of things recently and it's been
so much better. It's much, much, much better
now. Mm hmm.
I agree, but yeah, game changer. I don't even know if I I can
write normal Elixir code. I was going to say I don't know
if I can anymore, but I write it every day.
The day job keeps me current with, you know, Acto and Elixir
in Phoenix. And then all my side projects
are are now Ash. I will say it's much.
It's much nicer not to have to shovel, you know, shove the bits
around like I can. I can really envision what I'm
trying to do and really, really get down to exactly how I want
it to execute. And I don't have to write, you
know, all the little bits and pieces and, you know,
accidentally type something wrong like, you know, it's just
I, I find it is very liberating. It accelerates me for sure.
Yeah, I agree. Well, like one thing that I
think is the superpowers, just like I always look at my
database migration files when they're generated, but I I've
never had to alter them or add an index or anything with ash
like it just it always is, right?
Very intelligent. Yeah, so that's nice because I
feel like that's the missing link.
A lot of people like late lately I've been reviewing so much I
shouldn't be mean on my podcast. A lot of AI driven projects that
are just kind of like over engineered slop and you can tell
that an AI generated it. It's like like sure, it's good
code, it works, but it's like it's too much.
Like I know you didn't put any thought into this.
You let Claude engineer the whole solution and that's been
getting old. But one thing Claude sucks at is
database performance. Yeah.
So it'll miss indexes in the migration files.
It'll do all that. And like that's a great, like if
you're getting interviewed, guys, a great got you.
Like, OK, what happens if I, I need to query half a million
records off this? Like is it going to perform or
is it going to time out? Yeah, as a DBAI, fully support
this message. Yeah.
Check, check, check your LMS and make sure that they're doing the
right thing. So.
And honestly, I took for granted because I've, like we were
talking about before, like the average software business, you
have a couple 1000 users, right? Like you're never going to hit
performance hurdles or cliffs just because you can kind of
just write sloppy code and get away with it because it's just
going to work. You don't have enough data.
But then when I started working at collects, I had to Start
learning about, I had to be mindful about, you know, where
you place indexes, how you're using them and how you're doing
table joins and all these things because I like the very first
feature I released, I like crashed production and it was
very stressful, but I had to go in and fix it.
And I learned all about taking a deeper look at what was my, my
sequel query, What did it look like?
And was it what I expected? And it wasn't because we use, we
use App Synth, which is like draft QL, you know, and a lot
like. For the most part it generates
your queries properly, but sometimes it doesn't and you
have to kind of force it to build the query you want.
Yeah, I work with my day job, I work with 60 different dev
groups and they have all different flavors of Orams and
we have we have issues like the graph QL issues all the time.
Yeah, actually after I finish this call, I have a call with
someone who has a oops, I have a 600 million row table.
How can you help me clean it up? So that's, that's, you know,
it's the life of ADBA. Do you, I'm like, how do you
handle like, how do you handle that?
Do you back up the table before you start messing with things
and then? We figure out what they want to
keep and then we we have some online schema change utilities
that Facebook published a long time ago that we've modified
Facebook, you know, they do a couple of passes.
We do 3. So we'll take this very large
table, we'll copy the data out that we want to keep and then
we'll, and that can take a while, right?
Because you sometimes you want to keep quite a bit.
And then we do a delta and then we copy the delta over and then
we lock the table. We copy the last little bit and
then we swap the tables. OK.
All right. So you never officially drop the
the table you're fixing until you confirm that everything's
good. It depends, you know, sometimes
it's just a queue, right? And they don't care.
That's sometimes Wild West shifting, honestly.
There's, there's a you'd be. So it's, it's a lot of different
dev groups and some of them have, you know, are focused on
the, the size of their data and some are not.
Yeah. Yeah, yeah.
Touching a prod database always makes me nervous.
I always aim for the replica first guys.
You should you should try it as a full time job.
It's exciting. I don't know.
I mean, I'm sure, honestly, I'm sure it is it.
It's probably a lot of fun, but it seems like a lot of
responsibility. It is, you just have to be
careful. Yeah.
And you get, yeah, it comes with time.
Yeah, what? I don't know if you actually
answered this at the beginning, but what got you into the
database space? Like how did you become a
consultant for that? So I built this really large
data warehouse on my sequel of all things.
Back in the day. It was a Snowflake database
before a Snowflake was a thing. Snowflake's an actual design of
a database, not a vendor. We use a snowflake IDs, I know
that, but OK, basically because it was a bad architecture
decision and we were running out of primary IDs.
Snowflake is basically you take your data, you have a set of raw
data, right? And then you have derivative
tables that are around the Snowflake, right?
That are their derivative intelligence that you've derived
from the data. Then you take those derivative
tables and you sort of join them together and you make Snowflake
tables out at the edge, right? And so we did that with my
sequel back in the day with the what was at the time a very
large database, 3 terabytes. Which is one of the reasons why
I love observability so much is because I built that database
and we put so much data in it that it would take three weeks
to do like a column add, right. With observability with the
these new time series databases that we've we've come up with
over time, unfortunately didn't have them at the time, but you
know, it becomes a columnar format.
You can very easily change the data that's flowing in.
You could have new infrastructure that just
automatically starts getting scraped or pulled or whatever.
And it, it, it's really cool. I, I really find the, the, the
concept of a time series database just completely
fascinating because I've been completely, you know, scared to
touch any part of my database because it was so fixed.
Yeah, it would take three weeks to to modify.
Well that's wild. Like doing a column change or
anything. If it takes 3 weeks and then it
doesn't work the way you expect it to, then you got another
three weeks to test the fix. Yeah, it's a very, it's not
exactly what you'd call a tight iteration cycle.
Yeah, yeah, yeah, yeah. 1 oops cost you 3 weeks.
That's crazy. Yeah.
It's an expensive 3 weeks. It is, yeah.
Yeah. And that's, again, if you
compress it down, right, your data gets smaller and you make
it columnar so that you can easily change what data goes in,
what data doesn't, and it compresses it so nicely.
I'm just very fond of it. And I've always kind of wanted
to write 1. And so it's been a really good
experience for me to finally get to do that.
I read the the the papers that came out for Gorilla when they
first came out. You know, Facebook published a
paper on Gorilla probably 15 years ago and I found it all
very fascinating. And actually writing one finally
is is been quite. It's very rewarding.
Yeah, that's cool. Well, I mean, I feel like that's
the only way to learn is to build it, you know?
That's the primary reason I didn't necessarily want to
release it, but the performance was so good, like I had pretty
high expectations for myself. I ended up with the something
that performs relatively well. We're at least, you know, as
fast as Victoria metrics or probably.
Now you got the bragging rights. You have to.
The second fastest, Yeah, we're probably the second fastest
database as far as ingestion and and so that's awesome.
That's very cool. So yeah, and not bad.
Guys, he's only been in the elixir world for two years.
Two years? Yeah.
He's blowing our pants off. He's making us look bad.
That's awesome. That's so cool, and now it's
going to be officially an A hex package this afternoon.
Yes, this afternoon I should be able to publish the The Timeless
Phoenix. Go ahead.
How many years have you been writing this?
2 Yeah, it was I started. Oh, so you like started it as
you learned Elixir? Yeah, OK.
Yeah, pieces, you know, again, I read that that gorilla paper,
you know, back in the day. And so the first thing that I
did was gorilla. And of course along the way I
decided gorilla wasn't a good enough compression algorithm so
I threw away that code. But it, it did start two years
ago when I started writing the Gorilla compression algorithm
and, and you know, we use much better compression now.
It's faster. It's it's, you know, better in
every way than Gorilla, But but get started with Gorilla.
OK, that's cool. All right, I'm what?
What does your like supervision tree and things look like?
Like how are you handling? How are you handling that side
of things? Cuz we talk about, we talk about
how awesome Elixir is at orchestrating.
But what does that mean, right? That's a really good question
and something I kind of wanted to cover on this.
The nice thing about having supervision trees, it means that
you have layers of isolation for your application and you can do
things right that you wouldn't necessarily do with C or go,
right? So if you're, if you're
developing a, a time series database and, and go, you're
going to do that in isolation. If you are developing a log
database and go, you're going to do that in isolation.
If you're going to do that in isolation, if you're going to do
that in isolation, you're going to do that.
You know, for traces, you're going to do it in isolation.
You're going to end up with three different services because
you don't want one service crashing and taking down the
entire thing, right? But with Elixir, doesn't cost
you anything, right? So you can just throw all those
services in one, excuse me, with the supervision tree.
And then, you know, if if one of them goes down, it's just the
one that goes down and it comes right back up, right?
Yeah. Yeah.
We do that on the DHCP server. We do that on all of our our
services. We, we piled every one of the
DOCSIS services into a single Elixir app and we don't worry
about 1 going down and, and taking down the entire app
because it won't, right? Supervision trees are amazing.
We even have sort of sort of advanced supervision trees on
the, the DOCSIS app side where you can stop and start services
and it's all green threads, right?
So the services immediately stopped and they immediately
start right back up, which is pretty cool to see.
It's not like running a, a binary, right?
So. You're kind of you're, you're
running like a dynamic supervision tree and then
dynamic processes off of that as well.
Yep. Yeah, that's exactly how it's
done. Yeah, yeah, the yeah, the the
actor model and like just the fault tolerance of Elixir, I
guess. I can't say of Elixir of of
Beam. VM is freaking magical honestly.
I used to have so much respect. So I, I've worked in telco for a
long time, right? So we had five E switches and we
didn't have Ericsson switches. We did have 5E switches back in
the day. And we had a ton of respect for
those guys. You know, the, the, the five
nines, the 10, you know, 6 nines or whatever of up time that they
were able to obtain. And you know, they were always
in the data center and they were sitting there by themselves, you
know, just just doing their thing.
But I always had tons of respect.
I was a data engineer and they were the telephony guys, right?
OK. And you know, it's, it's nice
actually working with the software that they were working
with at the time, even though it's not the same, right.
It's a 5 E switch versus an Ericsson switch.
But but still switch technology is is phenomenal and and I think
we should not take it for granted.
It's really good stuff. I mean, we take it for granted
for sure. Yeah.
But like we were able to do that because of how well it just, it
just works. Just works.
Yeah. It's fantastic.
Did you like when you were saying you said 6 to 10 nines?
Oh yeah, they've up time like that's.
I think there's an Ericsson switch that ran for decades,
right? There's like probably multiple
decades of up time. Yeah, it's pretty crazy.
That's wild. What's crazy is like there's no
hardware failure. That's even more.
Well, they have redundant hardware.
OK, it's, yeah, it's the way it's designed.
It has redundant hardware, they can pull pull boards out if they
go bad and put boards back in and it's hot swappable.
OK, yeah, that makes. Sense it was, it was really
advanced stuff for the time, you know, and I, you know, that was
again, that was the beginning of my career.
So that was 30 years ago. Yeah, yeah, that's wild.
They solved everything in the mid 80s, really.
And then we just keep getting new problems that have the same
pattern, which is cool. Exactly.
Why? It's funny watching all the
banter on like X with all the LLM companies of like, what's
the best tech stack now for agents and all this and I don't
know. Are you familiar with Mike
Hostetler? Oh yeah, yeah, I.
He he likes to, he likes to get in there and get all
controversial. It's fun to watch.
I sit back, you know, I just like to be, I'll be the nice
face of Elixir and but it's fun to watch him poke, poke the
bear. Yeah, and that's why I, I agree
with everything. And he says I think Elixir is a
fantastic environment, especially for agents and stuff
like that. But unfortunately I'm on the
performance side, right? So I, I had to do some Nifs,
but, but yeah, I think the performance is, you know, 1/4 of
what you get like in Go, that's still very, very good.
That's 10 times faster than any interpreted language, right?
Yeah. So the performance is good
enough for just about anything you wanna do, except for maybe
build a database. Right.
Well, I feel like it's
fantastic. I feel like he would agree with
you there, right? Like you're using Elixir for the
orchestration and like, the process capability, like
spinning up thousands of processes like nothing.
Right. But you still mind things.
Yeah. Depending on what kind of data
you're pounding through, you might need an.
If you might need to use Rust, you might need to use something
that's just more efficient at, you know, heavy calculations in
math. And Elixir is a polyglot, right?
You could write Lua, you could write Python, you could write
Rust. You can write what's Zig, right?
Ziglar CC, It's it's whatever you want to do.
You can pretty much do it in Elixir and just peel off the
parts that need just a tiny bit more performance.
You just peel that off and do a nif, right?
Yeah. How do you, how do you go, how
did you go about like breaking down like what needed the
performance boosts like how do you how do you find those
problem spots and how do you tackle those problems?
Most of them were obvious, like you know when I started trying
to. So you, what you do when you
ingest metric data is you have a set of tags for this particular
piece of data and then you have a value and those tags that look
up takes time, right? That's, that's the, that's the
sort of the bulk of what takes time.
And so, you know, I first did it in Ed's tables, then I, you
know, and just, you know, a lot of different, I tried
everything. But but yeah, we first did it in
Ed's. And you know, that look up would
take time because it is such a kind of a long string, right?
And and so it was, it became obvious, you know, it was, I
don't want to say that I did some grand deep dive into the
beam and figure out exactly what the performance issues were.
I kind of knew. You just saw your bottlenecks
just with time like the query length or whatever.
I guess it would be a query. And I also have the ability so
and someday, and this day is not today, but someday, I'm going to
have the ability to already know what your metric ID is going to
be, right? So you have to derive that long
string into an ID and then store the metric.
And so if you just store the ID and the metric, I've gotten up
to like 13 million inserts per second, right?
So, you know, if you just do that and you have something that
collects the data, that's intelligent enough.
Sorry, my dog's about to bark. You have something that's
intelligent enough to collect the data in such a way that it
knows that that ID exists. You could actually insert it,
you know, 1313 million inserts per second.
But but yeah, so it makes it obvious, right?
If you can insert that at 13 million per second and you're
only at 3,000,000 per second. If you have to do that, look up
your bottlenecks. The look up right?
It's, it's pretty easy to determine what your, your
performance penalty is. And so, yeah.
And so unfortunately it was easy.
I, I wish I could give you some insight on, you know, how to
find performance issues in the beam, but it was for us, it was,
it was obvious. Yeah.
Well, if there you always want to know, if you want to know
everyone's tricks, right? Like, that's where you learn,
that's where you learn the juicy stuff, you know?
Don't have any tricks unfortunately.
How do you, how are you like pushing your boundaries and how
are you keeping those metrics? Like what kind of test system do
you have to like know what your limits are?
So you know I was doing, I did most of the development on a air
right AM 4 air. And so it was OK at sort of
judging performance. I did upgrade to the M5 Pro with
the maximum number of CPU's, but you know, lower, lower number of
GPU's because I don't, I don't need GPU cores.
So I could sort of test, you know, with, you know, best of
breed processors. I also have an AWS account.
And so I, I did spin up 192 core ARM instance and try this
against Victoria metrics. And then I also did a 92 core
A-96 core with against Victoria metrics and we did well.
We didn't actually beat them, but we did well.
And I really feel strongly that we're probably the second
fastest time series database. The time will come to be #1 I
believe in you. Yeah, eventually maybe we'll get
there. But yeah, it's, it's the
performance is pretty phenomenal and you'll see it when you start
to use your app, especially if it's, you know, you're not doing
millions of inserts per second. It the the query performance is
phenomenal. It's it's way sub millisecond.
It's like we're in like 10s of microseconds.
That's why. I feel like like running your
benchmarks is like the hardest part because.
It is when you have to do that, right?
Every little thing that you change, you have to run all the
benchmarks to make sure that you didn't affect it too much,
right? And so, yeah, benchmarks become
a big part of your life when you're doing database stuff.
Do you set up tests for like time benchmark, like speed
benchmarks and things like that? Like how do you know if like
something you just tweaked actually after everything up?
I probably should I, they probably should I, but it's,
it's all manual. You know, I'll, I'll finish the
session and I'll, I'll run a benchmark and make sure that I
didn't regress. I usually do it on a branch
because I worry so much about performance regression.
I'll do it on a branch. I'll, I'll make sure and test
before I merge. So I know exactly.
And then sometimes you just take performance penalties.
Like sometimes it's just, well, you know, this was valuable
enough that I'm gonna take a 10% hit on my performance.
But you know, it makes it a more robust.
Service. How often has that happened
where you've had to take the performance hit for something
like a few times? Yeah, a few times.
Yeah, cuz I feel like especially on the rest side.
Your whole, your whole thing though, like performance is so
important. So it's, it's a hard decision,
right? It is.
It's a hard decision and you know, again, my, it's, it's made
me, it's made it easier on me just knowing that embedded is my
my goal, right, at least initially.
So if embedded is your goal, then performance, there's no way
that any one of these apps, I mean, you could have a server
that's 192 cores taken 200,000 HTTP hits a second and it still
couldn't generate enough logs for you to cough right.
And so having that as a, your baseline and, and, and you know,
shooting for performance numbers that are so high, it makes it
easier, right? So you can say, well, I can take
a 10% to 10% performance hit on this and it still will be good
enough for 99% of the users. Right.
So yeah, yeah, that's cool. I feel like yeah, like your
benchmarking was probably the hardest process to set up too.
And I did lots of different kinds of benchmarks.
I, I would do like a sustained load on queries and a sustained
load on inserts and see how they affected each other, right?
Because usually with metrics databases, they're different,
right? So usually with a normal
database, which is what I deal with in my day job, you're 9995%
reads and and 5% writes, whereas this is 9995% writes, 5% reads.
And so it's a very different way to look at databases.
But I wanted the read performance to be very good.
And so I, I really did focus on that and, you know, kept the
read performance really good too, so.
That's good. Yeah.
I could see that kind of being ignored because you'd be more
willing to wait on getting data back to investigate issues than
obviously writing data because you need all your logs, you
can't miss any. But I also still had that in the
back of my head, right? I wanted that canvas where you
could drop your graphs on, right?
All of all of the graphs that are, they're relevant to
whatever you're looking at and, and grab that timeline and go
back and, and have every one of those graphs repopulate as
quickly as possible, right? Yeah, that's cool.
So I did have some, some performance goals in mind for
that. And I think everyone's, you
know, I love the the canvas, but I don't know that it's going to
be something that everyone embraces.
But you know, you always have Grafana.
You can always go back to Grafana and do your normal, you
know, trace, look up or your, your graphing or your dashboards
or whatever. But but I wanted it to perform,
you know, from a read perspective.
I was, I was, I was really hoping for really good
performance. Yeah, that's cool.
We just started using Grafana at the day job, so I'm like kind of
learning that. It's not too complicated, but
it's pretty cool with the amount of data you can have.
Yeah, it tends to be convoluted like it.
It has a lot of little knobs and buttons and things, and I think
that that that doesn't work well for a lot of people.
I tried for something different, yeah.
And I I've achieved it. Hopefully I'll release the the
server component with the canvas engine later.
And it will, it'll be hopefully people will like it.
We'll see. I think people will love it.
I I'm excited to get my hands on.
I'm going to put Timeless into kill switch today, this
afternoon. Sweet.
And I'll give it around. That's awesome.
I'll play around with it. Our first user.
I don't know we we got we got a couple people here just waiting
for it. Let's see, Daniel said.
Elixir plus C equals conquer the world, man Yeah CC is a good
that's a good happy. Lifter Yeah, there's a there's a
web server called Rocket that I developed specifically for this
project so that I could get my latency down for
Http://requests, and it's all C. And it's nice and it's.
Fast. Yeah, it's very fast.
It's 30 times faster than Bandit.
Wow, that's. From a latency perspective,
which is latency is what what I cared about, yeah.
Right. But 30 times, that's a lot.
That's a big number. Well, and it, you know, it's
just, it's just the latency, right?
So it's three times faster in requests per second or something
like that. And depending on your pull size.
But, but yeah, it's faster. Let's just say it's faster.
And the latency was what I was concerned about because I wanted
to be able to pull the data out as quickly as possible.
And so I just bypassed, you know, basically Bandit, any of
the HTTP, the plug, right? All that stuff is bypassed.
It just basically interprets your request, figures out what
you wanted to do and sends the data back as quickly as
possible. It's called rocket.
It's, it's released separately as an open source project as
well. But it's, you'd have to know you
need it, right? Because bandit's extremely fast,
right? There's not a lot of people that
need a a, you know, lower latency than Bandit.
Yeah, well, it's like you, you have to run into these limits
before you know you need to fix them, right?
Yeah, I was doing benchmarking and a lot of half of my
benchmarking is internal metrics, like I'll just push the
metrics directly into the database.
The other half is, hey, how am I going to do against things like
Victoria metrics and, and Prometheus and those guys, And
they all have Http://interfaces. And so I wanted it to be
relatively fast. And so I, I, I I thought it was
an important part of the project.
So, so all these systems you're using basically like just a REST
API for taking in data or not necessarily?
They're well established interfaces that I had to conform
to in order for things like Grafada to work.
And so I just followed those. I there's nothing.
Yeah. And it's, it's not rest it's
similar, but it's, it's there's prom QL, I think is the one that
I, I conform to the most. And so you have a certain way
that you query and a way that you push data in.
I see and and I just. I've I had to follow those
standards in order to work with Grafana and and other.
That makes sense. Open Telemetry pushes metrics,
has a certain format that it wants to send and so you don't
have a lot of. Would sockets ever be an option
or not really? Yeah, I mean it.
It'll hold the, it'll hold the HTTP connection open and just
keep. OK.
Yeah, yeah. Gotcha.
It's like web sockets, but it's not technically, it's just an
HTTP connection. I see, got it.
OK, excuse my slowness. I have to sometimes ask ask the
dumb questions. It's just like web sockets
though it it holds the connection and just continues to
send data or query right? OK, cool.
Yeah, that's awesome. Yeah, I, I think we can wrap
this up. This has been a lot of fun time.
The time goes past so fast on these things.
I don't know how it was for you, but yeah, me too.
And it was great meeting you. Do you have any advice for for
developers that are just getting into Elixir, have been learning
it a while? Just anything.
Any. Know your strengths, know know
what Elixir is good at. Really take advantage of that.
Don't necessarily pull off nifs because you, you feel like you
need to. I had very specific performance
targets and I, I felt like that that was necessary for me.
But Elixir is a very fast language.
I think you're going to be fine. And, you know, oh, ask
open-ended questions to your LMII.
Think that really helped me a lot, understand what is out
there as far as compression technologies and then zero in on
the ones that are most important for me.
And, you know, they have a lot of data.
They don't necessarily understand it as well as you do,
but they can get you to the right places.
Yeah, I love that. All right, well, this has been a
lot of fun. Thanks for joining us today,
Mark and everyone else. Thanks for tuning in and have a
good weekend.