This episode is brought to you by Beam OPS.
If your Elixir deployments are painful, or your infrastructure
is a mess, or you just need experience hands to help you
scale, these are your people. Beam OPS specializes in scalable
Elixir development, automated deployment pipelines, and
infrastructure migrations. They will help you work smarter
and ship faster. As authors of the Pragprog book
Engineering Elixir Applications, they know their stuff.
Check them out at beamops.co.uk. Secure your Elixir apps with
paraxial IO. From code analysis and bot
defense to dependency checks, they've got you covered.
Elixir first design with GitHub integration.
Elevate your security at paraxial dot IO testing.
What's up? I'm Jacob Litzow, your Elixir
mentor and welcome to another exciting episode of the Elixir
Mentor Podcast. This is where we discuss
everything related to Elixir, from interviews with
enthusiasts, pioneers in the community to innovative projects
and library shaping Elixir's future.
Today I'm joined with a collect's colleague, actually
Jason, to dig into a couple of his projects.
And I'm sure we'll ramble on with a bunch of other random
things, but Bedrock and beadwork are some exciting things.
So welcome. Welcome to the show, Jason.
Thanks for having me on bud. I guess like for people that
don't know you, do you want to just give us like a quick
background to like where you came from, what got you into
programming and like kind of like the start of your career?
Sure. So I've been a nerd doing this
stuff for just about 40 years now.
Now that I think about it, I got my first machine when I was 8 or
9. My dad fixed or repaired
computers for the military and so he always had parts and stuff
laying around. And we cobbled together a
Frankenstein machine from parts at the time.
And it was, it was really deluxe.
It had 512K memory. Every time you turned it on, you
had to set the clock, right. So because it didn't have a
battery, this was old stuff, right?
And had floppy disks. And if you you know, punched a
hole you could double sided. The power button actually killed
the power supply back then. Oh, yeah, no, no, it was legit.
And then and then later on I got one with like a turbo button
that like just changed the clock frequency.
Like it just made it go. It literally made it go faster.
And there were games that broke because, you know, it just they
had like timing loops and things like that.
So anyway, it was the dark old days.
So I've been writing software for a long time.
Badly at first, right? And then after you've been doing
anything for a while, you get a little better at it, I started.
Work, I know. Dude, there's not a you're not
doing, you're not improving as a person if you don't look back at
the stuff you did six months ago and cringe a little bit.
I mean, it's just the way it is, right?
So, so about 16, I moved out the house and got a job right in
software, kind of quit school on a Friday and had a job the
following Monday building things for machine control.
And I had imposter syndrome for the longest time.
You know, I just, I couldn't. Here I am in a room full of
people working and they're, you know, these old people who had
degrees and all stuff. I'm just some dumb kid.
And I realized after a while that like all the books that
they had and books for the thing I, I was reading them and people
coming to me and starting to ask questions.
And so I slowly dawned on me over the course of working with
some of these folks that like I was actually not maybe not
terrible completely at this. And this is something I could
do. So spent some years doing that,
moved through contracting various jobs, so many messes,
different companies, different places, all kinds of different
tech. I remember the web when it was
CGI bin and there's a little compilable, you know, little
programs and had Pearl and all kinds of fun stuff to do Web.
It's it's it's come such a long ways.
It was so dark. I only remember hacking my way
through really ghetto HTML sites with like no CS.
And write a passage well and, and, you know, having things
that suck like, you know, like tools that suck and machines
that suck and are slow. They really, it teaches you how
to, I mean, if you want to get anything good out of them, you
know, you have to work at it. You have to think about it and
you have to try and be clever or borrow cleverness from other
smart people. Yeah, You know, I was just
talking about that with a like a previous guest.
You almost have to just do embedded software, embedded
device stuff now for that because you need the limited
resources to become a better like programmer architect.
Well, it's true. So like I was a kid and I was
into mischief, right. And you know, phones and and you
know, modems and all that stuff was big back in the day.
This is like pre Internet adoption.
And so you could literally just call around numbers and hit a
modem and it'd be like open and then you'd get a prompt and then
you could start doing things. And there were, early on, there
were banks and all kinds of weird things that were just
open, right, people just so you build a piece of software that
would literally just go dial phone numbers, right?
And then when it got something that was they could recognize,
it would just put that on a list and then you'd go back and check
it out. Did I remember that this thing
when I was in middle school, my buddy and I would hang out in at
the library computer lab all the time and we discovered that
there was no security on the school network.
We could go into all the teachers computers and we could
change. We could change grades and we
were like, we were pissed because both of us had really
good grades and we didn't have to change ours.
And we're like, well, what are we going to do with all this
power? So we changed to Buddy's grades,
who was not doing well. Yeah.
Did you get like 50 bucks or something?
Because that's the way that's right.
Like hey bro we can jump you up from this D to well we'll do AB
minus we can't be unrealistic. Right.
And so, so The thing is, you know, you have these
limitations, stuff that kind of forces you to get better about
things. And so grew up with a lot of
that and you know, as computers were getting better and better,
I mean, it's crazy what you carry around in your pocket now.
I mean, compared to the things I played with when I was a kid.
And so, you know, there's, there's different waves.
You see the things rhyme. You know, you get the, the, the
original web wave, then you see the mobile wave and then you
ride the, the, the, you know, the other waves, the software as
a service wave, the whole, you know, now we're into AI, which
is a lot of fun. But every place you go, there's
stuff to do in different problems.
And you, you pick up things from, from, from the folks you,
you bump into. And so I've been doing this for
a while now and there's, you know, you see a lot of things
that rhyme, you know, so you see things that happen over and over
again, which is kind of fun, but lots of different places.
I mean it's like the the amazingness of like the BEAM or
Erlang too, Like the problems they solved back in the 80s are
recurring problems now with AI and agents and processes,
whatever you might need. Right.
You see, I mean, I mean the beam is, I mean to refresh people's
memory, it's like almost 40 years old, like it's been around
for a hot minute. And so you have these things
where like the, the underpinnings of it, some of the
stuff has changed underneath, right.
But the basic way that it works is fairly fairly, you know,
locked in. And you have these, this idea
where you have small processes, you have, you know, not shared
memory. So they can do whatever they
want with their little, their little corner of the universe.
And then they trade things with messages like this has become a,
a, a really powerful pattern. And it works well for all kinds
of the more modern problems, not just dealing with telephone
switches or whatever it was that they built it for originally.
So, yeah, I know it's, it's, it's, it's a really kind of
interesting model. I mean, you see different, it's
one of the things I like about different languages and things
is you can try on a different headspace, right?
Like there's a different way of thinking that goes with each one
of these things. You get the beam and there's a
certain way of thinking that goes along with it.
You got Go or Rust or C plus any of these other things, right?
And there's a, a, a, a, a, a just kind of a headspace that
goes along with it. And it's kind of fun to try them
on a little bit 'cause you can pick up things from one place
and bring them to another, right?
Like, I just had a, a rough project I was working on and I
brought Elixir style telemetry over to it because it made it
easier to, to, to get at certain problems, you know, and it's,
it's interesting that sometimes people don't play with things
that are outside of their normal wheelhouse.
And I think they miss out a little bit because there's
interesting ways to solve problems.
And if you don't look, you don't see, do you know what I mean?
I was better at that when I was younger and for whatever reason
now I just want to be a cranky old man and stay in my bubble.
I hear you. No, it's fun and it's nice.
I mean, immutable data, I mean, just being able to think about
the, I put that there, it's going to stay there and nobody's
going to mess with it, even if I have other processes and things
flying around. And again, the limitations,
right? Because you have this process
and, you know, I have to think about, you know, the, the way
that it manages information, because if I want to move thing
from one thing from from one process to another, I have to
pay that tax of going through the mailbox, right?
And things are copied. So this is, this is a, an
interesting problem because if you have a large piece of
information, you want to move it from one place to another.
It can be slow, but there are ways around this.
And we can get more into that when we talk about bedrock,
because that's one of the tricks I, I end up pulling there.
Yeah, yeah, I think Bedrock will be a fun conversation.
What? Before we get there though,
how'd you find yourself to like? What was your path to Elixir?
And like, what made you discover Elixir and start using a
daylight? 20/16/2017 I had a buddy who was
like neck deep in it, you smart guy, you know and he was really
into it and I didn't quite get it at the time.
It took me a while to come around to 'cause I'm like, OK,
you know, fantastic pattern matching.
Yeah. You know, it just didn't really
click in my brain, right. And then he showed me this thing
once that was like, really like parse GIFs or something like
that. And it was like this much code.
It was so cute. This thing, it just, it, it was
kind of this little haiku poem of, of, of, of software.
And it, it stuck in my mind. And then when I ended up having
to solve some other problems, it, it, it was a tool I kind of
reached for and I, I, I tried it out and it worked And then it
was crazy. I found people pay me to do it.
So that was, that's always fun. You know when you can do the
things that you want to do and and get paid for it, That's.
Yeah, I had a similar story. A buddy was just in my ear
telling me how great Elixir was and I was like, I'm sure it's
great. Like whatever.
And then I actually took it seriously because I was, I
started building a side project, my I, it was called perfect
form. It was like a, a fitness
marketplace and I was like, all right, let me look into Elixir
because I think it sounds like it'd be a great fit.
So I started digging into it and it was like before I even like
actually understood the power that I was learning, but I was
like, oh, I love the syntax. And then once I like wrap my
head around pattern matching and immutability, I was like, oh,
this is I just love it. It's like fun to write.
So I started writing my first like REST API, very object
oriented ish, but with paradigm like.
It wasn't pretty, it was horrible, but I learned a lot
and I rewrote a lot of code 'cause I would like get it
working. And then like, oh, this is like
so wrong, this is not Elixir. And so I'd fix it, correct it.
And then I decided to start a YouTube channel with tutorials
because my learning curve was so difficult coming from like
JavaScript and like object oriented background to
functional language. And I was like, I need to just
like help people get off the ground running and that that.
Makes. Sense.
One of the things I dig about it is like I find myself reaching
for something and I go dig around in either the ecosystem
or frequently just go dig in the beam docks, right?
You go look in the Earl dock, the Earl link docks.
And it's like there's this little gem somebody thought
about it 15 years ago and it's in there and you can just dredge
up this little toy for, for doing things like I was, I was
building a thing for doing file diffs the other day.
And wonder, I wonders there's a tool in there.
It's built into the runtime library.
You can you get the, the Myers diff algorithm is right there.
You just trot it out and poof, you can do file diffing line by
line. Lovely.
And so there's all kinds of toys like that.
And, and what's neat is when you, when you when you're
digging around in there, you can, you can roll your own.
It's really easy. And then of course, like I said,
there's all kinds of stuff to to play with, which is which is
fun. There's a so many rocks to turn
over. I mean, like you said, when you
you get into it and the more you get into it, you start feeling
like Oh well this is neat and processes and OTP look at this
whole thing, supervision trees and all kinds of power this
unlocks. So good fun.
It is fun. And then like, yeah, the more
you learn, like how to use supervision trees, Genservers
and all of that, it just starts compounding and you're like, I
can do anything I want. Right.
And then it's just a matter of having hours in the day.
Yeah, that's true. I don't have enough of those,
honestly. All right, so do you want to
dive into bedrock first? Sure, we can talk about it.
I mean, I think it makes sense to talk about the problem that
we're that it's trying to solve and then maybe loop into what it
does and that attempt at solving.
All right, I OK, I want to read your little like one liner from
your docs and have and have you and then have you explain it.
Yeah, no, just explain it to me like I'm, I'm five, you know?
Fair enough. All right.
So it's an embedded distributed key value store with guarantees
beyond ACID like OK, right, what's that mean?
It sounds there's a whole. Bunch of words in there.
It's got croutons and everything, right?
OK, so, so The thing is that you find yourself, I mean, and
they've solved this problem on the beam in in ways a long time
ago, but it's got real limitations.
You have things like amnesia, right?
Where you have a, a, a cluster of nodes.
And I want to be able to have, you know, records and things
that flow across them so that on one node I can do a thing,
right? And then on the other node, I
see the thing and this is great, right?
This is a lovely thing. Amnesia has all kinds of
limitations, though. You have to like a kind of a,
they didn't really imagine, you know, clusters that would scale
up or down over the course of the day.
You'd have a machine called Ernie and one called Bert.
And they would, they would play nicely together.
And that's what would happen. And so Amnesia breaks down when
you have to do things like add a node or remove a node.
So like dynamically scale nodes? Yeah, pain, pain, lots of pain.
Just Ouch. Hurts, right?
So that's one of the places where it's got sharp edges,
right? So it's it's not, it's not so
fun right there. There are other ways in which
like it can break. There's limitations just on the
amount of data, the people, there's all kinds of fun things
that that come in there that make it not really suitable.
So This is why you don't see a lot of people just, you know,
amnesia, even though it's there and you could reach for it,
right. So you got this problem where it
really is useful to be able to have a cluster of machines and
to be able to have like a consistent picture or something,
right? And then of course, if you can
then run a transaction on that thing, right?
So that you can be sure that what you're changing either
happens or doesn't, right? And then it works across the
cluster. And those machines don't have to
do a whole lot of things, you know to to to get at it from a
ergonomic standpoint. One thing I want to poke at,
yeah, before we like dive too deep into like how Bedrock
works, like on a distributed system, I think a lot of people
miss like how difficult it is. Like, for instance, like when
you're running a Postgres database, you usually just have
one big database, right? And.
One picture of the world. One picture of the world that
all your nodes are pointing at. So you still have that
bottleneck, right? Right.
Well, it's not even just what we've done here is, and this is
a good thing is when we're talking about a system where you
have a bunch of nodes and you still have like 1 database,
right? What you've really done is
you've outsourced your distributed system problems to
the DB, right? I, I, if I'm going to keep a
consistent picture there, I go to the DB and I ask it a
question, right? I want to change things.
I go to the DB and I, I put a transaction together and I do
it. And it's the place that deals
with all of that problem, that hard problem.
The downside is, is that there's only so much bigger you can make
ADB and they don't really horizontal is not a thing that
they do so well, right? They're just, it's not, it's not
the way that's built for a lot of these things.
And so you end up having to reach for all kinds of thing
other other property caching, people want to reach for
caching. But then you have this
distributed problem where I have a user and I just read it out of
the database and I make some changes to it and it's on node
one. The next request comes into node
2, right? I want to get at the same user
the freshest copies right here. But I don't know that.
And so I have to go to the DB again and get it.
And if I try to cache something in a place like Redis or some
other tool like that, all we've done is we've just outsourced
the problem again to another thing.
Right, it's the same problem. You, you, you kind of, I mean,
it's faster, but you can't get right.
And so now you think, OK, fine, well, I'm going to keep track of
this on this node, you know, and this user's right here right
now, and I'm going to let that be here.
And then this other guy needs to talk to that talk, read
something about that same user, and it now has to know where to
find it, right? This is the problem all by
itself. Like where are things just
keeping track and keeping that information up to date is
difficult. And then of course, if you're
going to deal with something like a cash, now you have this
problem where I've got new information here and it might
make a change over here and it becomes stale and now I got to
shoot it down. So there's all of these
interesting distributed problems when you have a computer, number
of computers with, you know, N more than one right now, all
these, there's different places that information can be and you
have to know where what's fresh, what's not, what's useful,
what's not. And then if you're making
decisions about money or people's financial stuff,
anything like that, you, you have to have the fresh stuff.
So This is why everybody just ends up going back to echo again
and going back to the database and letting the database deal
with it, right? Just, I mean, it's, it's a lot
easier to do that. It is it is it is a problem.
You have to pay. Attention, that's when it comes,
that's where the rubber meets the road.
It's like when you have a system that you've built and for most
things, I would say 90% of things that people are building,
this is perfectly fine. This is these are problems you
don't necessarily need to solve. But if you start getting, if you
have something that's big or growing, right, like for work,
work, we have a situation where you know, things are growing.
You have millions of people, you know, you need to be able to
split things onto more than one machine.
And now all of a sudden, you know, the way you built all the
way up until now doesn't work so well, right?
You run into all these problems head on.
And so, you know, the, the idea, the problem that Bedrock is
trying to solve is how do we take and make it so that we have
within the beam a consistent picture from node to node and
that things that conflict because we're changing them in
the changing the same thing in two places conflict properly.
But things that don't touch the same data, they can sail through
without a problem. Like you see what I'm saying,
Like user A makes a change, user B makes a change.
Those things are unrelated, right?
They shouldn't conflict. They should sail through nice
and quick. But if user A is changed here
and user A is changed here, then there needs to be some kind of
order. There needs to be some kind of
order imposed on that so that somebody gets, when they try to
withdraw money, you know, twice somebody gets an error and
somebody doesn't you know, we got to you got to sort that
problem out. And so that's kind of the
problem space that Bedrock lives in is what that looks like to
you is I've got this giant KV universe, right?
And I can which. Kind of like.
Keys and right keys. It's basically a dictionary,
right? When you have, you're just
doing. Oh yeah, it looks like a big.
Value, Yeah, OK. Yep, Yep.
And with some special operations for things like range scanning
where, you know, I want to start a key and I want to grab the
next X number of keys or I want to, I want to do a walk through
a set of keys. You know, things that you can't
really do on like a map, but the map is a perfectly good mental
model for how this thing works, right?
It's, you know, I puts and gets right really when it comes down
to it with with some fancy sauce on top.
So the mental model is easy. I have a thing I want to put it
somewhere and then later on I want to be able to get it back
right. And so that leaves a lot of
choice up to the to the user as to how granular they want to get
it. Do you want to put a whole
record in there? Do I want to put pieces of a
record? You get the idea.
But the important thing is, is that when I'm messing with
something over here and I'm and you're messing with the same
keys over here, we need to figure out who, how this is
going to work so that we're not stepping on each other's stuff.
Because you can imagine we have a key that keeps track of your
bank balance and somebody's doing a debit here or somebody's
doing a credit here. We want to make sure that that
key stays properly updated with the correct value.
And that's going to matter as to which order those things go in.
And we obviously, you know, don't want them both to go at
the same time such that you end up with a mess, right?
So that's, that's the general problem space.
That's what we're trying to solve.
And of course, by making it simple like that, I've just,
I've got AI can start a transaction, I can read and
write and then I can commit it and it either happens or it
doesn't, right. This is a nice, easy ergonomic
mental model for people to play with.
Yeah. And OK, you've made reference to
like foundation DB and stuff, right?
And that that is written in C++. They actually have their own
weird language that like transpiles down to C++.
And the reason they did is the the it's funny because
Foundation DB actually is built on the actor model.
So it's they're trying to do the same thing that we do on the
Beam with message passing and all of this.
I was going to ask, So what advantage do you get using the
beam? Like, it sounds like you would
get a lot more out-of-the-box just.
There are some places where the the beam is really sharp on this
and there's some places where that hurts too.
And we can get into that in a minute.
The, the general idea is the foundation, give these, these
really smart people a decade ago, decade and change ago,
they, they came up with this. They had the same basic idea
where I'm going to have a key value store.
We're going to have it spread across a distributed system such
that all of these clients could read and write and keep their
stuff straight and it would be nice and simple.
And if a machine goes down, it's not a big event.
If parts of the cluster go down, it's not a big deal.
Things come and go, right? Sounds lovely.
And it was, it was lovely enough that Apple bought them, right?
Apple bought them back in what 15 and had them for many years
and most there's a good chunk of Apple right now.
Cloudkit, all that stuff runs on top of foundation DB now and and
I don't see it going away anytime soon.
So again, the same mental model where you have key values, you
have transactions across them and that you can guarantee what
happens and what work, right. This is, this is a good thing.
Like I said, they built it on the actor model.
So you have these this idea of like little tiny processes,
right, that do one thing and they move things back and forth.
And if there's always a sound and like a really good fit for
some things, you know, it is so this this this model fits onto
the beam in a pretty it just kind of sneaks in there in a, in
a, in a, in a really nice way. So there's some wrinkles, you
know, because as we mentioned before, like there's copying,
you know, when you when you send messages back and forth in
between processes on the theme, like people don't realize this,
but things are quite right. There's the processes don't
share anything. So if I haven't a map full of
stuff. You also can't control the order
your messages come in. No, yeah, you have kind of a
rough, they have a priority mechanism.
Most people don't know about it or use it, but there's a thing
you can do, right? But yeah, it's there's a lot of
there's a lot of, you know, messages are coming through the
door. And so when I have a map and I
send you a message, it's nothing but just a map of things, right?
That map gets copied onto your other processes heap, right?
And then your process kicks in and now it has that stuff.
And so it's immutable, but it's also copied.
So if you, This is why some people don't realize that you
can't just ping pong big data structures back and forth
between processes. Hey, why is my stuff slow?
Well, it's because you're doing a silly thing.
You know, it's like, it's a little silly.
There is one exception on the beam, and this is binaries that
are larger than 64 bytes for, you know, nerds out there,
right? If you are tossing a binary back
and forth between two processes, it's free.
You're just passing a pointer, right?
That's reference counted. There's like a global heap for
binaries. Yeah.
So if you think about this now, you know, from a standpoint of,
like, I'm going to put together a transaction and it's got to
flow through the system, and a lot of things need to look at
it, right? And where it ends up, it's going
to end up on the disk somewhere. You know, if I'm packing and
unpacking this thing, it's going to be really slow.
If I send it in some kind of disaggregated form, like I said,
a big old map or lists of stuff or whatever, it's going to be
copied and that's going to be slow.
And but if I send a binary, right, that can go from process
A to B to C to D all the way to the disk with no copies, right,
Zero copy. How do you?
It doesn't move at all. How do you control state at that
point? So you got to be clever.
So like what you do is you just say, for instance, in Bedrock,
what we do is we take and the, the client actually puts
together the transaction. You have a little packet, right?
It binary encodes it like the little binary that's built.
And what happens is that's what it submits to the system, right?
It's a little encoded key value pack with just the stuff that
changed and what was read, version numbers, a few other
things, some details, but it's basically a little binary
package and it flows to the next process and it flows with no
copies. And so the next little process
in the, in the, in the chain looks at it and says like, oh,
well, this is what I got to do with it.
And then it moves along and so on.
And eventually this thing ends up on the disk.
And there are relatively few places, a lot of places that are
taking it apart and reading it right, but nothing that's
actually changing it. And so you don't end up having
to copy things that way. People don't realize it's
something like ETS, like if you have an ETS table and I put a
giant piece of data in it and I go to read that from another
process that's not free. It copies each time I read it
from ETF, it copies it right into my memory, right.
So if I go put 60 megabytes of something into AETF table and
then read it back in another process, dude, if it's not a
binary, you're, you're, you're making copies of that stuff each
time. So, you know, it's there's,
there's, there's some limits. We got Mike Hostetler in the
comments pretending he just learned about this today.
They said today I learned about passing binaries among OTP
processes. So awesome.
Same dude. I also just.
Learned it is it is true. I mean, but see, again, this is
one of those places where, you know, Harkins back to you go dig
in the docs and you peel back the layers of the beam and you
take a look at like how it works and and and what it does.
And I just started this once, Dude, it's good.
It's good. It's good, right?
Where does that go wrong? You know, I love books, you
know, and so The thing is, is that you, you, you learn about
these things and you're like, oh, wait a minute.
Well, if I know about it, then I can take advantage of it, right?
And that's, I think this is where it comes down to a lot of
different kinds of software. It's like you have to understand
the thing that you're using. And if you do, you can squeeze
and speed out it. So bedrock story is how do I
take and, and keep these things straight?
We can get to that in the details of that if you'd like,
but we make it so that also it's moving things quickly, right?
Well, so, so let's, let's walk through how how it works.
Foundation DB is what's called a disaggregated database.
In other words, normally you have this monolithic thing.
I submit a query to things disappear inside it, right?
And it's one big process that goes through and chunks through
all these. Well, most of us know and then
deal with like, right? With like breathing?
Yep, like breathing like this is this is, you know, we, we do it
without even, we're fish in a fish tank, We're not even
thinking about the water anymore, right?
We're just so used to it, right? This is, this is the way things
are done. And so these foundation DB guys,
they're really smart dudes. They, they basically said, OK,
well, how do we turn this problem on it's head?
How do we split it up so that we can make this problem into
something that we can do across all these computers and still be
safe? And what they figured out was
that you, you need to be able to guarantee an order between two
things. So if you're making a change and
I'm making a change, you know, there's two things.
Those two things can't happen at the same time.
We need to have a kind of a logical concept of what, what
goes before what. So they have this idea of this
thing called a sequence, right? Everything gets a sequence
number. Wait, wait, wait.
We might have. OK, we lost you for a second.
Oh, I I think your Internet just got a little laggy, but just
talk about sequence order from the start again, yeah.
Sure. So, so sequence order like we
have to be able to, to, to, we have to understand that things
happen in a particular order, right?
So I'm going to do this debit and then there's a credit.
Things might be different if I did the, you know, the credit
before the debit. Like it matters, right?
And we certainly don't want things happening at the same
time. It's OK if they appear to, if
they're not touching the same things, but for certain for
things that are touching the same pieces of data, they need
to happen in a particular order. So they realized this and they
said, OK, fine, we're going to, we need a concept of a sequence.
Great. So they, they, they have that.
Then then individual clients need to be able to read data,
right? So when I set up a transaction,
I need to be able to, I need to be able to read this stuff.
So what they do is they say, OK, well, when I do a read, I'm
going to do a read at a particular sequence number.
So first thing I do if I want to start doing anything is I get a
sequence and I say, OK, this is, this is the version of the
system, right, that I'm going to look at.
This is my snapshot. We can imagine that everything
is frozen in time, right At that point, right?
This is the point I'm going to look at and I'm going to go ask
questions. I'm going to say, hey, you know,
I want to read this key. I'm gonna read this key.
At that point, I'm gonna, you know, I'm gonna write something.
I'm gonna write it. It'll happen right after that
point. What does that do?
Like this just sounds like now you have to manage all these
copies of a entire database, OK? Yes, and we're gonna get there.
And it's really, it's really sick, right?
So here's what happens. So I'm, I'm reading.
I'm reading things happily, right?
I, I said, OK, all these things, I read it at a particular
version, so that's important. I read key A at this version,
key B at this version. Cool, great.
Now I'm going to make some decisions.
If we're doing anything useful, if I'm just doing a read only
transaction, it doesn't matter at all, right?
We're done. Like I've looked at my
consistent snapshot. Things aren't if I read the same
key multiple times, they're all read the same version.
Nothing's changing. I'm not seeing half of somebody
else's data. I see a consistent picture.
You can think about this in a sequel database.
We get the same thing when we create a transaction.
And depending on the isolation level, like if, if, if I have a,
you know, full isolation, then when I'm making changes, I see
my own changes, but I don't see other people.
I don't see your half baked, you know, import of something until
it's done right. So we have this idea of
atomicity. You know, this is the a, an
acid, right? So how Foundation DB gets there
and how Bedrock does it too is you have a version that you're
reading at? I'm reading, reading, reading,
and then I make decisions. So there's logic that's
involved. You know, it's I'm doing a
debit, right? I'm going to update other keys
based on the things I read. So I'm going to do a thing.
I'm going to read some data. I'm going to subtract 50 from
it. I'm going to come up with a New
Balance. The New Balance is what I want
to save, right? This is important.
And so now we have a thing that can conflict, right?
So when I have, when I, when I put together my little
transaction here, I keep track of all the things that I've read
and at what version and what keys I want to change, right?
I submit that to the system, the system Now, like you said, we've
got copies of these things. We need to reconcile this,
right? How does this work?
And So what it does is there's some, some some services called
the resolvers that are in this process.
And what it does is it they basically say, OK, well, we're
going to take these two things and we're going to check them.
And the question is, is did this transaction read or write any of
the keys that this guy needed? Right.
And if the answer's no, then they don't touch and they can
both sail through. They can happen in either order.
It wouldn't matter. OK, right.
So no follow me. It flows through and they.
Can both touch sail? On they can both touch the same
row or key. At this point, they.
Can read the same things, right? But if one, if somebody writes
something that you read, so let's say at version zero, you
know, you read a key A, right, version 0A, right.
And you and somebody goes along and while you're doing your
computation, somebody writes A at version 5, right?
So this happens. A has changed.
Now when we go to reconcile your your transaction, we're going to
say we read A at 5. We made decisions and you know,
all the decisions are based on the fact we read A at 0, excuse
me, and somebody changed it. So because there's an
intervening change, we know there's a conflict.
I see. You follow me.
Yep. So like, you know, so like you
think about it with people, right?
I, I, you know, you're basic. I'm going to give you some
information, right? You're, you're, you're, you're
going to make some decisions based on it, right?
And then somebody comes along and does something to one of
those underlying pieces of information, right?
You're, what you've done is now suspect.
It's not right anymore, right? Somebody beat you to it.
Two people pulling money on the same bank account.
Deciding how to do cash like that's a very difficult problem.
Yeah, sure. Perfect, right.
So like, you know, you have this situation where it's like, I, I
need to know, like the database needs to know, did anybody touch
the things that this, this set of decisions was based on?
And if the answer's no, great, sail through.
Awesome. That's most the case.
So the thing that the foundation DB people got that they
understood was that lots of these changes to the system
don't have anything to do with one another at all.
Most of them, in fact. Right?
If I've got a user over here and a user over there and they're
screwing with their own data, man, none of that touching.
Like. You're in your own, you're not
colliding. You're right.
You're doing your own thing. And like where what needs, what
you need to be able to do is you need to be able to figure out
when that's happening and then what to do about it, right?
And so their answer in this situation is really shockingly
simple. You basically do this.
You say, OK, fine. Well, we found out that I read
this key at a particular value. Jacob changed it, right?
I want to do something that was based on that information.
I can't do that. It basically figures out that
there's a conflict the first time first conflict and not not
which ones, just that there's a conflict it it just rejects the
transaction says no, can't do it, no good.
And then you now how that service.
Fetch the data or. Yeah, you basically redo it.
Now, what's kind of interesting is in Elixir you can just, you
know, I have a transaction, I pass a, a block, I pass a, a
function, right, that does the actual work.
It reads and does all the reading and does all the
writing, right. And then and it's done.
It's going to go ahead and submit the transaction.
This is all done by the machinery for you.
And if there's a conflict, what it will automatically do is just
rerun your function. I'm curious.
Simple like when you're running like a database transaction at
least like in a post address world like you can lock a row or
lock a table, right? How?
How would that work as bedrock? You don't, you don't.
And So what happens is what what happens is you basically things
either conflict or they don't, right is what happens.
And so, so if I'm, if I'm I'm going to go, you know, we're
both going to go start a transaction.
We're trying to take money out of an account, right?
You read the balance. I read the balance right.
We both decide that there's 20 bucks in there and you're going
to pull out all 20 and I only want to pull out 10.
But obviously both of them can't happen, right?
One or the other has to happen. So now you know, let's say
you're a little faster and you get yours in there first, right?
What happens is, is I see mine at the at the same or older
version. I my change wants to happen
after yours. What we see is that there's a
conflicting key, right? And then mine gets rejected and
so then my code runs again. I pull the balance, the New
Balance which I read is going to be 0 because you took all 20.
Bucks and you're like shit, right?
We got robbed. Well, and then I just, well,
then it, what, what happens is instead of a database error, it
turns into an application error where it's like, oh, sorry,
there's not enough balance in here, right.
And so it's it the code for a bedrock, you know, when, when
you're working with it reads really nicely because I get the
information that I'm basing my decisions on.
And then I do things and then the whole thing either happens
or it doesn't, right? And if there's a conflict, my
code runs again, right? I, I read the values and, oh,
this time there's no money in there.
Well, I just returned. Sorry, Jacob, you know I took
out the money. I'm all right.
So this is all happening. We have a distributed system,
all these nodes, right? Where, where does the, the, the,
the truth? Where Where does everything sit?
Because that can't Can you pick a master node or what's
happening? So what, what the, the, the, the
trick that foundation DB pulls and we ape is that they, they
realize that you actually don't need a tremendous amount of like
coordination once you've got the, the basic pieces of the
system set up and wired, right? So now imagine this I've got,
we've got this, this, this commit proxy.
This is the thing that sits in the in this is what I submit my
transactions to as a user, right?
I've got some number of them in the system.
They're well known if it's going to talk to the resolver, it
knows about the resolver. That's cool.
All these pieces are wired together and the information can
flow really quickly. What Foundation DB realized is
that that under the normal static state of a system, right,
things can flow. We don't have to figure out, you
know, is this guy still up? There's all these complicated
things that you that could go wrong, right?
So what they figured out is that instead of trying to come up
with all of these different compensations for all of these
edge cases, they do something really simple.
They say, OK, we'll run the way we're going to run until there's
a problem, right? So I, I'm a commit proxy.
I try to talk to it, it's broken.
You know, there's a problem. The commit proxy tries to talk
to the resolver. Resolver is broken.
There's a problem resolver. You get the idea that you're
going to pass to the logs. You know, there's a problem.
So what, it's what their whole trick was, is to figure out that
like when something goes wrong, to realize it, to have something
that's kind of watching these things to realize that there's a
problem. Mark the system is invalid.
Stop everything, tear it all down, put it all back together
again. Very OTP, very supervision tree,
right? Set up the supervision tree of
these processes again and then let it RIP again.
And so it's it's the model fits on the beam so well.
Yeah, No, it sounds like, yeah, I do have a question, but we're
going to take a quick break for ads and then I have my question.
All right, cool. I will I.
Will I will. Coding Elixir with AI or
building AI into your Elixir apps?
The Elixir AI Collective Discord is your community for both.
Link in the description below. Hey, I'm Kimberly with Elixir
mentor Jacob recently added recruitment services to help
companies hire pre vetted Elixir devs.
I help with company outreach, so if you're hiring, I may be
reaching out, or you can connect with us anytime at
elixirmentor.com. We're going back, all right.
So my question now is like still kind of going back to like where
is our central point of truth for state, right?
Because we have nodes and they all have to stay updated with
the current data, right? Or maybe they don't.
They do and they don't. So here's what happens.
Think about if I got. Let's think about the simple
case. I got one machine, right?
I got a commit proxy. It needs to resolve
transactions, right? If I'm doing this all on one
machine, I have a commit proxy. I want to hand it a, a, a
transaction. You're going to hand it a
transaction. It's going to try and sort these
things out, right? So what it's going to do is it's
going to handle that transaction to a resolver and says it was
anything in the last seconds touched this in such a way that
would cause a conflict. And if the answer is no, then it
moves on to the logs. And if it gets logged as
something that actually happened, right?
And then from there it goes to storage and then we're answering
questions again, you know, so, so I go from commit to
resolution to the log, to the the write ahead log.
And then the storage machines read from the log and then
project that for people to read questions again.
What is the difference in need? Like what the log state versus
storage? What are?
The two jobs there fair enough. And so The thing is, is that
like we mentioned that things have to happen in order, right?
There's, there's has to be an order to things and we have to
agree on it, right? So I, I go to submit a
transaction, that transaction gets stamped with the number,
the sequence number, right? It gets resolved when the
resolution happens. And if happy path, everything is
cool, it's going to go to the log.
That's where it's recorded for posterity.
So the log system is looking at this and going like, oh, I'm
waiting for a transaction greater than 3 right.
And then, OK, cool. I'm waiting for now 4, now 5,
now 6 now 7. And we're going to go ahead and
write those to disk. So the log's job is to take all
the transactions that have been agreed upon like the everybody
has these don't conflict. This is cool.
Their job is to write them to disk as fast as possible.
Just put it on the disk. And so as soon as that stuff is
written to disk, that's on the record.
That's that's the way things happen.
Now, if I have a multiple, multiple of these logs, right,
I, I send the same information to each one of them, It's
replicated onto each one. As soon as I get an answer from
all of them, then I know that that stuff is cool and it's safe
right? Now, the storage system, it
reads from those logs. It basically says like, hey, the
last thing I got from you was three.
Let me read from three on, right?
And it gets and it applies those transactions to it's in memory
state, right? So it's taking those things that
we agreed upon and it's keeping track of them and saying, OK,
version 5 is this change. Version 6 is this change.
Version 7 is this change. So when you go and ask a
question to the storage, you're asking it at a particular
version. Do you see what I'm saying?
Yes. Follow me.
So when we started off, first thing we did, I'm going to go
ahead and we're going to move money from one place to another,
right? We get a transaction number.
We say this is the read version for the system.
This is where we're going to read at right?
When I then I ask the storage server directly, hey give me
this key at this version it's going to do.
It and it doesn't matter if another version already exists.
Nope, Newark. It gives you the one picture
that you're asking. For yeah, OK.
Right. So it keeps track of and so now
you can't do this forever back. So Foundation DB famously has
this, the 5 second rule, right? And So what it does?
Basically. Very much the same thing, right?
So the idea here is that it keeps track of basically the
last five seconds worth of transactions.
And anything older than that, like if you go to submit a
transaction, it's older than 5 seconds, it just says no, let's
too old. I can't, I can't check it,
right? And so it keeps track of five
seconds worth of information. So this information is
constantly being flushed out. So you can see that like when
I'm doing resolution, the resolver only has to keep track
of five seconds worth of stuff, right?
The logs, that's posterity. And then the storage boxes only
have to keep track of five seconds worth of versions to
read right after that window, that stuff's on the disk.
It's there for good, right? You can you can consider.
That like you just dump everything after the five second
window. Yeah, it just throws.
So what happens is it flattens. You can think of like a good
mental model's git, right? I, I, each one of these
transactions is a commit, right? We have to agree on what order
they go in on, right? So, you know, I'm going to
commit, I'm going to commit, I'm going to commit, I'm going to
commit. These are each changing things,
right? And we, we can agree on the
particular order at which it happens.
And then what happens is that the five second mark things just
start squashing, right? It starts squashing down right,
right. So you're you're basically just
like, you know, you can, you can read it, but you can't like
distinguish a version, an individual version older than
the certain mark that that the five second mark, right?
It just like history just gets flat.
It like runs into a wall and just gets flat, right.
So like, does that make sense? Yeah, so is every node
constantly doing some sort of version message passing check
then? No.
And this is one of the clever things that they did.
They basically said, OK, we're going to have the sequence
number and it has to increase, right?
You know, it's always monotonically increasing.
So what they did is they took time in it's like microseconds,
like nanoseconds, microseconds. It's a time stamp.
And So what happens is the sequencer is we're handing out
these, these numbers, right? And they're time stamps.
So I can basically look without having to coordinate clocks,
without having to do any of this stuff.
I can just look at this version and say like, is this 5 million
more than, you know, the oldest 1?
And so things just fall off. It's, it's really truly
brilliant. I mean, the, the engineering
that these guys put into this is just top notch.
I think the problem was, is that like the material science that
the, the toys that they had to play with to build it in just
weren't really there to, to, to, to see their vision out.
So they had to like come up with this crazy stuff.
But the beam is a good fit today.
So you get this 5 second window, right?
Stuff that kind of falls out after that.
The storage servers only need to be able to answer versionable
questions, right for that 5 seconds.
And then anything after that, it's just like, Oh yeah, well, I
got the, I got the most recent value of that, no problem.
So they're not having to keep track of like every version
ever. There's a, there's a window.
Yeah, 5 second in space. So now did you copy that same
idea? Oh, same.
I, I stole the stuff straight up.
When you've got a good idea, you roll with it, right?
You, you take, you know, it's, it's, it is the, the, the height
of arrogance to think that you can completely improve on
something. I mean, generally around the
edges, but you know, when, when there's a good coherent idea,
right, you stick with it. So they have this idea where you
get processes like these pieces you meant.
We, we talked about the the players were talking about.
We got to commit proxy the resolver with the logs.
We got storage. There's a couple of bit pullers
around the edges we're not going to need to mention, but that's
the general idea. And we we wire them together and
once they're wired together, that's the system's runtime.
It's going to run that way until something breaks.
Now machine goes down, right, Something dies.
Now you got to think about like, well, OK, I could have all these
complicated retries. I can have all this, you know,
crazy stuff that's really hard to test because distributed
problems are hard. Or, and the answer they chose, I
think was the, the, the brilliantly simple one.
It's basically OTP. They throw away the process tree
and build a new one, right? So there's a, there's a, a, a,
a, a Singleton in the system of the cluster controller.
It's job is to just kind of keep an eye on things.
And when it sees a problem or a problem is reported to it, it
basically says, all right, this version of the system is no
longer valid. Can't, can't, can't process
transactions anymore and we're going to tear it down.
What does that look like on the Beam?
Is that a supervision tree or is that like a Gen. server?
How are you doing that? It is it, it, but, but it spans,
we're spanning the clusters, right?
Because I mean, the different processes can run different
places, right? But it's the same idea, right?
I mean, if you think about it, it's a distributed supervision
tree is essentially what Foundation DB built.
They say, OK, we got this node. It's a cluster controller.
It's keeping track of all these other guys.
When something breaks, we're going to tear them down.
We're going to stand back up, right, and then wire them
together and then we can run things again.
And So what they said is they said, OK, fine, if this process
is, we're going to make it simple, this recovery process,
we're going to make it simple. And we're going to make it so
that it's really just, there's some straightforward rules that
you can follow. So we tear the old system down,
we stand up new processes on the various machines to do this
stuff. We wire them together, let them
know about each other, and then once everything's up and running
right, we can launch transactions again.
And then things will hum along happily until something breaks
again. So with Foundation DP, this
recovery process is designed to be simple and reproducible and,
and, and correct. And so rather than trying to
come up with every edge case, they take the BEAM philosophy
OTP philosophy, which is like, oh, let it fail, let it, let it
fail. We'll just tear it down and
build new ones. So you can see how this fits
neatly onto Elixir and Beam, right?
So you got a cluster, there's a bunch of machines we got to
agree on who's going to run the Singleton, right?
So in, in foundation DB, they use, what is it not Raft the
other one, they use a mechanism for determining which, which
who's going to be the leader, right?
So they, they elect a leader. We have this, we have a, a
notion of that. And, and once we have that, we
can, we can establish A Singleton and then the Singleton
stands up the cluster and then everybody can run right?
In bedrock, we use raft. So the nodes, they dynamically
discover each other. They say, oh, well, you know, I
know who I'm talking to. We agree on a leader.
The leader sets up a cluster controller cluster.
Controller off. Coordinators.
Oh, you mean the the algorithm. No, it's how is it?
It's not raft. It's it's on the tip of my
tongue. It's consensus algorithm.
That's the one. It's, it's all, it's a funny
paper. The paper for Paxos is really
weird because the guy wrote it like it was, he's telling the
story about these people on the Isle of Paxos and the island and
there's characters and inhabitants and that's how we
wrote the paper. It's really quirky, but I, I, I
misplaced it. But yeah, they're using Paxos,
right? Which is up until Raft came
along, that was kind of like the thing.
Raft is like a simplified, more straightforward, easier to
understand consensus algorithm. And so the, the general point of
it is like, we've got a bunch of machines we got to agree on on
something, right? In this case, we're agreeing on
who's going to start the system. And once we have that, that can
say, OK, I'm going to go tell people what to do.
They're going to listen to me because I'm the guy, right?
And then once all the pieces are stood up, they're wired together
and then they just run until something breaks.
And then the idea here is that when something breaks, we tear
on the old system, we just do it again.
And so this is how, if you think about it, that's how a
supervision tree works. I've got a set of processes and
they got children, right. And when something, something
goes wrong in one of the children, we don't sit there and
try and figure out every little thing.
We let it fail. We let things die.
And then we have rules for how to build it back up again,
right? It's, it's, it's extremely
similar. So it fits nicely on, on, on
this model. So bedrock is foundation DB on
the beam and with some basic ideas moved around so that we're
we're not having to because I mean, they're all written T and
stuff. So you can modify data, you can
keep the immutable things are happening and you can do you
have to do things a little bit differently.
But the the compromises actually make the system run faster.
Because now one of the things that people try really hard to
do in some of these things is get to 0 copy.
The more time when copying things around, the slower it
gets and so zero copy is it's. Wonderful, right?
It's extra work that you can't control much of either.
It's just background. Right.
So the idea here is, is that the fast, the ideal number of times
I can copy something is 0, right?
And so if you're in the beam, if you make a binary and you set it
up properly, you encode all the little bits and stuff, you see
what I'm saying, right? You get it out.
It just kind of falls out for free.
And so now you get the system right.
So you got this idea here where you got this disaggregated
system. It's got this really simple
model for standing itself up, right?
There's a, there's a very clear path for information to flow.
It can flow in such a way that until it has to leave that
computer, there's no copying. So if you're running everything
on one note, it's moving from process to process to process to
process to the disk, right? Super fast.
But if, if you got, if you got to move it across the network,
moves across the network. But again, the minimal number of
times and so you have this, it fits well and it actually can
can go really fast. Foundation DB deals with this
concept of you have redundancy. So I want to make sure that my
data's safe. So you have these multiple logs,
things go to the logs, and then they go to multiple storage
servers. So that's how they manage it.
You can think of like everybody knows like what a RAID array is,
right? You have like 3 disks and it
writes, you know, every piece of information gets written to two
of them, so one can fail. You get the idea, right?
They do the same thing, right? So they read from the logs,
they'll have 3 storage servers, right?
Each one reading from the logs and they're all recording.
Any one of them can answer questions, right?
But if one of them dies, nothing's wrong.
Bedrock takes us a bit further. When Foundation DB was designed,
they didn't have things that were reliable like Amazon S3 or
GCS. You have these object stores.
Now I can put data out there and I'm pretty sure the universal
burn before I lose that information.
I mean, there's so many nines on the reliability of those
services that you just don't have to think about it.
Once you hand off a piece of data to GCS or S3 and they tell
you it's OK, it's good, you don't have to think about it
ever again. So what that does is if we bring
that into the mix now, now, now you don't have to worry about
having all these storage servers all over the place keeping track
of all this. You can just funnel the stuff
from the logs to something like GCS, you know, either locally
run or like that. And now, now that data is there,
that deals with the replication issue, right?
It's nice and simple. At the same time, it's also
available so any of the other nodes that want to read that
block of data, you can do it from there without having to
coordinate so. You can have any S3 bucket the
Oh yeah, the primary target. Yep, you just the data just get
dumped in there blocks of transactions at a time.
So as these things are moving along, you start off with my
individual transaction right at the commit proxy, those get
batched together into a bigger block, right?
And then when it hits the logs, those get batched together into
a bigger block. And then every so often those
blocks get sent to S3 storage or wherever.
And so you end up with these, you know, 60 some odd Meg chunks
that end up on storage. And then when you're going to
materialize them, you pull down these, you know, chunky but
manageable pieces of information that are, that are, that are set
up for easy consumption. All right, my next question is
why? Why would you use anything else
like? Why would I use Postgres
anymore? Like where does where does?
Like? It's a super interesting
question and Postgres or my sequel doesn't.
Well, again, these foundation DB and I all credit to these
people, right? When you really start picking at
it's a really under appreciated piece of tech.
The more you look at how this thing actually works, how their
thing actually works and, and, and the patterns that I
followed, it makes a lot of sense.
So you say, OK fine, now I get this KV storage, right?
I've got this. This is great.
What if I wanted to write SQL queries against that, right?
What if I want to do something like that?
I could store rows as keys in the system.
The indexes are the, I mean store rows as values in the
system. The indexes are the keys.
I might store row one at, you know, whatever table slash 1.
Table slash 2, table slash 3. I could write queries that would
go get these things. I could have indexes that are
just other ranges of keys. You could, they did as a as a
proof of tech, they took Sequel Lite and they took off the B
tree engine that makes you know, writes stuff to disk and they
pointed it at Foundation DB. The Foundation DB people did
this and you could have Sequel Lite that would actually write
to read and write to Foundation DB.
And then of course, you could run that on all of these
different machines and you got all of the benefits and
transaction abilities of Foundation DB, but you got
sequel too. And so one of their things, one
of their big ideas was that you have this KV layer and that's
kind of like the base substrate. That's the bedrock, if you will,
of what you would want to work on.
And then you, you can layer on top of this other kinds of
systems and they all then inherits the properties of the
underlying pieces. So if I were to build something
that knows how to do sequel and answer sequel questions on
Bedrock or Foundation DP, then all the transaction abilities,
all of that stuff I inherit for free.
But why? Why do you even need to do that
at that point? Like if you're.
Sequel friendly sequel's fun. You know, I mean it, it's, it's
a great way to be able to compose like if, if, let's say.
Good question. Once Upon a time in database
land, right before sequel, right, you had you had all these
systems where they were all different and you could answer
questions, but you like some of them.
You had to like follow nodes and read from one to the next.
And like these databases were complicated and weird.
And the sequel came along and made it so you can ask general
questions, you know, give me the sum of all this stuff where this
is that, right? That's great.
It's powerful, right? It's nice and and and and and
comfortable, but at the same time, you know, you do have to
make some concessions to it. There's a dude I want to shout
out to that that is working on an ecto layer for Foundation DB.
So you can compose ecto queries and it will read and write from
Foundation DB. So you get all of that
beautifulness, right? You get all of that with the,
with the sequel ability to be able to compose quotes and with
Ecto, you get all that comfort, right.
But if the storage model is handled and it's distributed and
it's replicated and you don't have to worry about that would
take not very long to port something like that to see like
me like a week. That's awesome and that makes
sense to me, but not but the need for like.
I would just adjust my daily procedures to not use you.
Could totally do that too. You know.
Here's where it's really cool. I open up a transaction, I do
some SQL stuff, I read and write some key values.
I save it. That's all one transaction.
Yeah. Right.
It's all one transaction. So you could actually have a
blend of the two and have that consistent picture.
A good example, you know, some people have won't name names,
but some people have a situation where they have a a primary
database. That's one system and then
you've got something like Oban in another, right And so Oban
loves Postgres and you know, other systems are not Oban and
you're not Postgres. And so they don't you can't
necessarily run a transaction across the 2.
So if I have something where I want to, let's say I, I complete
an order and then I want to schedule a job that happens
after that I have to do a little bit of a dance to make sure
that, you know, that thing goes through or the job insurance
get, you know, like the both things happened.
There's like you have to do a little bit of a dance to make
sure that works. Imagine a situation where you
could, you know, take your code that's used to run in Sequel.
You do that. You could you could have it in
both systems and have that consistent picture without
having to do anything special. It just works, you know, so you
can do some neat things. We have on top of Bedrock.
There is just got some PRS for it today.
There's a job layer, right? So I can submit jobs.
They get composed into keys. There's other processes that
know how to. Yeah, you see.
What I'm saying I need to just drop postgres and use bedrock
and kill switch I. Mean I I think I mean.
We put because I also have an. Effort into anything.
I also use Oban and if you're already integrating a jobs
scenario sounds like well. One shop.
Think about what you need for jobs.
Right. I need to be able to work.
On job scheduler maybe? Right.
So I need to be able, I need to schedule things that need to
happen. I might want to schedule them in
the future, right? So I have a key space where I'm
constantly grabbing the next job.
I want to be able to do that in such a way where two things
aren't conflicting that, you know, two job runners aren't
like fighting over the same stuff.
And it turns out that Apple has the same problem.
They have Cloudkit and there's a million billion things going on
in Cloudkit all the time that are scheduled jobs.
So they came up with a system called quick, right?
It is a, a, a protocol on top of foundation DB, right, by which
you can use the key value system and the transactions to do and
run distributed jobs. So now you can have any number
of jobs that are running, if they complete, they complete, it
gets scheduled and marked. If if they fail, then they're
not failed forever. Something else comes along and
eventually we'll pick it up after some lease timeouts.
You get the idea. It's like they have solved this
problem. I was able to take and the same
algorithm, they published a paper.
I took the same algorithm and dropped it on Bedrock as the job
queue system. And that's what you see there.
So you can run jobs, you submit them, it's very lightweight.
You can choose which machines are running the jobs.
It's done efficiently, it's distributed and everything is
working against the same picture as everything else you're doing
in Bedrock. Fun stuff.
Yeah. So you want to think in layers,
right? Like there's, there's, there's,
there's, there's layers to it. You know, you, I've got the
jobs. And now on top of that, maybe I
want to build a workflow system, right?
So now that I got transactions and I can do jobs and I can do
jobs, transactions. Well, now I can think about a
workflow or an agent or something like that that sits on
top of it. And now I can be.
Now you see why Mike is interested in in.
Easter Hostetler. I mean, MM has it integrated
into Jeeto, right? Yeah.
He's he's Guinea pig and things
for me. So what I was hoping when I
built this thing is that there's a problem set.
I mean, if you want to have a distributed cash, if you want to
have a, a, a situation where I've got all these machines, I
want to be able to, you know, look into cash, get a user
record and know that it's the freshest thing.
Bedrock or something like it could work really well for you.
And then if you're changing things, you, you know, you write
through, you do treat it like a write through cache, write to
the database, but write to Bedrock at the same time.
And that way all the other machines can just look at.
It on like a performance level, like obviously having a single
database, there's a bottleneck there, right?
Right. Is there ever an issue with like
too many nodes or like are you going to run into like syncing
or? Well, that's the beauty of the
system is if, if information flows in One Direction only,
right? I mean like so.
So I've got transactions, they get resolved once we're sure
about who who's doing what, then they get logged, and then the
logs can be read by anything. And it's always first time
served, right? And then you get rejected if
you're asked, Yeah. Right.
It's who, who because I mean, we can't know, right.
So it's, it's, it's however, who, whoever gets to it first.
And so whichever order they're assigned when they happen to
come through, it's gonna, you know, they'll, they'll they'll
be de conflicted and then one or one or the other will lose.
How would like fast like if you have like a really fast rights
or like an update to a single field would that would it just
overwrite like as the second one comes in?
Well, that's The thing is everything is de conflicted
before that happens, right? So you know nothing, nothing
makes it to the to the logs until it's passed resolution.
And resolution is the explicit step of just how do does this
touch this? If the answer's no, yes, it can
go right. And so there's some clever stuff
with batching and, and to make this whole process much faster,
but so that we're not constantly communicating between these
parts. But it's, it's relative.
I mean, it's fast. So I mean, on a single machine,
you're, you're getting things written at the disk and round
trips and, you know, microseconds under millisecond
easy. You know, the nice thing about
the model is, is it hides the latency that you would normally
see to something like S3. You know, S3's they're they're,
they're, they're vast. It's going to be.
It's going to be you. Can store data forever, but it's
a little slow, right? And so what's nice is the way
the system works is if things batch together larger and larger
chunks and then end up on S3. Well, and you're not.
Big meaty bits. You're not getting your data
from S3, your nodes are. No, no.
And what's happening is, is you can you have this idea of the,
the, the storage thing is reading, reading from the logs.
And the worst case goes back to S3 for it.
But basically it's just playing back transactions.
And you can have, because it's just playing back the same log,
you can have more than one of them.
So imagine this, I've got a cluster of machines.
I've got some hot spots where some machines are handling a
particular set of information. There's nothing stopping Bedrock
from spinning up a materializer. It's something that's dealing
with the storage and materializing that right there
on that note where the data doesn't have to leave at all.
So now you got one process that's watching the logs,
updating the internal state, and then other things are asking
questions of it. Right there.
You can move the information right to where it's being used.
Yeah, I feel like I wouldn't power.
Beam. To use anything else for any
distributed system. I think if we put some time and
effort and love into something like this, then you're
absolutely right. I, I, I, I would not reach for
other things. It's just right because it's
nice and simple to think about it.
I've got a function right it, it reads data, it writes data.
It's wrapped in a transaction. Done.
It either happens or it doesn't. How hard is it for me to tear
out Postgres and use bedrock? Well, funny you should ask.
So what I would recommend is if you were going to do embark on
something like this one, be a little careful because bedrock
is bleeding edge stuff. So sometimes you get a little
bloody. But you know, we put a little
time and love into it. It could be better.
The the idea here is that the general flow would be I would
treat it like right through cash.
So day one, the benefit you would see is that instead of
going to Redis or something else, you could say, hey,
Bedrock, give me user ID #23 right?
And I can be sure that user ID 23 that I'm looking at is the
latest freshest copy. When somebody changes something,
you would go ahead and write it to to Bedrock after having
written it to the database, right?
So the database is system of record at that point.
You still keep track of stuff there, but if the thing goes
through, then you write it to Bedrock and then everybody else
that's there. So it's a cache.
You can treat it like a cache. Now over time what would happen
is is your data would naturally just collect there would
naturally pick up the entire set as things are read from the
database. And oh, actually what you're
saying that's smart. Then you don't have to run like
data migrations or anything really.
It might be kind of fun. So like you could see an off
ramp for some of the existing tech onto something like this,
provided we can, you know, give people the same level.
We want to make sure that there's a storage system.
So you got to be serious about people's data and you want to
make sure that that that things are being dealt with properly.
So there's a whole bunch of testing.
I can't have a data failures right?
Because it has to be there when they don't expect it to work.
You know it has to always work. So that's one of the reasons why
I would recommend, but you could treat it as, like I said, like a
cash. So in other words, what happens
when the cash fails when a cash fails, right?
You just go do what you were going to do anyway from the
database, no big deal from the database, right?
So what you could do is you could use something like this,
you know, while kinks are being worked out to, to, to, to answer
those kinds of questions quickly, right?
So you can take pressure off your database.
And one of our conversations I like I showed you like a region
map, right where yes, really high lag in like Asia, obviously
because I only have one server here in the US.
But light only goes so fast. Now using Bedrock as like a
caching system, if I sprung up a note over there it would be
amazing. And the data's sitting there on
the note. Right.
And see Foundation DB contemplates this, right?
They have this idea where you're, you've got, you know,
multiple data centers worth of machines, things that span
different places. And as a result, information is
kind of the replication problem is dealt with for you that way
or the the system that kind of the features you get out of the
way the system is built end up with moving your information
around in ways that are advantageous.
Think about this. You know, you're, you're in a
global situation. I'm in, I'm on, you know,
continent A and I'm writing things and I'm mostly reading
over here on continent B. So now continent B, you know,
can pull from S3 or GCS or whatever, slower pace, but it
still gets there. It's, it's, it's, it's big
blocks of information that move quickly on fast networks that
are internal ones, right? And then you can answer your
questions because you can materialize that data over here.
So you might instead of having 500 milliseconds worth of of
lag, you might only have 15 or 20 or 30.
You know, I mean like it's a lot less and with without having to
do a whole lot of crazy coordination.
Yeah, that's. Awesome is the promise.
Well, I mean, whenever you're starting to hit 500
milliseconds, things get painful.
Like from a UX experience? Well.
Think about this. I mean, right, you got, I got
this data and if I got to ask a question and then I have 500
milliseconds before I can ask my next question, it's not fun.
I mean, that's, that's you just, you know, now all of a sudden
the pain of having to go back to the database is crazy, right?
But if you can keep copies of the information in multiple
places and keep those fresh and be sure about them, right?
You can be sure about how that you know that they're valid,
then all of a sudden the rules change.
So, yeah, I think that this kind of model, not only does it fit
the beam, but I think that when when you put the kind of rigor
in that you need for a storage system, we put a little bit more
rigor into it. We would have something that
people would really want to use. And I think you'd have to tear
pry it on my hands. I I was talking to a community
member who benchmarked Bedrock versus Rox DB, their NIF and.
And they were surprised. And it it blew it out of the
water. That's awesome.
Like. Yeah, yeah.
I mean like because you're past. Binaries around.
I never knew that. I learned that today.
That's brand new information to me.
No, like I said, I mean my, my, my, my problem in life is that,
you know, I got, I got in touch with autism, right?
So you know, I and I'm a tool user.
So, So what happens is I get something and I just have to
like, you know, I just, you know, autistically fixate on it
to the annoyance of my family until I understand something.
And then once I understand it, then it's like, like I know how
to bend it and twist it to my will, you know, so the, the beam
is no different. Like you get some I oh man, it's
just stuff you can dig around in there and find and it's great,
you know, so that was one of the gems.
I, I don't remember where I ran across that, but yeah, I think I
was looking at the, the documentation for like send and
receive, like the low level. And it's like, oh, well, wait a
minute. I don't know why that's going to
be useful. I'm going to tuck that little
fact away in my brain. And so go to do something like
Bedrock or you, you where you're moving information around and it
needs to be done in a way that you don't want to copy it.
Well, the natural form is now binaries.
So it's interesting that there's the the materializer, the way
it's built inside of bedrock, the way it like keeps track of
all the key values that's done with pages of information that
are stored in binary, even though they're, it's cheaper in
Elixir to just pattern match and decode these things.
Well, pattern matches are really fast to pattern match and decode
these things. And it is to keep them loose and
update them, right. So it's, it's kind of wild.
Yeah. Now it's it's.
It's an interesting model. I think it this is an important
lesson, why it's like so important to understand what
you're building on. Like you can't just know Elixir,
you have to kind of understand Erlang, and you need to
understand how the BEAM works and then when you do and.
Then the machine underneath it, right?
Like so like the, I call this, you have to have like mechanical
sympathy, right? You have to have sympathy for
the machine and what dumb ass thing you're asking it to do
because it's, I mean, computers, computers are fast.
They're they're ridiculously fast.
But if you ask them to do something in a stupid way,
they're going to do it, but it'll be done stupidly and don't
be surprised. Well, you and I have had
conversations around like I've struggled with like my sequel
performance, like with queries and things.
And we've talked about like, well, look at what you're asking
it. And it's just like, well, yeah,
obviously. Yeah, that's really dumb.
Well, that's the thing. It's like these tools have
different textures to them. Like my sequel, you bring that
up. We we've had our battles with
this right? And, and, and people are a lot
of people are in the Postgres world and Postgres is wonderful
and it and it is right. And I'm no certainly no my
sequel fanboy, but the tool has interesting properties and if
you use them and don't fight them, suddenly things become
really fast. One thing that Postgres really
doesn't like is it doesn't like lots of tiny little changes.
It doesn't like them, right? You, you end up with this
garbage collection stuff that it ends up having to do, but just
because of the way it was built. And keep in mind, Postcrest is
40 years old too. It was designed in the 80s and a
lot of that tech has has been replaced, but a lot of the
underlying ideas are still there.
It's got this garbage collection mechanism deep down in it that
runs out of, you know, runs out of numbers and has to compact
things. And it, it, it, it, it causes it
pain in Postcrest when you store a record, record 1 and record 2,
it does it with hashing. So there's no guarantee that
those two things of pieces of information are going to be
anywhere near each other within the system.
Well, my sequel is different. My sequel takes it what's called
index oriented storage. And so it says, hey, if I've got
a primary key and I got, you know, the next one up, those
things are going to be on the same page.
So you know, the thing we did with the attributes, you got
attributes for a piece of information.
I want to store them. If I take and I store them by by
key, you know, primary key and then like like an attribute
name, then I know because that's a compound key.
Those things are going to be like row by row by row, like
right next to each other. So when I do the common thing of
like give me all the attributes for X, they're all groups, not
so much. Like in our case, we have like
products and products have attributes, but we could have 20
attributes per product, but they're all on the same page,
yeah. They're going to be right next
to each other on disk. And so in in my sequel, that
kind of structure pays. It would be nothing but pain on
Postgres, right? Postgres, you have to take it
different. But see, The thing is, is that
people are like, oh, well, SQL, SQL, SQL.
And it's like, yeah. I used to think that way until I
met a guy named Jason. Well, you gotta, you have to
think about the what you're asking it to do.
I mean, it's not none of this shit is magic.
It's cool, but none of this shit, it's magic.
I mean if you don't understand it.
Yes, fair. Right.
So it's, you know, as soon as, as soon as as soon as you
understand something, it's it, it, it loses it's mystical
qualities a little bit. And that's a good thing because
then you can take advantage of it, right.
So we learned this thing about how mysical works and because of
that, you can take advantage of it.
That kind of thing doesn't work so well on Postgres, you get the
idea. But you have to be, you have to
have sympathy for what the machine what you're asking the
machine to do, right? And if you don't, you're going
to it'll probably work. Machines have a lot of memory
and they get a lot of cores these days.
So it'll, it'll work. I mean, Bunny quotes, but it's
you're not going to get something that's going to be
fast or simple or memory efficient or any of the other
things that you're after. You know, this is, you know, AI
has this problem. Anywho, so you, you, you bedrock
is neat. I wish that more people would
look at it. There's a a live book that you
can grab. People can grab it.
They can run the entire thing. Keep in mind there's no nifs,
there's no nothing. It's just pure Elixir code.
You can run it and it's sizes down to run inside of a, a live
book and it could run across a cluster depending on how you
want to set it up. So you can grab this live book
you can run through, I think I have a, what is it a college
courses or allocating seats to a room, you know, tickets to a
room, things like that and all these things, thousand processes
trying to fight over allocating seats and it happens in
milliseconds. It's nice and quick.
So you can grab it, try it, play with it.
If people want to use it, then I recommend you be a little
careful because it is new, but at the same time play with it.
And if we make it better, I think it's a killer weapon that
something you'd have on the beam that just people just don't
have. It's kind of like live use a a
killer weapon for elixir. You know, it's like you can you
could you could do some cool stuff, but you combine these two
things and now you're starting to get really fun.
So you can, right? I mean I already want to RIP it
out, but I know that's not the I want to RIP postgres out of kill
switch. But I know what you said is a
smarter approach. Well, and you know, like you
could try that first and, and see how that goes, you know, and
then like I said, your data is naturally going to kind of end
up there. Plus, The thing is, is like
anything you know, you're going to have to figure out how you
want to arrange your information.
Like. Storing things with like.
Here's the thing. Like my, like I, I, I don't even
have any users in Asia, but just knowing that my latency sucks
there it it bothers me. Yeah, You know what I mean.
Like a core level no I get. It dude, it's just.
It's one of those things where it's just.
Like like this isn't like this would actually be a really
shitty experience if I logged into Killswitch in Asia?
And and and I might someday. And if I did, I would be upset.
I. Would be pissed right?
But. But even if I spring up another
node and I still have a database in the US right?
So my data, my CRUD actions are still going to be slow and make
things feel shitty still. So like I need I need something
like this. I, I, I, like I said, there's a,
a series of niches that something like this would fill.
And I think that we have had our heads stuck in the Postgres.
You know, this Postgres is the way you know and date, you know,
monolithic databases are the way.
They're good. You can solve a lot of problems
with them. They're pretty awesome.
But there's a limit. And they also force you into
another subtle thing is like they force you into thinking in
terms of sequel, right? Which is sometimes a problem.
I would like if I could just, you know, I could get an account
and I could look at the number and then I could, you know, so
we put all these abstraction layers on top of it, but
ultimately we're still fighting with sequel to get some of this
stuff done. And so I have to kind of shape
my problem, my problem Sequel shaped in order to, to, to make
it work. And it would be nice if I could
just think in terms of data. Like I wanted to store this map.
I got a map of stuff. I just like to store it.
I'd like to get it back when I asked for it.
And I would like to make sure that when changes are made to
it, that I get the latest stuff and that when I make changes to
it, my changes don't stop on other people's.
That'd be great. So, I mean, and of course, a
whole bunch of problems get a lot simpler, right, conceptually
when you start thinking about things that way and because it
lifts you out of it. So there's a class of problems
that you can do it. You can.
You can do that. That makes sense.
Sequel has its place, yeah. You know, but I'm also like, I'm
thinking from the perspective of like most CRUD web applications,
you're, you're, you're not going to really have data conflicts
because you're the only user touching your data.
Right. Well, I mean, so and that's a
good, that's a good point. That's an observation.
It's important to understand. It's like, it's important to
understand your machine, right? And how it's working.
It's also important to understand the structure and
texture of your data, right? It's like, because I mean,
ultimately you're designing a container for something, right?
You, you designing, I can't build a bucket.
I don't know what a bucket is. I can't make a bucket if I don't
know what water does. Does it respond to gravity?
Is it heavier than air? Does it stay?
You know what I mean, Liz? Does it evaporate?
Like what? What are the properties of
water? And as soon as I know about the
properties of water, I can build a better container for it,
right? The when we're building our
systems, it's the same kind of thing.
You need to understand the properties.
So if I'm building linear or kill switch, I've got data
that's it's not a huge amount for any given person, but in
aggregate, there's a lot, right? And the data's not supposed to
cross. I mean, it would be really bad
if I were to able be able to look at linear and see somebody
else's tickets, right? You know, or pull up kill switch
and be able to see somebody else's data, right?
That's a bad thing. And so you have these isolated
silos. Some problems are not like that.
You have the good news on Facebook, right?
The. Good news with kill switch is
even if you got someone else's data, you couldn't decrypt it.
This is good. So I mean, we're gonna roll that
but but we're not gonna give it to them in the first place
because why? Exactly.
But if it happened, they can't read it guys so.
You'd be safe. The power of encryption.
Math math for the win, right? All deep, dark math nerds, dude.
So The thing is, is that that you've got, you know, if you've
got a data that's siloed, right, that's one shape.
I mean, you could run each and, and people do this where they
have sequel Lite, you know, like each split and they they Shard
it that way and they'll they'll do these things where they'll
take sequel Lite and they'll you know, it'll come down to the
node, they'll do things for it and they'll put the database
back. You know, there's things that do
this and then you have something like Twitter, you know, X or
whatever it's called these days where, where I have a thing
that's published, but other people see it, right?
So that you have you have information that's kind of like
semi public and it cuts across collection has this problem,
right, where I put up people's cards and other people can see
my collection, right. So it's a it's a different
textural problem where I want people to see the things that I
put up there versus I don't want that at all.
So it's important to understand your problem, right, and how,
how, how that goes together. So I think like I said, if if,
if people were so inclined as to play with it a little bit, I
think they would like the idea and if they like the idea enough
to submit some pull requests, we can make it really awesome.
So. OK.
So Dalton has a question. Does bedrock run adjacent to the
prod beam or within it? Within it, it's a Elixir
process, just like it's 100% Elixir.
It just runs on OTP just like your other stuff right next to
it. So if you were to run this on a
single node and you were to ask a question, the node you're
asking a question of is the same node.
It's a process hop. You're going to get your data in
nanoseconds. Yeah, it's so cool.
It's so cool. It's safe to sell the next
machine over. There might be a few
milliseconds. But it's safe to say you have to
come back on to talk about beadwork, because I knew this
was going to happen and we're going to talk about Bedrock for
an hour and a half. But I like, I love.
It are we at time. Well, we're not at time, but I
don't think it's worth like. Because I do.
I mean, we can touch on it if you want to.
OK. All right.
But I feel like it would be better to talk about beadwork
for an hour and a half. Oh, OK, I could go on and on.
It's up to you though. Well, I mean, let's let's tell
people what it is, right? So what the whistle a little
bit. So it's not a crazy idea and
it's not a new one. But you have a situation where
you have agents and they need to be able to keep track of things.
And all these agents they have like to do lists and they really
suck because when you hit compaction, the To Do List get
compacted. Bad things happen.
People work out of a markdown document, the agent forgets
where it needs to update the little check box and bad things
happen. So it's not a elegant system.
And so Steve Yagi came up with this idea of beads.
And it's a ticketing system that's kind of baked into Git.
And it's it's, it's slick. But now it's 200 megabytes of
stuff to download and it does all this crazy.
And So what I did is I took that strip.
Well, it's built with AI big into the dark factory idea and
it, it, it's got, I mean, it's got warts and legs and horns by
now. And so I practiced it.
It, it hit me one day when I was, I, I downloaded it the
latest version and it told me I needed to run BD doctor to get
it to work. And I was like, OK, cool.
So then I ran it and BD doctor told me to run BD doctor, which
was rage inducing. And so I practice a little rage
driven development and end up building speed work.
So there is a simple, light, reasonable system for managing
tickets for agents. It doesn't get, it doesn't in a
clever way that doesn't corrupt off your repo.
And you know, like I said, we could talk about it lots more at
some other point, but it's, it's a good tool and I think people
would like it if they knew about it.
So. Yeah, I agree.
It's pretty dope. I think what we should do, I was
looking at my schedule is trying to get you back on.
Wait, what month are we in? June 19th is my next opening.
Should we do a beadwork episode then?
At least I. Might even have something more
interesting to talk to than that.
Something cooking. Oh, man, All right, Jason
already always cooking. Dude, well, that's The thing is
like these days you got like bionic arms, right, you know?
It's bad. It's everything.
Like the busier I am, the more ideas pop in my head and you
have to be like, no, don't build that.
Do not build that. Focus on this.
It's going to come down to, I mean, in a world where anyone
can build anything, it's going to come down to style and taste
like, yeah, right. I mean, I could build.
I could build AI could build a house with purple polka dots.
But should I? We were just talking about that
with Mike, like it's, it's coming down to style and taste
for building good products because everyone there's, I mean
not everyone, but for the most part, engineers can read and
know if code is well written or not.
Like that's not like the highest of skill anymore.
People have a sense of smell, right?
Like, and you look at something and you can be like, oh, this is
smelly. Yeah, yeah, yeah.
It doesn't. It's just, it's just solving the
local problem over and over and over again.
And sometimes the aggregate is silly and people call it slop.
But let me, I'll tell you a secret, right?
Slop has always existed, man. Like, since forever.
People have been making crystal palaces of crazy.
Crazy. Shit, I always joke.
I was like they. Just do it faster now.
Before LLMS made it, we were copying and pasting dangerous
ass code off stack Overflow. Like now it does it for us,
right? Yeah.
You know, but but will it work next week?
That's the problem. It's AI doesn't have to deal
with the consequences of its actions.
I'm excited. I'm having Peter, Peter Ulrich
back on and he's been finding like all those CPS, yeah with.
That's that's exciting. Yeah.
Well, I mean, here's the thing. These things are they can focus,
you know, autistically on, you know, finding tracking something
down and that's it's, it's amazing.
So anyhow. So the tools used by the right
people can be pretty awesome, or in the right way.
It's always ever been, man. I mean, like, you know, it's,
it's always been that way. It it just it took more that
dedication and nerdiness back in the day, you know, you just had
to because you have to know how to make things.
Now you can just, you know, Leviosa it into existence.
It's a little a little bit different, so.
Yep. What am I down?
I don't. Know you get that look on your
face. I pushed a button but this is a
lot of fun. I really appreciate you coming
on bedrock, sounds amazing. I will send over some links for
the show notes and to underlying foundation DB.
If you don't play with my stuff, go read about their stuff.
It's super awesome and you might want to use their stuff if you
don't use mine. It's.
Good stuff. That's awesome.
Yeah. Well, thanks a lot for joining
me today and we're going to get you back on.
Hopefully you're free on the 19th of June.
Yeah, we'll see. I think.
I think we can make that work. Cool.
All right, bud. All right.
Have a good one. Thanks everyone for tuning in
and I'll see you guys next week testing.