Hey, I'm Kimberly with Elixir mentor Jacob recently added
recruitment services to help companies hire pre vetted Elixir
devs. I help with company outreach, so
if you're hiring, I may be reaching out, or you can connect
with us anytime at elixirmentor.com.
Secure your Elixir apps with paraxial IO.
From code analysis and bot defense to dependency checks,
they've got you covered. Elixir first design with GitHub
integration. Elevate your security at
paraxial dot IO testing. What's up?
I'm Jacob Litzo, your Elixir mentor and welcome to another
exciting episode of the Elixir Mentor Podcast.
This is where we discuss everything related to Elixir,
from interviews with enthusiasts, pioneers in the
community to innovative projects and library shaping Elixir's
future. Jamel, founder of Fire Zone
joins me today talking about building AYC backed open source
zero trust access platform with Elixir and Rust.
Welcome, Jamel, and thank you for joining me.
Thanks for having me. I usually like to kick these
things off with like a, just a, an introduction of like who you
are and a little bit about your background and what led you to
Elixir. Yeah, sure.
So let's see, I guess I got my start in in college.
So doing kind of web projects for for, you know, on the side
to make earn a little, earn a little extra money.
I think Rails was getting big around then.
This is the back in like late 2008, 2009, like like Rails 1,
Rails 2 days. And so, yeah, I just started, I
started basically on the on the on the rails train.
And and then from there, you know, after graduation I kind of
got into, I got, I got a job at Cisco.
That's where I got my start in, in security and, and, and sort
of always did the web programming thing on the side.
But at Cisco, I had, I, I worked there over 8 years.
I had, you know, four different, actually four different stents I
would like leave and then some other team would kind of bring
me back in. And then between each one of
those, I would do, you know, go back to sort of web programming,
web application programming and, and Rails.
And, and so the last stint at, at Cisco, it was during COVID
and, you know, I was like 4 years in and I was like just,
you know, kind of getting burned out.
And towards the end there, you know, COVID started and Cisco's
this company where it was hybrid at the time.
And so obviously, you know, everyone had to just go and work
from home immediately, right? And, and so, and so when that
happened, their VPN concentrators sort of fell over.
They got overloaded. And I thought that was
interesting. And so I was like, OK, yeah,
this is kind of the premier, you know, like one of the premier
networking companies and they're having struggles with this.
And so always wanted to, to, to find a project to work on Elixir
with. And, you know, kind of coming
from Rails and was just thought it was interesting.
And so that's how Firezone started.
So it's, yeah, over five years ago now.
But yeah, I, I, yeah. So went from security.
It's kind of like the, you know, the, the intersection of the
Venn diagram between, you know, web programming and security
and, and started Firezone. It's just a side project and
kind of went from there. How did how did you like land
down Elixir being the fit there? I, you know, that's a good
question. I, if I look back at my Git
history, like if I look at my GitHub, you know, my personal
GitHub, I'm like, I had some, some, some side projects there
going back to like 2015. So something piqued my interest
in it. I don't remember what.
It's probably like a Hacker News post.
And, and I, I do remember, so in, you know, we used Ruby and
Rails at Cisco on some of the projects I was on.
And I just remember spending, you know, not, not, not to throw
shade on, on Ruby or Rails, but I just remember spending a long
time kind of debugging some performance issues related to
the threading model. And and, you know, we figured it
out and and it ended up being something, you know, not, not
too hard to fix. But I remember reading sort of
one of the selling points, I think in back in the day with
elixir, I think Jose would talk about how it was like, you know,
kind of his, his, his story about how he how he, you know,
set out the salt, you know, set out to kind of think about
languages, you know, in a multi core, multi core world.
And so that kind of like resonated with me.
And I think, yeah, I just played around with it.
Always, always thought it was interesting.
And we, we had, yeah, Erlang was one of those like no one, no one
really heard about or used, right.
Like, like I didn't, you know, I studied electrical engineering.
We didn't hear about it right in college.
We, we had, we had a guy come on and give a talk, you know, I was
involved with like the networking or the, the I triple
E society. And we had a guy come give a
talk and he's like, oh, I built all this stuff in Erlig.
We're all like, what is Erlig like, you know, and, and so and
so when Elixir came out, I just thought it was interesting and,
and I started looking into it and, and so finally got a chance
to to work with it with Firezone.
In college I turned my back to programming because I was
learning C and Java and I absolutely hated it.
Yeah, I remember those days. Yeah.
I I don't, I don't know why, why they kind of shove Java down our
throats so hard. Yeah, that was a big one.
Yeah, For a minute, I was like in a master's program.
I was like, I'm not going to be a software, you know, software
engineer. And it was all Java.
And I dropped out and, like, built my own projects because I
was, like, so impatient. I wasn't, like, building fast
enough. And I hated Java.
Yeah, They're like, solve this hash table collision in Java.
And it's like, what does this do to it?
How does this help anyone? You know, it's very detached
from the real world, Yeah. I accidentally kind of like I
started writing Elixir for like a marketplace platform that I
was building in like 2020-2021 And I like kind of just picked
it because I was like, it sounds what I like, what I need, but I
didn't like actually understand the full benefits of like the
fault tolerance and like the actor model.
I didn't know what I was getting myself into, but it just ended
up like the more I learned, the more perfect of a fit it ended
up being for me. Yeah, I, you know, now that
you've mentioned that, I do remember we use device, right.
And so like I got, I was like, OK, this is great, right?
You know, in the real worlds, like my favorite gym, right?
And, and so when I yeah, so when I, when I, when I picked up
elixir, I was like, oh, it's by, you know, kind of by the same,
same mind. And I was like, OK, this is, you
know, this is something. I can, yeah.
I fell in love with the syntax before.
I fell in love with the like, the power of like the beam and.
Yeah, exactly. Yeah.
Yeah, I like it. I feel like Fire Zone is like
not not a simple undertaking. Yeah, product it is very it is
very complicated, unfortunately, so yeah.
So I, I, so I maybe I should get into kind of more of the details
of how, you know how Fire Zone was originally architected.
So, so big yeah. So a lot of mistakes were made.
I'll, I'll preface, you know, this, this story with that, but
so, so you know, it's a web app, right?
Like I like, I wanted to learn Elixir now, now and I, and I
wanted to, and so I, I don't know how much you or your
listeners know about Wireguard, but you know, kind of this new
VPN protocol, I know, you know, much simpler.
It's like. Most networking stuff.
There was a time I took two years of a Cisco networking
course in high school. Oh, I'm sorry.
And I did. I did well, but I don't remember
a thing of it. Yeah, yeah.
So, so before Wireguard came along, like the VPN protocols
had sort of been stagnating, right?
There wasn't really any new developments in the space for a
couple decades. And they're pretty heavyweight.
You know, I'm not, I don't claim to be a Wireguard expert, but
you know, the, the protocols before, they're pretty
heavyweight. Very, very, very complicated,
very large. So Wireguard came along as like
this breath of fresh air, right? Easy to configure, perform at
lightweight. And so I had friends here in the
Valley, I, I'm in Silicon Valley.
So I had friends like they're at their, you know, they're working
at their startups, like a hundred 200 people.
And they're like, you know, we need, we need a remote access
like wire guard's great. But like, it doesn't really fit
into the business, right. Like it's, it's, it's like very
bare bones. It's like configuring SSH
configs. You know, you, you, you private
key, public key. Here's your static config.
You know, you got to get on the machine somehow and, you know,
no failover like, and, and not to fault Wireguard.
That's just, you know, it's not meant to do that, right.
So there, so so I had one friend that was like, you know, just
build it AUI for Wireguard to generate configs.
I was like, OK, you know, that's and so that was Fire Zone
initially. And, and, and so it was just an,
it was just a web app and you just like, it would just spit
out Wireguard configs, you know, generate the, I don't even know
if I should admit this, but it would, it would like generate
the, the keys in the browser, like browser JavaScript.
And it was just a toy project. And so it was like a disclaimer,
you know, like this is, you know, just this is a toy
project. Ended up putting it on Hacker
News and it was on the front page all day.
And then like I was like, OK, And then, you know, kind of
always wanted to, you know, in the back of my mind like wanted
to do a startup. You know, I had left Cisco a few
months earlier to take a break and, and sort of recollect as we
do sometimes and. I've been there.
For yeah I've. Taken I've taken a year off from
touching a computer before. We'll have to talk about that
maybe another time. But yeah, they, they, they, they
accepted us. And, and, and so, you know, we
got into YC and, and the architecture was just like, you
know, not, not, not there at the time.
So what it was back then was it was only self hosted.
So you would just take this Elixir.
And so as you know, self hosting, like distributing an
Elixir application is like kind of a challenge in itself.
So we use this this package called Omnibus, which I think
git GitLab uses. It's like way back from the chef
guys, right? And it would just like package
up all these dependencies. I think it came out, you know,
it's from pre Docker days. So nowadays you just ship a
docker container for everyone, right?
I know, that's how I. Roll it.
Yeah, yeah, yeah. So, yeah.
And we added that later on, but yeah, it was, it was built with,
it would package up, you take it, you run Firezone, you got to
manage your database and all that.
And then it would still just spit out configs and you got to
have the Wireguard app to connect.
And we went through YC with that architecture.
And in the back of my mind, I was like, you know, this is not
for, you know, this is not really going to scale the
business. You know, it's not solving the
problems that businesses have when they try to take Wireguard
and bring it into their into their company, right?
So we ended up building onto that architecture for a good
solid. You know, you go through, you go
through YC. For listeners out there who've
never kind of been through an incubator program, it's pretty
intense. Cuz I'm like just like a
bootstrap, you know, small product guy.
So yeah, yeah, yeah, that's the process.
Yeah. So.
It's a different it, you know, it's definitely a different kind
of horse, right? Like, you know, you don't go
through YC to build a sustainable, you know, well, I
shouldn't say sustainable. You don't go through IC to build
like a bootstrap business, but that may seem obvious, right?
But it's it's to get big, right? It's not it's not to go small
and like. It's the IPO, right?
Yeah, it's a it's to have, you got to have an exit at some
point, right. So, so you go through YC and you
know, obviously you got to move fast.
And so when you've got this, like we had this sort of when,
when they say like, do things that don't scale, I wouldn't
recommend applying that to your product architecture, right, if
you can avoid it. And so we, we, we had this sort
of product and it was like growing on GitHub was like
getting GitHub stars, but it was just self osters.
And, and, you know, we, we ended up having a lot of businesses
use that banks and, and whatnot. And, and they would run into
issues and they would e-mail us and they'd find us and e-mail us
and be like, Hey, you know, we're trying to get high.
We're trying to like combined 2 Fire Zone instances on the same
database to like get high availability.
I was like, no, no, no, like what are you doing?
And so after about six months of just growing, right, because,
you know, investors want to see growth and, and, and, you know,
we kind of decided that we were going to go back to the drawing
board and actually build, build the product the correct way.
So, so it took a few months to do that.
Maybe I don't recall how long it was maybe 8 months, nine months.
And before we did that, we, we got all of those people that
were using it. I won't call them customers
because it, because it was, it was just free back then, right?
And we, we got on calls with banks and, and other businesses
and we were like, all right, what would you like to see,
right? Like what, what problems are you
seeing? And we did actual customer
discovery, right? And we had, we had the, we had
the kind of the privilege to do that because they're already
using Fire Zone, right? So we, we took, yeah, we took
lots of calls and we sort of collected everything and put
that into the new architecture. And so we, we, we, yeah, we, we,
that's the complicated thing that you see in our repository
today. But yeah, if you go look at like
the legacy branch, that's where we, we just like stopped
development on the other thing 'cause it was just, yeah, it
was, it wasn't gonna scale so. Was it already in elixir?
You just re architected things or?
Yeah. We, we, we sort of rebuilt
everything we knew, we knew we wanted to have a cloud kind of a
major cloud component to, to manage all the configuration,
right. So the, so with the new
architecture and then we know we need that we needed our own
client applications, right. So, you know, you need to do
like the user authentication and all of that tie that to the
keys. So we went back and we built all
of that. We kind of started with a Mac, I
think it was Mac OS client and sort of, you know, started
building everything in parallel. A lot of stuff to, you know, a
lot. A lot of code to write.
As you can imagine, and are. You writing a native apps for
everything or? Yeah, we have native app for so,
yeah. So I'll get into the, the
architecture a bit. So the so so Fire Zone is kind
of two, two components, right? It's a control plane and then
the data plane. The control plane is all
essentially Elixir. So that's, you know, a big
Phoenix app up in the cloud. And and then the data plane is
the client application on one end of the tunnel.
And then you have what we call the gateway, the VPN gateway on
the other end of the tunnel. And those the gateway's
completely in Rust. The client applications are, you
know, it's a Rust core and then you've got a on Apple and in
Android we do Kotlin and you know, Swift and Kotlin, right?
OK. Can you for for the listeners,
can you kind of explain the the reason?
Like I'm guessing it's for like computation reasons like why
you're using Rust? Yeah.
So that's a great question. So I think memory safety was was
kind of one of the first things that that that pulled me in that
direction. It was it was kind of on, it was
up and coming. So it was a little bit of a
risky bet when we chose it. This is maybe like 3-4 years ago
now, OK, but you know you've got to have a systems language,
right? You like we couldn't do, you
can't do, you can't maybe you can't, I don't know.
But I wouldn't recommend building AVPN tunnel with
Elixer, at least not yet, right? Because you've got to you've got
to you've got to interface closely with a system.
You got to control routes, you got to open, open what's called
the Tung device. You got to do obviously packet
encryption and decryption. So yeah, those need to be
performant. So, so you do need the systems
language and and so, yeah, Russ just seemed like a good fit at
the time. Yeah, yeah.
I've heard that a lot. I have never like I've only been
on like strictly Elixir like backends.
So I have never like implemented both and it's it's cool.
Yeah, I, I, yeah, my Russ skills are, I feel like I'm, I'm a
forever student in that language.
Like no matter how much I don't, I, you know, I don't get the
opportunity to, to code as much anymore, but and, and don't
really have the time to kind of learn as much as I would like.
But yeah. Well, our, our Rust lead, he
just did another podcast. He did a Rust podcast on Fire
Zone. Nice, cool.
Yeah, yeah. So he's, yeah, he's a, he's in
Sydney, down in down in Australia.
But yeah. We'll have to bring one of your
one of your Elixir devs over here too.
For a yeah. So I'm curious like you have a,
you have a Co founder, right? No, I don't.
Not anymore. OK, yeah, I was gonna be.
I was curious about like how how that relationship started and
like how you balance like skills and things at the beginning.
Let's see, at the beginning, I can talk a little bit about it.
So yeah, we, we had a mutual friend.
Let's see. So I, I had a friend and he was
applying to IC. This is like summer of 2021 and
he was like, he was going to do a security something in the
security space. And so he got referred to me
just to bounce ideas off of he's like, Hey, can you validate
this? It was, I think it was like
analyzing it was like doing static analysis of, of source
code for kind of like security vulnerabilities right before
they before they go and get shipped, right.
It's like, it sounds like such a hot thing now, but back in 2021,
you know, pre ChatGPT, it was like cutting edge.
And so he's like, Oh yeah, we're applying to YC.
And then a couple of months later when we did the, you know,
we put Firezone out there. I guess I put Firezone out on
GitHub and he introduced me to, to another guy.
And so we worked together and actually applied to YC together.
And his background was in growth and sort of business, right?
And he had worked on it at A at this company before.
Like I've made the mistake of partnering with too many like
minded same skill sets people and it ends up just like you
have no one to help you on the business market inside when
you're both technical. Yeah, there's, well, there's
like the time element and then there's like the skills element
of it too. And I guess they're related.
But yeah, it was very, it's very helpful to have someone to just
be like, hey, you know, are you thinking about this?
Hey, you know, are we getting, are we thinking about putting
posts here and here and here on Reddit or you know what not, or
we should organize our GitHub read me to look like this so
that it like, you know, ranks a little bit better in SEO.
And it's very helpful to have someone like that for sure.
So what happened was, I think when, when we went through our,
you know, so I mentioned we had this thing that that wasn't
really ready to sell, I would say, and it was going to take a
long time to rebuild. So when we went back to the
drawing board, it just didn't make sense to have, I think to
have really any, any business people, you know, on, on at that
point, 'cause it was like we, we needed all the engineering
talent we could get right at that point.
So, yeah, so, yeah, we, we, so, so now we're at a play.
We're still tiny. We, we shipped that.
I think we shipped the one at O that was like summer of 2024.
OK. Had a lot of, you know, as, as
you can imagine with any sort of one dot O product, you had a lot
of interesting problems to figure out and things to learn.
But yeah, I would say it's it's got that mostly ironed out
within a few months and tacking on features and what not, you
know, doing the whole product iteration thing sense.
Awesome. How do you how do you balance
like features versus like product improvement at this
point? That's a good question.
So, so, so my day is like I wake up, I see, OK, what fires should
I put out right, Like what fires need to be put out?
Customer support and any sort of critical bug fixes tend to take
priority. And with Fire Zone, you know,
luckily the elixir side is like has been completely trouble, you
know, like worry free, like any, any problems we've had with
elixir have been our own doing right, our own bugs.
We haven't hit like, Oh my gosh, you know, you know, the VMS out
of memory, like for no reason are like garbage collector lock
UPS, you know, like, you know, nothing like that.
No random crashes. It's just been super rock solid.
And so all of the problems that we face typically are either,
you know, like maybe the database got overloaded, right?
We didn't, we didn't structure a query the right way.
And or more more commonly, you know, you have these client
applications and we're doing all of this like we, we do this, we
do this thing called, you know, DNS interception.
So we see all the DNS queries on the system whenever you sign in
to Fire Zone. OK, to do that, well, requires
you to like read the system resolvers, right?
And some platforms. One layer is the DNS stuff
happening on. Is that that's that's in the
Rust layer? Yeah.
OK. Yeah.
So, so that's, so that goes through the, you know, the FFI
boundary and we've got, you know, it communicates the
depending on the platforms for Apple and Android, they've got,
it's a very sort of platform specific way to do it.
So we do that on the Swift and the, the Kotlin side and then we
send the system resolvers down into the into the Rust side.
And and then we know like kind of how to do this DNS
orchestration dance or DNS routing dance rather.
And, and yeah, so, so we so there's a lot of bugs that just
sort of come up because like someone's got like a, you know,
is running like a ADNS, another DNS program on their computer,
right? And there's like fighting over
who has control, right? Or someone's like, you know,
running another VPN. And, you know, they may not
because the people that are using Firezone meant much of the
time, like they're not technical, right?
They just they just have to use it because they're IT admin,
they're saying right, right. We need a VBN client, right,
Yeah, yeah. So they.
Don't know if they're running 2 VPNs.
They they sometimes they don't know.
Sometimes you know they don't. They thought they disconnected
the old one and it's still connected.
Sometimes it got like I'm. Curious, does like Apple's like
relay system does that affect Fire Zone?
That's a good question. I don't we've never had issues
with it. I believe that that is, yeah,
that's a good question. I'm not sure.
I think that I think because that's an Apple technology, it's
it's like, it might be, it might be like, yeah, it just plays.
It's like shielded offset, like a lower level maybe I I don't
know. I mean, if it's not an issue and
you don't know about it, that's probably a good thing.
Yeah, but also like typically, you know, Sally from marketing
or you know, whoever is like not gonna use the private relay and
then also their work on their work computer I guess at the
same time. So yeah, yeah, yeah.
So, yeah, typically the issues that we see it's it's not so
much other VPN software. I'd say most of the issues that
we see are like Windows, you know, a Windows laptop plugged
into like an, a dock. I won't name any brands here.
And you know, that you like the like the network card driver of
the dock is like, doesn't, doesn't properly respond to, you
know, some of the system calls that we make.
And so then we've got to handle that gracefully.
And you know, and, and it's like if Fire Zone goes down, like if,
if the user can't connect to their stuff, it's a big problem,
right? It's like, it's like your
Internet going out or like your, your, it's like where utility at
that point in a lot of cases. And so that's the stuff that we
really have to get right. And so that's where we tend to
have the most fires. I feel like those are also very
difficult bugs to track down. You probably have to have
specific hardware and like be able to replicate it on device
or not really you. So, yeah, this is a this is a
whole, Yeah, this is a whole, whole topic.
Observability, yes, is super critical for our client
applications and for anything the customer's running on Prem.
What we use now is Sentry Sentry's been great.
I love Century. Yeah, we're not we're not
sponsored by them or anything, but yeah, Sentry, Sentry, once
we added Sentry and and we turned it on, it just like lit
up like a Christmas tree. We're like, oh, we're getting
these different warnings in these places.
Oh, this user has this issue. You know, we never would have
suspected that, right. All of those things that you
think like, you know, your software is never going to hit
it hit like, you know, you see it with system software.
So, so sentry's been super helpful.
We we used to and now it's got log streaming.
So we can on demand. We'll, you know, if a customer's
having an issue, we'll will remotely enable debug log
streaming just for that user. And then we'll and then yeah.
And so we'll just see like, and then we'll see, we'll say like,
OK, go reproduce the issue and then we'll see basically trace
logs of all the packets coming in and out.
Yeah, we don't do that for everyone, but you know, it's
like a on a case by case basis when they're having an issue.
So the opt in to it, yeah. Yeah, having sensory and a good
logging system is so important. Yeah, it's, yeah.
It's been super helpful. So not to like, I know we kind
of were just going kind of down architecture and how things work
rabbit hole. I want to like kind of circle
back, like what Like when you apply to like YC and stuff like
that, Like I don't like your background.
You're kind of just a day job Cisco developer, yeah.
Well, I, I would. So I worked at Cisco over a
total span of eight years, but I worked there four times.
So I had an internship and then I had to go back to school,
right? So I left and then I worked as a
network security analyst on the incident response team.
And that was actually a really fun job and, but, but I always
knew like I wanted to try to start something, right?
Like I like I like I wanted to. Kind of like try the same bug
dude. It's like I so I, so I would
just like leave, you know, I would, I put in a notice and
everything and leave responsibly, right?
But like I would always just like leave and, and then, you
know, just start building, right?
And, and, and then I would take on various contracts, like part
time contracts, you know, in the meantime, right?
Like kind of in between, right. So I was always kind of 1 foot
in the startup space and then one foot at and Cisco and I
never took the full leap. And then we'll be like when you
get into YC, like they give you enough money to kind of like,
you know, you, you got to do it right?
Like you can't, you can't like have a new YC at the same time.
So, so yeah, it's, it's, it's a good sort of onus to to get, to
get started, but. I've I've I've always been the
same way like like 1 foot in one foot out and then like I'll
usually jump ship too soon and then just like add financial
stress to my life on top of using the.
Product oh and I've been there so many times and I gotta say
like I think yeah, definitely at least you know, just coming
coming from my my perspective like I wish I had done it
sooner, right like I, I'm in I guess I'm in my late 30s now and
like, you know kind of like I, I applied to YC in the past like,
you know, got rejected. I think a lot of people don't
don't get in their first time, but.
Man, I'm sitting here. I'm almost 41.
Oh, OK, not too far, but you know, I think the right way to
do it or I don't want to say right, there's no right or wrong
way to do it. I think what would have, what
would have worked better for me is to probably do a little bit
earlier and then yeah, and just go straight on and do and do the
YC thing or do some, some kind of incubator And then because
the financial stress thing was just like unnecessary.
That that's a big bonus for sure.
I'm like what, what was your experience like?
Do they on board you well or like how do you what do you know
what like what do you know what to expect when you join a
incubator program? That's, well, it's, it's a lot
of fun. Unfortunately for us, ours was
the last batch that was fully remote because of COVID.
But what I hear from other YC founders is that one of the big
sort of value adds or, or I guess pros they get out of it is
that they, they meet other people, right?
It's like the whole social aspect.
A bunch of people doing really cool stuff.
To you meet all kind of amazing, amazingly talented people, right
and from all over the world. And so we had a little bit of
that because we would do sort of informal meetings, like outside
of the day-to-day batch activities, but everything we
did was remote. So yeah, so I'm sure it's a lot
better now with the in person thing, but yeah, it was I have
to say like it was probably one of the best.
I mean it's a it's a you know, it's a it's a boot camp for
business, right so it's probably one of the best experiences I've
had. Obviously, you know, there's
some stress involved, you know, and you're not, you're not.
You don't want to be like, yeah, you want to try to have your,
your life like free of other distractions.
I'll say you don't want to have competing things going on.
I would kind of sabotage your your time in NYC.
But it was, yeah, it was, it was great.
Would I do it again? Maybe if I, you know, not having
done it, I would definitely do it one time, but you know what,
where I would go back and do it again.
I'm not. I have a huge fear of taking
outside money. I don't know why but.
So, yeah, I don't know how. Yeah.
So I have I have opinion, opinions here.
I won't share all of them. So I will say that Y CS great in
the sense of they're not going to tell you what to do, right?
Like they're like, you're going to get money, you're gonna, you
know, you need to work on, you need to work on your thing, but
you're not gonna get a call from, you know, from YC saying
like, hey, why haven't you sold? You know, why aren't you at
1,000,000 ARR yet? Right.
So, so the so yeah, I, I would say, yeah.
I mean, at least for me, that that part wasn't too bad when
you. The thing about YC, though, is
for, for those of you who I guess aren't, aren't too
familiar with it, you, it's a 10 week program.
You, you start and then each week you're sort of moving along
towards this like demo day. And the way the whole program is
set up and the way the demo day is set up is so that you sort of
raise a round of investment, right?
You, you raise, usually you're, you're, I guess your seed round,
but it's, it could be different for, you know, depending on what
stage your company's at, but the, the sort of like condensed
to, you know, to hit that goal, right?
And so, and that's what everyone's doing.
And that's what we did, right? You, you go and you raise this
money. Now when you raise that money,
right? So if you got 2,000,003 million,
4,000,005, whatever it is your, your seed round, you gotta start
putting that to use, right? You can't just be like, oh, we
need to redo our architecture for eight months, right?
Yeah. So, so just think about that
when you go through like try to understand that, that that's the
advice I wish I had given my, my, you know, my former self.
Like make sure you have something that or you, you at
least know where you're going or you know, where your product's
at before you raise that money. You don't have to raise money,
right. When you, when you go through
YC, you can, you can take the YC money and then and, and, and
kind of like, you know, decide when you want to raise your next
round. Just YC is a huge boost to to
getting that first. Yeah.
Like is it like milestone based or is it just when you get
accepted into the program you get a chunk of seed, seed money
essentially? So I think nowadays they give
you it's half a million and yeah, as soon as you're
accepted, they'll start setting you up and.
That's a nice kick start. It's yeah, it feels, feels
really good, Yeah. I mean coming from someone who
who tried to bootstrap everything himself.
Like it feels good. You get accepted and then you
just get this chunk of money to go.
You basically don't have to worry about making ends meet I
guess for the next few years. Kind of wild.
I like that. All right I'm gonna just quick
rate a sponsor and then we'll continue How do I let's see so
this is for gito. Sorry Mike, still haven't pre
recorded. So I'm just talking coding
Elixir with AI or building AI into your Elixir apps.
The Elixir AI collective discord is your community for both Link
is in the description below go to agent gito dot XYZ forward
slash discord go check it out. It's a good group of guys all
super obsessed with LLMS and all the coding.
So it's a it's a cool discord to hang out with and Mike
Hostetler, he's a cool dude if you guys don't know him.
All right, we're back. All right, let's see.
Oh I'm kind of like. So this stuff I feel like is so
over my head. Like DNS layer, whatever it may
be. Can you kind?
Of explain like what? What's like snownet and how does
it work and why? Why is it important?
Yeah. So Thomas is, is our lead on, on
all of that stuff and on on the Russ, on the Russ side and he
architected that. So let me let me kind of back up
a little bit. I guess I won't get into what
AVPN is. That's that's a that's a that's
a longer, that's a whole, you know, it's a longer story, but
so well, I think. We have like a macro like.
What is AVPN like? I'll yeah, so so the Internet,
yeah. So I have a post on our blog
about this that goes way back to the history of the Internet.
But so, so AVPN essentially at least in for for our purposes,
right. So there's, there's two main
types of VPNs. There's like a personal VPN
which is mostly too exposed or to, not to expose, but to, to
hide to prevent you from exposing your, your traffic,
your net traffic until it reaches some end of destination,
right? So it's like protects that first
hop typically if you're working from a cafe or something like
that, right? Just make sure all of your
traffic is default encrypted. Sorry to like derail this.
What's the difference between like AVPN and a proxy?
So a proxy's typically application layer based, so that
would be like an Https://proxy. So you've got your OSI model.
That would be like a layer seven thing all the way.
Yeah. So, so if you're using any other
application protocol that's not HTTPS, like if you're connecting
to your Postgres database that's not gonna go through the VPS,
that's not gonna be proxy. You could do, well, you could do
a Postgres proxy, but it's not gonna be your typical, you know,
standard Https://proxy. OK.
And then AVPN sits all the way down in layer three, layer 4,
sometimes layer 2. That's going to be your, that is
your, you know, that's going to just route and tunnel all
traffic, right. So it doesn't matter what
application is trying to, you know, communicate out there,
it's going to get wrapped up in in the VPN.
OK. Right.
Yeah, OK. Yeah.
So, so the personal VPN is typically concerned with
protecting your traffic right, from from prying eyes, I would
say. And then a corporate VPN is to
get is it does that as well, but it also sort of authenticates
you or ties your identity to what used to be, you know, the
corporate Internet, right. So you get into the perimeter.
So for these like hybrid workforces, you've got the
office network. So in Cisco, we had the, you
know, over in like Milpitas, right, We had the, the HQ you
come in, you're connected to the, you know, when you badge in
you're, you're in this sort of perimeter, so to speak, You're
in the protected zone and you're in the Internet.
And typically, you know, you wouldn't use a VPN then, but
then as soon as you went home, you would need some way to get,
since you're not physically in the office, you need some
virtual way to get to that network, right?
And so you're sending these packets through.
And so the, the corporate VPN is, is a way to put, you know,
it's like putting an envelope inside a bigger package and then
like sending that. And then it gets unwrapped and
you know, it's it that's, that's, that's all of AVPN as a
technology is packets and packets.
So, so those in a corporate setting, those packets to get to
the firewall, you know, the VPN, they they're unwrapped, it says,
you know, this is Jameel or, or whoever.
And then you're, it's like you're sitting on that Internet.
So how does this play into Fire Zone?
So Fire Zone. So, so the model I just
described is what's known as you could, you could say perimeter
based security. But as soon as you're in the
Internet like you can, you can sort of talk to everyone.
You're, it's, you're, I guess I wouldn't say authenticated, but
you're, you're, you're in a more trusted, coming from a more
trusted zone. OK.
The, the model that Firezone that we're, that I would, you
know, the, the, the vision for Firezone, you could say, or the,
you know what, what we call Firezone as a product is a zero
trust access. And so the term zero trust just
comes from this notion that you don't have this trusted, you
don't have a trusted, A trusted network anymore, right?
There's no, there's no perimeter or you're moving the perimeter
much closer to the resources that you're trying to access.
And, and so as part of that, there are many more, potentially
many more little VPN tunnels that you need to make to get
closer to those resources, right?
So let's say you've got a database server and a web server
and a, you know, a file server. So the old way of doing it with
the perimeters, you put one firewall, let's say in front of
it. You've got all of these things
in the, in the giant intranet. And then you've got one VPN
tunnel to that firewall. And it may be doing like full
tunnel, you know, into the old, you know, kind of typical cases,
you'd be doing full tunnel drive.
You're sending all of your YouTube video watching and
everything like through this tunnel.
And, and then you would get to the the intranet and then you'd
access those resources behind. And so with zero trust at at
least the way that we're approaching it is you move that
perimeter, which is run by the Fire Zone gateway, you move that
closer to the resources. So you'd put, you know, you may
have a gateway near the the file server, the web server, the
Postgres server, the database server.
And then those would be 3 tunnels, right?
Yeah. So so so so I'm I'm not quite to
where to what snow net is yet, but but.
But I'm learning a lot so. Yeah, yeah.
So, so, so you have and So what? So it kind of plays into, you
know, Wireguard helps enable this because it's so
lightweight. You can open even thousands of
tunnels, right. You couldn't do this.
You can do this with with a, with another VPN protocol, by
the way. So so you can't so you know you
can't micro segment your your network to reach each resource.
I'm I'm assuming that in like plummets like latency issues and
like it keeps speed up because everything has got a purpose.
Right. But the latency breaking out
traffic, yeah. So you've so instead of doing
like a full tunnel route to send all of your traffic right to
your corporate concentrator, you're doing what's called split
tunnel routing and you're sending just the intended
traffic through, right. So that's Fire Zone by default
is a split tunnel PPN. And so, yeah, why we, you know,
Wireguard sort of helps enable that.
And so, you know, from a, from a high level, that's really all
Fire Zone is, is like, you know, it, you configure the routing
rules, we call them other things, we call them policies
and groups and, and things like that.
But you, you know, at the end of the day, your, your client is
receiving a, you know, a routing table from Firezone or routing
rules from Firezone, applying them to the routing table.
And that decides like whether your packets go over here to
this gateway, this corporate gateway or this corporate
gateway. And, and yeah, so, so, so that's
sort of at a, at a high level, what the control plane's primary
responsibility is in Firezone now getting into Snownet and
sort of the Rust layer. So one of the big things with
that we learned with our with with that legacy product that we
had was so a couple things. So this is, this is, you know,
some, I guess more technical drivers, but you had to open a
firewall port right now Wireguard is, you know, it's
like provably secure. It's, it's, you know, most
people, I guess don't have a problem opening a port for, for
wire. Some people still want to
protect it, right? They, they want to keep it
closed off, keep it invisible. And so we had, we had and then
so that's, that's sort of #1. And then number 2 was failover
and high availability, right? So if you can imagine your
packets have to go to a destination, right?
Typically, like if you just in a sort of basic network, if you
open a, a port for a service, they have to go to that port and
typically there's a service behind it, right?
If you then want the packets to fail over to a different
service, right? Like let's say that server goes
down, your wire guard server goes down, you need something to
to steer the packets over, right?
Like, so you either got to put a load balancer in front of it, or
you've got to configure the clients and then talk to
another, another one, right? And so in Fire Zone, we, we
chose the latter. And So what this meant was that
we had to build, we had to build mechanisms to sort of steer the
packets and set up connections on the fly, right?
It wasn't just going to be, hey, here's your static wire guard
config. You're going to always talk to
this end point and. Because that's not guaranteed to
always be there. Right.
It's not guaranteed to always be there.
It requires you to open a port. And so, so that, that kind of
leads me to the second part of this, which is the the Nat
traversal or or hole punching, right.
So there's a trick you can do with UDP.
It works for TCP too, but most commonly used with UDP.
And actually on our call, we're probably using it as well via
web RTC. So it's a way there's a trick
you can do where you can, you can whole bunch connection from
from one peer behind a firewall on the Internet to another peer
behind the firewall on the Internet so that they're talking
peer-to-peer, right which? It is a video ninja, I'm pretty
sure, yeah. Yeah, it's, well, it's every
browser, you know, web RTC uses this, this bag of tricks, right?
So, so it's a, it's a collection of, of what of it's a collection
of, of techniques. It's collectively known as ICE.
So there's Rs CS for this. You know, we didn't invent this.
We so Thomas, Thomas is our, is our rust lead and he implemented
all of this and he calls it snow.
You know, we we named the night of the library Snownet because
it's it's, it implements ice, the ICE standard.
Oh, I like that, I think. OK.
ICE is, I think, interactive connectivity establishment.
OK. So, so happy to let's see I can,
I can talk about how that works. That might be a little too deep,
but but it, but essentially it allows you to establish a
connection and end 95 to 90% of the time it'll be peer-to-peer
just depending on the firewall or the Nat devices on either
side, if either one of them misbehaves, then you've got to
go through a relay. And so that that's the whole
thing. And so that that's basically the
purpose that Snownet serves. No, I think that's plenty deep,
especially just for a conversation.
You know, yeah, OK. Yeah, no, I appreciate it.
And I like I, I, you know, I saw these words, these trigger
words, and I'm like, I'm going to just sound like an idiot, but
I'm going to ask. No, that's OK.
Yeah, that's. Now.
Yeah, I, I, I'm in this stuff day-to-day, so.
No, I love it. I love hearing about it.
I'm kind of curious like, all right, how does like the actor
model and like, you know, the concurrency layer and all that,
how does that fit in? Like how does Elixir fit in
after the Rust layer? Yeah, that's a great question.
So, so, so that's a, it's actually, yeah, that's a good
segue. So I mentioned, you know, we set
up connections on the fly, right.
So the way the Fire Zone client works is let's say you wanna
talk to like google.com or maybe that's a bad example.
Let's say you know, let's go back to the file server example.
So let's say you're, you know, you want to reach this file
server and it's running at like 10.1 dot 1.1.
Let's say something like that. In the Fire Zone client, we'll
see a packet for that server and we'll set up a connection to the
gateway. Now the way the connection setup
works is we will send what's called a, we call it a resource
intent. So it's an API message from the
client to our control plane, and this is over a web socket.
OK, so that goes up. We run this through what we call
the policy engine, which approves or denies the request.
If the request is approved, we'll then talk to the gateway
and say, hey, this client's trying to connect to you.
And then Snownet kicks in and it'll do the dance.
Now, as part of that, the keys need to be distributed, right?
The wire guard keys themselves. So, so, so the connection will
be will be established on the data plane, we'll log it
basically in the control plane and all of that happens sort of
within about 500 milliseconds anywhere you are in the world.
OK, so where Elixir comes in is we do all of this with actually
Phoenix presence. So we have a web socket.
Pub sub essentially. Yeah, we use pub sub exactly.
And so cuz we've got these, we've got app servers all over
the world. So you may, as a client device,
my web socket may be connected to an app server in San Jose and
then the Gateways app server may be in, you know, Sao Paulo or
something like that, right. And, and so, yeah, so we've got
a giant Erlang cluster. And so whenever I need to send
this resource intent, this connection intent that actually
the keys actually get distributed through our Erlang
cluster over to the gateway. And so we use Elixir as our
signaling layer for ice as well. OK, cool.
So it's in the hot path of connection setup.
So it needs to be. Passing, you're just like.
Exactly. Yeah, exactly.
And, you know, and, and, and there's some, there's some logic
in there to like, you know, it finds a healthy gateway that's
running. And so that's how we do our load
balancing and our failover. So whenever you're trying to,
you set up a connection to, to something you want to talk to.
Yeah, we do it. We all, we, we handle that with
Elixir too. OK, cool.
Yeah. And so we do all of the
configuration, you know, if, if, if an admin goes in and says
like, hey, you no longer, you know, are allowed to talk to
this database server or what have you, we, we, we push those
messages down to the. You're really kind of just
you're solving like the phone system problem that Erlang
solved, but just at like network connectivity level instead.
That's a great way to look at it.
Yeah. That, that is it.
Yeah, that's a good way to look at it.
Yeah, that's pretty cool. I'm curious like cuz like I've
said, I'm like been part of just like small bootstrap, you know,
a couple 1000 users. I usually don't even need a
distributed system like 1 server is enough.
What does that look like for you?
Like how many clusters or nodes do you need?
That's in the world, yeah. This has been fresh in our mind.
So we've been we've been well, I wouldn't say we are taking we've
been we've been migrating clouds.
So, so this is all sort of fresh, but we, we've got so, so
because connection set up is in the hot, what we call the hot
path, right? Anything you slow down there is
going to sort of ruin the experience for your end users,
right? So you don't want to slow down
packets as much as you can, right, data packets.
So, so we try to make the connection set up as fast as
possible. And to do that, you know, we've
got to have app servers pretty close.
We want, we want to make sure that the total, the round trip
time from a client through an app server to its nearest
gateway is as short as possible. So I guess I yeah, so we're
moving, we're in the process of moving from GCP to Azure.
But really it doesn't matter which cloud provider.
The only important thing is that we have to use a hosting
provider that's got essentially servers everywhere.
Servers everywhere. And so they're hyperscalers,
yeah. Is it like auto scaling?
Like if you have a demand, you kind of spin up a node.
No, we, we, we don't really need that yeah.
We don't use we, we, we've got we just use virtual machines and
and we deploy with Docker. We just got, you know, kind of
basic Ubuntu, you know, so you're.
Manually spinning things up depending.
Yeah, we've, we've got a manual, yeah, 'cause I mean, our, our
load, at least on the application servers is quite it.
You know, we're not like Netflix where we've got, you know,
hordes of people just kind of coming online, like depending on
the region, I guess we do. But from a, from a general
cluster perspective, it's pretty even load.
And so yeah, rather than deal with the the churn of like
adding new nodes to the cluster and then like suddenly removing
them and like, like that creates some overhead, right?
Like, you know, and, and like the, you know, the, the, the
presence state track is great. Like we haven't really had any
issues with it, but we'd just avoid that if we can.
So we have a consistent number of of Elixir nodes, what we call
our relays that are built in Rust and those are those are
everywhere, right. Those are in every region we can
we put we put a relay because those are those are very
critical to to latency. Yeah, OK.
Yeah, that makes sense, especially cuz you just you
mentioned like 500 milliseconds and I'm guessing that you don't
wanna go above that cuz then. Well, that's just for the setup.
So once we set up the connection, then the client
talks directly to the gateway and then it's just whatever
they're they're linking with each other.
Yeah. So just kind of pass through at
that, then at that. Point, it's just like when you
go to like, yeah, you go to likegithub.com, enter, it's like
500 milliseconds. And then, yeah, we'll start, and
then the page will start. Yeah, that's pretty good.
That's awesome. I like I so collects.
We I'm a back end engineer there and we have like 4 million users
now. And so it's like it's the first
platform I've been on where like query, like as you mentioned,
query performance and things like it's a different, it's a
different ball game. Like I remember like week 1 of
joining the team. I I pushed out of my first fix
and I like crashed production all because I built a nasty
query and like wasn't used to dealing with like a ton of
requests in a second and then just like large data tables, you
know? And yeah, we, yeah, we've done,
yeah, we've done similar things. I mean, we.
So yeah, I could talk about that a little bit like, and So what
we've recently done is we've moved to having more read
replicas around near 1 of near each of these app clusters.
And they've been super useful because our first, our, our
heaviest query is when a client first connects and it needs to
get all of the list of resources that it has that it can connect
to. So that requires that's like a
join across like 3 or 4 different tables.
And we call it our hydration query.
From there it's cached. And then we only update the
cache from that point forward. But so those now hit the
replicas and so yeah, that that that's been a lot better.
But our architecture before this was a little bit different.
And so, so, so how it worked before was we had, it was we, we
have to continuously solve this problem of.
So the way that we, we do access control in Firezone is based on
what group you're in. So if you're in like, if, if
your company uses like Google Workspace or like intra right,
or Okta or something like that, you're going to have users and
groups. And in Firezone, we say you need
to create a policy to get access to anything.
And that what a policy is, is basically just a join.
It's a fancy join table between a group and a resource.
OK, So depending on what membership you have, like in in
which groups determines what resources you have access to.
And let's say that, you know, we do like an identity sync and you
got removed from you like you switched teams or something like
that, like whatever it is, right?
And you got removed from the DevOps group.
And so we've got to evaluate, you know, we've got to re re re
evaluate what resources you have access to.
And we've got to push that down to the client and push it down
to the gateway, right? Very quickly.
And so rather than what we used to do was like anytime each one
of these things changed, right? Anything that could potentially
change your, what we call your resource list and your client,
your list of resource you have access to.
Any time 1 of that, one of those changed, we would just go and
just rerun the, you know, the the four join right four table
join query and and we started to hit problems pretty quickly, as
you might imagine with that. And so we moved to using, looked
into a few different solutions like listen, notify, you know,
we use Postgres as our as our database and found out.
And then we discovered, I don't remember how I, I found out
about I, I think I, I don't remember, I think it was, I
might have been Superbase. I know Superbase uses this, but
yeah, so logical decoding, right?
Or change data capture. And, and so we move to that
system. And so we, we essentially we
take all of the updates that that flow through that we
previously would have had to query for and we push them out
that way instead. OK.
Cool. So yeah, on the subject of slow
queries, yeah. Well, we got add away in the
chat and now Peter Ulrich just joined I.
Don't. Know but one add way said wall
events WAL is. That yes, yeah, we, we streamed
the wall. So we've got, we've got 2
replication connections essentially we've got one to do
and handle all this called what we call the changes.
So it's, you know, change data capture, we take the change.
We, we, we have one consumer, it basically reads all of these.
So these are insert, update, delete events coming from the
database. And then we, we, we form the
struct out of that and we just broadcast the struct out to the
client and gateway. The channel pits, right the
channel project, wherever they're whatever servers they're
running on. Whoever's listening.
Whoever's listening, right? So we broadcast that out and
let's say it was like the resource got deleted.
Well, that'll get turned into a resource deleted event on the
client and, and yeah, Bob's your uncle and, you know, everything
stays updated. So the second replication
connection is actually for audit log.
So we haven't shipped this yet, but it's a, it's an up and
coming feature. But yeah, it's, it's a, you
know, you, you when you get these, you get this, this wall
processing, right? You get it.
It makes for a very, very nice audit log.
And so we can embed like subject data into there.
And yeah, you can just get a, you get a history of what's
changed in the portal where like, hey, did this user, well,
you know who, who added this user to this group, right?
Like, why do they suddenly have access to the production
database? So we we can track an audit log
using the the wall stream as well.
OK nice. Is that like internal or like
user access? Can see logs as well.
So there's two types of audit logs that we'll be shipping in
Fire Zone. There is the what we call like
the configuration change logs. Those are in the those are,
those are basically what's changed in the database.
And that's where this replication stream comes in.
And then the second flavour is what we call flow logs or yeah,
those would be like the access logs.
And that's like, what did this user access?
Like what are, you know, how many bytes did they transfer to
the, you know, when did they talk to this database?
And so that's at the. Those are captured at the
gateway. And then we, we, we, we emit the
metadata from there. Yeah.
OK, nice. All right, let's see 10.
I have some questions, but we'll see.
Do you are you using Phoenix or Live view in any part of your
project? Like, yeah, we, we, we are, we
should, yeah, we, we want to use it more and more.
So we've got, we do use live view pretty heavily.
And it's been awesome. We we've, let's see, we've got,
we've got like, so imagine we get, you have like a table view.
You got one administrator using the product on one on, you know,
using what we call the admin portal, using it in one screen.
And then you got another admin using, you know, using the admin
portal. Let's say they're looking at the
same screen or the same view, and then one goes and like
deletes a resource, right? So it's nice just to be able to
detect that and be like, you know, we, you know, we, we're
receiving resource events in this, in this live view.
And then we can say like, hey, the table table data has
changed. Would you like to reload it
right you? Don't auto.
You don't want auto reload it. We, we, yeah, this is like a we
had a discussion about this. We decided not to just because
you know, how many times do you like you're using a product and
then you try to click on something and then it just like
swaps out from. You.
Yeah. And then you click on the wrong
thing. Yeah, it I feel like that
happens to me like 5 times a day now.
So we don't do stuff like, we try not to do stuff like that.
That makes sense. We'll just.
We'll just show a little notification.
We show online, offline status in real time.
So when a client connects, we'll show like you're looking at the
clients table or the gateways table.
We'll show all of that stuff in real time.
Yeah, we're using presence, Yeah.
So those that live view module will subscribe to the presence
tracking for that. Yeah, it's been great.
I love I love live view. I know some people have harsh
opinions of it, but it's been perfect for everything.
Really, what are the What are the criticisms?
I mean, a lot of people it's just like it, they don't like
how like the access to the Dom and like things like that.
So they want like react and like more control.
But like I I'm not a front end guy so maybe I don't really I
have the same pain points or I refuse to accept I have pain
points and I just make live view work I.
Don't Yeah, live view seems like perfect for us, you know, full
stack engineers who lean back in, who are just like, I just
wanna yeah, like I have. The front end together, you
know. Exactly like I don't have to
write JavaScript. Mostly I will say we had to
think a little bit more about, you know, as with any, as of any
project or any web app these days, like if you're doing, if
you've got global users, you gotta think about latency and
like interactivity. So by default, just putting
something up with live view, you know, you, you could like you,
you may need to think about like how to handle interactivity and
what state needs to live actually in the browser or on
the app server or, and what actually needs to go back.
Yeah. It's a difficult balance right?
Like so. I recently just launched a a SAS
product called Kill Switch. Encrypted.
Document storage with dead man switches.
So you can like if you don't check in you can send documents
to people automatically, right? And but I really like so it's 0.
Knowledge. Encryption.
So I have a lot of JavaScript on the client side taking care of
encryption. And then just a lot, there's a
lot of JavaScript. And I feel like you have to have
that fine balance. You need that speed with
JavaScript and you have to be really mindful of what's taking
the trip there and back to the server.
Do you, do you, do you use live view for that?
I. Do I mean the JS side?
Yeah, the little JS, yeah. I'm I have a lot of JavaScript
hooks. Essentially, yeah.
Yeah, yeah, yeah. Yeah, it's been great.
I'm happy with it. Like I don't, I don't find any.
Limitations at least. Yet, you know, we'll see what
happens. Yeah, it's been.
We have so, so sometimes the Rust guys on our team will come
and, you know, to ship a ship a feature like we, we try to, you
know, encourage, you know, people to take ownership over a
whole feature, right. And so like to get a feature
shipped, they may need to make some changes to the web UI or to
the channel modules or something.
And surprisingly, they've picked it up like they've picked picked
up both Elixir and Phoenix in live view like pretty quickly.
Like they don't have any issues, you know?
Yeah, I haven't written anything in Rust, honestly, so I don't
know. Like is there is syntax similar
or not? Not at all.
Oh, Rust is. Rust is.
Rust is a bigger I feel like. It's a bigger language than
Elixir. That's just, that's just my, my
feeling, right. It's it's there's definitely a
learning curve there. Yeah.
You, yeah, you, you get, you get sort of beat into submission by
the borrow checker very quickly, but.
But once you get that. Once you get over that hump, I
hear I don't think I'm over the hump.
I'm somewhere, you know, still pushing, pushing that boulder up
the hill. But once you get over that, I
hear it's it's it's a good feeling.
Yeah, OK. I mean, I feel like the.
Rust devs, they love Rust, so it can't be horrible, right?
Or is it more of a is that a kool-aid cult?
I don't know. No, I mean, I think it has its
merits. Yeah.
But yeah, we, yeah, there's there, Yeah, there can be a
little bit of Kool-aid there, I guess.
A little bit of Kool-aid. Let's I'm so I, I know there's
like a lot of like AI and LLM fatigue out there.
What, what's your approach to developers using it in their
workflows and like, how do you feel about it?
So I don't think. From a so I so like we.
Encourage it. At the end of the day, you own
the code that you push, right? So if you're doing, if you're
doing or if you're doing a review, you need to make sure to
do a good pull request review, right?
You can't, you know, you can use, you can use it on both
sides. You can use it both for writing
and for reading. But at the end of the day, I
think it means you, we all, we all are reading more code now
than we're writing. And so some developers, you
know, they're, you know, feel better about that or some feel
worse about that. But yeah, I think the, you know,
I think the lever, yeah. Where the, the kind of the thing
to focus on is like is, you know, are, are you reading the
code that you're writing and like, are you still taking
ownership over it? And as long as that's the case
and you feel like that, you know, you feel like your
engineers are still embodying that, then I think, you know,
whatever helps them go faster. Yeah.
So Colex takes a like.
Our whole back end team takes a very similar approach because at
the end of the day, like whatever I submit, I have to
have ownership of. So if I'm submitting garbage,
that's on me. And and yeah, and if we're
wasting people's time P doing poll requests and reviews, like
that's not fair either. So.
You have to be. Very mindful of what you're
pushing up. Yeah, exactly.
Yeah. Like you know that you can't
you? Can't push up anything that you
didn't come over yourself, right?
Like you can't request review Yeah like with with Elixir
mentor, I do like. Pre vetted recruitment as well.
So like I'll send like take home technical projects, you know, to
people and I like I don't I mean if they specifically ask, I say
look, I don't care. Use your normal developer
workflow, like if it's using an LLM, great.
Like I want to talk about your workflow.
But then there's also like if there's over engineered parts of
the application, you better fully understand that shit
because we're diving into it. Because that is one thing that I
found like people will go over the top with like logging
systems and like admin and like extras.
And I'm like, I know you probably didn't write this or
think about this feature. You were just like, make it
better. Like make it more.
Impactive. Exactly.
So like if I catch you not? Knowing what code is in a
project you submitted, it's not going to go.
It's not going to look good for you.
Worse than just not having it, right?
Exactly. I'd rather a feature be missing.
Than have a feature rich, something you didn't write or
understand. I mean, yeah, yeah.
I mean, at the end of the day. Like all code is tech debt,
right? It's all going to get going to
get changed and be obsolete eventually, right?
So isn't that kind of sad to think about?
But then they're like, we're so happy when we.
Delete more lines of code than we've added on a refactor.
Yeah, yeah. That is true.
Yeah. We just did it.
We shipped a really a really large refactor recently.
We we had to refactor essentially our whole identity
model and it was like it was, you know, I, I'm gonna, I have,
I have scars from it, like PTSD. It was, it was, it was, I think
the PR, it was one of those PRS where you're like, I'll just
start this one off small. And then you're like, you know,
I can't really like split this up easily.
So I'll just like add into it. And then it grew to be like, it
grew into this thing where it's like, all right, we're gonna
have to do this in an atomic cut over way because it was like
shipping AV 2 of something, right?
And so we had like 7080 thousand lines of code at the end of the
day. And then we just had to merge it
in and then schedule the roll out for that.
So I'm curious, I like, do you have a staging?
Environment. I feel like, yeah, we do OK,
because I feel like that like is that.
Difficult. To kind of mirror production,
how does that work? It's not difficult at.
All, it's all just terraform, you know, So you, yeah, you
just, it's just a renaming of things.
But we have it identical to to production and it's, it's
essential for us because you know, you know, like I mentioned
earlier, broken code we if we're down for 30 seconds.
Then. We start hearing complaints,
right, like just 3030 seconds, like or a minute, right, like
people. Yeah, it's so, so there's, so
there's yeah, you've got to, you've got to make it reliable.
And yeah, we, we have a staging environment for that.
OK. And then like you mentioned,
like a rollout. Plan do you slowly roll changes
out to so we do blue greens yeah we we do we.
Do sort of, I guess you could say blue-green.
What we do now, it's evolved over the, over the, over the
past year or so in a few different times.
What, what we do now is we ship Docker containers to a, a pinned
image. So let's call it, you know,
whatever our imagery repo is production, then we've got a job
on the on our VMS that looks for new images.
So when it finds them, it'll pull it down and then it run it
spins up the image, but in isolation.
So it's not actually connected as a service yet.
And then it'll run a series of health checks on it, right?
So it'll join it to the cluster. It'll make sure you know, so
there's you know, it'll it'll make sure that everything's good
and proper and then it'll start returning 200 from its I guess
we have a we we do a red easy endpoint and then you kind of
don't have to do, you don't have to deal with hot reload.
From the Elixir side, yeah. So that I mean we've talked
about it. The, the way that we've
architected our, our, our control plane is such that it
can go down or it can get reconnected without really any,
any packet loss on the data plane side.
So those are kind of like that. I feel like I need to implement
something similar like right? Now I have like a few seconds of
intermittent connection, like when I push a change to
production. Yeah.
And it's because I'm just, you know, I'm, I'm basically just
Docker composing a building. And then so it goes down from,
oh, it restarts. Yeah, yeah, yeah.
So, So what? I do.
So yeah, this is actually fairly simple to do.
What you could do is you could spin up the container on
different ports and then you have an IP tables, IP tables
command that just swaps the ports with Nat.
And then now the packets are gonna and then you spin down the
old container right, 'cause so that way the, yeah, it's, it's
been, yeah, it's been pretty reliable for us.
Just a little trick you can do. And yeah, but no, we've we've
talked. About doing the hot code.
Reloading thing. I mean, it's nice to know that
if we really need to hold the socket open, the web socket
open, we can we could do that in the future and the.
Language supports it, right? The.
Runtime. Yeah, that's another like.
I've never dealt with it. Like, I've read some blog posts
on it and maybe I'd just go that route too because it's pretty
cool. Yeah, yeah, we've got.
It's just a lot of state. I I'm sure too.
Yeah. We've, we haven't done it
either. It'd be great to talk to someone
who has a lot of experience in that regard.
Yeah. Who has production code?
And they're using hot reload. Reach out, I'll have you on.
Yeah, because yeah. Like I've heard like.
Once you understand how to set it up.
It's relatively like. Easy.
But I don't know if it's like the most straightforward thing
to get functioning. Well, I don't know.
I mean I, I think, yeah, I mean I think the the setup in the
boilerplate is not. The hard part, right?
It's like, you know, you, you, you do a mixed release, you get
the the bundle and then you just send it out to the server and
swap it over. I think the hard part is to
reason about, OK, you have this old state, right, that your
application's in, it's running and now you've, let's say you've
done a refactor where you've just, you know, you've added new
fields to a schema module or something like that.
You know, like now you've got to like get it into this new state
and then so every. So it may not be possible prefer
that balance like yeah, so. You've got to migrate.
I think it depends on the module, right?
But you've got to migrate, you've got to migrate state over
yourself, right? I see.
Because you're not. You're not.
You're not taking the supervision.
Tree. Down and then bringing it back
up right that stays that all stays online.
So each module needs to know how to migrate state.
So like yeah the bigger the changes the more.
State demand. So do you write like I'm, I'm,
I'm wondering, do you just write some like migration files for
the changes Then yeah, I wouldn't know.
Yeah, I think. I don't think they're migrations
in the sense of database migrations.
State migrations, not database migrations I see.
I yeah, I think now we're getting onto onto.
The edges of my Elixir knowledge.
But I think now, I think, I think you've got a right.
Yeah, yeah, I think you write, you write hooks for this and
they live in the modules. I could be wrong on that, but
yeah, they're definitely not other separate files from what I
know. Yeah.
Now I want to kind of dig into it.
I'm curious. But the, you know, the, the sort
of, it's like the Holy Grail, right?
You don't even drop a website connection, right?
It's like everything stays connected.
Yeah, which I kind of would like like from a.
From a live view. Experience on the front end that
would that would make it incredible.
We we do that is 1 big area. Where we do, where we are
interested in using something like that.
But even then, the workaround I think we'll be employing, which
we do a little bit are ready of is when you ship a new,
basically just drain the connection.
So like when you ship a new release, pop up a little
dialogue. If whoever's got an active
session, just pop up a little dialogue saying like, hey, you
know, the application's going to be upgraded in 5 minutes or
something like that. They got to, they have like a
countdown, they can finish whatever they're doing and then
they can, you know, if they're filling out a big form.
Yeah, 'cause, 'cause like, yeah, this whole, the whole hot code
reloading thing is like, you know, from from what, what what
I know about it. I it seems like a lot of it, it
just seems like it could get very complicated very, very
quickly. So if you can avoid it, it's
like metaprogramming. Like if you can avoid it, just
just avoid it. You know, dude, metaprogramming
hurts my brain every time. Yeah, I.
Can like I can like I always. Feel like I have my head.
Wrapped around it and then I don't you're like quote UN
quote, like like where, which, which one am I?
In yeah, like I I did like a elixir.
Introductory course and I touch on metaprogramming and even
through that course, like I struggled through it like like
I'm teaching it and I struggle through it.
This is like it's very hard to stay focused on what you're
actually. Intending it is.
Don't get me wrong, I think it's awesome.
It just really hurts my brain. Yeah, yeah, yeah, we, we've
recently. Yeah, we've done a lot with just
kind of trying to get rid of some of some meta programming in
our code base, just just with the with the action of like
LLMS. So that's that's one thing.
Like it's hard for LS PS to kind of like integrate with, right?
It breaks your go to definition, right?
Like so we've just sort of set a rule in our code base to avoid
it as much as possible. And and so then the places that
we do use it, it's like very clear and very beneficial.
Like it's a clear benefit to have it in there.
And then, you know, the, the pro out the pros outweigh the cons.
But yeah, we were doing it. We we used to you.
Know we we were trying to be clever.
In some places and and just ended up just making it
explicit. Yeah, yeah, sometimes cleverness
and over. Engineering just hurts you in
the long run. Yeah, yeah.
I. I'm curious, is there?
Any like other than you said, you mentioned your logging
system, are there any like big features you can talk about that
you're excited about implementing in 2026 or let's
see? Let me take a look at our road
map. We do have a public road map.
We try to keep it mostly up to date.
I mean, we're. We are open source so.
So there's not much there's, you know, there's, there's, there's
not much I guess on the product road map that is closed off.
But yeah, I mean, we've got the the audit logs.
Features of is a really. Big one, other products in the
space don't quite give you the level of detail that the this
like logical decoding can can give you.
So you know just every insert, update and delete, seeing what
user you know is attributed to that and whatnot like.
What you know all the way down to their.
IP address. So we're really excited about
audit logs. Another one is so we we just
like I mentioned, we just shipped that IDP refactor we do
now we sync your users and groups.
We refactored how we do all of that.
We're going to so we built the sync engine for that.
We're going to use that sync engine to then pull in like
your. So pull in your your
organization. 'S.
Device asset list, so this is for like MDM mobile device
manager like Intune, Kanji, JAMP, any of those ring a bell
IA little bit pull in, so we'll be pulling in your.
Devices basically. Your organization has an asset
list. We'll pull those devices in and
then we'll be able to authenticate that.
Hey, this, this user's connecting only from an
organization owned device, right.
So we call that MDM better MDM integration.
We've got a few on here. Those those are kind of the big
ones that we're focused on now finishing our cloud migration,
although that's not really user facing feature, but gotcha.
I'm curious like with with open. Source is Is your full feature
list available open source or do you pay all parts of it so?
The, the full product is, is, is open source.
Now some of the purists out there may take issue with with
that. So we, our control plane, the
Elixir stuff is elastic. So if you're going to run it,
you can run it yourself or your organization unrestricted,
right? No, no feature gating or
anything like that. The reason we did that was
because we used to be and then the, the rest of it's Apache
two, we used to be fully Apache 2 and we had some issues with
like people taking that and just running it and selling it on, on
the cloud marketplaces. Oh gosh, of course.
And it's like, OK, but that was like that.
That's fine. Like, that's one thing.
But then we would get the support request for it, right?
And so, so that's like we're supporting someone else's
business at that point. And so we, we thought like, you
know, going forward, we'd still like to make the code open
source and available and kind of like free to use as long as
you're not trying to sell it, sell yourself.
How do you mitigate though your policies?
Like that, like I feel like that would be difficult alone.
We had, I mean, we, we find we. It's funny.
Because like we typically find out because they're, they're
not, you know, they, they may take our control plane and go
host it, but they're not taking our client apps and like
rebuilding them. You can't take Firezone like re
rebrand it and distribute it on the Apple App Store.
Like Apple will typically catch that, right?
Yeah. OK.
So, so unless you're, unless you have like an enterprise system
set up like MDM, well, they probably see the binaries are
exactly the same, right? And then they're just, well, a
white label over it. Yeah, Like I, I, yeah, I that's
a good. Question.
I don't know if they would detect it like that, but but
they typically what happens is like they're just using our apps
And so we get all their sentry logs and we're like, OK, you
know, like we, we, we, this happened actually with one, with
one entity and we had to send them, we had to send them a
letter like, hey, please stop. You know, like, you know, you're
violating our terms or our our license.
So, but yeah, it hasn't been, I mean, I think most.
Players in the world are pretty good act are are are good
actors. And you know, no one's gonna go
blatantly try to violate your yeah.
Do you find like since like I like?
You're AVPN service it's like based around security.
Do you think making it open source is essential for like for
trust? That's a great question.
I. I personally do.
I guess it's not always cut and dried, right?
Some, some people, I guess, still believe in this.
Like security through obscurity. Yeah, I mean it helps, but
you're probably more. Secure because you get scrutiny
it it could it could go both ways.
I mean, we definitely. Get drive by researchers sort of
being like, hey, I found a problem in your product, but I
won't tell you what it is unless you pay me, you know, some some
bounty right and and so we you know, we're stocked too
compliant. You know, we get, we get pin
tested and all of that. And so I will say overall it is
a, you know, you, you can get people out there who might find
something and never report it. But I think you, you generally
get, there's many more people out there who will find
something, either report it or you know, we've yeah, they'll,
they'll, they'll report it. Or generally by having it open
source, I think you you. You tend to squash those.
A lot quicker right? Do you get random contributors
or is it mostly your team? It's mostly, it's mostly just
our team. Yeah, we, I mean we.
We welcome contributions. Before you're going to go work
on something really big, though, probably check in with us,
right? We've had, we've had a few
larger PRS that we, we couldn't accept because they just weren't
in the right direction for where we were going to go.
Or we were going to be, you know, they were on some area of
the product we were about to completely refactor.
That happened recently. So but like bug fixes and things
like that, we've had, we've had a number of and yeah, super
welcome. That's awesome.
All right. So being a.
Single like SAS founder like no employees.
I'm curious what what do you do as far as like marketing and
getting eyes on your product? What's like the best Ave. for
you? That's a great question and it's
something I need to do. More of right it's AI feel like
we technical founders don't technical founders it's well
it's. Like yeah, the product.
Has to work right so a lot of our a lot of my time is making
sure that everything you know the grease the big figures are
greased when I'm not doing that we we typically will we do a
number of so we we. Have blog posts?
That we'll write from time to time that some of them have
landed well, some of them have been complete flops, but we get
a good amount of inbound and we get a good amount from actually
our GitHub repository. Oh, nice.
OK, that's actually so, yeah. So we'll be spinning up a number
of different. Marketing efforts.
I guess in this year 2026, so, so stay tuned.
But yeah, a lot of this, a lot of a lot of the stuff that we've
been working on, especially on the data plane side, I think has
been really interesting for for that community.
And then in addition, you know, better kind of detailing some of
the stuff we've been doing on the Elixir side.
So we want to write some some blog posts going over all of
that. What do you think?
What lands better there? Like very technical.
Blog posts or more feature based or like we do, we like what
you're doing. We've been doing these.
Devlog posts. That have been.
Interesting. Yeah.
And and yeah. So, so it's nice just to look
back on the previous month and go like, Oh yeah, we did work on
that and we shipped that. And you know, here's how, you
know, we we should probably remind, you know, such and so
that this was shipped because they were waiting on it, right?
It's a, it's a nice little ecap at the end of the month.
That's a good idea. I should do like a dev log.
I so I just started doing blog posts with kill switch but more
and more like feature related or why you would want kill switch
in your life kind of thing. But maybe more dev based ones
that explain how the encryption works and things like that might
be beneficial what you could do that we've started doing
recently. Is just do AI.
Don't know if you're working with anyone on on Killswitch.
But. We've started doing week weekly.
Well, you maybe you could still do it, but we've started doing
weekly vlogs. So we'll, so I'll get on a call
with someone else and at the end of the week and we'll just talk
about, we'll block off 30 minutes and we'll just have a
casual chat about what we worked on, what we were worried about,
what issues we faced. And then it's in a, and then
it's in a video format, right? You don't have to think of, you
know, you don't have to type it all out, pick your favorite LLM
tool, transcribe it right. And then you've got, and then
you've got content for your, for your.
I like that. I mean, I could just like, talk
to myself. About what I did that week for
Killswitch and do that. I like that.
Yeah, yeah, it's a good idea. Do.
Do does your blog? I haven't checked it out but.
Do you can people log in and interact?
Can they comment like, Oh no, not, we don't have that.
Yeah, we it. So the website is something I
built in like 2 weeks I think it was like a couple summers ago.
I was like, oh we need a launch, we need a website and it doesn't
have any of that stuff. No, it's my I'm not throwing any
shed. My blog post is the same way.
Yeah, it's just you read it. Yeah, Yeah.
And then I was pretty. Proud I built a pretty.
Cool like dynamic? Site map so I don't have to
update it with my blog post. Slugs.
Yeah, site maps are critical, you know.
Do you have? Your your site optimized for
what do they call it the LLMS text?
Yeah, the. There's that one, but there's
like SEO. For it's called like Geo or
something like that. Oh no, I don't have that.
I may I may I may be getting that.
Acronym wrong. It's.
Like SEO for LLMS. Let me see if I can find.
This generative engine. Optimization Geo.
Yeah, I put an. LLMS dot text route in I don't.
Know if that's what this is? Let's see.
Just tell me more Google. Dang it, so much to learn.
Now I got to do something with this.
But yeah, it's it's wild because you start hearing everyone's
just. Using LMS to do research now and
what's funny is I actually used. Claude to.
Find I have an affiliate partner program now and I found the
Stripe app with Claude because I was like, I need to be able to
pay people, you know, worldwide and that's like not it's not an
easy thing to track because you have to give them all the proper
like tax paperwork and things for like the EU and then the
that's and they handle all that, which is kind.
Of nice and I'm using. Dubdub.co OK, is this is your?
Selling direct to consumer. Yeah, yeah.
OK. Yeah, Yeah, yeah, that.
Yeah, they're a. Little that you use a little
more. Lenient if you're doing like B
to B transactions, I think. Oh, are they?
Yeah, you just yeah, it's a different tax.
I've had some people bring up, I don't know if it's.
Worth my my time yet? But like a what would you call
them? Like a a.
Financial advisor was like, he wants, he wants a plan that
would allow him to kind of like oversee and control clients
stuff. And I was like that like defeats
the purpose because my point is like you don't have to trust any
third party. And so I don't know, maybe
there's like. A.
Way I have like. Different vaults of files and
you can give people access to specific data, but not all your
data. I don't know.
I got to figure that out because the second you start.
Sharing like. 00 knowledge encryption, it's no longer that
right? Because now you're you're
allowing access to the encrypted file.
So you have to kind of you unwrap your private key
essentially. And it just doesn't.
Work the same way and it it opens things up but I don't
know. So you're using 00 knowledge
encryption. For for.
Yeah, yeah, yeah. Is.
It is there like a. JavaScript.
Library for that that you've. Been working with yeah kind of.
So I'm using JavaScript. And I I generate like a private
key. I'm I'm doing basically like
what 1 password and bit warden do.
And so basically your password client side will unwrap your
private key. Like I wrap the private key
client side, send it to my server and store it.
And so I can't do anything with it.
But then you get it back, you can unwrap it, encrypt, decrypt
your files client side. And then it's all like
relatively safe. And then the, the share links
were the hard part, right, 'cause this is where you kind of
like you kind of open up a layer of vulnerability.
But I I basically create a share link with a new encrypted key
that you can remove access to. So let's say this share link
goes out through a deadman switch to a client.
It was an accident like you don't want it to hit that
beneficiary. You can revoke that that off
token essentially. And so they lose.
Access to the file and all that. But the caveat to that is both
parties can now read your encrypted file.
So, but you can remove it. You can remove access, but
there's a balance. It's kind of, but I tried to
follow best practices and I went down the rabbit hole of like,
how does 1 password share vaults?
How does this work? And yeah, we're all doing a
combination of symmetric and asymmetric.
Key encryption, yes. Yeah, exactly.
Yeah, Yeah. But.
I kind of like I went into. It not knowing that.
Much. And I learned a lot about
encryption, so that's cool. It was like a service I.
Wanted for myself, I was like I want a dead.
Man, switch and I turned it into.
A product is it? Which can I ask?
Which cloud it's running on? Are you running on a on the
cloud or are you using like fly? I use.
I use Digitalocean. I run on my own droplet and just
docker compose. Yeah, yeah, yeah.
And then everything is stored. On their S3 buckets.
Encrypted so I can't open any of it.
Nice. Yeah.
And that was like, that's, that was my.
Whole goal, I was like, if I ever have engineers, I don't
want any of my engineers to be able to go and read a user's
files, you know? Yeah, you say that.
Now until you get. Support issues that are.
Like ah lost by login slash browser storage slash browser.
So that is, that is a concern and.
Right now I don't expose the private key to the client and I
do like recovery codes. I generate recovery codes that
they can download on registration.
And I have considered I may be revealing their private key to
them in that same step and give them a way to save it.
Because if they have that, they don't even need my service to
decrypt their files. So I could still allow them to
download their encrypted files and then if they have their
private key, they could still decrypt it.
I don't know though. I feel like that adds a layer of
complexity that I don't really need.
Yeah, you know the. Let's see.
The at least. On Apple.
Systems, you know you've got these if you're using, so it's
not like a cross-platform, cross browser solution, but you can
get into the you know you can, you can integrate with the I
guess the secure enclave or key chain access to to help out with
all of that stuff. So you're not.
Having to do it in. In JavaScript, yeah.
Yeah, that's. An option I've also like batted
around the idea of creating like an Electron app version or some
version where you can locally have it to kind of protect that
side a little bit. Little menu bar applet that's
like running with a browser extension.
And yeah, yeah, I think that could be cool.
And then just like. You just get a local
notification for check insurance and just another layer.
So we'll see. I have a lot of ideas.
It's just where does my time go? Is it open source?
It's not. Should it be?
Yeah. I think it should you.
Should you should open source it?
Yeah, you can do little pieces at a time.
Just do like, you know, just do the web, the web UI first, OK?
My like my fear is. Properly open sourcing things
like making sure everything is is clean.
Do I have any? Default.
Keys in there that I shouldn't you should.
I mean those should be. Those should be easy to.
Find, yeah, those should be easy to find.
But yeah, you won't get, you'll get.
I'm sure if you've got any major issues, someone will come by and
fix it for you. That is true.
It's kind of like you get a free audit, right?
Yeah, but I mean nowadays you can run.
You can, you know, there's a million different audit tools
you can run like code QL and yeah, yeah, for sure.
Some of the stuff like so below I think is another one just for
basic I I use paraxial on it too.
And that is so blow. OK, so as far as I know.
Everything is secure. Yeah, yeah.
I mean. Like I think you could.
Also just put a big disclaimer and say like, hey, you know,
don't, yeah, maybe don't offer it as a service.
I don't know if you're offering it as a service now, but it is a
service. Oh, it is a service now.
So. Yeah.
So you. Yeah.
So if you put it out there, yeah.
You, you probably, you know, you could, you could also get like
an independent review. That's not a bad idea.
Yeah. I think like the hard part.
Is people have to trust I'm. Handling their data in the right
way, right? Yeah.
So maybe like the third party review would be a good
disclaimer or you know, a good review.
To have. Have you looked into like the
the whole trusted? Computing movement with to to
because like you can have your. Because the problem there is
like you, you put your code, put your code out there in a repo,
but there's no real guarantee that what you're running in your
service is the same, right? Yeah.
So if you're looking to build trust, that's a good need to.
Yeah, I'll have to go down that because I think it's like.
One of those things you have to prove and people have to know
you're doing it the right way. But yeah, cool.
All right. I really appreciate you joining
me this. Was a lot of fun.
Yeah, it's great. Fire Zone is a cool.
Product, it's over. My.
Head, but it's. Impressive.
Like it's cool to. It's cool to see something I
feel like at that scale working so well.
Like, it's cool. And yeah, thanks for having me.
Yeah, it was a lot of fun. Yeah, I appreciate it.
Thank you so much. Jamelle.
And. I have all your contact info in
the description and tomorrow after this process is on YouTube
for 24 hours, it'll be pushed up to all the podcast platforms so
people can catch this conversation wherever they want.
Awesome, the future is now. And for everyone that tuned.
In Thank you so much and. I'll see you guys next week.
All righty, bye. Testing.