This episode is brought to you by Beam OPS.
If your Elixir deployments are painful, or your infrastructure
is a mess, or you just need experience hands to help you
scale, these are your people. Beam OPS specializes in scalable
Elixir development, automated deployment pipelines, and
infrastructure migrations. They will help you work smarter
and ship faster. As authors of the Pragprog book
Engineering Elixir Applications, they know their stuff.
Check them out at beamops.co.uk. Secure your Elixir apps with
paraxial IO. From code analysis and bot
defense to dependency checks, they've got you covered.
Elixir first design with GitHub integration.
Elevate your security at paraxial dot IO testing.
What's up? I'm Jacob Litzo, your Elixir
mentor, and welcome to another exciting episode of the Elixir
Mentor Podcast. This is where we discuss
everything related to Elixir, from interviews with
enthusiasts, pioneers in the community, to innovative
projects and library shaping. Elixir's future.
Thomas joins me today about building systems that don't hold
you hostage and keeping knowledge that makes your team
irreplaceable from ever walking out the door.
I'm sure we'll talk about plenty more, but thanks for joining me
today, Thomas. I'm excited to have you.
Thanks for having me, looking forward to this a bunch.
Yeah. So do you want to just give a
like a quick version of how do you end up as head of
engineering at level all and like kind of what's your journey
there to that to to this moment? To this moment, I have been with
Oval all for three years. I started as a senior, moved to
staff and then we had a departure above me who wanted to
return to being an IC and it was just kind of like a natural fit
that I slotted myself into. I'm a Co head of engineering
with another gentleman on our team and we've been tag teaming
us into success. Awesome, very cool.
I recently I met you down at X Max.
You did a talk down there and I know like we've talked in the
past, we Elixir community is small.
I like that. And you're incredibly passionate
about like deploying your own code, not getting locked into
solutions that make it hard to be flexible and kind of like
adjust your needs as, you know, fast-paced start-ups change.
What's kind of your experience with deploying production
applications and what made you not like solutions that exist
out there and kind of do your own bare metal deployments?
Of course you can call it passion, some people may call it
delusional. It's really like AII want to
keep all available options open. And I feel like when you are
deploying to a pass like Heroku, RIP Gig Elixir Fly, while they
take care of a lot of things for you, they also tie your hands at
the exact same time. And that's not necessarily the
flexibility that people may want.
Hands off is great until you run into a problem.
And last year we ran into a problem.
We were deployed, we had a couple services on Fly and then
our main production application was on Gigalixer and we had a
bunch of traffic and we could not throw enough hardware at the
problem to settle all the traffic.
We took the bar and slid it all the way over the that node that
we were on at Gigalixxer. We had multiple, but it like our
bill was $7000 a month. And that just seems nutty to me
because I know with the shadow of a doubt that our application
doesn't need that, right? We all instinctually know that
Elixir doesn't. Elixir's really powerful, but it
doesn't also need a really powerful machine.
I feel like most Elixir production applications could be
hosted on a Raspberry Pi and be totally Vine.
But I think that's where the lie of performance and cloud kind of
conflate because you are paying for inflexibility and bad
performance. And I know that's a hot take.
I'm going to hear about it, but it's what I feel so.
Hot takes are the best takes man.
Sometimes we migrated away from Fly and Gigalix are still in the
cloud at Google, but using Google directly, we are as bare
metal as you can get on AVM mix release.
There's no Docker, there's no nothing like that.
Raising instance templates and Migs.
I deploy a little bit differently, but the first time
that I heard of this deployment mechanism for Elixir was
actually Log Flare. There's a blog post on how they
deployed and so I kind of mimicked it, but I do a whole
lot more in our cloud in it than they're doing.
But I'm happy to say I finally got approval for actual bare
metal. We need to wait till our Sock 2
audit window closes in the middle of June and then we're
out of there. We are going actually bare metal
on rented hardware getting out of.
I have no idea what that was. We're, we're going to actually,
yeah, I guess not. It's not surprising to be
honest. We're, we're going to take that
actual leap. If you, if you've paid attention
at all, I, I feel like there's a kind of symbiotic relationship
between Elixir and Ruby. But if you paid attention to
what 37 signals talks about all the time with getting out of the
cloud, exfilling, that's what we plan on doing too.
The, the insanity of the amount of money that we're paying for
like hosted Postgres and things like that.
An $1800 a month server with 64 physical cores, a TB and a half
of RAM, 16 terabytes of storage. That's lickety split fast.
I mean, we pay $6000 a month for our databases.
Across all databases. There's more than one, but we
pay for 30,000 provision IOPS. If you're on an NVM me drive,
you're getting millions of IOPS just because that's a massive
difference. So yeah, you lose the benefit of
point and click backups and things like that, but.
When you say like you're gonna rent space, like, are you
actually gonna just rent a server or do you rent Rackspace
to put your own server in? Like what does that look like?
We're we're going to rent the the server, the company that
will go with this called latitude.
It's Latitude dot SH and they, they were really big and the the
hosting of blockchain initially, it's how they got started is
hosting really powerful servers for Solana.
But Equinix Metal, which bought deploy.net, they bought it, they
ran it for three or four years and as acquisitions go, they
just sunset it. And so Latitude is been picking
up the baton where Equinix Metal I guess left it and they are
growing like crazy. They have a ton of available
hardware. They've got things that where
it's not you're just not running just a computer.
You can set up your networks. You can immediately deploy 5
second bare metal deploys like that's.
That's pretty wild. That's amazing.
I'm I I'm inspired by your just the mixed release and getting
rid of Docker. I haven't pulled the trigger of
getting rid of Docker in my own deployments, but I really want
to. I should probably spend the time
to do it. It's it's an interesting choice.
Docker provides you a lot of ease.
You set up your container exactly the way that it needs to
run. Logging or metrics are really
easy because there's like a single outlet for it.
I feel though, that the overhead, which is crazy to say
because there's not that much, it's just not worth it.
Beam really likes being on hardware.
So I wanna I wanna give it it. And I feel VM is like when
you're running in the cloud. AVM is already kind of, you
know, it's hypervisor. It's not exactly bare metal.
But that's where when I was running benchmarks to convince
upper management last year to move out of Gig Elixir, I was
load testing our application there and I was comparing it
against a 2011 Intel Mac mini old.
And I was beating the pants off of whatever gig Elixer was with
a piece of junk Mac mini at least 10X the load on a $7000
Gigalixer node. And I'm not here to like throw
shade at gigs lickers or they they obviously provide a great
hands off pass for people. I I like what they offer, just
like fly. I really like them.
But for our needs, we just need to be different.
I feel like a lot of people say, you know, if I build it, traffic
will come yes or no. But our traffic is very bursty.
Our application is for education and you'll have a classroom of
students who all use the product at the exact same time.
Or in the case where you're like no traffic or all traffic like
you don't have to. Exactly.
So we right now, as the school year is coming to a close,
people are doing all the testing and things like that.
Right now is the worst time for traffic for us because we won't
just have a classroom, we'll have an entire school sitting in
a cafeteria or an assembly like in their pack.
And then it won't just be 1 school, it will be 12 schools in
an entire district. So you'll go from zero traffic
to 50,000 concurrent users all at once.
And that thundering herd problem is kind of hard.
Yeah, that's crazy. So rather than worrying about
like there is no time to say oh we're gonna auto fail on memory
or CPU or request per second and then wait for the server to come
up, we just have to be over provisioned all the time in
order to handle it. How do you, how do you handle
that? Do you know when like events are
going to happen or like a influx of users?
Do you know or like, how do you prepare for things like that?
Sometimes, sometimes our sales team can give us a heads up when
an onboarding is happening, but most of the time it's kind of
like, oh, all of our charts are going up and to the right, I
guess something's happening. Yeah.
And then do you do you auto scale or do you just have to
have nodes ready we? Have auto scaling turned on
right now in Google but we are over provisioned so much that we
rarely hit it and some could say well that's a waste.
Sure not using docker, not been packing your compute could be a
waste too. But that extra few $100 a month
of overhead for our entire businesses like stay up ability,
I I don't feel like is a thing like no, of course you're just
going to pay the extra few $100 to have an over provision
machine and not worry about it. Yeah, I, I feel like the trade
off is probably worth not having the headaches.
Of course, like over a couple $100 when you're talking about
server bill, that's 6 grand, you know.
Exactly. Yeah, it doesn't really matter
at that point. What do you do?
How do you, how do you like, I know this is just like a lot
more difficult of a problem and a lot of people don't talk about
it, but like sharding and scaling databases are it's not
an easy thing to do. Do you just have one really big
database or do you actually have?
Do you actually distribute your your data as well?
Today it's all one big database. It's the worst type of database
to have in the sense that if there's anyone listening who's
in Operations or ADBA, they'll know what I'm talking about.
Very high read and writes. You have both sides of the
problem. Our entire database is only like
200 and 52150 gigs big, but our wall file is like 8 terabytes.
Oh Dang, why? Well, it's because we have so
much coming in and out and and editing and changing and
updating all the time that like it's always being touched.
So we haven't sharded yet. We haven't done anything out of
the blue. We have failover in place, but
we're not even using Reed replicas.
I just don't feel like we need to.
I was just going to ask if you do Reed replicas, but no.
No, they're, they're only there for hot spares.
I when we go true bare metal, I could look into it, but so far I
don't think we I don't think we need it.
Live view is actually the the thing that uses the most amount
of like resources. Or compute if you want to.
If you want to be like super honest about it, there's just
like. That of internal sockets.
Yeah, exactly. Yep.
But we are moving away from live view going into API only mode
using a view JS front end because we have iOS client now
and we don't want to dual right API and live view.
That change is a little bit different.
I like live view, I don't really want to leave it, but at the
same time it's like it's kind of a necessity now and I don't feel
like we're going to leave. In fact, we'll probably be able
to de provision a little bit because live view is pretty
heavy and when we're just going standard controllers, it won't
be a big deal. I see.
Do you are you gonna keep live view at all for like admin
dashboards or anything like that?
Or are you completely removing it?
We're completely removing it. OK.
That's a big, big step. You you use Ashe framework as
well right? Yes.
Nice. And does that, does that help
you at all with opening up like API endpoints versus not and how
are you? It it did, but here's another
hot take. We're we're getting rid of ash.
Oh, interesting. How come?
Yeah. We, we, we, we jumped in two
feet first and just started running as fast as we could with
Ash. The problem is finding people
that understand Ash and it's interesting.
If you talk to Zach or anyone from Olympic, they will show you
a bell curve. Of I'm having Josh Price on next
week actually. Great guy.
I've talked to him quite talked to him quite a lot.
I sat next to him at an elixir confluence and he's he's he's
hilarious. The The Bell Curve talks about
you and the progression of being a developer between new and
super senior. And they talk about how the
people in the middle who are not quite senior but are very
dangerous, will push back a lot against Ash because they're
like, well, I can write this better, I don't need a
framework. And for the most part they're
wrong. But where, where Ash is wrong is
there are so many of those sorts in the Elixir community that it
becomes very, very hard to hire for it.
I can see that. And so there's people who come
in who are gung ho about like, I don't know Ash, but I'll learn
it when 80% of stuff is written in Ash.
Like I, we don't, we just don't have time for you to learn it.
That's it's, it's just a pure business CTO level decision that
you just have to contend with. And so although Ash has helped
us a bunch with going fast, we have to pull it out because it
hiring is just too hard. Interesting.
I feel like I have like I've, I have a bunch of developers that
like always ask me if if they can write my like recruitment
projects in Ash, which is kind of funny.
Like the people that love it, love it.
People who love it, love it. That's a really good way of
saying it. And I feel like there is a, if
you were to draw a Venn diagram of people who love Ash and just
passionate people in general, they're like right on top of one
another because I mean, I, that was me at the beginning and I
still love Ash. I just don't.
Love the idea of that for. Hiring.
Like when I first discovered Ash, I was like, oh, I love the
idea of this, but I was running into like I got frustrated
because I was moving too slow learning the DSL and I would, I
ended up like ripping it out of a couple of my projects.
But then I reinvested my time into learning it and I'm using
Ash again across like most of my projects because it I do enjoy
the speed. Like especially like being the
lone developer on like my own projects.
You can move so fast. Absolutely.
If you know what you're doing, it's very quick.
If you don't, it's an exercise in frustration.
Yeah, yeah. No, for sure, it definitely is
because you're like, I know how to do this without ash, but how
do I do this the ash way? Exactly.
And so I feel like when people start to try to convert stuff in
their head, they'll be like, oh, I need this and this and this.
And just because of how Ash is written, very DSL heavy errors
are not as good as they could be.
If you're not doing it correctly, well then you're
going to hold the phone wrong and it's your fault and you're
going to go slow. Yeah, Yep, yeah.
I ran into that a lot. Like I was like, dude, I know
how to just, you know, I, I'm trying to think I just went
blank on what I was about to say.
Oh yeah. I was like, I was building a
multi tenancy app and I was like, I know how to manually set
this up and I freaking couldn't get it to work right with Ash
and it was so frustrating. And then it was literally a one
line change and like Zach helped me and I was like, are you
kidding me? Like so like once you unlock the
power, it's amazing. Absolutely.
I mean, Josh may talk about this next week, but Olympic has a Ash
APS, ASH Premium support. It's a thing where you pay
monthly to a limbic and they'll create a shared Slack channel
and you can create tickets for the people who write ASH.
We did that and it was incredibly helpful because
there's sometimes where you're like, well, this is ambiguous.
I could do this a couple different ways and ASH, I don't
know what's best or one of the big ones that we found is if you
have really weird esoteric queries where you you need to
like a good example is you need to reach inside of a Jason B
column to pull something out. You don't want the whole column
because you know, it's gonna be like a mega of Jason.
Ash by default trips over itself because it's like, oh, let me
load the whole thing and then let me go select the thing that
you want. It's like, no, you're I don't
want 1000 megs of junk coming out of the database.
I just want the one title of whatever I want out.
So questions like that are really good for APS because
although the community is helpful, their Discord is
helpful. Going directly to the people who
write it is a lot more helpful. And then if you run into like a
bug, they'll confirm it's a bug. They'll fix it immediately,
almost same day or not signing you up for anything Josh.
But if there's some feature that is just basically missing, like
I wish I could do this, it gets added to the road map and and
merged in pretty quickly. So for those of you listening
who really like Ash that want to take it to the next step, not a
shell, but I would go with APS because it's really helpful.
That's awesome. So how do you feel about ripping
it out of your project? Are you sad about it?
Mixed feelings. Totally mixed feelings.
I have to change my hat. If I put head of engineering hat
on, it's kind of a no brainer because of how hard it is to
hire. If I put my developer hat on,
yeah, I'm sad because it does a lot of things.
It's declaratively magical, as Zach would say.
It reminds me of Rails in a lot of ways, although there's
circuit Breakers and peel off points everywhere.
If you need to do something custom, it allows you to do it.
So I, I, I'm, I'm sad, but I think it will be a good change
for us because at the end of the day, again, putting the head of
engineering cap back on every team is going to deal with
churn. People move on with their lives,
want something different. And when you lose developers
that understand the code base, understand how Ash works, you
got to hire someone and train them up.
And it's like, it's just one more thing that you have to
train these people on. And it's like, we, we just don't
have the time. We're a tiny team, including our
project manager. We're 11.
That's that's not a lot of room to to handle churn every every
year. Yeah, how, how do you guys feel
about like Claude code and LLMS these days with ash?
Are you are you using it or not really?
Our team has gone through ebbs and flow with AI in general.
Last year I feel like we leaned in really heavy and then it bit
everyone in the butt hard because I don't think last year,
like early last year, the models just weren't ready for that.
And so we we pulled way back and said stop doing this.
You can use it to augment what you're doing, but don't, please
don't add a bunch of AI generated nonsense into our code
base. But over the last few months,
the models have been coming so good that I feel like it's just
an accelerator. And as I explained to our team
and now the general public who's listening, there should be like
a one to one relationship almost with your planning phase as your
coding phase. You need to spend an hour
planning, going back and forth, making sure you know exactly
what that model is about to write before it does.
I don't really have a problem with one shotting things because
of that. And I think when people read on
X or and blog post about these developers that we look up to,
like DHH or or Mitchell who's doing stuff with Ghosty or
anything like that, how can you one shot stuff?
I've never been able to get my model do that.
It's because they're planning endlessly and making sure that
every line of that plan, it's like, yeah, that's exactly how I
would write it. Because once it matches what you
would do, then it's going to do the thing that you expect.
Otherwise you're going to end up with hilarity.
The amount of times you get with a an unplanned model writing
nested IFS conditionals inside of a with macro is like Oh my
goodness, who would ever write that?
Well, no one would, but AI will if you let it.
I, I found it doesn't do it so much to me anymore.
And I maybe it's because my planning phase and like my like
my agent MD is just more organized, but I used to like,
for whatever reason, Claude just loved conditionals and like
nested if statements over pattern matching and I always
had to yell at it. I was like always use pattern
matching like over, you know, a messy block like this.
And I, I feel like I pretty much can one shot like new features
now. And I mean like I do spend a lot
of time planning it and like I get a detailed markdown file of
my feature that I want to implement.
And then I don't know if you've have you heard of beads?
Yes. So that whole project has kind
of gone to shit and it's like kind of garbage.
They've tried to make it like too fancy in my opinion.
And actually one of my Co workers, Jason, he he built his
own version of beads now and it's called bead bead beads
work. And it uses like it uses git and
stuff and will create its own like unattached branches.
And you can do it's like it's awesome.
And I use that for extending my context.
And then I just make like a feature that I want to do.
It creates it as an epic and then it breaks everything down
into issues. And it's pretty awesome.
I enjoy it and like, yeah, I one shot things all the time and it
just keeps track of its progress with with issues.
So it's pretty sweet. The planned phase is really
important, I think for people that are new to AI who can't
figure out how people are using it.
That's that's the secret sauce they're missing is don't, don't
assume the AI is going to do the right thing without planning
because it won't, it will not write it like you would.
And that's what we're paying you for your brain, not its brain.
Right. Well, like, I feel like it's
really important, like you need a language understanding, You
need to understand how your project works so you can
properly coach the. Like I look at Claude as like my
little junior developer and knows how to write code right
and knows how to write really good code as long as you guide
it properly. And yeah.
Absolutely. I feel like the inexperienced
people with AI are actually the most senior people around
because they're like, it's got to take my job.
Well, you could think about it that way.
Or you can think about how you've got the best rubber ducky
in the world. I've had really angry, like
probably really smart senior level engineers that are like,
how dare you, like, you know, just chill AI and then say that
everyone that's not using it is going to fall behind.
Like they don't like opinionated like opinions like that, but
like it's, it's a new tool and it's entered our lives and I
don't think it's going anywhere. So like if you don't know how to
use it, you are gonna fall behind 'cause you're gonna be
relatively a lot slower at getting stuff out than like a
mid level engineer who knows how to use AI joining your team.
Of course, yeah. If you wanna dip your toes into
it without having it write everything for you.
The suggestion that I've given people is use it as a a really
fancy To Do List and if anyone out there has executive function
issues, because it seems like everyone around here has ADHD.
It's really easy to queue up multiple things that you're
working on all at the same time. If your project or your company
has multiple services or multiple applications.
What I typically do day-to-day is I've got multiple things
running all at the same time and I'm not waiting for AIAI is
waiting for me because by the time I've done something, I'll
switch to a different project or a different branch or something
and then go do something. It's, it's a way of always
knowing what's next because context switching is really hard
for all of us. Sometimes if you get, I feel
like this is a difference between an IC and A and
management, but I get bugged all day long, every day, pulled into
meetings that I wasn't aware of, etcetera.
And so I, I don't really have the time to deal with the 10 or
15 minutes it takes me to get up to speed.
But if there's a markdown file waiting for me saying, oh, hey,
you did this, this and this, we were going to work on that.
How about you do this next? Great.
I it's like a built in. It's a built in somebody behind
me, behind me coaching me on what to do next.
And I that's a great way. If you don't want to like just
go all in with it coding for you, great.
Don't use it like that. Use it as a coach.
Use it as someone who's like cheerleading you on.
Yeah, rubber duck with it. Like make a plan like you don't
have to let it write your code. I mean for like real redundant
here. I'm going to put this on hold
for a second and we'll be right back in just like 30 seconds.
Hey, I'm Kimberly with Elixir mentor Jacob recently added
recruitment services to help companies hire pre bedded Elixir
devs. I help with company outreach, so
if you're hiring, I may be reaching out, or you can connect
with us anytime at elixirmentor.com.
Coding Elixir with AI or building AI into your Elixir
apps? The Elixir AI Collective Discord
is your community for both. Link in the description below
back on. All right.
So to go back to like a management position, like I feel
like once you're like not just an engineer at a company, you're
not just focusing like feature at a time, right?
You're bouncing all over the place.
So it's probably nice to have the ability to just have
markdown to do lists like, hey, this is where you left off and
welcome back. Absolutely, especially.
If you're working close with product and you know that
there's a feature that's scheduled on the road map for
six months out and you really need to start thinking about the
changes that I'm making today, Are they helping or hurting what
I know is coming six months down the road?
That's really easy to just take that nugget out of your head and
shove it into an agent, say remember this.
And every time we talk about a feature that is anywhere related
to the thing that's in six months, remind me or say, oh,
you know, if we added this column here right now, we would
be collecting this data for the future.
So I feel like there's just so much low hanging fruit here that
for people who are really against AI writing code, great,
don't use it for that. But it is a great personal
assistant. You know, our, our team at
Collects, we, we lean pretty heavy into AI, but like into our
workflow. One of my favorite things that
we've done is we created a skill called Neckbeard and you, you
know, you can call hey, neckbeard, can you just go go
through the code and tell me exactly how like our
authentication flow works And it will go through it.
It's not allowed to write any code.
All it does is give you a summary and then it will
footnote actual modules that it's referencing at the bottom.
And dude, it saved me so much. I could have just written a
feature and I like forgot something, but someone's asking
me a question about it. And I just, you know, call up
our neckbeard and he, he'll, he'll run through it and give me
a summary of how everything works.
And it's like the most powerful thing that I use.
And it's not even writing code. It's just evaluating code for
me. I, I hear things like that and
then I start thinking like the people who are so against AI
could be using that stuff and it's just going to accelerate
your workflow. And I'm not as big on AI as I
would say most of our team is. I just know that there is a new
tool. It's not going away.
It's the worst that could possibly be today.
It's only going to get better. So you might as well learn it
because I think he will fall behind if you don't.
Like a good example, like especially like in a role like
yours where you're, you're jumping around, you get invited
to meetings last minute. Like if like just for example, I
just like implemented like an OTP feature, right?
And I had a question come in on Discord as the front end team is
implementing it. They're like, hey, what actions
require OTP? Like what mutations are we going
to hit? And like, off the top of my
head, I think I can remember most of them, but with looking
through and verifying in the code base could take me like an
hour, right? But asking neckbeard, I get an
answer in like a minute. And I have every single thing
that I just added an OTP requirement into the middleware
for. Like, dude, that's, that's huge.
It's big, yeah. It's real big that I I want to
borrow that idea or steal it. It's amazing for it, yeah.
I'll, I am more than happy to share this the, the markdown
with you and you can edit it for your, your own project.
But yeah, it's, it's the it's my favorite feature of using
Claude. Just explain this feature to me.
How does it work? And then like well we have one
that we have like a coverage 1 and I can like be on a current
feature and be like hey, where is my test coverage low on this?
How can I improve it? And it's only skill is to find
low test coverage with. Testing.
I feel like it's amazing because you can say, what edge cases am
I not thinking about? And it's like, oh, here's like
40 different things that you obviously didn't think about and
you're like, oh, of course we would have run into these things
in the future. Go write test for all of that
stuff. Oh, hey, now we're doing TDD and
stuff broke because we weren't thinking about that when we
wrote the feature. Let's go fix that and be a
little bit more defensive in our code.
Great. You just helped yourself by
asking a question real easy. Yeah, Yeah, It's so awesome.
I used to hate writing tests, and now it's, like, enjoyable.
Like, I feel like writing tests, though, it requires almost more
attention to detail than the actual code itself now, because
you don't want just garbage tests going in that just pass.
Because I have, you know, every once in a while, I feel like in
the last couple months, not so much.
But when Claude just writes a test to pass and you're like,
that's not going to fly. It will absolutely be lazy with
testing or it will mock stuff where it's like just returning
OK's no matter what happens and you're like bro, don't do that.
So sometimes you have to catch it on itself.
Yeah, 100%. I, I kind of want to go back to
your transition from past systems to like your own, you
know, your own bare metal server.
What was that transition like? Did you have downtime?
How did you handle it? How much time did you have from
getting off of Gig Elixir to your own your own box?
Oh boy, I had one week. Less than one week, actually.
I had five working days. We ran into a problem.
It was on a Monday. That was the third Monday in a
row that we were having issues with the amount of traffic.
It just so happens that Monday around lunchtime is when the
state of New Mexico uses our product and we fell over.
And when our upper management are fielding phone calls from
the people in charge of education in New Mexico.
We have to fix it. We can't, we can't deal with
going down or anything like that.
And so it was a week and that was a very, that was a very
challenging week. It was myself and another
developer who acted as as my second and we stayed up all
hours and just moved everything. And at the time, I think we're
at 3 or 400,000 users and 0 downtime.
The only downtime is if you got caught potentially in a
situation where DNS wasn't being authoritative.
And we had pointed it at two things and one thing.
And it's like maybe if you had like a really slow ISP that go
really catch things, you would have downtime.
But realistically we had 0 downtime.
That's impressive. That was because that's one of
those. Situations.
It'd be a lot easier just to down things and you know, of
course sold it back up, you know?
And I would, you know, I was given permission to do that.
And I wanted to take it as a personal challenge to say, how
can I take services from two different past companies and put
it all into Google without any downtime and just a hard cut
over. And I don't know, maybe I like
pain, but it was it was fun. I'm sure people are like, why
would you want to like just 100 hour we can stay.
I I had a lot of fun. That's good.
I don't. Think I don't think anyone would
do what we do if you weren't having some sort of fun.
Right. Well, I think.
Being a developer in general is incredibly hard and challenging
and frustrating. If you're not a team of one, if
you don't have your own product, if you're inside of an
organization, everything is your fault, no matter if it's good or
bad. Tensions run high, things are
needed last minute or three weeks ago, and it's it's hard.
But if you weren't having fun, and if you didn't enjoy that
challenge, you'd go do something else.
What I think is like a good learning experience, like before
I worked at collects, I didn't really experience like a large
user base, you know, tons of queries per second, things like
that. And you, you kind of like when
you're in like a start up mode, you only have a couple 1000
users, traffic's not high. Like, you know what I mean?
And you always think like, I'm gonna get to a point where all
these growing pains are gone and you forget that there's like a
whole set of new problems you have once you start having to
scale. And I feel like like level all
like are you, can we talk about the most recent like growing
pain that you've had that you're working on right now?
And it's another like good experience of like kind of
vendor lock in problem, right? You want to explain the issue?
Yeah, absolutely. On Monday, another New Mexico
onboarding and we we fixed our systems last year.
So we're extremely fast. But that unlocked a new problem.
A new challenger has arrived and no, we got rate limited at auth
0. So when that happens, your users
don't see your login screen, they see AI don't I don't
remember. It was so cheeky and cheesy.
It was like, oh, this is a popular application wait a
minute till you can get in or something.
And yeah, we immediately heard from educators like, hey, we're
we're reaching this error screen and it's like, Oh, no, like
that's something completely out of our control.
And that was Monday, so. That's a great way to start.
The week, what's what's, what's the plan?
What do we do? We immediately spoke to customer
support and I mean, we pay all sorts of money to all zero.
Anyone else there knows that they are extremely expensive and
we pay for enterprise and enterprise support.
And luckily we were able to get some answers pretty quickly, but
the answers were ambiguous at best.
Well, here's why it happened. OK, how are we going to fix it?
Oh, well. But here's how.
This is why it happened. Great.
How are we going to fix it? Are you going to raise our
rates, the limits? What are we doing here?
And we have yet to get a good answer out of them.
The biggest problem that we have is with Albuquerque Public
School system owns public IP space and all of their students
come out of like 4 IP addresses all at once, Dang, across the
entire district. And so it's a lot of users
coming out of a single IP. That's, that's an attack
according to most systems. When you have 10s of thousands
of people just crushing on a single IP address, like, yeah, I
I would probably turn you off too.
But that's just the way that our traffic is very bursty.
And so, I mean, this is a real live issue that as soon as I'm
off this call, I'm going to go back and fix.
But we decided to completely RIP out all zero and put in a
different all system in a matter of days because it's Friday.
It's got to work by the end of the day and to make sure that we
can be ready for Monday and we don't have 3 or 400,000 users
anymore. We've got over 1,000,000 in our
database. It's a lot to do.
That is how do you so you're switching to a new service and
have they don't rate limit or they are willing to make a
custom solution for your circumstance.
So how's that working? Yep, we are moving to Fusion
Auth and I've used Fusion Auth in the past.
The nice thing about their product is you can self host it
OK so you can give it all the hardware, You can give it all
the hardware you want. So back to, you know, no vendor
lock in your yourself in your solution exactly.
We own the database, we own the software.
It's just there, we control what how much that's load balanced.
It's right now being deployed in the same mechanism as our Elixir
server. So there's a template and a
managed incense group. It also has auto scaling turned
on. So I'm just not worried at all
about of of thundering herd problem because we'll auto scale
on that. That is a big part of vendor
lock in. And so some people would say,
but didn't you just move from one vendor to another?
Yeah, sure. But for the price, could we
handle what we need to handle? Not at all.
We we aren't just doing username and password here.
We have lots of people with username and password or e-mail
and password. We also have a ton of people
coming in through OIDC, SAML. We don't use it today, but skim
might be a thing if you, we say students and educators,
districts, states, but really, if you squint, they're
enterprises. They use enterprise software.
They expect an enterprise connection, they expect
directory sync. There's a lot of things that we
do that are actually enterprise everything.
And although, yeah, we could write it ourselves, it's just
not worth it because we have so many other really cool features
that help students out that we want to get to.
Yeah, yeah, if I feel like, like that problem alone is just
something you'd wouldn't even worry about and then all of a
sudden it's just knocking on your door and it's a huge
emergency because no one can get in your system, of course, from
that IP at least. I never expected it.
And I think on on one of the calls that we were on with on
zero, my CEO said something and it's really true.
Well, we should have known the rate limit.
And it's like, oh, you're absolutely right, that's my
fault. We should have known and I did
it so. And it's really low.
By the way, default rate limit is 500 per minute per IP
address. Which is it's not going.
To work at all for us. Right.
Well, how many students are in a school district?
Couple thousand at least. Albuquerque has 25,000 there.
You go for for for 888 through 12th or something like that.
They've got a ton. There's a ton of people in that
district. Wow.
Yeah, 500 won't do it. Yeah, No, we were.
They told us that we had 60,000 requests in a 20 minute window.
So some of those were retries because they couldn't get in.
But it's like, yeah, we had 10s of thousands of people trying to
get in and they were essentially told no.
And it, I can't say it wasn't our fault because of what my CEO
said. It makes total sense.
I, I should have known. And I feel, I feel really bad
about that because that's something that I would have
highlighted when we moved to Google last year.
I haven't seen for months. Like, I'm really concerned about
the performance of gig Elixir. I'm really concerned about the
performance of fly. Can we please move to something
that is a little bit more scalable for usage?
And essentially I was, I was told not by the CEO as someone
else, it's not a problem until it's a problem there.
That's a conservative way of looking at it.
It's not a problem until it's a problem.
But when it's a problem, it's a really bad problem.
When you're dealing with unavailability, that's the worst
type of problem you can have. So I felt like I was seeing the
future with our hardware, but I never once thought we're going
to get rate limited by our auth provider, never thought about
it. I mean, I think this is just a
good example for everyone. Like especially like junior
level people that are learning, like you're never going to see
it all, know it all until you experience it.
Like a lot of times you learn your best lessons through a
really hard, hard way of learning.
You know, like, this is a huge pain point for you, right?
Your whole week, whatever you had planned for this week got
thrown out, right? And this is your main problem
now and you had to solve it immediately.
Absolutely. And it being in the management
side of things, it's not just the code level, it's getting on
sales calls, it's walking the contract through, making sure
that we're OK with terms, sending our red line version
back. Like there's so many steps to
things that if you're just an IC, it just happens magically.
And it's like, there's a lot of stuff that we do to just get
prepared to sign a contract, then we sign it and we still
have to do all of the work. So there's it's a good point of
like, sometimes you don't know until you don't know.
I'm sure a lot of people out there would say like, well,
don't beat yourself up about it. You didn't know.
It's like, yeah, you're right, but I wish I would have known.
And that's just me trying to be as best as I can be at what I
do, knowing where the pitfalls or the things that could go
wrong exist. But yeah, fusion auth it fusion
auth. The integration so far has been
going pretty smooth. I'm a little tired tonight.
I was up till Oh, you look great.
I can't. Even tell thank you.
The I fake it really well. I think the the the the biggest
challenge of just between the two, if anyone cares about the
tech is all zero makes some really interesting choices where
they're erring on the side of ease of integration.
So they do things like their view SDK as storing tokens and
local storage. It's not the most secure thing
there there there are some risks involved.
They're low, but there are risks there.
Fusion auth is like, well, why would you even take that risk?
Don't do that. Store them as session HTTP only
cookies. Then you don't have to worry
about JavaScript having access to your token.
OK, but then that necessitates a rewrite of how our API off flow
works in multiple stages. And it's it's one of those like
it's not it's not as easy as like, oh, we'll just turn this
one off and this one on and away you go.
There's there's some hard things that you have to contend with.
And then later today I will we already we already have the
export of users out of all serial, but when we import them
got it. Make sure that the B crypt hash
goes incorrectly, gets converted correctly.
There's a lot of, there's a lot of, there's a lot of minutia
that you got to make sure it goes correctly in order for
again, I, I want to hit deploy and all of our services roughly
at the same time and I don't want anyone to experience
downtime. That is a personal goal.
How do you do you have a staging environment set up where you can
at least test this? Yeah, that's good.
Yep, we will. We will test it very thoroughly
before it goes to. Production, I'm always like, I
feel like it's becoming more and more common where people push
straight to prod these days. And I just think like, that's,
that's brave. It it's, it's brave, but I also
feel like I'm dating myself that that's, that's what we used to
do, right? You would live edit over an FTP
connection. I mean, I'm not gonna lie, I
have, I have changed code on the fly in an IAX instance, of
course. But, but you know, I don't like
to do that stuff. It's not ideal, but I like to be
able to. Of course, yeah, we our code, I
mean we deal with education and students and our, our PII is
very valuable because we are bound by Copa and FERPA sock
two, we have a lot of regulatory bodies who oversee what we do
and don't do. So we can't be as cavalier as
other people. Everything is security around
here and we take it really seriously.
But there are definitely times where it's like, man, I wish I
could just pop into a a server and just edit this file because
like now I gotta wait for the whole CICD thing to run and all
that. That was, that's a good point.
Actually. That was one of the biggest
gains from moving from the multiple passes to what we're
doing ourselves is we used to have, because of the way that
Gigalixer used to run where it didn't use releases.
It's different now, but when it was in its build pack era, it
would take close to 27 minutes or so to from merge to
deployment. And that is if you have a hot
fix that needs to go out. That is an excruciatingly long
time. Yes, it is.
We're down to 5 or 7 minutes now and when we go to bare metal I'd
like to get that down to 30 seconds.
Does that include like your CI like running tests and
everything 5 to 7 minutes? Oh, OK, no.
Well, so CI will run on the PR and once it's good when you
merge, it doesn't rerun the thing that it just ran.
We don't we don't do that. I feel like that's a waste.
So when you merge. Into main I actually I should
change that in my processes. So many people do it because I
feel like someone wrote a blog post, you know, 10 years ago
that that's what you're supposed to do and everyone does it.
But it's like, but you just ran the stuff.
It's. Really annoying the way to the
yeah. Of course, yeah, no, get rid of
it. The only thing that it should do
is if you're. If you're on ACD system, it's
going to deploy to staging or production of any on what branch
you've just merged into the. Way it goes, Yep, I like, I like
that. I'm trying to think how long my
tanks to deploy right now, my kill switch product, at least I
bet it's about. It's probably like 10 minutes,
honestly. Yeah, yeah.
I want to get it down to 30 seconds.
Our view applications take about 5 to 8 seconds to deploy, which
is great because it's just moving some files.
But I want to get I want to get our elixir down to that we are
I've I haven't decided if level all is going to open source this
or I am personally, but I'm working on a a deployment
mechanism for bare metal or bare VM.
Have you written any like blog posts about it or anything?
I haven't. I am so terrible with like
personal brand stuff. I'm just I'm bad at it.
I should start a blog and write about it, but the software
essentially is a, it's a, it's a server agent set up where the
agents are running on your deployment targets.
So kind of think about like Ansible or Kamal or something.
But there's instead of having just a a singular server that's
just doing things, the server agent are talking to each other
over Erlang. It moves files that are
deployable over Erlang and the agent is declarative so the
server will say this is what you need to do.
The agent is like oh OK I'm going to change from this
version to this version. I'm stuck there.
The agent will do its own stuff to deploy so that if the
connection is severed between server and agent, it doesn't
matter. It can continue to run and
typically people would reach for Go for this, but I want to do it
in Elixir because I want to take advantage of Erlang talking to
itself. So there's no APIs to write,
you're just remote calling functions.
The files will happen over Erlang, very similar to how
flame works on fly. Yeah, I that's that's when we
when CICD will run the CI portion will run all the tests
and create the the packaged build.
Once it hits CD, that server will take over.
Tell the agent, agents like, let me download this, spin it up,
change the system D file around it.
Away you go. It should be about. 30 seconds.
That's awesome. Very cool.
Yeah, I want this. You got to get this this open
source project out. Absolutely I will.
I'll let you know when it's ready.
It's like that'll be my push to get off docker.
I, I think once I and I thought really long and hard about just
doing it for ourselves because there's definitely some
particulars that I think we will make.
And so I actually had a discussion a really, it was like
a multi day discussion with AI on like, what are the merits of
doing this internally versus open source?
And a part of that is like, well, do you feel it's asking me
this? Do you feel like you've consumed
more than produced for open source?
Yeah, that's my entire career. I've edited a couple of things
here and there for open source, but I feel like they're like
small peanuts compared to how much I've actually benefited
from open source. It's ridiculous.
It's what I've done is a rounding error.
Yeah, my only open source. Now.
Tutorials. Exactly.
So I want to, I want to give back and so I hope the AI helped
me think about like where to be opinionated and where to allow
for customization hooks or plug insurance, but still have you
know what, if this doesn't work for you, that's fine.
It's not going to be for everyone.
And that's basically what Kamal is.
You, you want a Ruby based Docker deployment system, use
it. It's amazing, it's great, works.
Great. What do you, how do you handle
like your database when you're just, you know, using mixed
release? Do you just have have like
Postgres installed on the server and it's just there or how do
you how do you handle that? Are you asking this like what
happens when you go to? 'Cause like I'm thinking about
like I spin up a Postgres instant in a Docker container,
right? Yeah.
So how are how do you handle that on like a bare metal
deployment? Run it yourself and Postgres.
Postgres is one of those things that people think it's big and
scary and you're going to lose all of your data if you run it
yourself. No, no, no you won't.
It can happen. Disclaimer, data loss is a
thing, but no, you won't. Postgres is unbelievably stable.
Modern hardware is unbelievably stable.
As long as you're backing up, I just really wouldn't worry about
it. I feel like most people think
their application or their project is it's got to have 900
nines of uptime. No it doesn't.
Yeah. Look how many times you know
Azure or Google or Amazon has problems and then half the
Internet doesn't work anymore. Stuff happens and it happens at
least once or twice a year. If they can go down and it's OK,
then you can go down and it's OK.
That's not your goal obviously. But for Postgres, I really stick
to as simple as possible. App get install tenant to a
version. And then there's a couple of
tweaks that I'll do If you Google Postgres configuration
performance tweaks or whatever, there's a lot of blog articles
on like how to change the config from whatever bases to something
that's a little bit more reasonable taking into
consideration how much memory and cores the machine you're
running on has. And then the biggest tool out
there for making sure that you have backups and if you want to
get fancy a hot spare things like that is Oh my goodness, I'm
blinking on it. PG backrest I think is what it
is. That's that's something that I'm
I'm blinking on the names, completely forgot the name of
the people that that run that. But it's a great system where
you you have a wall file that's continuously there.
You can do nightly backups to multiple S3 buckets or Google
buckets or whatever just to make sure that you've got multiple
backups like it. It you just, I don't know, you
just don't worry about it as. Much I I should like?
Things are less fragile than you realize.
I think that's the biggest thing.
Things are less fragile. Well, I feel like databases are
just robust systems too. Like they're impressive pieces
of code. Absolutely.
Think about all the garbage queries that are getting written
by AI and the database is like here you go.
Imagine writing like a database right?
And you pretty much just have to accept everything and not break.
Yep. That's amazing.
It is. It is.
My database backup is. I have like a like I do like a a
postgres dump and then I push it up to S3 all through just like a
bash script. But I could probably clean that
up and then I have a Quran job. Just run it every day.
Yeah, for the most part it's good enough.
If you need point in time recovery wall files the way to
do it, PG backrest will be able to help you.
It's real easy to set up there, and when you go bare metal you
don't even have to separate the database from where the
application runs. Put it on the same server.
Talk about latency overhead. Just none.
Just there. That's actually pretty cool.
I need to do this. I need your open source project
too. We do too.
It's our sock. Our Sock 2 window closes mid
June. I think that's when we're going
to start migrating into bare metal, so it'll be between now
and then that it will get written.
Awesome. Very cool.
That's exciting. I like that.
All right, let's see. All right.
I kind of want to talk about like what you think like like
founder mode thinking means to you as like being a leader
inside of a company versus just being a developer.
Like what I know we've kind of touched on this a little bit
with like the Oauth problems and rate limiting, like things that
you have to think about that other engineers don't until
they're told to think about it. Like, was that a hard shift for
you? No.
And that's just because of my
career trajectory I guess. This is my first W2 job in 17
years. OK.
Yeah, I was a consultant before, had my own business.
So there's a lot of, there's a lot that I bring to my W2
position that maybe a normal career trajectory person
wouldn't just because when you are the, when you're in the hot
seat running your own company and you have, I mean, I've run
plenty of companies, but when people are dependent on you,
like you can't just say, oh, well, it'll happen unless you do
it. So there's, there's things that
I just, I think I handle myself completely differently than
other people might. Do you think it's a personality
trait or is it a learnable trait to be more in management leader
position where you you just absorb?
You know what's the word I'm looking for?
I don't know. I hate when I go blank live on a
stream, but like you kind of just absorb whatever job
requirements you have to in that moment, right?
And I, I've met some people that are just really good engineers.
You give them a task, they get it done, but then they also
can't like high level think and like, you know, think six months
from now as well. Like that's a skill in itself or
just feature planning, product planning.
A lot of people lack those skill sets I feel like.
I want to say that it's probably teachable, but there's a part of
me that thinks it it might be teachable, but you might not be
good at it because you might not want to do it, period.
I am the most motivated person ever when someone tells me no or
you're not good enough. You can't do that.
And I'm now on a warpath to prove you wrong.
That's personality from when I was a little kid.
And I feel like, I feel like that is a, there's a lot of
people that are not like that. You can't do that, OK.
And then life goes on and they don't care.
And sometimes I, I, I have envy for that because it's like, how
can you not care? Oh my gosh, I've got to go have
an Internet fight now 'cause someone said I was wrong.
But that's that innate sense of fighting for what you believe
in, fighting for yourself. Proving people wrong is why I
feel like I'm successful when I put a, a view upon everything
that's on the table, whether it's immediate need six months,
a year down the road. I'm looking at all things from
all angles, all the exact same time.
And I'm proud and I'm like, I'm proud that I can do it.
I pride myself on the ability to do it, that it's like, no, I
have visibility on everything. And what's interesting is that
wasn't just now that I transitioned into this role.
That was day one three years ago.
I just as just as a normal. I see.
That's why I'm sitting where I am now though.
Like when people can trust you and rely on you and when you say
don't worry, I got it. And you've proven yourself
multiple times that you can actually deliver and bring
receipts, then yeah, you are afforded opportunities in your
career path that otherwise wouldn't be there.
And I think that. I want to say that there's a lot
of people out there that may say, well, I'm just a developer,
I'm just gonna stay in my lane, don't do that.
Yeah, I think, I mean, I think it's really important.
Like I'm thinking back like I, I was late to the game in the
software world. Like I didn't start software
development until I was 30. So like I spent like 10 years of
my life in manufacturing and I was, I was an engineer, like I
was building automated systems. I was doing like ladder logic,
you know, but drag and drop things.
Like I didn't really understand software development or anything
above it. So I was self-taught and I
learned iOS development first. Like Swift was my first language
and I didn't know that much, but I got, you know, I was like very
motivated and I got my foot in the door at a startup and I was
just willing to, I think like I was a little naive to like,
like, you know, the Dunning Kruger curve.
I was at the top of Mount stupid.
But I think that benefited me at the beginning because I was
willing to jump in on any problem and I would solve it and
get it done. And it like accelerated, like I
got promoted very fast within this company because I would
just take responsibility and ownership over anything that
fell on the floor was handed to me, you know, And I think that
helped me. I mean, I learned the hard way,
like I was very naive with security.
My first encounter was our app had a man in the middle attack
and I had no idea what that was. And I learned very quickly what
it was and then how to fix it. And then I kind of got obsessed
with being better and like, I hated the fact that someone got
like one up on me and, you know, could circumvent my security on
my app. And then I just powered my drive
to not let that happen again. That, I believe is the perfect
way of describing founder mode mentality.
Is that total ownership of the problem?
It's it's I see it. What am I going to do about it?
I want to get better so it doesn't happen again.
I want to understand it so that I know what I'm doing in the 1st
place. And for people who are at the
top of Mount Stupid, founder mode mentality is going off the
other side and learning all the things that you don't currently
know to get. Better.
Yeah, Yep, yeah, yeah. I think there's like there's
definitely a benefit to the whole the Dunning Kruger curve,
right? Because you get to a point like
when you first start building something and you're like, Dang,
I know it all. Like I can build an app, I can,
I can build a REST API, you know, I can, I can do CRUD
operations. And you think that's it, right.
And then you start having to learn how the production side of
things works. How does deployment work?
How does security work? How do you secure things and and
then life goes on and you just keep realizing how little you
know and then it humbles you a little bit, but you still have
all the the scars from learning all those things.
For a long time, I always worried about what if I don't
understand what I don't know? What if I never come on the
other side of Mount Stupid, as you put it.
And I think there's a sense of freedom when you when you
finally understand that no, imposter syndrome's never going
away, bro. No, it's not.
You're just gonna feel it. You just have to embrace it, and
in fact, if you take yourself less seriously, then it's OK to
be wrong. The worst part to be the worst
part of that chart is when you are confidently incorrect and
you, you just know better and, and no one's gonna tell you
otherwise. And I feel like that's when we
were talking about AI and super senior people.
It's like that's them. They just don't know because
they're unwilling to learn. And for, for founders, no,
you're gonna be wrong. And that's a part of it.
I tell my kids all the time, like, oh, oh, and they, they'll
say, well, I don't know how to do this.
I'm frustrated. Great.
You don't know how to do it. And that's a part of learning.
Understand that you don't know. Understand you're going to get
the problem wrong. Nobody gets everything right
100% of the time. That's an anomaly.
It doesn't happen. So be OK with the ebb and flow
of your own success. You're gonna fail.
And that's a part of life. That's a part of your career.
It's part of everything, especially if you're doing your
own thing. You're, you're, you're a true
founder and you're your own company or you're leading A-Team
or if you were a junior IC, you're going to get it wrong.
And that's OK. It's part of the problem.
It's part of the. Process.
Yeah, I, I was part of a, a, a pretty funny conversation at my
gym yesterday, actually. And one of the one of the
people, like young, young girl in college, right, complaining
about, you know, exams and how hard school is.
And then, you know, one of the one of the older people that
runs a company and all of that goes, oh, just wait till you
graduate. It only gets harder.
Like, dude, it's never over like you're, you continue learning.
You'll have to learn more. You have to learn at a higher
level and just get ready. Strap in for the ride.
Level all as a company I feel like is walking on to the set of
suits where everyone around you is like Ivy League educated,
unbelievably smart. Everyone is there for the
mission ready to help America with post high school choices.
And then there's me, someone who dropped out of college and
became a developer and I always felt the sense of like, man, it
feels weird to be around all of education all the time and, and
whatever. So it's interesting that you put
it that way from the lady from your gem, because it's like,
sometimes you just have to know, yeah, after college it gets
harder. Or when you're in your adult
life, it's just harder. Since then I went and, and while
I was working at level all, I finished my undergrad and got an
MBA. So I I fixed that problem
sometimes. Do you?
Feel like you needed to do that or was that just like personal
con? Yeah, I was going to say like.
It's a it's a personal thing and I feel like I feel like that's
my personality is like I, I need to, I got to collect them all.
I've got to accomplish every possible thing that I want.
Yeah. So it was really important that
I, I don't want to just walk around the office even though
we're remote and be like, yeah, drop out.
No, not when people are coming from Harvard and Stanford and
stuff. It's like, no, I'm, I'm going to
go do my thing. And I really wanted my MBA to
say it wasn't enough just to get the undergrad.
I wanted the full thing. Yeah, I always think like
hindsight, I like, I tell myself this all the time.
Like I probably wouldn't have gone to school, but then at the
same time, maybe I would feel like you and you're like, well,
I need to accomplish that. Like I do have a piece of paper
that says I'm a I'm an engineer. That's pretty cool, you know, So
I don't know, like, am I using today?
And maybe it's a lot, maybe it's not.
I feel like the math side of things like calculus and that
stuff sneaks up randomly and you need to know how to do it right.
But it's not right. It's definitely not a daily
problem I have as a software engineer.
The NBA program is really interesting.
Because there's a whole side of HR.
And laws around that, that I was just unaware of.
So that was a really interesting learning experience.
But the for the most part, I've run multiple businesses.
I've done accounting and taxes and books since 2009 for
multiple of them. That on the job experience
obviously is a lot more valuable than what you'll learn in a
class when you're actually doing it and understanding PNLS and
all that stuff. But I I talking about.
Founder mode. It's like.
Yeah, owning the problem. If you feel like an imposter.
Because you're working with a bunch of wicked smart people
with college degrees, and you work at a company who's, like,
one of the focus of the software is to get kids into college,
like, and you don't have any like, come on, let's fix that
problem. So I did.
Yeah. Fair.
I like that. That's cool.
Yeah. An NBA could probably benefit me
as well, just there's a lot of gaps in my knowledge on the
business side of things and it's just school of Hard Knocks.
Currently it's and then when you look for.
Help out there, I feel like when you're on.
YouTube looking for a business help.
You never know who's trying to sell you something.
And so I think that's the biggest problem right now.
Everyone wants to sell you. Like I don't coaching like yeah,
well, why are you selling coaching?
Like are you literally just a coach?
Like is the only business you've built your coaching business?
Because they're not. I'm not interested.
Yep. And I feel like.
Yes, what? What gives?
You the qualifications to be. A business coach.
Where are all of your successful businesses?
It's like people who are Uber successful in business aren't
coaches, they're not businesses. You want to just be their friend
and take him out for a drink and.
Talk to him, you know. Yep.
Yeah. I always find it funny when
like. I get a like a.
LinkedIn request and it's like a 20 something year old kid
selling coaching services. I'm like, bro, good luck, all
right? I respect the hustle, but wrong
audience, Yeah. Like it's not, it's like it's
like if you're hiring a personal.
Trainer, are you going to hire someone that's like your same
fitness level or worse? Probably not.
You're going to want someone that you're like, oh, I'd like
to look like them. Absolutely.
I always laugh at the Super huge NFL coaches.
It's like the least you could do is eat right and get in the gym
with your with your team. For real.
Like dude, come on. I have a big problem with that
and I I feel like you see a lot, especially in like personal
fitness, you see a lot of like coaches that don't look like
they, they follow what they preach, of course, which.
Kind of leads back. To.
The importance like I. Like you got your degree right
because you felt like you need to practice what you preach like
at level all. That's kind of that's the point
exactly. With my with my own children,
college is really important. Daddy works at a company that
helps people get into college, but Daddy doesn't have a degree.
Like, yeah, yeah, do that. Yeah.
Fair. Well, you could.
You could. Yeah, but I could I?
I get why you don't want to lead by.
Example right? I think that's what makes.
Really good. Leaders inside of technology
companies is the people who will get into the trenches with their
developers. Lead by example, be be a player,
coach, get in there, get get your hands dirty.
Because just orchestrating from afar is a really good way to
isolate yourself. Not not wanting or willing to do
the work that's required to get something done.
It just causes a lack of respect for you.
If you're telling someone to do something that you won't do
yourself, there's a problem. Yeah.
Like at Collects, our CTO.
He's always in the trenches with us.
He like, I'll wake up to messages from, you know, 3:00
AM, like, like, so you know, he's just always there and it,
it, it helps a lot. I feel like it makes you respect
people at a a new level, you know, of course, absolutely
like. You're engineers, you're up till
3:00 AM. Fixing a problem, right?
Like probably every day this week because it's a pretty big
problem and everyone sees that, right?
So it's good. I, I I.
Am blessed to have such. An incredible team of people to
work with. It's it's fun.
It's I don't not not every not every company or not every
engineering team has this and we have a really good thing going
because we have incredible world cast people.
Yeah, that's awesome. I like when you like the people
you. Work with and you feel like you
like I like I personally, I work with people that are way smarter
than me at least like that's. My.
Opinion right, And that's the room I want to be in.
I want to be in a room of people I feel like I'm learning from
constantly and that's a cool, that's a good team to be on.
So I know just probably in the room.
Find a different room. You're in the wrong.
Room, right? I feel like this might.
Be a difficult question for you right now just because you're in
the midst of a like a, a current fire.
But like, what do you foresee being like your next big hurdle
or issue that you're going to have to deal with and you want
to deal with it in a like a clean manner.
Like you're growing, you're a growing company.
What's the next like hurdle that you kind of see coming?
The biggest one that we're going to deal with.
Is. Our code base, just how it was
formed over the last four years, decisions that were made.
The product that we have today is nothing like the product that
was started four years ago. But foundationally we can only
make so many changes here or there because it was envisioned
as a different product and the product a year from now is even
way different than today. So we are having to do massive
rewrites of the core functionality of what we do and
doing that in a way that it's not downtime as much as it's
like. Resetting.
Progression of students progress through our application.
We don't want to have to. Say, yeah, we moved away from
whatever we're doing before all the stuff you did before it's
all gone. You can start again.
And it's like, that's not not ideal.
So that's going to be a very hard problem.
Is. You're talking about shifting
tables and columns around, adding, removing.
But you still have to restore the old state of things at at in
some way. Exactly.
At least there's a. Benefit that are the core
functionality of what we do. If you start and you start with
us in 6th grade, you're out at 12.
Or if if you start with us as a college student, you're out in
4. There's, there's some, there's
some niceties there because people.
Typically have a time. Frame in which they'll use the
product and then hopefully you're successful with our help
where there's like a bounding box of like, well, if we just
take care of this time frame, we can put everyone else in a
different box completely, but make sure that we have
compatibility for those folks. I see.
And then it's kind of like you only have to maintain it.
For a certain amount of year window too, because then all the
all the old. Stuff.
Moves on, right? People move on with their life.
They graduate, they stop using your software.
Exactly. Yeah.
So I think that I think that's a blessing.
There is that we we have time boxes, but definitely the
biggest problem is just looking for that word, but couldn't come
to it. Yeah.
The other is is another vendor lock thing.
Are the way that our the the content that teaches.
People how to do things and. Our software is controlled by
Sanity. Sanity is a.
It's basically a data lake with a.
React front end you want. To think about it as like an
enterprise version of WordPress in a way that that that's
holding us back a little bit. You can do infinite amount of
customizations inside of sanity, but the problem is, is that we
don't own our content. It gets synced to us.
And because we don't fully own that we everything we do, you
have to change inside of sanity and our product.
We have to make sure that that never breaks.
We have to make sure that as we progress forward with whatever
we're making from product choices that we're always
including sanity. And I think the next big hurdle,
which won't be any time now, I know probably next year, is how
do we replicate what we're doing in Sanity but in our own
product? Or how do we make our connection
with Sanity as flexible as possible so that the way that it
works today isn't holding us back for the future.
Yeah, Fair, do you think? It's pretty common practice.
Like, you know, you start, you're.
A fresh start up. You don't know if you're going
to be around next week, right? Do you find like the more
established you become in level all the more you start building
your own systems to get out of the vendor lock situations?
Do you think like that's a pretty normal process,
Completely normal? It's so easy to say the things
that we do like don't get. Vendor locked, Build it
yourself. When you're in a business that
is successful, has a trajectory, you're not worried about the
day-to-day. There's a sense of calm because
of that. If you're in a true start up
where you where you don't sell, you don't eat, I would be taking
every possible shortcut to get the product out in time.
Don't be concerned about vendor lock because you might not be
here in six months. Then it doesn't matter.
But there's there's. A tipping.
Point like with almost anything, consider a product like segment
or mix panel. After a certain amount of time,
both of those companies will come to you.
And say, hey your MA. Us are pretty high we you should
go to enterprise because it will be cheaper well the alarm bells
should be ringing in your ears that well they.
Want to lock me under a? Multi year contract because they
know it's starting to get painful to pay these bills and
we can write our own thing. So I feel like it is a natural
progression no matter what. And even SAS companies that
cater to start-ups even know that because it's a part of
their sales cycle. They know that once the pain
starts happening, yeah, we're going to we're going to spin up
our own click house data database, throw analytics in
there and point Metabase at it. And hey, that bill just became
freer. So it happens.
Yeah, yeah. I think it's important for
people listening. Like if you're building
something new, worry less about most of this conversation, but
now it's gonna happen. Like I think it's good to.
That's the goal. Road map you you want.
To have these problems. Right, they're good.
Problems is a problem you want to have.
These are growing pains. They're the best kind, yes.
You will never get it right at the beginning and that's OK.
You want to get, you want to get the product in people's hands as
fast as possible. No matter what, you're going to
be vendor locked in certain ways just happens and the that's the
from my perspective, the hardest part is.
Getting your product in people's hands because I'm no salesman, I
just want to build things, but you sometimes have to put that
that sales cap on and and honestly for people who are.
Developers who want to do founder mode and and do
something like kill switch, it's hard to, it's hard to have that
point be understated at all. Just because you build something
doesn't mean people will come. Well, there are competitors out
there that are 10 times worse. Than your product may.
Have 100 times of sales because they have a marketing team
behind them. It's nothing against you.
It's that you want to build, you want to be in the code, you want
to do deployment, you want to do all the fun stuff that comes
with tech challenges. But marketing and sales and SEO
and ads and all that is a completely different realm that
people are really good at. And sometimes you might need to
bring someone on either a part of your team or pay the money
because as the adage is, it takes money to make money. 100%
demand driven advertising is a big, big deal.
Take true classic T-shirts as an example.
They're. Selling a pack of T.
Shirts for what, 30 bucks? Something like that?
40 bucks. At one time they were spending
$70 in order to make that sell. Why would you ever?
Why would you ever do that? Well, because they need to.
Get their pipeline. Running and then it started to
go down. Now they're only at $10.00 or $5
or whatever it is, but it's a multi $100 million company
selling T-shirts and they almost exclusively had that success
because of Facebook advertising. They're a behemoth now, but they
had to spend the money to get their name out there.
And then there's a tipping point that happens.
So it's really hard. I see a lot of people who will
build in public and launch their product and then six months or
12 months go by and they have their oh shucks, we're going to
spend it down probably because they have 4 1/2 users.
And it's like, well, it's because you cannot rely just on
a personal brand unless if you are someone like DHH or
something, right? Like a hard lesson for me that I
just recently. Learned too is like, I do have
an audience, right? I have a extra mentor, but my
audience is not the people buying my product.
So I need a new audience, you know, and that was hard.
You know, you're you always hope it will get you a little further
like what you have, but it doesn't.
So you have to pivot and you have to move and it can be
demoralizing, but you still got to keep going.
Of course. See, for you this is unsolicited
device. Don't hold me to this.
But there are so many like wealth conferences or real
estate conferences that happen all the time and and people want
to go and learn about real estate or how to how to do
wealth. I could easily see you giving a
talk on a panel or something about your product.
Because if you're interested in wealth, you need to protect
yourself. If the worst thing in the world
happens to you, everyone there would immediately be like, oh,
that makes sense. My family needs to have access
to all of my stuff. If I get hit by a bus, it's
amazing. So that has nothing to do with
tech, nothing to do with technology.
It's, it's just breaking out into a different sector of
people who are actively thinking about I need to do wealth stuff,
right? And that's another one.
Like I was actually just. Writing like a a case study blog
post for someone that wants to post By the way, getting
backlinks on other websites is really good for you as well.
So I'll, I'll take the time and I'll, I'll do this stuff.
But one of the things that kind of like brought forth to me is
like my. Competitors.
Like there's a couple competitors out there that are
doing like very similar things to me, but they're not, they're
not my problem. My problem is like the
procrastination of my customers, like, 'cause no one wants to
think like, oh, I could get hit by a bus tomorrow, I could die.
I should have this set up now. Everyone's, everyone's like, oh,
I'll worry about this in like 6 months.
So they're not immediate customers, right?
You need the, you need to like get that urgency, that fire
under them. And that's the difficult part.
I just had a flood of. Ideas of like stuff.
That you could do for the procrastination stuff, because
yet. Yeah, like, that's the same
thing with life insurance. Oh, I'll take care of it next
year. No, you won't.
Not if the worst thing in the world happens.
Yeah, exactly. Yeah, I should actually just
reach. Out to people selling life
insurance and give them affiliate links yeah or people
who specialize in. Doing trusts or wills?
Yeah, same thing. Yep.
I was like I'm trying to like figure out my angle but I want
to like like. LegalZoom or other platforms
like that, like those are, those are my, that's my clientele
right there. Everyone that's thinking about
that. Absolutely, yeah.
Yeah, got a break out of the. Comfort the See, that's the.
Problem as a developer. You want to stay in your comfort
zone. It's talking with other
developers. It's building products and now,
you know, now I need an audience of non developers and no one
wants to do that. I have to talk about my product
not from a technical perspective, from like a use
case perspective, right? Yeah, it's hard.
But fun. It is fun, I mean.
Like I every problem. That I get to solve.
I think is fun, right? You just have to.
It's a new kind of problem, yeah.
All right, well, we've been talking for a.
While we can. Probably wrap this up.
Do you have any unsolicited or solicited advice to give the
viewers? Oh, let's see.
You will never know what to do at first if you never give
yourself the opportunity to fail.
Yeah, some people ignore their gut because they're too scared
to make a choice. Your gut is a powerful thing.
Listen to it. You may be wrong, and that's all
right for. For founder mode.
Mentality. You got to own the entire
problem start to finish, and ignoring it is not going to get
you where you want to go. Yeah, but I don't think it gets
much. Better than that.
One of the funny like not to continue this even longer, but
as technical founders, if you're building something and you don't
want to do something, you'll procrastinate with building more
features. You have to stop doing that.
I I have the same problem. Like sometimes I'll just want to
relax and be like, I'll work on that feature I've been wanting
to build, but really I should be reading Reddit threads and
posting on there and being like, hey, this is what I do and I got
I'm learning. Like Reddit is actually a great
platform to advertise. You just have to do it really
smoothly. Otherwise moderators are really
quick to ban you. I've been learning this lately.
Can't just sell sometimes instead of sometimes instead of
adding. Features.
Go back and work on the features and make the ones that you
already have better or curate them.
If the feature has too many options, remove them or add more
options. But just continuously adding
features and not pounding the pavement and making sales
calling people it. No amount of features is going
to help sell a product that nobody knows about.
Yeah 100%. I love it all.
Right. Well, thank you so much, Thomas.
We will let you get back to your current fire and I hope it's
completely extinguished today and you can enjoy your weekend.
I hope so too. I don't particularly.
Like fires. But I love the challenge and I
find them fun. So it is what it is.
It's a it's a good challenge and it's like one of those things
you don't think. About until it happens well, now
hopefully more people will think about it yeah rate limit I do I
do think you're. Like a very unique situation.
For like hitting rate limits like that too, right?
Because everyone's essentially going through the same proxy or
same network, so it's the worst shape of traffic you could.
Possibly have which is a. It's a fun challenge.
Yeah. Yeah.
Interesting. All right, Well, thanks a lot.
This was this. Was a lot of fun and I'll have
to have you back on. I think we can we have even more
to talk about. So absolutely I can't wait.
Thanks a lot. Yeah, everyone have a good
weekend testing.