Hello everybody, I am Shinobi technical auditor at Bitcoin
Magazine, joined by Seditid, Bitcoin Core developer
contributing to the Colonel project.
So Seditid, there's a little bit of a long tangled history to the
Colonel project. Years ago there was the Lib
consensus project. So attempting to actually take
all of the consensus rules and move them to a totally external
library that could just be called used independently on its
own like any other software library.
And that project kind of got sunsetted in favor of the
Colonel project, which as opposed to trying to become a
full library is looking at just isolating kind of the consensus
logic in the Bitcoin Damon so that you can call it
independently or like granularly bit by bit from that program
running rather than having to go through the whole validation
flow as its implemented in core. You kind of want to talk a
little bit about why all the efforts switched from the Lib
consensus project to Colonel and kind of the the logic and the
reasoning for that? Yeah, sure.
So I when it started out, the initial goal was to just isolate
a very small subset of the entire consensus logic and
specifically was only the script interpreter that was surfaced
through the original consensus library.
And some projects did integrate this, but the traction it got
was pretty limited. And this had a multitude of
reasons, but I think the main one was that it just didn't give
enough leverage to the programmers using the library
and it's feature set was also pretty limited.
And the main problem with expanding the scope of the
original library was that the code within Bitcoin Core just
wasn't suitable for that at the time.
So because everything was so very tightly coupled with
completely unrelated functionality like the GUI or
the wallet was basically impossible to surface more
validation logic, that just did one thing at a time.
And yeah, it just took a lot of time since that original, the
Bitcoin consensus release, to get the internal code into a
state where we can now finally add a bit more, let's say,
flexibility for the developers using the library and features
beyond just verifying scripts. And he was specifically asking
before where like the cuts between LIP Bitcoin consensus
and the new LIP Bitcoin kernel happens.
And the main difference there is that the LIP Bitcoin kernel
functions are stateful, so they maintain system resources that
are allocated on the hardware itself.
It has all the database functionality that a typical
Bitcoin node would have, and servicing all of that through
the original LIP Bitcoin consensus architecture was
basically impossible. And yeah, that's why the step
had to be taken to deprecate it. So it's just, you know, because
consensus entangles so many things outside of script is just
like unpredictability in terms of how trying to apply that
library, interacting with alternate implementations of
consensus critical things like the UTXO set or a block database
like that, that kind of issue. But you you can isolate the
script rules, but you can't actually guarantee that the rest
of everything that needs to be touched in the consensus process
will behave properly. Yeah, exactly.
So I guess it would have theoretically worked if you
would have been very, very careful to like patch out all
the unrelated functionality with like stop functions that you
manually copy in. But once you're doing that,
yeah, it's too much of A hassle to release, right?
You you could just take the current code wholesale if that's
what you're doing anyway, so. At that point, you're not
achieving the goal of having a known, consistent, reproducible
consensus rule sets or implementation that you.
Use exactly the The entire point is that you don't have to
autismally interfere with the code every time you want to just
verify a block, right? We just want to give this one
function that does the logic correctly and everybody can just
call it in exactly the same way and get a reproducible result.
So what what was kind of the original impetus for, you know,
Lib consensus and now the Colonel project?
I mean, Bitcoin Core is unquestionably the dominant
client on the network. Like any serious miner is either
running core or running something very tightly related
to core in terms of custom behavior or code base changes.
All the major businesses exchanges are running core.
Like, why put so much time and effort into trying to strip out
and isolate the consensus rule so they can be used somewhere
else as opposed to just continuing to focus on core as
an implementation? Yeah.
So I think initially the emphasis was purely on a
philosophical level where we don't just want Bitcoin Core to
dictate everything to the entire ecosystem and kind of prohibits
other node implementations from ever receiving the same trust
level just because they kind of gate keep the validation logic.
But even beyond that, the kernel has like significant benefits to
the project because it's really clearly demarcates what actually
is validation codes and what isn't.
And yeah, making things nice to use from the outsides will
automatically also make it nicer to use for our own codes and
allow us to also experiments more freely with it, write our
own tools and utilities with it. Yeah.
So it gives us a whole lot more flexibility and also confidence
with reading the codes and doing reviews of, yeah, sometimes
pretty complex validation changes.
So how is implemented kernel impacted or affected the work
towards multi process separation?
Like that's that's something, you know, for a similarly long
time, core as a project has kind of been trying to clean up the
code base so that things are nicely isolated.
Could potentially in future actually be separated into
completely separate programs that talk to each other as
opposed to, you know, 1 singular code base.
But it's it's been a long road, I think a lot longer than Lib
Consensus and the Colonel project going from the giant
file that Satoshi left everybody to like the much more modular
clean code base that we have today.
Yeah, definitely. I would say the time spent on
either, probably more time was spent on the multi process
project cumulatively so far. Initially the efforts that went
into separating the code for multi process were also really
useful for the kernel, but by and large that split is kind of
done now. Like there there are a few areas
where we definitely still need to separate a bit more cleanly,
for example for the indexing code, but that doesn't really
directly affect the kernel anymore.
So the kernel is already like 1 sub components of the innermost
multi process binary or like the Bitcoin nodes multi process
binary. And most of the organization
that takes place now in the kernel project is within that's
Bitcoin nodes multi process program already.
So there's not really much of A symbiosis between the two.
They're just yeah on separate tracks and don't really
interfere with each other. Yeah, just kind of a little work
overlap, I guess in the early phases, Yeah, Kernel, OK.
You know, I think kind of the the original goal of, you know,
Lib consensus of Colonel project was to kind of enable other
implementations to just take cores consensus logic and then
build out all the rest of an implementation themselves,
knowing that the core thing that matters is going to be
interoperable with the rest of the network.
Practically speaking, like how far from that stage is
development for the project? Like do you think that would be
a safe thing to do now or is there kind of still some more
groundwork to be laid before we're really at that point?
I think it's a fairly safe thing to do now.
We are at the point where there's I think two or three
projects which are now trying to build out a complete full nodes
around the kernel library. My main hesitation still with
that is the splits and the end of phases for that are still
pretty immature. Like probably want to wait a few
more release cycles before we say OK, we now officially
released this. It has been used successfully by
I don't know maybe 1020 other projects.
No major bugs have been reported in like half a year, because
obviously we really can't mess this up once it hits yeah,
production. So yeah, we definitely have to
be very careful with that. So most recently, I think 2
weeks ago, Floresta, one of the Utrixo implementations,
integrates with the kernel or the subset of the kernel
library. Yeah, that's the project
Vintium's funding in Brazil, right?
Yeah. So they're they're running it
now, which is pretty cool. And they've also committed to
working with us a bit and giving good feedback.
And we also try to upstream changes that we're planning to
roll out Bitcoin Core soon to them.
Yeah, that's been a good symbiosis.
Hey everyone, it's Shinobi and I'm here in front of the Chicago
Fed to talk to you about the core issue of Bitcoin Magazine.
The last few years have been a bit of a communication breakdown
between developers and people using Bitcoin.
Bitcoin exists to be an alternative.
To this institution. But to do that, it needs people
to actively maintain it. If you actually want to hear
from developers themselves, how they approach their work and
what they choose to work on, go to bitcoinmagazine.com and get
yourself a copy in the court issue.
All right. And I guess you should probably
try to keep this higher level. But you know, I, I guess for
some of the curious viewers out there, you think you can kind of
like walk through the different validation steps that were in
the code base and isolated in the kernel now.
And I don't know, maybe kind of talk a little bit about which of
those was maybe the toughest to kind of isolate or was sprawled
the most across different sections of the code base?
Yeah, I can try to do that. So on a high level, you get a
block from the network and then you pass that to various
functions in the kernel library and those functions then
validate the block header. So check the proof of work,
check the time stamp, the previous block that the Merkel
root is actually calculated correctly with all the
transactions in the block. And yeah, once that is done, it
commits it to disk or writes it to disk router.
And once that is done, it validates all the transactions
and all the scripts from the transactions.
So it it checks the signatures of every single spent output in
the transaction. So yeah, that's basically like
the high level overview of of what the logic actually does in
terms of what was most challenging was probably the
step between writing a block to disk and doing the actual
validation. There were like a couple of
callback hooks back into the GUI and some extra functions that
called out to the boiler to unnecessarily.
It's not really too complicated stuff, but I that I had to deal
with personally. But also have to say at that
point that when I took the project over three years ago
now, it already was in the fairly mature shades.
And that preceded yeah, all the multi process work before.
It's that cleaned that up some of the work that Corey Fields
did on the old lipid con consensus originally, some of
the work that called on did. Yeah.
So it's really been this long process over.
Yeah, probably more than a decades to to get to this point.
Yeah. You know, it's, it's kind of,
you know, hard for me to reconcile looking at like how
much work has gone into things like this in core.
You know, like Lib consensus, the Colonel after that multi
process, like all of these things that are decade long
projects at this point with the explicit goal of opening the
project of making it easier for more people to do different
things with it or to integrate it into something in different
ways. Or even, you know, with kernel
itself to explicitly build something from scratch entirely
around that core consensus logic.
That is really what matters. You know with this attitude
nowadays that core developers are like some ivory tower group
from above just trying to force their will on the network or
retain like some position of power, influence.
When the work I see is things like this, trying to do the
exact opposite. Like actually trying to give
away like some of that power and influence that they have and
make it as easy as possible for other people to pick that up and
actually do something with it. You know what's what's it like
working on something like this in that environment?
Yeah, it's, I mean, honestly, I don't really get that criticism.
So one, one of the things I've heard repeatedly is those might
actually centralized power further around coal because now
we're extending our control into other projects.
But I think that's even further from the truth because the the
only thing that we're really establishing here is an
interface and everybody can tinker with the actual
validation codes as much as they want to.
And the cool thing to see over the past year of development on
it was that a few people actually showed up and did
exactly that. So they only wanted this
interface that they can rely on and reuse for their own
projects. And they ended up implementing
some of the new op codes, some weird tap tap script stuff.
So yeah, this is, I mean, it gives me a very good feeling and
it makes me very hopeful for Bitcoin as a whole.
If we can get this project out there and just allow people to
do whatever they want with it with the confidence that they
have this interface, yeah, they can tinker with it and probably
have a bit of risk doing that. But All in all, they can build
on this 1 1/2 decades strong foundation.
So yeah, that's pretty cool. That's that's kind of just
confusing logic to me that something like Colonel would
give more power to Core because like the the underlying dynamic
here is like to do something completely from scratch.
Like unrelated to core is the risk of consensus
incompatibility. And that's only a risk because
of how big core is. And you know, they're the
dominant implementation on the network.
Almost everybody economically relevant is running that.
And when you open the door like this to build entirely new
things knowing they're compatible with core, you're
opening the door for anything to come in and compete and gain
network share without that risk of a consensus split.
And at that point, if it comes down to it, if, if something
else using the kernel were to actually, you know, gain
adoption, become a majority of the network, like at that point
it's just just another repository.
Like those other projects can simply fork it and do whatever
they want with it. Like Core has no power just
because they originally implemented that code base.
Like just click fork. And if everybody's running some
other thing built around it instead of Core itself, like
Core literally voluntarily obsoleted itself in that
scenario. I just, I don't.
Get the argument that that's entrenching core more.
Yeah, I don't buy it either for the same reason.
One thing that I can see playing out maybe is that people will
just stick with us because we have all this tradition that
we're building on top of. But I mean, as long as that is
well earned and it can't be switched away from easily, which
definitely can if we deploy this common interface that everybody
just uses anyway, then yeah, this is just a pure win in my
eyes. Yeah.
I mean, it's, it's definitely an important project.
You know, I think, I think to kind of like go back to the
lineage of like live consensus and then kernel and kind of
project out to the future. You know, the next ideal step
would be a formal specification, like an actual like bit for bit.
This is how the protocol should operate.
This is how it should evaluate data.
This is the result that should be output in whatever conditions
to provide the same degree of guarantee that the kernel is
aiming to without any actual codependency.
Like that is a really sticky topic in in this space.
In terms of the practicality of that, you know, whether that's
worth the effort, whether that's something that's even
potentially possible. Like what are your thoughts on
just the issue of a formal specification?
For yeah, you're definitely right that it is sticky topic.
I think it was one of the first topics that were this custom
bitcoins history from the very beginning.
So we have been going a bit in circles around is it better to
have a specification or is it better to just have the codes be
the reference? Yeah, you have to re implemented
bug for bug if you want to and that's going to be very hard
because the code is difficult to read and it's pretty
complicated. Yeah, looking out into the
future, I think I would like to see a specification of Bitcoin
eventually. And I'm kind of hoping that the
kernel can be a tool to make that easier so that we can break
all the validation logic into like separate calls to the
kernel library itself and have these calls be super, super
granular where you just do a single check per valid per
validation function. And then you just let the
interface for each of that validation function.
And what it exactly does be the specification for Bitcoin?
I think that would is probably the cleanest way to achieve any
kind of formal description of Bitcoin just because you have
this guarantee that you don't introduce a bug when you
translate from the reference clients to MM.
Hmm. Yeah, it's really, it would have
been hard enough as just a distributed system, but with the
mess that the code base was beginning as the worst problem
to go backwards rather than having started with a spec and
like move towards an implementation which is the
norm, you know, when you really look at things.
Yeah, but that's not what we have, right?
Yeah. We have what Satoshi gave us.
Yes. All right.
Well, yes. Any you know last thoughts on
the project or your work over the last few years?
You want. To voice.
Yeah, I'm, I'm pretty hopeful on what it will enable for the
ecosystem at large. Like I was hesitant I think a
year ago or so we had the feeling that it didn't really
get traction. But now we have like 3-4 bigger
projects completely external to Bitcoin Core actually using it,
actually integrating it into the stacks.
Second, one of the ARC implementations now uses it in
their test framework. I can see a the future where the
Lightning implementations will use it to either check their
scripts for their transactions or do some pre validation that
their transactions that they create actually pass the policy
rules. Yeah, that's just so much that
can be built with this. And I think just putting it out
there, ensuring that it is as easy to use for developers as
possible, that's going to take a lot of work still, but I think
it'll be very worthwhile. I couldn't agree more.
Thanks for sitting down and talking sedative and I hope you
all enjoyed. Thank you.