You need to actually store all that state somewhere.
And for a billion users, it's actually going to be a lot of
state. And this is usually everybody
talks, you know, talks about GPS.
But actually one of the bigger bottlenecks right now across the
space is state. And so obviously no single
computer, no single kind of node can process all that can
validate the whole network. Now you can rotate this,
validators validate what chart all the time, every block they
can be. Actually, you know you can
randomly select set of validators and they are able to
validate this because they don't need to sink into the Shard.
They don't need to process every single transaction that ever hit
that Shard. They only need to look at this
specific block. The data availability and
consensus are merged together into a single single mechanic.
The essence of it is you you want to have a design that is
not dependent on the underlying hardware itself improving at a
certain rate to be able to service user demands.
You want to have mechanisms beyond that type of scaling.
This episode is brought to you by Gnosis.
Gnosis builds decentralized infrastructure for the Ethereum
ecosystem with a rich history dating back to 2015 and products
like Safe Cow Swap or Gnosis Chain, Gnosis combines Needs
Driven Development with deep technical expertise.
This year marks the launch of Gnosis Pay, the world's first
decentralized payment network. With Agnosis Card you can spend
self custody crypto at any Visa accepting merchant around the
world. If you're an individual looking
to live more on Chain or a business looking to white label
the stack, visit gnosispay.com. There are lots of ways you can
join the Gnosis journey. Drop in the Gnosis Dow
Governance form, become a Gnosis validator with a single GNO
token and low cost hardware, or deploy your product on the EVM
compatible and highly decentralized Gnosis chain.
Get started today at Gnosis dot IO.
First One is one of the biggest node operators globally and help
you stake your tokens on 45 plus networks like Ethereum, Cosmos,
Celestia and Dydx. More than 100,000 delegators
stake with Chorus One including institutions like Bit, Go and
Ledger. Sticking with Chorus 1 not only
gets you the highest years, but also the most robust security
practices and infrastructure that are usually exclusive for
institutions. You can stake directly to Chorus
One's public note from your wallet, set up a white label
note, or use the recently launched product Opus to stake
up to 8000 ETH in a single transaction.
You can even offer high yield staking to your own customers
using their API. Your assets always remain in
your custody so you can have complete Peace of Mind.
Start staking today at Chorus .1.
Hello everyone, welcome to Epicentre.
Today I have an amazing episode lined up for you.
We're talking to Alex and Ilya, the Co founders of Near, which
is a in production sharded blockchain with a lot of value
riding on it. Specifically, we will cover how
near sharding actually works, try to build it concept by
concept into an integrated hole, and then understand where they
are in their journey to implement sharding.
Alex and Ilya, welcome to Epicenter.
So at this point, we've kind of done a few episodes with both of
you. Maybe you could give you know a
short introduction to your to your backgrounds.
Hi, yeah I think at this point my background is all near.
Everything that was before is irrelevant.
But yeah, I I was working on the sharded database called Mem
Sequel before near for five years which is right now it's an
in production sharded database and and before that I was at
Microsoft. Yeah, I mean my background is
actually in machine learning. AII was at Google research prior
to our startup adventures and I was one of the class.
I was in a paper that introduced Transformers which is technology
powering ChatGPT majority and other AI advancements.
And then with Alex, we actually started originally near as AI
company and realized we need SAS, cheap, easy to use, easy to
build on blockchain because we wanted to use it ourselves for
data crowdsourcing and some other data use cases for our AI
company and adaptivity to that in 2018 and yeah, focusing on
that ever since. Yeah, let's get into sharding,
blockchain scalability. So what is sharding overall and
why has it been a generally difficult target industry wide?
I think I mean maybe broader question is if if you imagine
having a billion users coming in and using blockchain as a means,
payments as a mean of tracking ownership as a way to kind of
coordinate resources and efforts.
You imagine that you need to have kind of a few things
happening, right? One is you need to actually
store all that state somewhere and for a billion users it's
actually going to be a lot of state.
And this is usually everybody talks, you know talks about GPS.
But actually one of the bigger bottlenecks right now across the
space is state and kind of its growth and so.
So that's probably problem number one.
Problem #2 is, I mean you have billion users try, they
transacting. There's a, you know, hundreds of
thousands of applications. Obviously there's a lot of
transactions flying around and and a lot of processing that
needs to happen and kind of people want to put more and more
complex smart contracts and complex logic indices.
So you need to have throughput, bandwidth and processing power
to do this. And so obviously no single
computer, no single kind of node can process all that can
validate the whole network. And so the way to support this
is one way or the other, partition this computation,
partition this storage and partition this kind of bad list
to receive this. And so people have kind of
historically been talking about sharding because that's how the
two companies doing this, right. So Alex mentioned, you know,
doing this kind of in web two world, then SQL single store now
is used by Fortune 500 companies.
You know Google and Facebook have their own solutions and it
kind of seemed reasonable that this is an approach that you
should take in blockchain. Now there are a lot of kind of
problems that arise when you actually move into a
permissionless setup, kind of compared to a permission setup
that usually left two companies to deal with.
I also want to add, Ilya said. It's obvious that you need to
share processing. I don't think it was obvious to
everybody until recently. So there were multiple block
chains which were like the favorite thing they were, they
like to say was like, look, look at Visa, look at how many
transactions Visa processes. And that's a world scale.
Obviously, we can do it on a single computer, right?
But as a user, you don't use Visa frequently, right?
You use it like 3 * a day on a good day, right?
And so finally now generally when Neo launched we made
multiple bets which were not obvious to everybody.
You know like when near from day one we had that named accounts
where you can rotate keys and and we had charding and it was
an obvious to everybody that it's useful and now suddenly
very scalable block chains which are not charted yet you know
congested and they have no way of getting any more performant,
right. And and similarly like you know
when it comes to account abstraction like a theorem right
now is switching to to that, so, so now finally it is becoming
obvious to everybody that those were correct decisions.
Yeah, I mean to give an example, right, like any, any kind of a
single, I call it single node blockchain, right, which is
something that every single node in the network needs to process
every single transaction and store all the state, right.
What it means is as soon as like let's say that that work has a
high capacity, they also have a huge state growth.
They have kind of limit on how many transactions they can
process because of the bandwidth and execution.
And so at some point there will be more demand like if if and it
is very natural thing, right, the price for the transactions
usually based on supply demand. And so while transactions are
cheap, don't you know at some point there will be more demand
because they're so cheap to even just spam spam it to try to get
some financial benefit from because the transaction fees are
so cheap. And so when that happens, right,
you don't have a way to expand capacity, so your your prices
start to grow for everyone, right.
And so, so this leads to now you kind of pricing out people who
are using this blockchain originally for normal use cases
because of the spam and kind of people trying to run arbitrage
for some other application. And so that's kind of just the
principle where like any single like kind of state machine,
right singular thing machine will get over on and start
rising fees. Right.
So kind of Solana is the, you know, contrasting example.
Of course, even Ethereum and Bitcoin are based on the idea
that the miners or the validators and even the full
nodes of these system have to process every transaction that
is happening in the system. Solana has taken that idea and
said yes, we have a bunch of validators at the 1800 or
something like that currently, and every transaction that goes
to the Solana system has to be processed by every one of these
validators. They are assuming that, OK,
these validators can be placed in data centers where networking
bandwidth is very high, which means a lot.
They can ingest a lot of transactions from the network at
a very high rate. They also assume that the
machines are very performant, so the work of accounting can be.
You can assume that each machine can handle lots of transactions
do do their, do their accounting work and then Solana would also
assume that. OK, maybe the history doesn't
need to be stored by these machines, they only need to
store the currently. What are the different accounts
and what what balances they own and what they they kind of like
a project like Solana assumes is the improvement in in compute in
terms of like bandwidth, processing power, every
resources it kind of double S on some time scale so some
resources double on 12 month time scale.
Another one doubt might download three-year time scale, but
because of this, doubling the capacity of the blockchain would
keep growing at a certain rate and the hope is that the user
growth is actually slower than the doubling rate of the
underlying hardware and therefore you continue to have
like a cheap blockchain. Whereas in the near case like
near the opposite approach where we where you say user growth can
be way faster than the improvement in hardware.
So fundamentally you need to move away from the paradigm of
every validator or every node needing to store.
First of all what are the balances of every account in the
system are, and then you also need to.
So basically no single machine may have a complete view on what
the balances of every account on the near system is.
And then it might also be the case that there's some machine
and there's some transaction on the network and that transaction
happened, but this machine is part of is is is part of part of
processing that network, but it never actually executed the
transaction itself. And so the essence of it is you
you want to have a design that is not dependent on the
underlying hardware itself improving at a certain rate to
be able to service user demands. You want to have mechanisms
beyond that type of scaling. Yeah, I would add that as I
said, it's not even about users, it's it's actually in in in a
way it's sadly the economics, the economics of this blockchain
isn't such that you need to have kind of a way to expand capacity
otherwise if you want to maintain low fees because that's
that's at some like at some level there will be such like
saturation where some subset of people are willing to pay higher
fees because they need planning like they try to capture some
economic value from an exchange token, launch whatever trading
and that in turn increases price for everybody else, right.
And so so Solana and and some other like kind of high capacity
networks even though the idea that like hey we we have to have
capacity and it will grow over time.
The reality what happens is since that it gets slotted by
transactions that are all trying to hit kind of the same economic
opportunity and extracted value. And so that people are willing
to pay way higher fees than folks that are potentially using
it for you know other use cases right for payments for example
and others. And so that's kind of the, the
point is and this is to add to the side that like in the state
grows and everything requires validators to continuously
expand their hardware even it just to continue maintaining the
network. So, so I think like to me
actually like past like 3-4 months have been a really great
validation and then this is not just about salon base and other
like kind of even you know bases centralized sequencer, right.
There's a single server effectively.
But if the Nets cannot catch up with all the transactions that
they need to process. And so that's kind of an example
of just like as soon as you have enough economic activity, you
starting to get kind of this slot of transactions trying to
capture that. And you don't have any way to
either isolate it and add that extra capacity for everybody
else to to do this, right. And so example I like to use is
imagine Netflix. You go to Netflix and first of
all, like in the CDO ecosystem, it would ask you to choose which
data center you want to watch from, right?
Do you want their arbitrary data center or optimism data center,
a base or you know, blast. And then when you go there, it
says like actually, first of all, you need to bring your
money from the other data center where you have your money if you
want to pay for this movie, pay for watching the movie here.
And then second one is actually you know because somebody else
watches a very popular movie, you cannot watch this movie
right now at the lower price, you need to pay more.
So that's kind of the current stage right is and what we want
to do is you know you go you can pick any movie and you watch it
and you pay kind of fixed fee, right.
That's like it's predictable for everyone.
And so to do that right. Similarly how NASA need to use
Amazon that kind of scales under the hood right and is able to
build more data centers that had of the demand that cats lakes
has kind of similarly you need a network that is able to scale
with demand and kind of you know in a way you know you have the
supply demand curves and so like you want to flatter the supply
curve such that even as demand grows you kind of can maintain
the their fixed. Fees, right.
So this is the distinction between burst capacity and like.
Average capacity in a sense where like a a system might only
be using like X capacity is sometimes in a year, but then
suddenly like one application or the old system might require 5X
or 10X and that burst might happen very quickly.
And what you're saying is essentially that if the if the
scalability properties of a system are only dependent on the
underlying machines that the validators use, then that cannot
change very quickly to adjust to burst demand.
Like certainly lots of demand comes in, machines can't be
changed across the whole network that fast.
So there needs to be like some other mechanism where a burst
happens and the system is also able to somehow respond and
being and be able to scale dynamically on a on a shorter,
shorter time horizon. And this is this is a property
nearly every blockchain kind of like lacks today, which is why
you have gas congestion. Yeah.
And so specifically in last months, we went from 4 shards to
six shards, increasing our capacity by 50% because we had a
couple of applications who had like massive growths.
We have heart which grew from zero to 5 million users.
It's like over a month, million daily actives within a month.
And so we started to having this actually congestion to South of
shards because of that. And so instead of just OK,
everybody's now paying higher fees, your network added more
capacity. So yeah, let's get into what you
know what. What are the different
challenges with with building building a sharded Shard
blockchain? So there's a couple of them.
First of all, there are certain changes to the user experience
because since nobody maintains the state of all the accounts,
if the transaction has to touch multiple accounts, something
needs to be done about it, right?
And it's a, it's a, it's a very large design space and generally
we refer to those transactions across short transactions and
that's that's one big challenge. The second challenge is in the
network, in which every node processes every transaction.
If you're if you have a node and you're at a particular state,
you have very high certainty that every transaction was
processed properly, because you literally processed each and
every one of them, right? In the worst case, you can be in
a situation where you're looking at a network which is not
canonical, right? So maybe some other part of the
network believes that another set of transactions happen on
this one, right? But that's a very different
problem from someone literally, you know, like making something
that doesn't make any sense according to the rules of the
chain. And so in the target chain,
because every node only applies a subset of transactions, you
need some checks and balances which ensure that that what you
see is actually a correct state that results from the properly
executed transactions. And when you start digging deep
into that problem, into the problem of ensuring that
everything is executed correctly, then you you start
facing another problem where in order for like for almost any
mechanic you can come up with that ensures that everything is
executed correctly. Maybe with an exception of ZK
proofs like. Once we start digging, today we.
Will we will see it? You need to be able to access
certain information in order to perform the validation, and that
information could be made unavailable by malicious actors.
And so you need to have a sort of native you need to have a
native mechanic on the blockchain which ensures that
certain pieces of data are available to certain
participants and cannot be concealed right.
And so those two challenges are like one of the biggest
challenges that exist. There are there are some others.
One interesting one would be that because the state is so
big, you need to have a way for people to either synchronize
state very quickly or work without synchronizing state.
So I I think those four would be the most interesting ones.
Right. So like, yeah, essentially, like
imagine yourself like being an accountant of the near
blockchain. It's it's a massive data
structure. You only have a small part of
it, the transaction comes. So the first problem is OK, you
might not be able to process the transaction fully yourself
because you only have a part of that entire data structure.
So you can maybe make some changes to that part, but then
the transaction might hit OK. Now it needs to do a change on
some other part that you don't have and that's a completely
different accountant. So you need to process you know
some only a part of the transaction and then it needs to
be like that that rename beta needs to be kind of handed over
to some other party and then they do that and and so on.
So like that's that's one, that's one issue.
Second issue is if you're only handling a part of that data
structure which has contains all the account balances, you
receive that data from some place and then how do you even
know that that data is genuine, right?
So that's that's the problem of stateless validation or like how
do I know that this data I'm receiving it's actually
processed correctly in the past? Then?
Kind of like, OK, if there's any mechanic to know that any kind
of certificate that would tell me that this was processed
correctly in the past, then the generators of that certificate
kind of need to have had access to that data in the 1st place.
But if that data wasn't there to the generators of like these
certificates, then then they won't be able to generate
certificates. A certain data needs to be kept
in a state where you can reach it in order to do something with
it. Yeah, so, so a couple of
comments. So the first one would you, you
mentioned that you need to process part of it and then send
it to someone else. So it's also important that that
message you're sending is not getting lost, right, but it has
to be delivered with certificates.
You mentioned that you need to be to have certain data to
create certificate. It's also in many situations the
case that you need certain data to be able to verify the
certificate. Actually, like in Near in in one
sense like there's a lot of complexity because fundamentally
like kind of like the accountants in your system do
not have, may not have access to the full data and so that leads
to a lot of complexity. But in a different sense, Near
is Near is similar to other block chains in IT that it's
just one data structure containing accounts and like
smart contracts and their data. And this is exactly the same as
kind of like Bitcoin and Ethereum where it's you're
dealing in the end with a single data structure.
NIR is managing the data structure in a different way.
But ultimately it's it's a single data structure like
that's that's a unifying property across all of these
systems. That's right.
Yeah. So yeah, you can think of NIR as
like a very large mapping from account IDs to state.
A big difference from other block chains is that the account
ID is not your key account ID. You think of it as like a domain
name, right? So let you know.
I would be Alex dot near. And it's not just a convenience,
it's not just something that is easier to convey to other
people. It's also like if you if you
think about it, if if the key is what identifies your account,
then if for any reason you have reason to believe that the key
is compromised, your account is gone, right?
You have to create a new one. You have to try to move the
assets. Not all the assets are movable,
right? Like you can imagine an NFT
which is not movable by design, an NFT.
You know like the first user of this feature, it's not a movable
NFT. That NFT is gone forever, right?
So near. If if I have a reason to believe
that my key is compromised, I just change it, right?
It also allows you to to do to like auction your account.
Like like your account has a particular state that you think
is of value, you can literally sell your account.
There are services in there that allow you doing that right?
And then this massive state of mapping.
At the end of the day, it's still a mapping from an account
to to state, similar to Bitcoin, Etherium, Stalana, any other
blockchain, but that state is short.
Just just to be clear, not Bitcoin because you J exos but
yes Etherium so like that every account based blockchain.
But yeah, you know, like you DXL, you can think of it is as
even less persistent account than than just a key.
And so this state of all the accounts, it's split into
multiple subsets, right, into multiple sets.
So today we split by by name, right.
So there would be a contiguous set of accounts that lives on
Shard one, and there is a contiguous set of accounts that
lives on Shard 2. And.
That those boundaries are not they not immutable through the
through the life of the blockchain.
And as a matter of fact they did change multiple times.
When NEO launched, it was a single set.
NEO launched with a single Shard, right?
And then it was split into four and then in the recent months it
was split two of them were split twice into two again.
So it's and in the future that will be dynamic.
In the future of the system will be changing the the boundaries
as the you know like as the load changes automatically.
So I mean, maybe one way to imagine it is like in your like
in the postal system in your city most likely like your city
is kind of like divided into these different regions, each
with a different postal code, right?
Where I live there called yeah PLZ or something like that.
And so each kind of like region of the city will have will have
a number and the city will be partitioned into various
different postal codes and you'll have like a post office
in every every code essentially. And Can you imagine like in near
it's it's taking that data structure, you can think of that
as the city and then breaking it down into like these these
areas, these shards. And the dynamism is like if you
have like a region in the city that has a postal code and
suddenly lots of letters are being sent there and they're
like, oh, now actually we need 2 post offices, then maybe they
will divide the region in the city into into two different
regions with two different post office with two different
numbers. And in practice that does happen
right? Like postal codes change over
over long horizons and in near similarly like OK, the whole
blockchain is broken down into like the shards.
The definition of a Shard can also change in order to kind of
route around the capacity demanding in in some way.
If you think of near today, you can think of it as there is a
you know, we all observe what happens in the city and how much
mail goes to every post office, right?
And at some point we realize that for a particular post
office, there's a lot of mail coming in and it's, you know,
it's getting harder for the employees there to handle it,
right. And so there could be a proposal
that says, hey guys, let's build another post office half a mile
away and split it like this, right.
And then there there's a separate entity which is
validators of the network which which either choose to, you
know, to go ahead with this change or not.
And it's sufficient percentage of them, which I think is 80,
you know, wants to go with this change, then it's then it
happens, right. And you know, if you think of
near of the future when the name of your sharding is there, it's
slightly different. It's, you know, as mail starts
coming into the building, another building just
spontaneously pops in without without any human being
involved, right. It's entirely.
More like that building splits into two and.
Moves away. Exactly, exactly OK And your
wall appears, yes, to building separate or like on a on a set
of rails. And that happens without any
involvement from any natural intelligence, right.
It just it it just occurs and then it's at some point, you
know two post notices become, you know it's it's getting chill
there. So they can, you know come
together and it all disappears. Yes.
Yes, exactly. But, but I think the important
parts here right? As you said, zip codes changes,
which means like when you're sending your mail, you need to
like. Now whoever is in the new zone
needs to update everyone. They're like in a new zip code,
but. So on here, you never see the
zip code on here you say I want a mail to be sent to Ilya,
right? And Nir figures out the zip code
itself. You don't need to know it.
That's the beauty. Yeah.
And and to compare with this with other approaches like
subnets, roll ups, etcetera, I mean in a way they're trying to
emulate the same behavior, right?
It's like oh you know this this roll up is too busy.
Instead of launching there, you know you can spin up a new roll
up and now everybody can go there and you have more
capacity. But it's a very, you know, not
just manual and expensive process, right.
I mean it's like each roll up you know cost is at least
$1,000,000 a year just to run between sequencer explorers,
RPCS etcetera, all the infrastructure.
But it's also now every user, every developer, every sort of
contract of us actually trying to use it.
Now it is to figure out how to go there, how to bridge there,
how what gas tokens is used there etcetera, right.
So it's it's a huge load on the whole kind of understanding of
the network process which we are actually addressing with kind of
chain abstraction and chain signatures as well because we do
believe this is kind of a unit like what we're trying to do
with Near is a universal problem, right.
It's like the capabilities of the network should be able to
change dynamically and everybody should be able to route things
without thinking about the underlying infrastructure.
But on near we kind of solved it in a very like direct way by
having this kind of namespace that is common for everyone and
using that to route like our transactions and messages mail
between between different participants.
Yeah, that is so cool. I actually I actually own meher
dot near and and yeah I've never needed to think about what chart
it is on right? So to me I I only need meh dot
near and it's journey through time maybe like it was processed
in Shard #1 then Shard #3 and then it's changing and I never
need to know about it right. Like that's that's that's like
really. Yeah, I think it, it wasn't
Shard. Yeah, Shard 0, then Shard 2 and
then Shard 3. Now it's on probably short 3
right now, but yeah, you don't, you definitely don't need to
know about that. And like even I am like you
know, there is a way to look it up.
But we actually don't show it in Explorer usually because I mean
some new explorers show sometimes, but because we don't
actually want people to know because it's it's irrelevant
information. It's like knowing which which
exact computer in on which rack in AWS is, you know, providing
us with this interface we're looking at right now.
Yeah, yeah. So maybe like an interesting
imagination for Near is because, like, ultimately it's like this
human memorable names at the bottom.
And maybe, you know, like each human memorable name actually
corresponds to a person or a company because that's how the
world is partitioned. And because like the Near system
breaks into shards, you can almost imagine like a virtual
connection of people that are transacting their business on a
Shard at a certain point and these are the users of of the
Shard itself. And of course like as the Shard
boundaries change, the collection of people transacting
on a Shard is also changing. But maybe if you imagine it as
like being you know stationary or constant for a certain while
which is true for near. We can we can think of this like
virtual collection of people in the in the near suburb on a
sense that's transacting on a Shard.
And then the on the other side corresponding to a Shard you
need kind of servers or validators or postman in our
early analogy that are kind of processing the mail that's
coming to that particular area or Shard.
I guess like one of the question starts to become so in any
blockchain network you have these validator machines or
minor machines that are ultimately like kind of like
this postman or accountants of the system that are doing the
processing. And even in near I would imagine
OK there's a set of validators that are that are found through
proof of stake. Now these validators sort of
like need to be assigned to shards that hey you go and
process the transactions in in this in this Shard you other one
you do it there and how does that process work?
So I I first want to to correct slightly the first time I I
think the 2nd analogy was good, but the first analogy was not
quite correct because even though Shard boundaries do not
change as frequently today as they will be at some point, it
it it has been the case from day one or near that two accounts
residing on the same chart has absolutely no significance for
those two accounts. So I think a better analogy for
that would be not the post office but like cell towers,
right. We could be next to each other
and I can call you and we will be you know, served by the same
cell tower or we could be into in different parts of the world
and I can call you and other different cell towers but we
have no benefit. You know, like I, I will never
know that you're on the same cell tower and I will never
care. And it is the case of Mir that
if you and I are on the same chart and they send you money or
you know, like we transact on some smart contract or we'll if
we are in different charts, from the user's perspective, there's
no difference in, in experience, right.
So you know, fees are not affected, the performance is not
affected. You know like like you know,
sharding is completely obstructed to it, right.
And so there's no incentive, for example, to try to be on the
same chart. There's no incentive to grant,
for example, account IDs or the contention was at the same, you
know, accounting in the same. Chart.
When it comes to the second analogy, you can think about
this way. You can think, I I like, I like
going in multiple steps and effectively saying, you know,
let's say we're designing a new blockchain and we want it to be
sharded right on. How do we ensure security when
not everybody processes every transaction?
And the first idea would be, let's say we have a massive set
of validators, right? So we set the, you know, the
minimum key to be relatively low and we say we have hundreds of
thousands of validators or even millions, I don't know.
And then every Shard, even though it has only a subset of
valid daters, it still has a massive set of validators,
right? So it's.
So we have a million total, 100 charts, and every chart has
10,000 validators. Then you can say, well, if if
you sample them randomly and we relatively certain that the
total set of validators has up to a certain percentage of bad
games, right, we say we believe that the total set has up to 25%
bad games and not more, right? Then you can do the math and you
say, well, if I sample 10,000 of them, then the percentage of the
bad guys exceeding 33% is so unlikely that we can consider it
to be impossible either to be like, you know, 1 / 10 to the
power of some large number. And then you say, well, and
because there's no more than 33% of bad guys in the chart, we can
just assume that they adhere to the protocol that any state
transition they approve of, Like if it has, you know, as a
particular percentage of signatures of those people who
are validating the chart, then we believe that that's the
transition was valid because the number of bad guys is limited
and the good guys signed off. So it's, you know, good to go.
It has practical issues. We don't actually have a million
of validators and we do want to have more than 100 charts in in
the limit. But it has a bigger problem
which is the contract of a big guy or a good guy is very
abstract. At the end of the day, everybody
who's on the blockchain, you know, they want to make money.
That's the that's the ultimate goal.
You know majority of validators, I'm sure there are some
validators who are there to build the, you know the
decentralized world of the future where everybody's a happy
corgi owning their data but invalid.
The majority of them are there because you stake money.
So or rather like you have people delegate to you, you you
keep the percentage, you make money, right?
And so correspondingly we should think about the security in the
presence of the bad guys who will try to corrupt other
participants, right? And people talk.
There are ways for people to talk, you know a lot of
validators are just sitting on telegram, right?
And it makes sense for them to be in the same telegram groups
because they they run into issues.
You know, like the network is too slow.
The validator, they need to know that they need to operate the
validator so they all in the same telegram channels.
They all easily reachable if the bad guy wants to come and say,
hey guys, I want to do this act of being a bad guy and I need
that percentage of validators to cooperate, right?
I can DM each of you and say this is how much money I'm
willing to pay you because there's something for me to gain
from the blockchain game going down.
It could be some minor extractable value.
It could be that I'm, you know I'm the Solana investor and I
just want you to go down and Solana to go up etcetera right
and so. The system needs to be designed
in such a way that a very large percentage of the validators
could be corrupted and, you know, incentivize to do
something bad. And we need to correspondingly,
let's say we have 100 or 1000 of validators and we have on the
small sets subset of them in every Shard, We should expect
that almost all of them will get corrupted, right?
Or even all of them. And so the system needs to be
designed in the way that, yeah, there are those, you know,
postman in the post offices, but it could be that the bad guy
enters the post post office, gives all of them, you know,
$1000 and asks them to do something bad, you know, like a
route and a mail which was not actually sent by the originator
or something like this. And so we designed systems in a
way that that is prevented or made very, very difficult to
execute, if that makes sense. So not only mathematics of it.
But if you if you take that that scenario that you sketched out,
first there's a million validators and then I'm
sampling. What I mean by sampling is, you
know, it's a huge set and I'm taking 10,000 and I'm choosing
randomly because the blockchain ultimately needs to choose
randomly and. Then if I get the.
Set of 10,000 my my mathematical intuition says that if it's like
25. Percent of that.
Set of million is 250,000. They are malicious in some way
and 750,000 are kind of honest. If I'm choosing 10,000 randomly,
my odds. Of.
Kind of choosing a set where the majority, maybe 66% is
malicious, means 6666 are kind of malicious and the rest are
good guys. I would think like that would
happen pretty frequently, right? Like if I'm if I'm if I'm
creating these sets like once every second, I would think
every six months or every year I'm going to get a set which is
which is full. Of full of malicious.
Now so if you have. If you have 250 out of a
million, then even sampling 1/3 will never happen.
If you sample 10,000 out of a million, then even then sampling
1/3 would not happen. It's extremely unlikely.
Oh, it's. Extremely unlikely.
So the so the my my my physical intuition is wrong.
And if I'm sampling 10,000 out of a set of million and I keep
doing that, keep creating new samples once, you can do it like
billions of. Billions of billions of times.
Like you can do it every nanosecond for a billion years
and it will not happen. We actually had the.
Math in. One of our original papers,
right. And it was I I don't remember
like that exact constants, but yeah, it was like 8:50 or. 20
minus. Studying something like that,
Probability so like longer than universe exists kind of thing.
But to an extent, it doesn't. Matter because we still like if
your intuition. Was.
Correct. Or if my math was wrong, which
is possible, it would only make the.
Situation worse. Right.
But we're saying we we're saying, hey, let's say the math
is such that and and and I think it is right, let's say the math
is such that it's very unlikely to sample a large percentage of
bad guys. It still doesn't matter because
good guys can become bad guys when they sufficiently incentive
us to be bad guys. And and then in the world of
blockchains, incentives are often such that you can, you can
benefit a lot by by corrupting, you know, a set of validators.
And so you you would every now and then the validation will
choose to do, will choose to do so, right?
And. So correspondingly you.
Want to design a system where even if the set of validators in
the chart is corrupted, right? Either either due to sampling or
because they were corrupted, what we call adaptively right?
So after the fact, then the system still operates reliably,
and so there are. Multiple you know there.
Are many ways of ensuring that and it's it's the area of
research that has been developing and this is where in
sharding we we use. You know this is the the biggest
user of Ziggy proves in sharding.
Because you know, if I can just prove that my transition is
correct, then the problem is. Solved.
Right, instead of sending a bunch of validators, if you can
just have a proof that says everything is correct, that it's
and the problem is solved. But maybe building building
this? Up from the from bottom up,
right. I mean it it starts with so kind
of we discussed right. There's there's account space,
right. You know, think of it as a city
with people and you know we have the cell tower.
So when people, you know call each other like for, for, for
use of the of the analogy when people call each other in the
city, right, Kind of bounces to the cell tower and then goes to
another cell tower to kind of connect them and so.
So the first thing that we need to do right is to ensure that
every second, every transaction is recorded, right, ordered and
kind of across all. Charts.
And that no, there's no way even for if everybody is corrupted in
that chart to be able to kind of change that order and and for
other use cases also, you know, potentially introduce something
that's not valid, right. And so that's called data
availability problem. The near had kind of data
availability from from 2019 then the design Nightshade and then
you know as other approaches like roll ups etcetera started
becoming more popular, they also needed data availability and
that's kind of where a lot of the current data availability
offerings are coming to market like a very short.
Primer would be, you know, like let's use roll ups as an
example, right? So there's, let's say this
hypothetical optimism and they they do transactions, but they
do not use zero knowledge proofs, right?
Actually don't know if optimism is planning to use zero
knowledge proofs, right? And so corresponding we're using
zero knowledge proofs for fraud. Proofs, I see.
Interesting. Interesting.
And so they check in the, you know, the the, the state route
to Ethereum every now and then. And they say, you know I applied
all the transactions correctly. And if you think you did not, I
posted just enough of cryptographic information of
like you know, a gestation that that if I did something wrong,
you would be able to come and prove that it was wrong, right.
And if you do that, I will, I will lose a lot of money.
And so I have a strong incentive not to do so.
Right. And so and so then you observe
the the roll up and you see that some transaction was applied
incorrectly. You can go to a theorem and say
this is a transaction. It was applied wrong and this is
the proof, right? But you can only do that if you
actually see the transactions, right?
So if the if the if the roll up was able to operate in such a
way that the validators cannot see the transactions, then the
roll up would be snapshotting something.
But nobody can prove anything wrong because nobody sees what's
happening. It's like you know it's a filled
room. So data availability is
effectively this concept of ensuring and proving that the
transactions that you claim to apply and the state on on top of
which you claim to apply those transactions are all visible to
everybody, right. And and that's something that
Near had from we were either first or.
Second to. Have it like I I don't know when
I don't remember when polka got launched and whether they had
the data will be with you from day one.
OK, so you. Have like a Shard.
And then there's a. There's a set of validators that
are processing transactions in the Shard.
Let's first get an. Intuition of like.
This. Set of like let's say imagine.
Like Shard 2 or whatever the set of validators at a processing
Shard too. Is it like a static set or does
it keep changing, changing with time?
Dynamically it's? Changing with time so.
The idea is and and so there is like what's right now lives and
what's been lives for for a few years and also what's we
launching with status validation.
I'll probably just. Start with about.
Status validation just for for ease of explanation.
So the status validation there is a two set kind of two roles
that eval that it can play and and all of this is repeatable
once. One role.
Is so-called chunk producer. Which?
Is somewhat similar to what you enroll up school sequencers
right to this. The node that receives the
transaction orders it in the block in the chunk, in your
case. Responsible for their Shard.
It sends out this chunk to others as well and then executes
the transactions. And receives the kind.
Of the result and so importantly, kind of where comes
in the data availability when when the chunk producer.
Sends. Out the the chunked information
they include. They kind of so-called erasure
coded, which means they replicate this information in
such a way that, you know, they send it to other cell towers
such that even if everybody who's in, you know, servicing
this cell tower is offline, goes malicious, etcetera, other cell
towers can completely replicate everything that happened in the
cell tower. So that's kind of what data
availability erasure coding is. So there's a chunk producer now.
There's a small set of chunk producers, kind of similar how
you know there's single sequence are usually on roll ups, but you
don't want that because of censorship and reliability now.
For validators. Actually have a different story,
right? So what?
Expansion you have this. Adaptive corruption problem,
right? So if you have validators which
is sitting there for a long term for a long time, it's it's
possible that you go and say like hey, if you're in this
Shard and I see you in this Shard for example, or I can
bribe you to you know do something for the shark and so
and then you need fraud proofs. And fraud proofs are complicated
and kind of require additional timelines.
And so with the. Status of additional.
Say actually the chart producer not just produces kind of the
transactions but also includes all of the state required to
execute this transactions. And so that's so-called state
witness. And so.
Then any other. Node in this in the network
right can receive this block and execute it without knowing
anything else about the network except kind of like client of
the of the network. So you receive everything, kind
of you. Need to validate that if you
apply this transaction and you have the state and the state was
included in the previous block, then the result of this and you
can confirm that and kind of send the confirmation and so
that's kind of I mean in a way it's you know kind of stateless
validation is ability for any node to come in and say like hey
I'm ready to validate. I don't need to sync synchronize
the network state, I don't need to maintain the state in on my
desk right. Which again reduces the needs
for the validator node requirements they can.
Just like make sure that. Everything's OK.
And so now you can rotate this validator's validate what chart
all the time every block they can be actually.
You know you can randomly select set of validators.
They can be overlapping. You can select you know out of
the million you can select you know 100,000 per Shard and
they've evaluated 10 shards for example each if if there's
enough capacity or or any kind of parameters and they are able
to validate this because they don't need to sync into the
Shard. They don't need to process every
single transaction that ever hit that Shard.
They only need to look at this specific block.
So that kind of opens up a lot, you know, both kind of, you
know, you can imagine I'm actually really excited that
potentially, you know, not probably this year but soon
enough somebody can open up a new tab, you know type in the
URL which has kind of a validator node in in a browser
and actually start receiving blocks and validating them
because again like you don't actually need anything else and
and browsers have web assembly to execute the the transactions
embedded. So that's kind of.
The like. Very very light, lightweight
validation that you can rotate all the time.
So. Like I'm imagining.
This state. Witness is as more like.
So let's say like there was block N and.
Then the transaction. Came in and the chunk producer
made n + 1. But is it the case that?
For every transaction, almost like for every transaction that
they processed in in that block, they are creating individual
witnesses for each transaction. So if you take transaction one,
they'll say, OK, transaction 1 modifies these two.
Accounts these. Two accounts had this balance
previously after the modification.
The result is these two accounts have this balance now.
So our state witnesses this was. What was before?
This transaction was processed. This was after the transaction.
Process. But the data that's being
supplied is only those two accounts that are hit by the
transaction. And so for every transaction you
are just creating like these like these bread crumbs, these
bare minimum amount of info that is needed to kind of validate
that transaction and that's the state witness for that
transaction. So every transaction that comes
in the block of the Shard by these this chunk producer entity
is kind of broken down into these individually verifiable
pieces. And then those individually
verifiable pieces are scattered across all of the validators of
the Shard and they can kind of do the job of verifying these
individual pieces one by one. And because each of these piece
can be verified individually you that's why you are able to run
the validator in your browser because.
While your browser. May not be a powerful.
Machine, It can still validate a few of them.
Yeah, yeah. That's, that's right.
So. Is it the case that?
This chunk producer that's kind of like more a more long.
Lived. Role and then the validators of
a Shard is like a more short lived role.
Meaning as a validator I'm doing some verification in this Shard,
then few seconds later in a different Shard than a third few
seconds later than a different Shard.
Like that I'm constantly switching as a validator but as
a chunk producer and few seconds later is a little bit.
Is a little bit consulting. We produce a block every second
but. Yeah, it's like every second,
every. Block you can be valid in a
different Shard because there is actually no difference.
Like you, you don't actually care which Shard it's on because
for you it's just a block result like a set of transaction to
resolve the information you need and so that's why you can rotate
validators now every second. I mean there's like networking
requirements and you know data information propagation, but in
principle, yes it can. It can rotate every second and
then for chunk producers they rotate every epoch right now,
which is 12 hours or 12 to 14 hours and there.
Where? You because you need to actually
sync the state. Like if you're moving to the
next chart for the chunk producer, you actually need to
know everybody's balances, right?
And so you need to actually download all that, make sure
it's correct, consistent. And then kind of while you're
downloading, you actually need to now receive new blocks and
apply them as well. And so you're kind of doing 2
jobs in parallel. You validate it, you kind of
chunk producing the Shard you're in and you're getting ready to
produce Shard that you are the next time.
And so that requires kind of more sophisticated setup.
Right. So, OK, so then you have.
These validators that are constantly jumping.
From Shard to. Shard validating some small
pieces and when is it the case? In like when I'm validating a
certain piece, I'm also adding my signature saying yes I
checked it and it's correct and every transaction and it's it's
witness is kind of getting more and more signatures or
attestations on the validators and that's how you're building
up trust, but it's not. Accumulating over time.
All the signatures happen on the same block, right?
So, so every block you need to do a sign off and it's not, it's
not the case that everybody signs of every block, right.
We we just need a certain majority.
And you know, like because blocks are created every every
second and the validators are running on the relatively
community hardware, sometimes you will miss a signature and
like if you go to an explorer you will see that like nobody
has 100% up time, people have like 99%, right.
But yeah, but the idea is that, yeah.
There's a. Set of validators.
They validating a particular block, We know whom we expect to
sign off on the block and then you know a majority, a
percentage of them signs off and the block is created.
You can look at it and say, well, that many validator signed
off at this point. I know which charge they.
Were supposed to validate. I know that unless someone
corrupted them in .1 milliseconds, you know it's
There's a relatively high certainty that the state
transition is valid, right? Because on the.
Other side, the mathematics says that when I'm sampling these
validators randomly, my odds of getting a completely bad set of
validators. If 25% of the set in the bigger
network is corrupt, it's kind of low.
So because of of that sampling, you're able to kind of trust
that OK, while these validators are considered or you are
sampling them constantly every second you are changing the
samples, but because the prob. You.
Calculate the. Probability of 1 sample being.
Being malicious like. So 66% extent or something has
been very low, you are able to trust kind of like the
signatures on the witnesses of your transactions and be be sure
of the state of of a particular Shard and.
Meanwhile like these chunk. Producers, when they are
producing these blocks, they are also forwarding the data
corresponding to these blocks. To a set of validators.
Now these this this other set of validators may be different from
the validators that are checking.
The witness the. Witnesses of the transactions
like they don't need to be the same set and.
So so. There's two things that
happened. One is.
You I. Mean the the.
Validators. That are receiving to check the
witnesses that actually received the data as well, right?
Because they actually can validate the transactions.
But also you want to send to other shards as well in case you
know this whole Shard of sale. But also like just you want to
route outgoing messages and so we're actually combine the
message routing data availability.
And. Consensus into kind of one
process where let's say you have you know let's say like you
withdrew money from my account on Shark One and then you're
sending money to Alex within Shark Two.
So now there's a message going to Shark Two saying you know you
should debit Alex you know 10 year.
And so now that message is not just a message, it also includes
a so-called eraser coded part of the transaction data that this
Shard was producing. And so kind of this process
ensures few things. One is everybody you know then
goes and when they actually signing and confirming their own
information and and sending the their approval, it also
confirmed they received the needed chunks, the needed kind
of parts from other shards. And so that's also provides us
guarantees to this this message delivery from from Shard to
Shard, it provides the data ability guarantees and it's all
kind of, you know integrated into the consensus messages that
are being sent by validators to each other to actually
accumulate the BST consensus let's say.
It's worth mentioning you there's an.
Extra role called block producers right.
So there's an actual is the blockchain, is the near
blockchain So it's not like because often when people think
of sharding and you know many sharding blockchains do work
this way. People think of multiple chains,
right? So every Shard is a chain.
It is not the case on near. On near there's only one block
chain and and there are block producers creating blocks on the
chain and when but but those blocks do not contain the actual
transactions, right? Or or like logically they do
right? Like you can think of a block
exactly the same way as you would think of a block on
Etherium, where it has like a header, consensus information
and a bunch of transactions with a difference that while
logically transactions are there, physically it only
contains information about what we call chunks, SO1 chunk per
Shard, or rather up to 1 chunk per Shard.
And physically, the block does not contain those transactions,
it just contains the information about the chunks that were
produced, right? And at every particular block,
some shards might might miss a chunk, right, because there's a
particular chunk producer responsible at every particular
moment. It could be a flying, it could
be, you know, busy, etcetera. And so a chunk could be could
could be missed, right? But if the chunk is produced,
what happens is that the chunk producer, when they produce a
chunk, as Ilya mentioned, they erasure coded and they send a
part of it to every block producer, right?
And the block producer would only sign off on the block if
they have a part of the chunk that that that is intended for
them. And so this is where consensus
and data availability are sort of mended together, is that, you
know, in order to reach a consensus on the block, 2/3 of
all the block producers waiting by stake have to sign off on it,
right? It's a BFT consensus.
If there's no 2/3 of signatures, the block, you know, we wait
until we have, right? We we favour safety over
liveness. And correspondingly, if we
cannot get 2/3 of signatures, we will stall, right?
But we and if you have 2/3 of signatures, because the block
producer would only sign off on the block if they have their
part of every chunk in the block, right?
Then you know that 2/3 of the block producers have, for every
chunk included have their little part.
And the eraser code is such that you need 1/3 to reconstruct any
chunk, right? So as long as you believe that
no more than 1/3 of the block producers is malicious, and if
you have 2/3 of signatures, in the worst case all the malicious
actors are included in those 2/3.
So you have 1/3 malicious, but you still have 1/3 honest and so
you can you can reconstruct any chunk, right?
So every chunk is available to everybody, guaranteed if you
have consensus reached on the block.
So data availability and consensus are merged together.
Into a single. Single mechanic.
Right, so like, this is wicked. Cool.
And it's also like hard to understand because this is
actually unlike any other system where data availability and
consensus are usually like like two very separate processes like
whether you go from the Ethereum to Celestia to all of the.
But in near it's almost essence right you you want.
To it, it sort of begins with with the your goals, right?
And then and then we were going back right.
So the goal was to have crossword transactions and
generally communication to be with a delay of one block,
right. So if I effectively, if in a
particular block a transaction initiated and it wants to do
something in another Shard, we want that to happen with a very
high probability in exactly the next block, right?
And so if the data availability was separate from consensus,
that would be extremely hard to ensure, right?
Because we need to be certain that data is available as of the
moment. When the block.
Is produced as opposed to that being a separate process, right?
And similarly Ilya mentioned there are three things which are
merged together, a data availability, consensus and the
message passing right. So together the chunk that is
now totally available and the moment of consensus being
reached also contains the messages that needs to be routed
to the to another chart. And it is designed in such a way
that it is ensured that by the time the chunk producer or that
other chart is producing the chunk, they not only know that
the messages exist, but they also have them right and so they
can immediately act upon it. And moreover they have to act
upon them, right? So a chunk producer, the chunk
would not be valid if the chunk producer did not act on the
messages that were sent from another sharp.
It could be that they don't act upon them immediately because
they because of congestion, like you mentioned, everybody sending
receipts to the same sharp right.
So that is automatically handled, right?
It could be that the receipt was not processed immediately, but
the receipt was acknowledged immediately and it's put on the
queue immediately. And most of the time it is also
acted upon immediately because congestion is there.
You know, most of the time there's no congestion, right?
But it is all. Part of the same.
Process where we ensure that if something happens in block M and
that something wants something else to happen in another shirt,
that something else will very likely happen in N + 1.
And maybe to to use like. ECDM and roll ups and allergy
here that you know you have optimism producing a block,
right? That block immediately sent to
ECDM validators who included and as well as they include every
other role. Let's say we have our optimism
arbitral trying to to send money directly between each other,
right? So like both of their blocks
need to be included at the same time immediately in the same
ECDM block, right? To guarantee data availability?
Because now the kind of our like let's say optimism sense
something to arbitrary connect. In it but.
If it's just that right, then arbitrary needs to read out the
state from from the CDM, right? Which adds extra latency.
And So what happens here is you assume that optimism and
Arbitrary are that those sequences are also validators in
the network. And so you send them, you kind
of optimism order and send their blocks to each other, right.
They confirm it and kind of those such stations are now
allowed to progress forward the blockchain.
And so like as if all the roll ups themselves formed a CEO
consensus, right and we're sending information to each
other directly and in turn that allows to optimize for latency
and for kind of this cross short communication because everybody
talking to each other directly and but rely but then you know
sending confirmation to the to the whole United system.
So I'll I'll try. To state this in kind of my own
understanding of how it works. So so it's like we need two
different views. One is kind of like the Shard
view and then when we need like the global view because there's
a global block. So you know, we we went to the
Shard view pretty well. It's like there's a huge set of
validators, Some of them are assigned to the Shard for.
For a block and then then we. Reassigned to other shards, and
they're kind of like validating the parts of the transactions.
In the Shard. Now these validators some
sometimes they also get the responsibility of being these
chunk producers which are more like long lasting entities for
12 hours for particular Shard they are.
They are producing like OK for this block.
This is the set of. Transactions.
And here are all of the witnesses, witnesses for it.
So these are like long lasting entities.
So you could have like a short lasting role and then you could
also get a long lasting role as a as as a validator, right.
So that's kind of like the local view of the Shard.
Now in the network there's like a global view where there's
actually a single blockchain with like block after block
after block like like Bitcoin or Ethereum, but what that block
contains is not the transactions themselves, but the chunks,
probably different shards that are sort of accepted.
Post. Consensus as being correct.
Like, OK, so here's block N it contains chunk chunk 1 from this
Shard, like chunk X from this Shard, chunk Y from that Shard,
chunk Z from that Shard and so on.
And it just contains what chunks the networks is is considering,
like finalized. And so all of the validators are
trying to build this blockchain that just contains contains
chunks and. The.
Validators are kind of like signing off on that on that
block containing chunks. They are trying to add their
signatures to it and their logic is something like what they are
checking is corresponding to each of the chunks that are part
of the block. Did I get the the slice of data
in order to ensure data availability for the whole
network? So as a validator, when I'm
signing off on a block, what I'm checking is OK, this block
contains these chunks. If it contains these chunks, I
should have received the data corresponding to these chunks.
Do I have it in my hard drive? Yes.
OK, so that's one sort of validation passed.
And then I sign and like all of the other validators are signing
that. So fundamentally the network is
coming to agreement that we have the data that we are going to
need to reconstruct any part of the actual transactions in the
block in the future. That's the thing people are
coming to consensus. So in a in a normal like Bitcoin
like the when a block gets finalized, the network comes to
consensus about these transactions that are processed
in near. When a block gets finalized,
network is coming to consensus about the validators
collectively agreeing that. They all have.
The data needed to reconstruct every part of the block should
the need ever arise. Is that right?
Is that intuition right? And they also.
Like in the world of. Stateless validation.
We also have the information that each of those parts were
reconstructed and validated by by a set of validators, right?
Right. Not only that, the.
The that the data is there that can reconstruct everything in
every transaction in the chunks, but also the data corresponding
to that every. Transaction.
With its state witnesses was. Validated by a.
Certain number of validators of the shards where the state
witnesses originated. So it's coming.
To consensus. About data that's yeah, that's
really interesting. It's really cool.
Yeah, I mean. The big the big benefit.
Was and I mean we do get this question which is like, you know
why ECDM kind of shifted from sharding to roll up architecture
and I mean. First of all, obviously not.
Talking for anybody individually, ECDM or ECDM is
not an agent themselves either. But practically speaking, to
design something like this right, you need to build
everything from ground up. You see how consensus, data
availability, message passing and kind of validation
correctness all layered in into one system and that requires
kind of this like approach and as well as the VM itself, right?
Because VM now needs to be aware that compared to you EDM where
everything is kind of available, right, you can always say like
hey, give me a state of that other account, like tell me how
much tokens does you know Alex have in, in the case of the
charted system like that requires a cross chart message,
right? That requires saying like, well
go find where Alex lives, right? And you know and ask him how
much tokens he has. And so you need to design kind
of everything from scratch, from bottom up with this
understanding given the goal we had, which is like you know,
ultimate horizontal scaling that is hidden from the users and
developers in many cases. And so for ECDM like they, they
can do not have such a luxury, right.
They have a working system, extremely valuable, extremely
kind of integrated everywhere. And so they needed something
that is kind of an evolution of their existing system versus a
complete rebuild right from scratch.
And I mean roll up architecture, you know, we use it as an
analogy because a lot of it is similar.
ECIDIUM provides the availability and consensus roll
ups are able to communicate with each other, but they need to go
through Etherium pretty much to kind of validate and and settle
before a message can be passed. And so a lot of it is kind of
spiritually the same and it's just because of the kind of
legacy of like well now each EVM is the whole separate universe
of accounts, right? And so now you I am as a user
have account that every chain right.
I don't have a singular balance now that I can use everywhere.
Like all of that is kind of a legacy, you know how do we kind
of upgrade the existing system into more scalable platform.
So that's kind of really the, the biggest obviously you know
in a way better that we had which was like starting from
scratch and kind of designing it.
Was this the luxury of the clean slate?
Yeah, luxury of the. Clean slate is.
Is what you had, right? Right.
So as a. Validator.
Sometimes I am. Taking on this role of being a
long. Lived Chunk.
Producer for 12 hours in a particular Shard.
I am constantly taking. The role of validating the state
witnesses of a Shard and I'm being assigned from Shard to
Shard to Shard and then. Every time a kind.
Of like a block is produced in the global main network what
I'm. Signing off on.
Is corresponding. To that block.
I should have received some data to backup the data of the state
of the entire network. The data of the entire network,
I should have received some small chunk of it.
I can identify it, have I received it.
And then I should not only should I have received data, but
I should have also validated some of the witnesses in the
global network or in the Shard. Have I done that?
And if I have done that, I sign off.
And if a majority signs off, that is when Near says OK, we
have consensus over all that the data is backed up properly and
all parts of the update had witnesses and they were
validated by enough, enough, enough validators and therefore
like this block is correct and like then that's done and then
the chain moves on. Yes.
Yeah, that's right. And.
At the end of the day, all of that complexity.
Boils. Down to you know, like there are
checks and balances that people do, but at the end of the day,
all you care about as the observer is do I have signatures
from the sufficient set to validate, and if I do, I know
that they have done all the work, or at least they claim to,
to have done so, right? But that's.
But that's a whole set of validators, right?
So for them to be corrupted, you need you.
Need to corrupt. Like a whole big proof of stake
system, which, you know, yeah, it has its own, you know,
design. There are certain design
principles that don't allow it to be corrupted without a
massive amount of money being lost.
And yeah, and so that allows also, you know, other chains to
easily create light clients, right, So.
So near itself. Encompasses a lot of processing
internally. It could also be a part of a
bigger ecosystem because building a lifetime for near is
a relatively straightforward process.
And near having you know general purpose WASM machine can can run
any light client for any blockchain that allows a light
client and that allows you to have two directional bridges
that effectively say as long as the light client of both chains
is not compromised. And light clients are very hard
to compromise despite the term light client like you know
compromising. In the theorem.
Light client or near light client is.
Extremely hard. Right.
And so then you say, well for as long as those are not corrupted,
Near is also part of the bigger ecosystem of of more.
Chains. So I guess.
Like every creator, when they make something, they end up
having like there's usually certain element of the system
that they wished was better. And then do you have those?
Like what? What for you individually?
What parts of Near are you kind of unsatisfied about in the
design? What I mean, there's few things
that we're still. Finishing up or like.
And and still on the road, I think.
So we mentioned dynamically sharding, right.
So right now as Alex mentioned, right, this is more of a kind of
a governance, technical governance process.
But the benefit with the stateless validation approach is
that because we rotate validators, you know, every
second, now you can actually change the number of shards for
validators very quickly, right? Because they they again, they
don't really care how many shards in the network, they
don't care which Shard, they just received the block.
And so the benefit? Here if.
We have sufficient kind of redundancy on chunk producers.
We can split the chunk producer group and say like you know as
of next block you don't care about half of the chart that you
were in, right. So comparative you know to to
where we are now, where we need to like everybody agrees you
know new client has been drawn, validators spit it up.
You know, this shortly happens, this can happen literally
instantly where everything like, OK, now we split into two
shards, you know, now I'm ignoring just half of the
transactions at that and they still receive because people are
still routing to me and I'm just processing this half And then on
the junk produce production side and then on the valid side, you
know, now you're just assigning. Yeah, sampling just from a
different for the listener, this is the.
Cell tower splitting into two dynamically, or the post office
splitting into two exactly and this can happen.
Like immediately, right, which is you know huge, huge benefit
for the spike in usage where you know sub applications just went
you know viral and esteem and or you know talking claim or
whatever and you can like hey let's just pull it out into a
separate Shard let it you know go nuts and then maybe merge it
back in like a day or so. So that that's that's you know
definitely huge kind of benefit of debt and and for context I
mean status validation is something we published in
February and I mean been building last year and should be
launching within the next few months and then we're going to
start working on dynamically sharded now.
From my. Perspective.
There is a few. Few things that.
Beyond that that would be that's what we should be working on.
One is and and now Alex actually worked on some designs for this
earlier is needless check production.
So right now there's one chunk producer at a time that is
responsible for producing chunk right.
And I mean they kind of rotate and they randomly assign, but
still you know if that chunk producer is offline now we have
a gap on that time slot. And you know when you drop in
GPS you have you know high latency with the users.
And so the idea definitely is they can also be.
They can also be deduced. Which is, yeah.
Something that happens on other networks right now?
Not not. Not because we sound.
Resilient to that in Just bad Guys shows another network to to
do those, not ours, but it can happen eventually, right?
So also leaderless chunk production consensus is
something we need to implement and then we have a a a pretty
clear way of achieving it. So leading by leading.
This chunk production SO when you say that I'm kind of like
reminded of algorand where where the the essential idea is like.
When you think of a network. Like like you when you think of
like a Cosmos network, the block chains rolling.
Along. And blocks have been produced.
Everybody knows who should be producing block n + 1, right?
It's publicly known. And if they fail to produce, the
network waits and then. Realizes fails to produce.
And then the network also knows if that guy fails to produce the
block, then this other guy should produce the block.
And there's kind of like a almost a line, a queue made of
like who has rights to produce. And so that's only what you mean
when you were saying leader. So if you have a Shard, you have
a multiple validate like multiple chunk producers, but
which one exactly produces the chunk for this block?
It's currently. Known.
In near just like Cosmos, but what you would like is a system
where there are N chunk producers and then when a.
Certain block rolls along. One of the chunk producer
realizes they have some. Kind of winning.
Lottery ticket and they can produce the chunks.
So this is no, this is not so you're explaining algorithm.
And it is not. I'm explaining algorithm, so is,
is. Is that division for lead now?
There's still a leader in what you?
Explain. So the only difference is that
the leader is not known in advance, right.
So partially solves the problem. It's much harder to give those.
For example, you cannot give those someone you don't know,
right. And so by the time you know that
they won the lottery ticket and we actually do the the way we
think of it is similar. So an algorithm, you actually
don't know in advance that you won a lottery ticket.
You, you know like like a fact. What happens is that everybody
looks at their lottery ticket and sees the number, and the
highest number wins. But you don't know if you have
the highest number because you don't know the numbers of
others. If you did and others did, then
it would be no different from Cosmos.
Then everybody would know, right?
So instead you say, well, I have a sufficiently high number for
me to believe that I might be the winner.
So I will produce the block. I will publish it, maybe someone
else will publish. And you know, given my number
was 97 out of 100, there's a good chance that I was the
highest, right? But maybe there was 98.
This approach has a has a minor problem that still multiple
blocks will have to be broadcast, right?
Because like effectively either the threshold is too high that
every now and then nobody will broadcast a block because nobody
want the ticket above a certain threshold.
Or it's like sufficiently low that multiple blocks will do the
broadcast but in our. But in our case, something
interesting to think about is that at the end of the day, the
chunk will be broadcast to everybody like a little part of
it, right? And so everybody will have to
receive. But within the network, within
the, within the set of the chunk producers, what they can do,
they will have to send the chunk in its entirety for validation
or like with with the validators of the of the chart, right.
So you can think of a system where on the high level what
happens is that every single chunk producer generates a
chunk, not just some people who who are beyond certain
threshold. Everybody produces A chunk and
then everybody stands to every person in the Shard.
So like every chunk producer every you know, like eraser
coded part. And so at the end of the day the
network overhead is not chunk size times number of
participants, It's still proportional to the chunk size,
right? But but now every but now you
have as many chances, you have chunk producers and then you
still do the lottery ticket. Then you review your lottery
ticket and if you want your chunk is is accepted right?
But there's no issue with the choosing the threshold and they
be spending the network with multiple chunks that have to be
exchanged in its entire team. So that's the high level idea,
right? But the high level idea is that
now you can literally never a chunk will be skipped, no matter
you know, like as long as there's one person you didn't
do, the chunk will not be skipped.
And it's, it's a slightly, slightly improvement.
Over. An AL grant idea where you, you
know, have a threshold and but still exchange the whole block.
Yeah, this is really. This is really.
Fascinating. So I think the essence of the
design being that. In Bitcoin or in?
Ethereum When When I? Have a block.
I have to forward. That.
Entire block to everybody, and there's a certain like diameter
of the network. So I'm a validator and there's
some validator to which I have the worst connection, like
there's multiple hops and that entire block must be now sent
across the diameter of the other side to that worst other
validator. But in near almost, it's like I
have a more efficient. Microphone or.
Megaphones I produce the block I.
Cut it into lots of pieces. And I only have need to send the
small pieces to all of the validator.
So there's some validator to which I have the worst
connection, but I only have to send a small piece to through
that. Worst.
Connection through that diameter of the network and because of.
That because. Because like this, this
broadcast of is only piece piece wise and this broadcast is kind
of like efficient. You can afford to have a design
where in a Shard, all of the chunk producers produce a block.
They broadcast their pieces and like.
Then they compare OK, which of these pieces has some highest
amount of randomness and that becomes like the canonical chunk
for for that block. Yes, yes, that be a.
Chunk of the block, Yeah. And it's like depending like
that person I have the worst connection to depending on why
they need my block, right? If they just the block producer
and they just need to to sign off, it's sufficient for them to
have just that little piece. We don't need to reconstruct the
block or a chunk. But if they do need the whole
chunk, it's still more performing than sending a whole
chunk to some, right? Like I sent little pieces and
then that person will collect little pieces from different
entities, right? So it's still a faster way to
propagate information and and I guess the advantage here for us
in building that leaderless consensus, sorry, leaderless
chunk production is that we already have a concept of this
erasure code, right? We already sent small pieces, we
already have the mechanic to gather them, right?
So that's it's much less invasive change.
Therefore, a network where today blocks are being sent fully
right, so for them to implement the same feature would be
implementing a lot of new mechanics.
Well, in our. Case it's just plugging into
existing machinery I I wanted. To touch, So you asked.
A question which I think is interesting and and nearly
covered it from a very different perspective of sort of, you
know, a. Something that.
Design wise is not great and we would like it to be different.
I I think something I thought a lot from the day we launched
near is that the accounting model for sharding is quite
different, right. Like you cannot have like flash
loans for example easily because accounts on.
Different. Shards and everything takes a
hope, right? And we've been trying to solve
this problem like we were trying to find a way to have atomic
transactions since day one with many different designs and it's
a drawback of sharding. But what's interesting is that I
think slowly we're coming to to realisation that long term not
having sharding is not an option, right.
So in essence the highest throughput block chains today
which are not sharded are getting congested and that it's
only that much they can squeeze more, right?
Like they they work day and night to remove all the sub
optimalities, but at best they will squeeze out another 20%,
right? Let's say 50% that will get
congested again like that optional blockchain today is a
fraction of what we want it to be to consider the whole
ecosystem to be successful, right?
And so sharding will have to happen.
And when sharding has to happen, people will have to deal with
this disadvantage of of this different account model.
And so Near in this case is positioned extremely well
because from day one every application near was built with
that account model in mind, right?
So we have tools, we have understanding developers in the
ecosystem are used to working in this set up, right?
While in the rest of the ecosystem people are still
operating in this atomic transactions mindset which they
will have to abandon eventually. Because at some point if their
application is to scale and applications they depend upon
are to scale, they will not be able to maintain this atomic
guarantees and scale to the user tried.
So they will have to abandon this mindset and we're
positioned uniquely in the sense that everything that is built on
Near this, you know, reach ecosystem applications is built
in the future proof way. You know, as we create more
charts, every application in Near gets to take advantage of
that, while for any application that is built on, you know what
we call synchronous. Runtime.
Then then they will have to to do right their applications I I.
Think that that actually the really.
Interesting part is because near is kind of a synchronous in in
the way every contract, every account and every contract is
designed to be independent. It actually doesn't.
Matter if that. Account is on near or not and so
that's why kind of a lot of the chain kind of abstraction ideas
becoming because well it. Actually doesn't matter for like
for. REST finance like an AML on near
doesn't matter, the token is a year on the stadium and based on
Salina like you can actually deposit any token and the rest
side answers will be able to handle it and so.
So like generally speaking nearest design with every like
every contract, every account is kind of handling assets that are
living on like in a synchronous way, right.
And obviously it's good to have them on near because you get
like one second communication time.
So like the latency is very low and you know that counts faces
is nice but it can be living somewhere else and it's like if
we have kind of a message passing way of sending this then
you know near smart contracts know how to deal with that And
all our standards like here C20 on analogous standard is
designed with school backs and kind of that's the passing mind.
And so again like compared to right now in the VMS they tried
to figure out how to do cross layer two communication.
The challenge is really lies and like all of the standards.
So I can imagine trying to say there's a 20 from 1 chain to
another like there's no standard address space or anything that
supports that. It's also synchronous like
expectation is that that transaction will execute same
block but actually it needs to you know be scheduled, message
passed settled, you know send somewhere else that revalidate
etcetera. So, so really that's kind of the
IT it, it was a very non trivial trade off, right.
And and we kind of I would say had a had a period of time where
we're like did we do a right trade off kind of but but right
now yeah we're seeing like we're literally seeing the validation
of our pieces kind of throughout, yeah.
So. The point being like something.
Like Flash loan where in the near design it's hard it's not
that flash yeah. It's not like Iron Man Low.
Yeah, yeah, it's. Not that flash.
So at some point in 2021 or something like that, it must
have seemed that, oh, Ethereum has flash nodes but near wooden
and that's a problem or is it perceived?
Problem. But when you look at like
Ethereum in 2028, which is Ethereum plus all of these L
twos and L threes, you can't have flashlights across their
entire ecosystem and you can't have flashlights across near.
So it's fine like like it's a. It's a downside, but not really
exactly like the modern. G 5/20/28 will be what what
people been building G5 near the past 2-3 years and as Ilya
mentioned, this entire. Ecosystem of L2's and L3's will
also be part of near ecosystem because because of chain
obstruction, right? Any account on any chain can be,
you know, with a very thin abstraction layer perceived as a
near account, just with a higher latency, right?
So so like a near account to near.
Account will be one second always but near account to
optimism account. It's going to be exactly the
same protocol on near site, but the latency will be higher
because there's communication between them.
Yeah, I'm not going to chain abstraction.
Because I feel like we've already covered so much, and I
did cover chain abstraction with in the previous episode with
Ilia. So I mean avoiding that because
things things start to become only too complex.
Curious listener? Can Google near chain?
Abstraction. Yeah, I I just.
Wanted to to to give an understanding why like like
chain obstruction didn't come from nowhere.
Chain obstruction was the mentality we took was near when
we designed near it's just like now we expanded that to kind of
all chains but it's still kind of you know the sinking we put
in into designing near is still there.
Now how do we apply it to the whole that three again assume
like you can think of the optimism just being another
Shard of near and this is where actually like you know for
example ZK proofs it and Aglayer what Aliga is working on is all
coming together because like if we can unify security if we can
kind of provide kind of common security layer then on top of
this you know and. Again, we have.
Near DA, we can actually settle DA of other layer twos.
Well, now they're actually not that different from other shards
of near. I mean there's differences in
sequencing and production. And so like there's things to
handle under the hood. But again we can kind of extend
our layer of abstraction that we provide to users and developers
to kind of cover up that and say like actually you know if
there's decay proofs and data availability, we can actually
like say security is the same and now we can message pass and
we can do gather species this way and so.
So that's kind of the idea is like you know how do we, how do
I apply the same methodology and and kind of user experience,
developer experience but but then expanded back more to the
rest of it 3. So like from.
My perspective, you know when I look at like Ethereum Ethereum's
road map and then like the near and near road map, one of the
things that stands out to me is in Ethereum the road map is
based on like scaling where these L twos and L threes.
But the relationship between ether, the asset and the core
asset of the L2 can be synergistic in at times but it
it can be non synergistic at other times right Like so.
So it's like the L2 pays the main Ethereum chain for certain
services and the service intended is usually data
availability. But it can be the case that the
L2 generates 100 million in fees, but it only pays 500,000
in fees to the main chain. It's this this relationship.
Is kind of like great if the L2 is kind of building a.
Completely new. Market that Ethereum never had
right? Like imagined?
I don't know, some some AI decentralized app comes and the
L to capitalize on it built it. They got 100 million in
transaction fees and paid 500,000 to Etherium main chain.
It's great for the Etherium main chain because there's a new
revenue stream coming. It went 500,000 but it's new.
The interesting case becomes when some app that was massive
like that's like massively popular on the Etherium main
chain generating millions in fees ends up thinking it's
better I migrate to that L2. And so they might be making like
10 million in fees on the main chain and then they migrate to
the L2 and then the fees get cut to a million and then Ethereum's
only making 100,000 on it. So, so it was making 10 million
in fees and now it's only making 100,000 in fees because the
L2's. Ecosystem exists.
And they're kind of like the relationship is well from the
Ethereum ether holders perspective.
That isn't so ideal, right? Because you're you're losing
adapt that might have been cultivated by these EM network
over years and then now it's kind of like migrated away.
And in practice this has happened with something like
DYDX. But what's really cool about
near is like this sort of system doesn't exist like the shards.
Like the relationship between like, near and the shards is
kind of the near token. Kind of like.
All the revenues made by all shards and kind of there is not
in order to. Scale it.
Doesn't need to have these complex economic James be
present between kind of. Like a main chain.
And an execution layer which is very much there in Ethereum, and
I believe that this will be, this will become a relevant
feature of Ethereum's ecosystem. Politics in the.
Future and NIA will just not have any of it.
Cool. So yeah.
I guess we can we can keep it at that and it was it was great to
have both of you on the podcast. Maybe we should have.
Another one to discuss on how Alex is planning to use recent
developments in the AI technology and what he's
building there. I was going to say that it's not
a coincidence. That AI stands for Alex Amelia.
Yeah. I mean, thanks.
Thanks for having us. Obviously this is a highly
technical topic that I think it's it's been really hard to
explain in general. I mean, we've been trying to do
this for years now, but. There kind of the.
Core idea is also like it was really hard to prove it out when
you just launched because when you just launched you don't have
anything so there's no users. So there's no need for sharding.
And I think like we was going to have that problem in Web 3 where
everybody was claiming scale, but until you actually have like
real world you know, massive user base to actually transact,
you know, just like a a general improvement was enough.
And I think only in last probably like 3-6 months we've
seen you know on near for example kind of multiple kind of
million user. Applications.
Launching right near right now somewhere between 1.5 to 2
million daily active right, which is more than any other
blockchain right now. We have more transactions
usually than all layer two combined at least on some days
and so like that that's kind of where this started to prove out,
right? And for context, we're still
under Solana transaction numbers, so Solana counts.
Solana counts. The consensus transactions.
So it's. I don't know how we compare.
To the to the actual member. Yeah, but.
But generally speaking, yeah. Like we have more data active
users than Solana and most of the days Tron because Tron is
kind of second the biggest right out.
But again like this, this is the point that you know as we
started to see kind of this growth like we started to see
you know congestion and sounds of shards.
And the idea was that, you know, we can just expand capacity
without increasing fees versus every other blockchain,
including Tronic chips. Really good example because the
transaction fees went from being very cheap to actually now being
like, you know, 30-50 cents for their users even though they are
running, you know, it was like a small subset of validators, you
know? A a kind.
Of modified Indian chain. So I think like that's kind of
where we're starting to see these things play out.
And obviously again it took a while right to ecosystem to
mature the applications to to you know build and launch as
well As for them to gain users. But now we're starting to see
this story really playing out and you know on real sweet.
It's it's exciting and it's also kind of tell now is really good
time to tell the story and kind of explain how it works.
Cool. Then I'll catch you again on.
Epicenter in the end, Alex, thank you.
Thank you for being there. Thank you.
Thank you for joining us on this
week's episode. We release new episodes every
week. You can find and subscribe to
the show on iTunes, Spotify, YouTube, SoundCloud, or wherever
you listen to podcasts. And if you have a Google Home or
Alexa device, you can tell it. To listen to the latest episode
of the Epicenter podcast, go to epicenter.tv/subscribe for a
full list of places where you can watch and listen.
And while you're there, be sure to sign up for the newsletter so
you get new episodes in your inbox as they're released if you
want to interact. With us guest.
Or other podcast listeners, you can follow us on Twitter and
please leave us a review on iTunes.
It helps people find the show and we're always happy to read
them. Well, thanks so much and we look
forward to being back next week.