This is epicenter episode, 471 with guests, Harry grief, and
been Fielding from Jensen. Welcome to epicenter.
The show would shocks about the Technologies projects and people
driving decentralisation and the blockchain revolution.
I'm Frederick a handstand today. I'm speaking with Harry grief
and been Fielding the founders of Jensen.
Jensen is an AI blockchain project, that is looking To
enable you to buy a i compute in a decentralized manner and we
will get to that in just a second before that.
I will tell you about our sponsor this week, though, our
sponsors Tallyho and open source wallet, redefining the wallet as
a public. Good with Tallyho, you can
safely connect to defy and web three with everything you need
from metal mask. Plus a lot more, you can view
your nft Xin, what it across the theorem polygon out.
Amazement a Bertram. There's also no need to manually
at these networks. They already come plugged in
tally, ho has the best edge of support around built.
By a community of developers that listen to users swap
between Assets in a wallet at a fraction of the price and
conveniently view. All of your account balances
across multiple networks, with our new and improved portfolio
tab. Currently, they're running a
campaign called gyoji SSE, why da GC See a layer to Adventure
that rewards users for exploring the arbitrary ecosystem with
Telly Hope from now until December 2nd.
So hurry up, rich fun. So Albert room with one of their
participating Bridges and claim your trusty space Dock and ft+
be entered into a giveaway for rare blueberry Club nft had to
their blog at block tortelli dot cash or their Twitter, telecast
for more info. Tell you who isn't just building
a wallet that work steady. Ho is building a wallet web
three can believe in Visit Tallyho to Iraq today and to
download the wallet and join Over. 150,000 people in signing
their Community pledge. Okay.
Fantastic guys. Hang on.
Let me let me order you a bit. Put myself in the middle here.
Fantastic. Harry and Ben.
Thank you so much for joining me.
Hey Federica! Thanks so much for having us.
I think we said this on Twitter, but we both been longtime
listeners of the epicenter podcast.
So really pleased to be here. And I think that's so good to
hear. How are you?
And Ben, I'm tell me about yourselves.
What are your backgrounds? And what did you do before
Jensen? Sure.
So yeah, I guess my backgrounds in machine learning research.
Mainly. So I did a PhD in deep learning
focused on neural. Architecture search as a problem
which is essentially searching the space of deep neural network
structures to find one that kind of most performant for a
specific task. So did a PhD in matte finish
that in 2019, and then moved up and moved into the startup world
and co-founded a data, privacy startup.
So I've got quite a kind of strong interest in individual
data, privacy kind of data sovereignty and things like
that. Did that for a couple of years.
And then joined an accelerator program in London called the
entrepreneur entrepreneur, first, which is where I'm a
hurry. And we kind of went down the
rabbit hole of what we're building with Jensen.
Yeah, and I'm on my side, my backgrounds and applied
econometrics, so kind of a fusion of economics and
statistics. I was sort of introduced in
machine learning during my post-grad doing my Master's
Degree whilst while studying econometrics and fell in love
with it from there. I just thought it was so cool to
be able to essentially quantify everything.
The kind of next step for me was leading data research team and
they I start up in London so whereas been comes from over.
Of technical kind of academic, background lines from one and
the applied side. Commercially got to the point
where I really wanted to build something in the space.
I saw a lot of issues with respect to scaling and yeah,
join the entrepreneur to First accelerator met then for anyone
who doesn't know what the year for entrepreneur.
First accelerator is it's been described as love.
Island meets Shark Tank. So join as a individual and then
you find a co-founder and then they kind of invest and you saw
sort of pre idea met then shared a similar vision for the future
of a I shared a similar sense of humor.
So yeah the rest is history. So it seems like you both come
from a fairly extensive a ID planning background.
What moved you to kind of marry this entire thing with
blockchain? Good question.
It wasn't a sort of instant thing.
It happens over a relatively long period of time, to be
honest and essentially, it was technology-driven.
So we knew we wanted to build massive scale, AI infrastructure
and essentially as we were doing the research to figure out how
we could make this, the absolute maximum scale.
We realized that in order to do that, you need to have a
trustless layer. Essentially, you need to be able
to unite compute without To do centralized onboarding of new
providers because at that point you end up with an
administrative kind of like scaling limit and we don't want
any limits. So we went down the kind of the
road of verifiable computation research until we hit that kind
of block of the always has to be a trusted third party that has
to be this judge or Arbiter when you're checking a computation
who makes a kind of consider a decision on whether something's
been done correctly, blockchain represents a way to kind of
break that and do it. Vai Vai by consensus
essentially. So a large group of people can
do it without having to nominate a single person to make the
decision. And that was the light bulb
moment for us where we said, like this has to kind of be the
next step for AI to get the scale that we want.
Like planetary a i scale. The has to be this kind of
consensus later introduced and blockchains the way to do it
before that. Interestingly we were kind of
blockchain Skeptics to an extent.
We hadn't kind of dived into the space before we'd sort of day,
taking the typical technical path of Of saying read only
database can do the same thing that for I Won't kind of dive
into it. But I know for me personally,
realizing that kind of trust layer was was an absolute light
bulb moment. It was when I realized the kind
of actual power behind it and got very into the space.
Yeah, interestingly that and I shared a lot of the kind of
ideals that you see Champions kind of in The Wider
decentralization scene. So we both were very like large,
free speech maximalists. And we kind of a lot of the a
lot of the kind. Of censorship stuff that we saw
with Snowden and things like that.
We we bonded over prior to even talking about blockchain.
So it kind of felt almost like obviously we should have started
in the blockchain space but we didn't interestingly right
before making the switch, we were trying to do Federated
learning which is an area of deep learning where you train
lots of models across distributed data and then
combine them to create a kind of metal model that can learn from
all the data sources and we were doing that with banks.
So the the kind of Of realization for other for me at
least. Well as that, there's a much
bigger problem with accessing computes or essentially just the
processors on which the models could be trained and to do that,
you need a decentralized kind of method of trust and that's
basically a blockchain. Okay.
So basically it's kind of the platform and decentralized
incentive layer that kind of did it for you in terms of that
form. Yeah.
And in terms of moving this to a blockchain Maybe let's do many.
Let's talk about AI fast before we kind of go into what Jensen
exactly does as a as a blockchain protocol because most
business of this podcast will be familiar with blockchain to a
certain extent, but AI is not so much our user cup of tea.
So let's talk about the state of AI today as an outside.
It kind of seems like it's totally on fire.
Fire. I mean with GP T 3 and T PT
Force concurrent going to come out soon.
I think and then things like Dolly and mean it just it's
completely mind-blowing. And can you guys talk about the
advances in AI in the last couple of years?
Absolutely. Yeah, I think it's interesting
being in the kind of AI space and watching this explosion
happen because the sort of AI and machine learning space, over
the past seven years I guess has basically been a series of mini
explosions. So this one is just kind of a
next one in the in the sequence. But I think to The Wider world,
it's one of the first times they've seen it actually create
real impact, and create applications that people see the
value and essentially. But yeah, I think Deep learning
fundamentally has been the big change.
That's kind of enabled all of this.
It was when I first started my PhD, the Deep learning kind of
explosion was just happening. It just started it just kind of
Taken computer vision as an area by storm.
They'd shown that essentially using a deep neural network, you
could blow away all of the benchmarks set by sort of manual
computer vision methods in the past.
So very, very I'll try it. Very, very briefly computer
vision before that used to be kind of manually defining sort
of Otters over images and then figuring out how to detect lines
and things. And then you would have to
Define this filter to detect the kind of line that you're looking
for and textures that you're looking for and is a very manual
process. Deep learning essentially just
came on the scene and said we can do all of this straight from
the data and that was such a huge change.
It took away all of that kind of expert knowledge, that was
required and just allowed somebody with enough compute to
design, a kind of relatively simple model, apply it to a very
large amount of data and then just have the outcome that they
want. What we're seeing now is
essentially the kind of building on top of that building models
that can do even more and then crucially getting them to the
consumer or to the developer who doesn't necessarily know the
specific problem that's been going on for years, but image
and Ali G, PT 3, Etc, of really kind of fast track that.
I don't know if you want to speak to some of the deep
learning stuff as well. Hurry, yeah.
I think whenever we kind of talked to crypto crowds about it
at conferences, we always do a kind of sharpener.
Iran, the distinction between three terms so the AI machine
learning and deep learning because you're used essentially
interchangeably but they're quite different and the best way
to think about it. Our series of kind of like
circles which are like a matroska doll almost wear on the
outside of the big on the outside.
You've got Ai. And Ai and bi the loosest
definition possible with many people will disagree with this,
but the losses definition is, it's just programming machine to
do something. So, you know, kind of washing
machine, is in a sense, a narrow version of artificial
intelligence. And you tell it to do something
and it kind of programmatically does it, or it works out how to
do it, machine learning kind of came into the scene, much more
prominently in the kind of 90s, and the kind of early 2000s,
where in you instead of having has been set expert systems for
you say, you know, if this then this you use data to advance.
So it's really work out, the kind of probability with which a
certain decision will be made, deep learning takes that
concept, but allows different kind of Concepts, be modeled
much. More with much more kind of
fidelity. So it kind of has hierarchical
feature representation which means that the way that the
model Works, different parts, learn different things.
If, for example, if you're the classic example is if you want
to recognize handwritten letters and your network typically
pushes the image through lots of different layers.
Each layer will kind of pick up something like a kind of all
this. This kind of number has a has a
close look in it or as a stem and then we're over time and
over Some kind of computational cycles and lots of tweaking.
The model will be able to generalize any new image.
It sees to one of these kind of categories, you know, a number
between 0 and 9. So that's basically the
distinction between a IML deep learning deep learning is where
you see all the kind of big breakthroughs coming in.
So all the things you mentioned, then gbd3, Thalia search for
stuff like stable, diffusion, all thats deep learning and the
story for deep learning over the past, kind of, I guess, you
know, I guess it's like 20 years. 16 2015 has been
transformed, our models, which are a specific type of deep
learning model that have been very useful for things like
large language modeling. I think what's crucial as well
as a kind of more social point is.
If you told people, you know, at the beginning of the 2010's that
they'd be able to essentially generate a comic book, which is
in really kind of convincing with really convincing art.
Just from a series of text prompts.
I honestly don't think most people would believe that's
possible. Particularly the kind of
consumer grade for like a normal person, just to be able to type
text prompts to create a comic book in the next few years.
The kind of same order of magnitude jump is going to
happen. So in the 2020s, the ability to
sit down in front of say, you know, Netflix and instead of
picking a movie which has been, you know, pre-made you simply
enter a text prompt and you're like, you know, I want to see
free technologists talking a podcast for an hour, you know,
about about AI or something. And with other kind of prompts,
and maybe like a kind of set of initializations, you'll be able
to generate an entire movie, which you can then kind of steer
or maybe a different place if you want or as a final point,
maybe you have the same story but you can change the genre of
the story. So you could turn something
like, you know, I don't know, Halloween into like a sci-fi
movie, where you can change Jurassic Park, it's a love story
or something, all by changing the using same script that
changing the kind of rendering, Lots of exciting things coming.
In my opinion can't Kenny talked about kind of the Paradigm Shift
behind this. So basically I mean if you look
at like old-school programming it's a lot of deterministic if
this then that and so on. And and in my understanding and
admittedly the, this is a very layer understanding you kind of
you use like some sort of neural network with, you know, like
complex connectivity. And where exactly is.
I mean, to Do people exactly understand how decisions in a
neural network actually reached? Is this something that can could
you kind of Transport this? But I mean, obviously, you don't
use real neural networks, right? So basically everything's in a
regular computer your, you don't have to go to like the bio lab,
although, that would be not. Yeah, I'm not sure whether that
would be terrifying or fun. Um, but basically everything's
anyways in a computer. So basically kind of your
modeling like in a different system that's kind of more
interconnected and more flexible and maybe you can, maybe you can
kind of qualify how the system you're modeling with your
regular computers different from just, you know, just giving the
computers prompts. I mean yeah sure.
I think the black box sort of nature of Learning models is
just down to the absolute size of them.
At the end of the day, you're still tracing a path through a
series of kind of decision points in the, in the network.
It's just that path is absolutely enormous and it's
hard to kind of Link the weights or the parameters within that
model down to exactly why they're that sort of value.
Because they've come to that value after being fed millions
of samples and you can deterministically, you could do
that, you could track every single update, but the size of
data that you would end up generating, would be absolutely
enormous. I think it's the sort two things
that I see happening. As we kind of go through.
This one, is the Black Box. Nature, is sort of falling away
a little bit. As we start to understand more
and more about the models that were building deep learning as a
kind of research area is sort of gone through a, an interesting
fast period where there's been a lot of experimentation that
wasn't driven by the sort of Of the research.
It was more driven by seeing what we could get out of it.
So we throw more data at it. We try out new architectures and
we just see what happens rather than starting from first
principles and designing this thing and knowing exactly how it
works. So, it's been that kind of
exciting period where everything has been very black box.
I think a lot of the gains that happened there or something to
sort of starting to slow down a little bit and we're seeing
people revisit those architectures and sort of check
and say, why does this work so well, let's dig into it and
that's kind of prove it out. Out.
So, in some ways that kind of curtain is lifting.
The other thing that's happening which is a bit more
controversial, I guess, is the shift in people's perspectives
as to whether a kind of computational system needs to be
fully deterministic or whether we can live in a probabilistic
world. We live in a probabilistic world
as people that's kind of self-driving.
Cars example is probably the clearest where, when we're
driving around, we accept that there are kind of stochastic
events that happen and that the can be small accidents in the
can be issues that Button with a self-driving car system.
We don't accept that at all and we say that this has to be a
fully completely deterministic process.
I think one of the challenges that the self-driving car
industry, as hard as been an assumption that people would
just accept that probabilistic mechanism, applied to
self-driving cars, and they haven't, but I think that will
change. And that's the probably the
controversial but as we as a society go towards actually
allowing kind of probabilistic computational systems to exist,
alongside us. Not sure if it'll be an easy
road but But I think it'll happen.
Yeah, thank you before we dive into the current landscape.
There's one term I have come across often kind of in
preparing for this episode. Also maybe that's a question for
Harry because you already talked about the different kind of
machine learning, deep, learning artificial intelligence.
So basically there's this time of artificial general
intelligence is said different from the three terms you already
talked about. Yes.
So it's a term which was popular.
Yes, I believe by at Bangor so who's on?
AI researcher and entrepreneur. The idea of AGI is similar to
also the singularity so it's the idea that you get human level
intelligence from a machine. So you have right now, what you
might describe as a kind of like artificial narrow intelligence
for by machines are good at doing certain tasks.
So for example, machines are very, very good at detecting
certain types of cancer from from medical scans.
Right, so pattern recognition, yes, yeah.
But kind of scaling that up to general intelligence for by a
machine, can be good at doing a task, which is kind of may be
simple to humans, but actually quite difficult to reflect in a
kind of computational. Like, can you give an example
prediction space? Yeah, good example, would be a
machine being able to walk through a crowded area in a
smooth way. Whilst being able to essentially
make get a discrete assumptions about all the The inputs around
it. It's one of the reasons that I
can't believe I can't remember the level of driverless cars.
I think it's like maybe level 10 or something.
It's one of the reasons that driverless cars do really well
on the motorway because it's a very kind of, it's a kind of,
it's a problem was to humans might feel quite complex but
it's quite like a simple sort of mathematical problem because
there's not much variation, but when you take that same car and
you put it kind of in a city street in Rome, you know, going
over cobbles Sakura squawking out in front of everything, it
becomes extremely I'm Lee difficult.
So it's kind of yeah, some of the stuff which we think is
really kind of difficult like being really good at chess.
It's actually quite easy for a machine but some of the stuff
that we think is really easy like being able to kind of walk
down the street or, you know, being able to like I guess
certain certain kind of things in conversation, like, you know,
understanding looking at some of the tire body language and
looking at everything of someone saying, and being able to kind
of withdraw an emotion from that, there might be, she won't
be good at various things like Estimation, you know, how
someone sitting but combining that all together and making a
kind of decision, it's quite difficult.
So yeah. Artificial general intelligence
basically means a model for a set of models or a system which
is able to essentially be as good as humans at everyday
tasks. Critically.
The kind of Advent of AGI leads to artificial super intelligence
because it follows that once a machine is kind of mastered
everything, a human would reasonably do.
Their rate of kind of marginal Mastery, over tasks moves a lot
faster than humans because of as a kind of function of both the
kind of complexity of their model and the amount of computer
available to them. So if we threw all the computer
in the world at a model which is already a human level, it's got
much more energy than the kind of normal human does and it's
also got infinite lifespan and it's also got a perfect memory.
So horribly Your Perfect Memory. So it kind of that's where you
kind of get into the realm of kind of Science, Fiction, horror
movies but this is what eat on is afraid of Yes, you hear a lot
and these kind of examples and you here also kind of what one
of the kind of Pathways that people are at least I estimate
will kind of take is, there is the kind of fusion, of humans
and machines. So, for example, if you have a
kind of brain-computer interface or brain machine interface BMI,
and you're able to essentially augment your lived experience
with, you know, machine kind of inputs that machine, learns from
all your kind of the way. Your brains working and firing
it learns patterns, you're helping it train is going to
helping you train your own brain and that's going to help speed
up that process as well. It raises a kind of you know,
Treasure Trove of ethical could you know, kind of issues.
But the yeah that's basically definition of AGI and then
subsequently a SI artificial. Super intelligence.
Who is super interesting. So let's look at what the
landscape currently looks like, right?
So basically say I want to In a, a martyr.
Where do I buy a? I compute?
So, I mean, I could just get an instance on AWS or I could run
it on my local machine. So kind of walk me through
through the options. Yeah.
So it really depends on the scale and model your training.
If you're a kind of student learning about AI, maybe you're
an undergrad, you typically just use AWS or for small enough
models. Your local machine.
As you can say, the next level up, you might be a kind of
Startup. You've just burned through your
kind of hundred K of a and credits and you're kind of
looking at the kind of marginal cost of training models.
You might go for an on-demand. AWS instance, you might go for
something more kind of fixed. We're going to permanent which
is typically cheaper when you have booked them and in advance
but the reaches a certain point when you're training models that
use a kind of are experiencing enormous cost in AWS or B, you
can actually achieve the scale required.
Terms of gpus. So you just get kind of limited
by AWS in terms of scale at that point, you see companies go in
house. So in our, in our kind of
research prior to raising our last funding round bed, and I
spoke to about 150 machine, learning, researchers, and
Engineers, a variety of places from going to find companies to
startups to Academia and whilst a lot of academics at top
universities, have access to kind of clusters and large, can
high-performance compute. And people, it's a We have
access to the fear research lab supercluster there, which is
biggest a, I lost him. The world, most people in our
experience didn't manage to get the scale that the that they
wanted. And one of the ways that some of
them kind of dealt with that would be, they buy gpus
themselves and they bring them in house and then we manage
them. And we heard all these horror
stories about people like in south of England, having a spare
bedroom with a fan in it and loads of gpus, it's a good, I
like a bum, like a Bitcoin miner upstairs, and like, also people
who would have them in their office.
Then it's a bit of a kind of fragmented Market.
However, basically, the bottom line is, if you buy the gpus
outright typically it costs less marginally over the long term to
run them. And that's a function of
basically not having to pay the sorts of 65% ish, premium for,
or should see margin for accessing Amazon ec2 instances.
So, those that's kind of cloud, local, or kind of get in your
own cluster. There's also high performance
compute if you're in Academia, You have access to that type of
compute but then again, it can be bottlenecks there.
There's other kind of options. So for example, if you're a kind
of, I guess the neverland organization and you're wanting
to solve our highly parallelizable computer, science
problem example, that would be like folding at home.
You can, you can access volunteer, computer networks
using things like boinc from from Berkeley originally.
It may be a lot listeners. Will remember things like seti
at home. Well, it's not, you know, it's
not It's not machine learning. It's kind of just on a
processing signals. It's a really good example of
grid Computing, reaching very large scale.
I think, right now that folding at home, which is kind of
successor has the largest kind of compute volume anywhere in
the world even greater mystery book and supercomputers like
'fuck a coup. So yeah, to summarize you have
you kind of go from your local machine onto the cloud maybe via
High-performance cluster at University or and then
ultimately back off the clouds, taking it back on program.
The goal of Jensen, as a segue is to give everyone access to
the same kind of compute scale that the people who currently
have on Prime clusters can achieve and crucially to do.
So, in a way, which allows Fair access so kind of it's not can't
be turned off by centralized entity.
There have been projects like this in the blockchain space
before one of Very, very old by blocking standards projects.
It's got em. I believe they actually did
their SEO and 2016, which is basically, like 50 years ago and
block him. Yes.
So how does that Jensen compared to Gollum?
Yes, great question. So, we've been given kind of two
axes. The first one is the kind of
thinness of the protocol, so to speak.
So, Golems are General compute protocol, you can do lots of
things on it, and we are, Finn protocol more similar to like
render protocol. If you want the kind of analog
there where we do one thing and that's training, machine
learning models. The the second kind of point is
on the kind of scalability of the verification.
So what we see in a lot of the kind of earlier projects is a
tendency to use things like reputation or to use kind of
less Byzantine tolerant or fault all, and I should say, methods
of replication when we looked at those.
Those kind of those architectures for verification
systems, be just didn't work for us.
As people who train machine learning models, we just
wouldn't, we wouldn't have enough faith in the results.
Doesn't mean that you don't work.
It just four of his comfort purely for kind of machine
learning. We just weren't such a villain
when we had conversations with kind of web to machine learning
people, they kind of agreed. So for us, the goal was to
basically take a lot of those initial learnings around, how do
you kind of position? Our you protocol in this kind of
world crypto world, but do it crystalline away which is only
for machine learning. So you can make super, you know,
optimizations around the the kind of speed and that the cost
of the protocol number one. But number two, how do you can
reach a satisfactory level verification right now, that
verification and consensus pieces, really like the vast
majority of our time and energy. You know it's it's the question
and we had a good initial stab at it with Our kind of inaugural
light paper but we've expanded on it since since then I don't
know been if you'd add anything to that.
Yeah. Probably just to emphasize the
kind of general purpose approach that most people before have
taken it's quite an attractive one.
You want to get the biggest Market you can possibly kind of
get to. So saying we do general-purpose
computation any scale, any kind of computational problem is
attractive at first, but you fall so quickly into the two
traps Harry mentioned the first drop is the very first Vacation
problem. It's very, very, very difficult.
Our thesis is, you have to narrow and that we will have a
big sort of set of thin protocols at the bottom of the
kind of the decentralized infrastructure stack if you
think about AWS. But in web three, we think all
of the kind of functionality that exists.
There will be ported over and it exists is this sort of
hierarchical stack of things getting closer and closer to the
user as you go up and on the bottom is protocols, like
Jensen, protocols, like render token where you do want Specific
type of computation really efficiently with really strong
verification. And then, on top of that, you
can have the kind of general purpose computer networks that
fall back onto that. So that's our kind of vision for
the, the decentralized infrastructure.
I think, as part of that, when you launch is one of those then,
protocols, you have a much easier job in initially
targeting your market. So, our Market isn't doing kind
of like chess simulations and things like that.
That just building machine learning models, that's it.
It can be really sort of attractive to say, I We could
just do this extra thing. We could do this extra thing,
maybe we could attach ourselves to an existing sort of thing
that's quite popular right now. Maybe we could generate ftes
things like that but I think when you do that you split the
mindshare massively in terms of product and people don't know
what you are for Jensen. We will always be very clear
that where machine learning compute if that's what you want.
Then you come here, if you want something else, you go to a
different protocol, maybe it falls back onto Jensen at some
point. But fundamentally, that's all
kind of we are. And I think they'll very long
term of it is we're behind the scenes.
Since we're just like HTTP, but for machine learning compute to
an end user and a developer, you won't even know that Jensen
exists. All, you know, is that the world
has changed. And now, when you train a
machine learning model, it goes out somewhere and it gets
performed by someone in the world through a series of kind
of apps and apps, and things until it eventually sits on the
Jensen protocol. We think that's the kind of best
way to provide this compute to the world is via that kind of
hierarchical infrastructure where we gradually go more and
more behind-the-scenes. Ali-A, dad had one final point
to that which is there's where we think about the kind of
properties that the network has to have it needs to be targeted
towards machine, learning engineers and researchers.
At least have the verification piece but crucially on the kind
of permissionless site in needs to have that level of sort of
censorship resistance but also kind of an agnostic relationship
with Hardware. So in the kind of, I guess deep
learning Hardware space, you know dominated by Like Nvidia,
there's companies which are doing their own proprietary
Asics, like Google through TP. Use sensor processing units or a
graph core and a good one with their IP use intelligent
conscious processing units. What kind of trap is I feel
which some protocols not even in the kind of deep learning space
of gone done before is shipping like proprietary Hardware.
So, the idea that, you know, I think I may be a good example
for General compute. I've listened to that are the
epicenter session with ICP and Affinity where they have their
own boxes, basically, and they're sold by them.
That's actually very attractive to us.
The idea that you can essentially ship your own
Hardware because then you all the kind of issues you have with
sort of, you know, rerunning proofs in a way, which is
deterministic or hashing Etc. Lots of our gets solved, but
crucially, it creates us a choke point of centralization.
So one of the kind of rabbit holes, we've seen some other
kind of Computer articles, go down is the rely on certain kind
of how would you say kind of secure enclaves?
So certain like secure Enclave like Intel sgx where they're
like, you know, we run we can run you computations for you in
a way which are kind of private. But you only have to use the
specific chat which is manufactured by the specific
company. And you know, it's only rentable
and these specific services and it just it doesn't hold true to
the decentralized, be false in our opinion.
It also doesn't scale. Well, currently at least.
Yeah. I mean, if you look at what
appears to me about Jensen's offering most aesthetic, kind of
it can use resources that are currently lying fallow.
And I mean, this would not be the case of you actually had to
buy a dedicated piece of Hardware to kind of partake in
the network know. Yeah, exactly.
I think like I said it's really attractive to go that route from
a A perspective because it's so easy.
But I think it intersects with one of the biggest things that
we think about when designing our verification system, which
is how what assumptions are we making, and how we can straining
the system, because essentially we have to make some assumptions
and we have to put some constraints in, but a constraint
like that to us is massive. It's huge.
We don't want to do that. Unless we.
Absolutely, absolutely have to there's other things that we can
do, we can sort of narrow, the space of devices in in a
temporary sense or in a permanent sense, we Can look at
certain manufacturers, we can look at certain libraries that
are provide determinism and things like that.
But every time that we make any decision like that, we make it
very deliberately and I think it's quite easy to jump over
those in the rush to ship something, but if you're going
to build the network that we want to build that kind of takes
the entire world and turns it into an AI supercomputer, you
have to be very deliberate about that and maybe it takes slightly
longer, but you've made it generalizable and that's the
kind of Step change essentially. Most 0 or 1, if you make those
assumptions, you won't reach that kind of end State.
It's I think it's sort of fits on three axes, this product
assumptions. There's research assumptions in
this technical assumptions and essentially, you have to balance
all of those things, which makes it, I think uniquely tricky.
You have to have kind of voices, of each of them, equally kind of
valid in the company and that's something that we've focused on
quite strongly with, with hiring and things like that.
Just making sure that we don't accidentally overweight a sir,
Certain kind of area. I think the some protocols we've
looked at before, who've fallen into traps.
There's some traps with research where you can go down a, let's
make the most formally verifiable system, we possibly
can. And then you never ship
anything. And then you can go the other
route where you make their kind of flashiest thing that an end
user will like you ship something really quickly.
And in previous startup terms that will be fantastic ship.
It it breaks build it again in the web three world not quite as
good as it breaks isn't just a little thing anymore.
It's a Problem. So I think it's sort of a unique
area web 3 where you have to walk this.
I think of it like a ridge where there's really attractive
looking paths that go down either side but they're not
attractive, they quite quickly, drop off the cliff and we're
being very careful to stay on that Ridge.
Quit before we dive into the ins and outs of the protocol itself.
So gentle its own layer, one block chain in principle.
It could have also been built as a Dap on another chain.
Why did you go the layer one road?
Yeah, it was it was a big question for is at the start.
I think, like we said the the sort of blockchain world for us
was all about tack. So when we, when we entered it,
we were quite sort of deliberate about it.
We looked at all of the potential ways.
We could build it. We made a massive lift list of
pros and cons and we kind of navigated through figuring out
what the again, I guess, the constraints and assumptions were
for each each one, we quite quickly moved from layer to
layer 1 because we wanted the freedom to kind of change
things. Is on the layer, one side
essentially, the consensus mechanism.
We didn't want to be constrained by certain smart contract
system. We wanted to be able to do as
much as we possibly could, because we knew this was going
to be a big sort of open-ended problem.
Essentially being a layer. One allows us to do a lot more
work on the Node side than we would otherwise be able to do.
I think if we'd built in the VM, which you could absolutely do,
you could build what we're talking about there.
You'd be very, very constrained by what you can do in solidity
essentially whereas Building in Rust for us, we can do certain
things we can fall out and do some machine learning processes.
Maybe we can do some tensor processing, things like that,
that just wouldn't be available to us within the evm.
It was a, I guess in a nutshell it was a future proofing thing
for us. We don't want to constrain
ourselves early when we don't understand fully why we're
making those constraints. So we kept it as open as
possible. And fundamental we also believe
in a multi-chain future. We think that the future is true
multi-chain. It's not sort of ecosystems full
of chains. Its Will change interacting with
each other with a kind of generally, agreed messaging
protocol. I think we've seen some
movements through the ecosystem having their own messaging
message passing and now I'm moving back into kind of General
message passing. And I think realistically, we're
seeing the multi chain future, sort of play out.
So, we're quite pleased with that kind of that so far.
So you're looking at building this, as a parent chain, why
this on substrate in the polkadot ecosystem?
So we're not, we're not fully. Certain weather will be a pair
of chain or not. Yet, the substrate decision was
essentially the technology. So when we looked at everything,
we looked at the sort of Frameworks that we could use in
the libraries that existed from Attack perspective just what was
nice. What had sort of the best
technology built-in and substrate came out on top for
us. We weren't blockchain people.
We were machine learning people. We came in knowing that we
wanted to like stand on the shoulders of giants, if you
will. We don't want to rebuild
consensus from scratch. We want to use whatever the best
one is and then carry on with building the machine learning
stuff that we're focused on and substrate provided.
That tours, as a way, to very quickly, iterate build up the
chain and then get on with the off chain stuff with enough,
flexibility to change it when we need to.
So the kind of frame sub system allows us to quickly get
something running but then if we need to step in and completely
change it, which is really attractive is written in Rust
were fans of rust as a language. Edge it just kind of made sense
from that perspective. It's interesting.
This was a year and a half ago and the kind of to that came out
on top or Cosmos and substrate and essentially substrate one
because of the tack and the kind of nice libraries in the
developer tooling and things like that.
But yeah, in the power of chain decision is one that we
essentially will make later as a bit of a cheat answer.
We can be at para chain, we could not be a pair of chain, we
don't need to decide right now. So essentially we don't if the
System starts to fill up with things that we can interact
with. So if there's like storage
layers in there, if there's sort of, sovereign data layers, and
things like that, that we would want close ties with, then maybe
it makes sense. If they Exist Elsewhere, then
maybe it makes sense to kind of Bolt IBC, on and exist in the
wider world. But yeah, yeah.
So you guys should look at Solutions like TC also.
So things have kind of allow you to kind of have a legacy
Operating system, that kind of hooks into a blockchain for
provable compute. It's a super interesting.
As you can say, sounds interesting, I've not come
across it before, but yeah, we'll definitely check it out as
you have an insulator. So let's dive into the protocol.
So there's a couple of participants in the Jensen
economy. There are submitted service
verify as and whistleblowers. The submitters are the people
who actually wants work done. So let's say let's start in the
beginning. Let's say I'm a submitter.
What kind of AI problems can I submit a my constrained in any
way? Yes, currently you constrained
by your AI problem has to use gradient, based optimization at
some point in the in the computational process.
Basically, we use portions of the gradient calculations as
part of our proof system. That's not necessarily set in
stone. I think as a hurry mentioned
earlier, we've got our light paper, which is public.
We're iterating on that internally and there are lots of
kind of things in play, essentially, but right now, it's
great. Nice optimization.
We use the signals from that as part of the verification
mechanism, what does gradient based optimization means so to
me as an AI Noob. So how would I know whether a
problem falls into that category or not sure.
Yeah. So I guess fundamentally, if we
think about a neural network, it is a big set of players that
have parameters in them. And those parameters are
essentially just real numbers that could be millions billions
now trillions of those numbers in there.
But fundamentally, they are the kind of deciding factor in the
output of the network and the training of the network is
setting, those two realistic values that allow data to go
through and Trigger the kind of outputs that you like at the end
of the network. So you go through lots of sort
of matrices, layers of these real numbers.
It sort of it changes the, the current input as its work going
through. And then you get the output that
you want. By all of those changes that
have happened, you need to update those numbers to reflect
the output that you For a certain input.
And previously way way back in the day that would be done
manually. So you maybe not with a neural
network, but with kind of certain systems you would set
those using expert knowledge and then you would know that when an
input goes through, you would get the right output.
There's also different ways of setting them programmatically.
So you could imagine a super sort of naive way of just
randomly setting, all of the parameters running, a sample
through checking, how far away from the realistic sample, it is
and then just doing random ones. Again and then doing a random
search, essentially until you make a smaller error value at
the end and then you just keep decreasing that error value.
You can do other strategies where you do sort of more
targeted updates and there's lots of ways that you can do
that. Gradient based optimization
talks about essentially, what was the big change for neural
networks and deep learning which was showing that you could
essentially use the gradient or differentiating, the parameters
of the layer with respect to the arrow as you go through the
network and you can use the Chain rule to apply that all the
way back through the hierarchical network that hurry
described. Essentially, in that way, you
get the position on the hill of loss, if that makes sense.
So, if you modeled the loss as a in like euclidean space, you
would see it as this kind of really bumpy area, where
somewhere there's a big dip and at the bottom is where the loss
is really, really small. And you're trying to find that
dip getting the gradients for each layer of essentially shows
you for that layer. Where you It on that surface and
what direction you should go. And so you use the gradient to
say, hey, we've got a massive like Drop here.
Let's go down it. So the direction that we want to
update the parameters in is this way and we want to update them
with this sort of size of Step because this is really steep or
it's not steep. So we want to make a big jump or
a smaller jump and essentially that's it.
You're just navigating this huge bumpy surface looking for a big
dip and the gradients give you sort of a position on that
surface so that you know which direction To go in and it was a
huge leap because that signal that direction is kind of really
clearly useful rather than just taking random leaps, all of the
space and figuring out. Hey, I'm on the top of a hill
now, or hey, I'm at the bottom of a trench, you know where you
are, you know, that you're on the side of a trench or that
you're on this weird flat bit and you need to make a big jump
to try and get out of the flat bill, something like that.
How do you know this? How do you know there's only one
Trend or how do you make sure you're on the right?
Train ch right because basically if there's lots of trenches you
kind of you want to end up in the Deep First one.
You don't want to get stuck on a mole here, right?
Yeah, I mean you want to go to Mount Everest.
So basically how do you make sure that basic?
How do you know how how you how low you can go?
I how how you can go with your mother very good question.
That's one of the big, big problems in deep learning
itself. Essentially little lots of
techniques for doing that. The very, very simple answer is
assume it's convex and then you don't have to think about the
being any other. Yeah.
Obviously in the real world, it doesn't work like that.
Essentially there's lots of sort of regularization techniques
that happen in deep learning training and make it a really
complex thing. It makes it more of an art than
a science because a lot of people have their sort of little
tricks that they do, there's things like within learning rate
schedules. So you'll use a learning rate
set, the magnitude of the jump that you'll take in that
gradient space but your you can use certain schedules to sort of
Decay The Learning rate, make it smaller over time, which means
smaller and smaller jumps. So you don't accidentally jump
over kind of trench. But in the same case, you can
suddenly randomly introduced a huge jump, which just allows you
to know that. Maybe if I am in a global
minimum, maybe I'm at, if I'm in a tiny trench here and there's a
massive one over here, I'll just do a huge jump, I don't know
where I'll end up, but it should, it could be better if
not, I'll probably roll back to where I was before.
So there's lots of techniques like that, that sort of more
trial and error than they are sort of deliberate.
But Like I said before, they're becoming more deliberate over
time. So now that people have
introduced these regularization techniques, Dropout, norms and
things like that. Now, people are looking back at
them and saying, hey, did this work for the right reason?
Or was it just some weird random Quirk of the model architecture
that made it work here? And can we kind of figure out
exactly why it works? But yeah, it is, it comes down
to an art more than a science, to be honest, it can be very
frustrating. So now, I understand that
gradient optimization problems are what I should submit in
terms of, I mean, can you can you can you talk about like real
word problems, Robert problems and say which ones are gradient
optimizations and which one, which ones aren't just, so I can
get like a feeling for what kind of problems I should be able to
submit Yeah, I mean the simplest way is thinking pretty much
every neural network is uses gradient based optimization.
There are other problems that use it as well, but within
neural networks, all of the kind of big steps that we've seen all
the big changes have been neural networks recently, so it's a
logical place to focus for us whilst also allowing this big
space of other places. So any optimization problem you
could theoretically, as long as it's differentiable and we can
use the chain rule to flow back. It could use gradient based
optimization and Use other optimization techniques with the
grading as a signal in there. As long as you're calculating a
gradient, we can use, it's useful, but yeah, fundamentally
it's all neural networks. Every maybe two three years,
somebody comes out with a paper that says, hey, we're training
neural networks with evolutionary optimization that
doesn't use gradients, and it's better.
It's never better. It's better in a really
constrained system and it never takes, hold not to say.
It never will do. But so far gradients of Managed
to stay pretty kind of solidly at the top.
Okay. Then I have turn the question
around. What I've what kind of problems
can't I submit? So what what what problems are
not solvable or not? Where's Ava were with neural
networks? Hmm.
Good question, expanding at the question your own networks in
general neural. Networks are generally quite
data hungry algorithms. So if you have a have a problem
with very low data volume, Good example.
That would be like, I might be wrong about this, but some of
the kind of toy examples which are used to teach people to do
machine learning like the iris dataset and stuff.
Are you have like a very like you could actually have a
spreadsheet like a hundred rolls and maybe if I think like seven
or eight features, I don't think that intuitively.
They're like, well, suited to neural networks are typically
better handled by like statistical machine learning
techniques. So I think data volumes, one of
them. There's also just certain
Certain types of neural networks which is very large.
So like fitting them on edge devices can be a challenge, but
in terms of like the actual, I guess, when you say problem, if
you think about the kind of, what's the type of thing, you're
trying to predict in the world, there isn't something which
immediately comes to mind, that neural networks aren't are like
explicitly currently, and always will be bad at.
I don't know if you've had any tuition there.
Been. I guess you can think of a
neural network as a universal. Action approximator.
So theoretically, it can do all of the things that you would do
with other methods. I think, like Harry said, the
reason you would not use a neural network would typically
be down to data volumes where with a statistical machine
learning mechanism you method, you could get a better result
essentially and then you would you wouldn't train that using
gradient based optimization. But fundamentally you could do
it within your own network if you wanted to.
It just might be a bit worse. Okay, good.
So I understand that. I can submit almost any
questions. So basically say do I why I
asked for an entire program or I mean do I ask for like a dolly
kind of output can I say like I want a picture of accountants in
hot air balloons over a waterfall and there should also
be a rainbow with scorpions on it and it'll do that for me.
Oh can I ask I'm betting? I'm reading this car and I need
an AI to drive that car. Can you deliver that AI?
Is that kind of the Both within the scope or do one of those
fall out of scope. So I guess it's more like you'd
use Jensen to train, the model itself.
So what you would do would be you'd think I want those things,
I want to be able to create my scorpion rainbow kind of image
generator from the text prompt scorpion rainbow and which I
love and you'd you'd build a model which, you know, receives
a text prompt and then converts that text problems into into
images and then you would have the training data.
Which facilitated the kind of learning of that model and then
to magenta Network, you would submit model data and then some
hyperparameters which determine you know, like badminton League,
The Learning rate, schedule things like that.
How do you how long you wanted to train for?
And then you're kind of the artifact you receive.
From that training process, the kind of product you get, is the
train model and then that model can, then you can then post that
and then you can submit scorpion rainbow.
Yeah, how do I decide, which untrained motto.
How to use? That's a brilliant question.
I think there's kind of two ways of thinking about it, so it just
kind of emerging and highly kind of popular concept around
foundation models currently, which is, you know, you get like
a big company like open the eye or something like mid-journey or
something, and they build the base model.
And then you take the base model with you, your training data,
which might have lots of rainbows scorpions in it and
then you train the model on that and then the output of that.
And then I'm always very good at, you know, approximately
Amazing that those outputs. That's kind of option.
One and that's the most common for people who are quite compute
restricted which is the kind of theme in the industry just now
second option is you build the model but they were be whatever
wise build from scratch and I don't know if you want to talk
about that then from our Yeah, I suppose a lot of our thinking
comes down to the foundation models approach because we think
it's the kind of the future of the space.
My research back in my PhD was specifically on essentially Auto
ml techniques. So the idea of allowing somebody
to optimize that that model structure and find the best
model structure without necessarily being an expert,
that's another way of doing it. And you've seen that sort of
happen within like AWS Sage maker for example, where they
and gcps. A few clouds as well, where they
build in some Auto ml techniques to say to a developer.
You don't need to know the specific machine learning
architecture because essentially, we can just see
that as something that's trainable as well.
And we apply an optimization technique on top of that.
Jensen is a protocol. Can have that if you wanted to,
we would see that as something that you would build as adapt
that would use Jensen. And that dap might Implement a
evolutionary optimization technique or something like
that. It would submit the sort of
individual architectures, it wants to train and Best to the
Jensen protocol, it would have them trained and then it would
iterate on the structure and it could build up the kind of the
model that you want. And that's a bit of a theme in
the way that we think about Jensen as being purely machine,
learning compute. All of these interesting things
that exist around it. We would love to see build out
as an ecosystem essentially. So all of the nice things that
you see on stage maker and gcp, we see as being additional
things on top, I think it could be very attractive to build them
yourselves but ultimately, it's a, it's a trap, but yeah.
A on the foundation models we've seen because of the compute
problem like how he described, we've seen people take
Foundation models, from very large research papers that have
been spent maybe 10 million dollars in funding in order to
kind of trial out all of these different architectures and then
they publish a new architecture and say, hey, this is the best
in the world at doing these three computer vision tasks and
then you take that, you use the vast majority of that
pre-trained Network. That cost Millions to train, you
would add some layers on the end.
You Some layers off and then you train those layers on a smaller
set of data and you have a kind of usable model for that and it
generalized, loads of information from the first kind
of training it did. So you call that pre-training
and then you've got fine-tuning and that's very kind of classic.
And in the Deep learning space, one of the things that we find
particularly difficult with that is the bias that gets introduced
in that pre-training. So, one organization doing that
on a proprietary data set or on a dataset that they haven't.
Closed features about means that when somebody else comes to use
it, they don't know what's going on because of those black box
issues that you mentioned earlier, it can't kind of you
can't go back in and say why did it make this decision?
The solution to that, in our minds isn't to go fully
deterministic and kind of get rid of the black box.
It's to open it up to everyone. And say, hey everyone train,
this Foundation model, so design it together, we train it
together on an infrastructure that nobody owns and at the end
of the day, we have a model that we can all use.
That's kind Global and hasn't necessarily been biased by a
specific company's cash of data that they've kept back and they
don't want to tell you what's in it, and things like that.
So, once we've got those kind of global Foundation models, then
anyone can come along and say, I'll take I'll find the hash of
that model on the Chain. I know it's been trained.
I'll pick that spot. And I'll continue training from
there on my data set for my problem or task.
And then I'll have a model that I know at least is as biased as
the entire global population. Rather than Being as biased as a
company in California. Okay so basically until we have
the global Foundation model maybe we'll talk.
We can talk about how you plan on kind of delivering that
later. But before we have that, I kind
of have to decide on one of the commercially available ones and
I have now submitted my my problem who gets to work on it.
Do servers need to kind of prove is some prerequisites.
And basically is it one solve a Ma, can you Paradise this?
That sure I would say it's at the task level.
It's one solar per task but a model can break out into lots of
different tasks. So typically when large language
models are trained, it's interesting.
They've kind of been built in a way which maxes out the current
Hardware at their time of creation.
So you know they are designed to fit chunks on you know certain
the video processors Etc. You imagine a similar thing
happening across the network. It's complicated.
Created by the fact that there's heterogenous devices on the
network. But essentially for any given
task, you know, supplier of compute.
So if I have a verifier or worker, they have their ability
to basically say, I'll take that from the mempool and then they
are randomly chosen from the pool.
People who say that they'd like to take that task.
So everyone can do it if the model and the data can't fit on
your device and you said that you know, it And then it follows
that there's deliver lightly be a penalty there because it's
kind of clogging up the system but essentially if you're if a
task and fits on your machine then yes your ability to run it
is essentially just determined by a verifiably random function
which is the likes you from a sub, a subset of the available
minors or workers, I should say, how do you verify the what kind
of capacities the miners have? So busy, if I say I have like a
16 core GPU and 400 gigabytes of RAM.
How do you verify that? Yeah.
So it's essentially in the verification of the computation,
they won't be able to do the computation if they don't have
that compute device essentially or that capacity.
And when they come to submit their proof, when I gets
checked, it will be found that they couldn't do the
computation. Okay.
There's a little bit of a sort of question there and how big
you make a task, because You, if you made a task in an enormous
piece of compute, then that would kind of be an issue.
Because you could quite easily Das the system by set grabbing
lots of tasks and saying, I can do these never doing them
wasting everybody's time and kind of money and things like
that. So it really is a decision
exactly. It feeds into that decision on
the size of a task and that's there's lots of other things
that feed into that. There's a parallelization that
you mentioned as well. And how you split the tasks up
into the most optimal structure. At the end of the day, we're
doing a lot of research on Figuring out what that should be
based on the constraints. When we launched our test, net
will do it based on the kind of practical aspects as well.
When we see how this actually works in the real world, we're
very conscious that it's easy to kind of divide Define this in
the kind of perfect system and say, yes, this is the best size
of task. And then you go out launch a
test, that someone does something really weird and you
realize that you have to completely change it again.
So it's part research part. Let's just see how it functions
when we get out there essentially.
How do so basically if If I get a specified model and the
training data, how can you make sure that I've actually done the
job right? Because it's very much not
deterministic. So it's not like, you know, you
can make me do a hash and then the hash will tell you whether
I've done it or not. How do you build in checkpoints
into this into this process? Because otherwise, I could, I
could just, you know, pretend to do the work and then kind of,
you know, this was, this was a lazy moderate kind of didn't.
Do the work, it kind of, maybe stupid.
I don't know. I've done it but it just
couldn't be taught. Essentially, that's the big
challenge that verification system.
It's a huge Challenge and I think the simplest most secure
solution to say is a zero knowledge proof of the entire
computation. Essentially.
That's sort of what you think about.
In X years time, we should be able to do any computation as
part of his own knowledge proof. And then we can you can check
that proof to say. Definitively whether someone's
done that computation or not? Don't you need a new second for
each given computation? Yeah.
So right now, that's that's the case.
And it's horrible to try and do for machine learning work.
The computations are massive. You need a DSL for defining
circuit, with respect to a machine learning computation,
it's horrible. Essentially, our approach is to
have a hybrid between the that and a probabilistic mechanism.
We sort of follow some principles.
Suppose in work called proof of learning by Nicolas paper knots
Group which is it was a paper within the machine learning
World, essentially, showed that using the path through gradient
space that we described before, you can sort of create this
certificate proof using checkpoints and that space that
theoretically it's just as hard to generate a realistic-looking
path as it is to just do the work.
And then using a kind of financial irrational assumption
on all of the Ben's you can say they would just do the work.
Essentially, there were issues with that paper and kind of
flaws with the with the ways that check things but
fundamentally what it showed was using a stanch Ali, random
auditing scheme on top of a path, through gradient space,
you can build up a relatively robust check.
And essentially, we take that one step further by introducing
zero-knowledge proofs at certain steps, and on top of the kind of
global loss of the model, just add another definitive kind of
proof on top. And we package all of that up
within a game theoretic mechanism.
That looks quite a bit like true bit from, from way back in the
day with staking and slashing solving the verifies dilemma
with like random jackpots essentially with whistleblowers,
and that's the full system. Although I'm aware.
That is a, just a big word, vomit of things.
So, happy to dig into bits of. Yeah, maybe that's, there's so
much unpack here, so, maybe kind of that's back up to true it.
So I think lots of people kind of, remember prove it.
It's kind of this. Basically, it lets you do large
computations of chain and then base, you can prove it where the
binary search on Shane. If anything's is that, is that a
fair summary? Yeah, I think, essentially
that's, that's exactly it. So, true bit prove that you
could take a very large computation that wouldn't fit in
the evm or would be absolutely massive and really expensive, do
it off chain and using that challenge mechanism and that
search that you described eventually, prove it on chain
with the chain, doing a tiny Ian, we take that same
principle, we apply it on top of some of the sort of certificate
proof stuff that we mentioned before.
So if you applied that to a full machine learning training job,
you'd be searching forever like it's enormous.
So we distill that down into a kind of a smaller proof that is
still representative of the larger IE rather than doing the
full thing. You do one in 100 checkpoints or
something like that. You've already reduced the size
by 100, then you go into that challenge mechanism.
There's also some work in the machine learning Space again
which is applied the true B mechanism to neural networks.
But rather than using virtual machine instructions for that
search, use a graph and you Traverse kind of Merkle tree,
graph of a neural network graph, essentially of operations and
you can do that at different granularities.
So you can do it at Native operations.
Like within Pi torture tensorflow on a convolution and
then you can step into that convolution and do the Matrix
multiplications that are involved and then you can step
into that matrix multiplication and do the individual.
And of floating Point operations that are involved, it's quite
large overhead. So it requires you to do that
big reduction before you get to that stage.
But once you do it provides, that crucial link, that goes
from random off chain participant to full consensus of
the chain with the chain running, something and that,
links back to what we said earlier about being a layer on
versus a layer. To, as a layer one, we can also
increase the size of computation that the chain can do.
So if we make the chain to a matrix multiplication and that's
okay, then we get to kind of skip that step in to a matrix
multiplication doing following points and things which is quite
nice at the end of the day, it's constraints and assumptions
again, you increase the hardware of the validators and things, so
you've got to be careful but we like the flexibility and being
able to kind of change all those levers and things.
I totally get that. So, I think basically, kind of
fixing the, the block gas limit and kind of maybe repricing some
opcodes. And I mean, obviously it gives
you it goes a long way, right? Maybe that's talk about the how
the blockchain itself Works in just a bit but there are two
more parties in the process. So there's the verifier will
actually make sure that the check points have been checked.
T' and then The Whistleblower who makes who make sure that the
verifier actually operates correctly.
Can you go through what their respective roles are?
Yes. So the verifying The
Whistleblower have a relationship similar to the
verifier in the kind of worker in the tribute paper.
So essentially The Whistleblower solves the verifiers Dilemma
problem which is the idea that you won't necessarily about you
know, verify Mark, unless, you know, you can be simply expect
there to be warkworth kind of, you know, catching is being
wrong and being rewarded for. So the question blower
essentially checks that the The verifiers Works being correct.
There's also incentivized to do so by force Terrors from the
verifier. So, the verifier wasn't from the
true bit paper right there. Yeah, it kind of like that the
dogs at the baggage carousel, where we see if they don't find,
you know, any drugs in their hand Let's put, you know, like a
suitcase with drugs. So, you know, they don't end up
depressed and you know, stuff working the dogs need dog
treats, occasionally the that's basically the kind of the kind
of thinking there, so it falls. The basically, the solver does
the work. If the work is incorrect, the
verifier shows it as being correct and the list of Laura,
can they confirm? It's incorrect that then goes
back onto the kind of chain which we can talk to in In terms
of, you know, being being verified on chain.
But essentially, it periodically and also kind of the rate at,
which is kind of linked to the security of the system, I guess.
V, the verifier will show up an error on purpose to twist a
blower, which keeps The Whistleblower wanting to be the
kind of engaged with time, as well.
If there was a blower, does find a problem, the play a game
pinpoint protocol, where we narrow down equal to down the
computation to a single Kind of point in the, in the kind of I
guess you could view it like the miracle tree of computations for
that area of the neural network and then that goes to the the
chain for, for arbitration. That's the kind of the the
version of it it kind of plane way that we originally had has
been mentioned earlier. We've Advanced on it in a couple
of areas after a basically closing or sit around and doing
more research work. But yeah that's the verifying,
it was Civil War. So tell me how that fits into
the blockchain as a whole. So obviously someone has to
build blocks, there has to be I assume this is some kind of
shaking Network so has to be staking token and how does all
of this fit in with the Jensen prodigal?
Yeah. Essentially it's a vanilla to an
extent substrate blockchain, we use the proof of stake.
Grandpa babe consensus mechanism.
Validators. Just kind of doing things in the
normal way that they will, and all of the parts that Harry
described. And I described earlier happen
off Jane, they're all kind of off chain participant's doing
portions of work and kind of being incentivized by the fact
that they've staked through a kind of normal staking palette
within substrate, but been a smart contract, it could just be
submitting, a certain amount of tokens and then there will be
rewarded. When that work, ultimately gets
checked. Now, the kind of game theoretic
Faculty here is making sure that all the staking potential
slashing amounts and the reward amounts all add up.
So that there isn't an incentive somewhere for somebody to either
be lazy or to do something that is malicious essentially.
So it gets complicated. When you add more participants
in the kind of having the Whistleblower.
There is an additional participant is annoying because
it's over complicated but it's crucial for us given the size of
the computations to have it there to assure the honesty of
the verifier. Essentially, it's not certain
that we'll always have to have that we do.
Keep thinking about ways that we can potentially remove the
Whistleblower. There's certain zero knowledge,
proof techniques, that mean that we potentially could.
But we don't want to get ahead of ourselves essentially.
So right now, it kind of looks like what's described in the
light paper but we're chipping away at each bit of it to try
and simplify it. We think, if you look at the way
that other protocols have gone in the past, there's a tendency
to launch with a complicated system and then once you get it
out there, realize that you can simplify buy it and we're
expecting to go through that essentially.
We kind of saw the same thing with polka dot on the fisherman
mechanism that sort of got removed after the thing had
launched and it was out in life. I died.
One other point. They're just on the kind of our
augmentation of the vanilla kind of substrate shown.
There's an issue in the verification system as we
originally proposed it and also as it currently looks for us
state-of-the-art whereby if the the data which is being used To
perform the initial kind of smart from the solver is removed
or made inaccessible. Halfway through the verification
process, you reach a kind of standoff.
Because at the barrel fire, can't access the data.
Then there's a follows that they can't verify some kind of data
availability solution that kind of plugs into it precisely.
Yes. So we are.
We built that in to on top of the old kind of substrate.
So we have a proof of availability.
P of a is this. Kind of what we've got adopted
internally layer, which is Erasure encoded etcetera.
And basically provides what we couldn't find in The Wider, kind
of storage layer market. And if anyone's listening to
this huge building that space and this does exist, I would
honestly be fascinated to see it.
But essentially a layer where in you can lock data for a period
of time in a way, which is pinned on pinnable for that
period of time. And verified on chain that the
exist there and threw that on our we've it's too expensive and
are we've. So are we is the answer but the
cost for if you think about, you know a terabyte of training
data, being stored Forever on our.
We've it just doesn't work when the kind of alternative is like,
you know, storing it on S3. So, yeah.
Should I should also caveat it has to be an expensive.
Yeah, but yes, just on that with our we've The reason we need it
is is for the data, the training data, but it's also for some of
the intermediate, like proof data, and that doesn't need to
be around for very long. It could just be 20 seconds.
While we go through a certain number of block block, like,
what releases or something with our we've, we don't need like
200 years of storage if that thing, but that's what we're
paying for essentially. So if somebody has really short
term but with the guarantees of our we've so bringing the price
down because it doesn't need to be 200 plus years.
That's what we want. Basically, we just haven't seen
it anywhere. You should go.
You guys should talk to Ali's. I mean, I weave the storage
rent. This would be, you know, this
might be a thing. Yeah, it's like a sort of Perma
web that are temporary web. The temperature at the Tempe
Perma web. Right.
So I assume that there's going to be agents in token somewhere
in this eventually, tell me about that.
Sure. So the Jensen token
fundamentally is required for the attack essentially,
everything that we've just described assumes that you have
this token built-in that can be used to stake /, provide
rewards, Etc. And also maintain the consensus
of the system itself essentially.
So use kind of a small inflation amount to pay out validators,
and then be used in that game theoretic mechanism to allow us
to kind of guarantee that Financial irrational kind of
assumption over the entire system.
I think crucially for us, that's what it's for.
And that's like the only thing it's for we're very, very
deliberate to say it's a technical thing that we need,
but we will bring in when we technically need it.
And not before we've seen what's happened with with kind of
utility tokens in the past where people have kind of launched
them too early. And then it's a distraction for
the team as a distraction for everyone.
People aren't buying it to use it in the system, they're buying
it for other reasons. We don't want any of that.
Ideally, I mean, It's easy to say that it's hard to kind of
see what will happen in practice.
But our approach is to essentially delay it as much as
possible and then quietly bring it in when it's required to
maintain that consensus and pay out the participants with the
game theoretic mechanism. Essentially, yeah, it's critical
to note as well that along with some other kind of early movers
in the Deep learning crypto space.
We're very much a minority with respect to the rest of a deep
learning community. There's at least in our
experience, quite a lot of skepticism about crypto more.
Broadly, I mean, bed and eyes but histories kind of Testament
today, you know, we were skeptical and we've obviously
from a technological and kind of ideological point.
We like it, I think it's the right way.
However, when we initially provide, when the network
initially, launches we anticipate that the majority of
the earning users will pay in Fiat and it will just simply be
swapped into tokens, the solvers, and the people
participating. On the supply side of the
network will facilitate with with tokens.
And we have huge interest from a lot of the kind of old fiery M1
miners who have lots of gpus who want to attach them to
something. But yeah, it's crucial that
there's a kind of the kind of crypto the scary crypto words
like Takin her kind of removed from the the end then deep
learning and machine learning users, which is exciting because
they're ours, this is one of those use cases which really
Bridges. The Two Worlds of web to and web
three, you know, there's our economic rationale for this
existing. There's the technology now to
enable it. Its existence, it's Dow.
Almost like a kind of execution question to a large extent, you
know, how do you just hold you get people comfortable with the
idea that I'm not using Amazon? And, you know, there's a kind of
variable price kind of concert happening here with the talking
and how the obfuscated that as much as possible and crucially
how do you have to skate it? Wait, which is decentralized
because if you very easy just to stand out some centralized API
front end which you know, just yeah automatically converts the
tokens and a centralized exchange somewhere but that
becomes that brings its own problems.
So yeah, I just add that to the pot.
So what is the roadmap look like for you guys, there's going to
be a test net early next year. Yep.
Test net. Early next year.
It won't be incentivized. Kind of torture.
Earlier and it's more to pick up the kind of do two things.
Firstly to kind of battle test, some of the, some of the tech
that we've been building internally and 22 going to get
feedback on the usability of it. Overall, that will be that
precedes our kind of incentivize test net which will be which,
yeah, well, essentially, you'll be able to train models, kind
of, you know, in anger on it. The the rate at which we move
is, It's something that we talked about a lot, you know, we
could ship something very very soon, which doesn't really give
us any meaningful feedback, but it kind of looks good because
it's like all you kind of shit something and we don't want to
fall into that trap because yeah, there's there's been lots
of things which have kind of come and gone for us where there
have been like incentives to kind of ship something super
early. So even kind of you know earlier
this year there was lots of kind of hype around the idea of doing
generative nft arts and we could have kind of provided like an
inference. And for that, you know, really
quickly, but we decided that it kind of, It kind of blows it
out, step of her principles, you know, it doesn't solve.
The big problem is not really on the way to solving the big
problem. It had lots of kind of other
things we have to build. So yeah, I guess I'm trying to
say is we're not like in a kind of immediate rush to believe
something. Tomorrow, we die were really
something, which is Meaningful, which takes time giving a
fundamental. Some of the stuff is,
particularly the zero-knowledge stuff which is, you know, which
is it was a pretty involved in terms of time.
Cool. Thank you guys.
Where can people go? To learn more about Jensen,
yesil. Jensen.
Do AI, is you're going to primary source?
We have a discard, we don't have a telegram group, but this cards
were a lot of chat happens. We're also hiring just now so if
anyone's listening to this and is interested in, you know,
building a permissionless, deep learning compute protocol than
we are, we're it and I guess. Moreover next year we should be
hosting as your knowledge machine learning Summit.
So if anyone's Particularly interested in that kind of
crossover. There's that and maybe as a
final point, maybe more for traditional kind of deep
learning or machine learning people coming was listening or
sponsoring the the New York's conference in New Orleans next
week. So, we'll bet I will both be in
Louisiana at the conference attending the talks flying the
flag for a crypto and yeah, if anyone's there, we were more
than happy to chat. Super good.
Thank you both for coming on. This was Super interesting.
We look forward to kind of seeing how this plays out with
them. The test net and the mainland
fantastic. Appreciate it.
Thanks dropping songs been a pleasure.
Yeah, thanks so much for having his own really enjoyed it.
Great questions as well. Really interesting.
Thank you. Thank you for joining us on this
week's episode. We release new episodes every
week. You can find And subscribe to
the show on iTunes Spotify, YouTube SoundCloud or wherever
you listen to podcast. And if you have a Google home or
Alexa device, you can tell it to listen to the latest episode of
the epicenter podcast, go to epicenter, .t V /, subscribe for
a full list of places where you can watch and listen, while
you're there, be sure to sign up for the newsletter.
You get new episodes in your inbox as they're released.
If you want to interact with us guests or other podcast
listeners, you can Us on Twitter and please leave us a review on
iTunes helps people find the show and we're always happy to
read them but thanks so much and we look forward to being back
next week.