That's exactly what's going on right now, which is basically
that service based architecture SOA, you know, we're we've
switched from this idea that we had to go ahead and originally
it was 1 central command to understand what the budget was.
And now that got pushed down to developers because the
developers could actually then care, understand and help
control that token. Novice is the exact same thing
100%. I will have you know I'm still
waiting for the coffee from Taiwan that you promised you
were going to send me after that blog post you.
Wrote and I'm and I'm glad you're recording it to basically
make sure that I'm held. Accountability one O 1 Yeah,
this is. No, no, you're right, I'm wrong.
The problem is that I'm actually not going this summer,
unfortunately. But if when I do, yes, there is
a specific set of farmers and a specific set of places that I
want to go to. And it's actually interesting
enough related to agents and we will get into that when you're
ready to go. But it's actually related to
agents, my friend. Are are there some of your
agentic workflows that you've got going on finding different
micro fincas around the world? Not just that, but also which
regions in Taiwan versus which regions of Japan or Korea
specifically for macho and shit like that.
What? How?
How are you getting agents to do this research?
Is it just like exploring the? Yeah, it's exploring.
So so basically, OK, so we're probably jumping ahead of her,
but like I'm using omnigents, Polly.
Sorry, Debbie, excuse me. That's the one where basically
they debate each other, right? And so this case is basically a
my ChatGPT, you know, 55 versus my Claude Opus 47, right?
And they basically debate each other.
And so basically I start, I, I already have enough like agentic
memory and skills, MD's and all this other shit that provide
like a lot of huge background on what I'm specifically looking
for anyways, right? So if you go to my blog either
like the daily jeely.com blog, the personal one, you'll
remember that there's this one asinine long Taiwanese blog
about Taiwanese coffee, right? It was supposed to be Yeah,
Yeah, exactly. Yeah.
It's supposed to be small. And then it turned out to this
basically be this long diatribe that went from like, here, all
the great coffee shops in Taiwan, to like, let me go visit
a farmer in Taiwan, in Nanto Province, in Nanto County,
specifically to go ahead and understand how he's going to
beans and how like. Yeah, like, and how used to be
Bidlon leaves, right? And then how the Taiwanese
government was switching over to convince people not to do grow
bin Laden leaves because bin Laden is a it's like a like
cigarettes, people get super addicted to it.
But also the leaves are huge, so the water actually degrades the
land underneath it. So the Taiwanese government's
trying to convince people to grow coffee.
Well the this is already too long, but the TLDR version of
this is that Nanto County is actually famous for growing tea
and 1st before growing coffee. So I've been actually using a
homogene to find me specific regions in Nanto to grow matcha.
Wait with the intention of you going there and then growing
matcha or? Partnering or working with the
farmers, working with the farmers in that area to grow the
matcha. And so some interesting tidbits
is that the oxidase when you, when you working, when you pick
the leaves, right it, it could oxidize really fast.
So you actually have to steam and dry them immediately.
So, so the problem like Japan and Korea, they both have the
infrastructure set up that next to where the leaf picking is
actually happening, they can immediately deal with the
steaming and drying OK and of the leaves.
For example, this is a good setup for coffee because coffee
in the coffee regions, they know there's like bourbon style,
honey style, whatever else, but it's all about drying the coffee
beans after you pick them and how do you do it correctly,
right? And so the for oolong and other
types of teas in Taiwan, they're all set.
That's nonto's famous for that, for that producing amazing like
for example, Juan tea, like they're, they're great at that
stuff. But for teas that oxidize fast,
like they, they don't have the infrastructure yet.
And so the idea is to actually ensure that they have the, the
steaming and the drying technology there right away #1
but then is there an area in Taiwan that actually can
actually handle the how to deal with oxidized tea?
Basically, like once you've tried, how do you process it?
There's only one area in Taiwan that actually knows how to do
it. It's a it's a, it's a region
called Sung Chao, which is near Banchao.
Banchao is like a, a major city in Taiwan.
Yeah, OK. And so ironically, that's an
area that I've been trying to convince my family to move to
like in other like to stay in Taiwan just because I love the
area a lot. It's got, it's got a lot of
great cycling, dude. Like the, the cycling there is
gorgeous. There's a lot of really, really
painfully painful hills. It's amazing.
So, so, so I'm basically getting the agents to debate with each
other to get me to dive in. And so it's showing me which
areas of social I can work with specifically, which with
farmers, which small distributors I can work with to
basically figure out how to go ahead and have them process the
oxidized tea as well. So it's like, if I'm actually
ever decide that I want to go into that business, then the
idea is like, yeah, like obviously it's a side business,
but as you can tell, I'm just a tiny bit OCD stuff.
Like that? Just a little bit, yeah, yeah,
just just a little bit. So so because of that, then the
idea is like the one of my fun projects when if I'm not cycling
would be like, OK, let's figure out how to actually like grow
high quality matcha. So obviously Japan has the lead.
Korea's infrastructure is amazing in typical fashion, the
Korea does. And so for me, I'm like, huh,
can I do this in Taiwan? Because the the final tip is
Taiwan Nanto elevation is typically isn't high enough to
grow some of the very interesting coffee beans.
Yeah, it does. So for some reason it works even
though it shouldn't exactly. You have the theory that that
also can translate to matcha, matcha.
And now with these agents debating each other, that's just
giving you all the context that you would need.
If you want to pull the trigger and say, OK, I'm going to go do
it now, I've got it, all the information I need, I can point
another agent at it and say read through this context and then
give me what I need. Yeah, so I save obviously all
the all the like, it's the old like build skills or build an
agentic memory to store all the stuff.
So I have a reference point because there's only so much
context you can compress before the thing blows up, right?
But don't forget it's like things like, OK, cool, now I
have a general idea of what's going on.
But then how do I for seek argument?
My mandarin's horrible. OK, let's be very clear.
It is disgustingly bad. OK, my at the time 5 year old
when we're in Taiwan makes fun of my Chinese accent and she's
it's fair. It's fair game.
It's not even like no, that's true, right?
Because my my Mandarin sounds like I'm speaking like
Cantonese, like I have a Cantonese accent Mandarin.
And mainly because I've loaded most of my Mandarin by watching
Jackie Chan movies that were dumped into Mandarin.
And so they had a Cantonese speaker speaking Mandarin.
And that's how I actually learned a lot of my Mandarin.
I wish I was joking. This is completely freaking
true. OK, so, so in other words, no,
I'm, I'm bad. OK.
And so now I need to go ahead and mess around with agents to
go ahead and read Mandarin properly.
And a lot of it's handwritten, not digitized, because we're
talking small farmers, small distributors, right?
So this is when I start switching over to the the
DeepSeek models, except now here we get into this, the DeepSeek
models and all the Chinese open source models are great, but
they're for simplified Chinese, not for traditional Chinese and
traditional Chinese because there's actually there's
actually more context. So you actually need more tokens
in order to process the same amount of Chinese characters for
traditional Chinese versus simplified Chinese.
OK, I forget the name of it. Those are really good.
Cool researcher, she had done all this research about creating
Taiwanese specific models. All right, LMS and how that it
was actually required more context required more tokens to
process this in essence the same amount of words.
And so there was some, I forgot who did it, but there was some
person was actually trying to do like, no, no, we'll translate
the, the traditional Chinese to simplify Chinese run against the
LMS that way and then translate it back.
And because, because then they'll be less tokens required.
And the translation actually is relatively straightforward.
It's just a direct mapping in a lot of cases you don't like need
thinking. In that case, it's like, no,
here's a simple vibe version. Here's the traditional version.
So I'm actually still testing that out.
I, I screwed up my selections of those models.
Why? Because my Mandarin sucks.
I think that's the thing here. But the idea is that I'm, I'm
trying out different things like that because then that's the
whole point. Like how do I then work with the
traditional farmer, the traditional distributor, with
the ones that actually have real longevity and real knowledge of
the topic, right. And so, so part of the reason
I'm bringing this up, by the way, is just because a lot of
people talk about how AI agents are removing jobs, right?
And, and there's obviously validity of that statement by
every stretch of imagination. But I keep trying to remind
people like, yeah, but it's like every other technology we have,
it also introduces new jobs. It also introduces a way for us
to rediscover things that we forgot using this small example
of like this, my obsession with matcha or my obsession with tea
or coffee in general, right? No, just any obsession I have,
right? Now you can just ask an agent to
go do these things that you probably would have never gotten
around to doing. No, because think about how much
research you can think of how horrible my Mandarin is, right?
And so instead I'm having fun being a geek again, But then I'm
also connecting with old farmers.
I'm connecting with old distributors.
Like this is fucking sweet. Like, so in other words, I get
to go ahead and work with people that normally I wouldn't even
have a clue to where to even start now, right?
Which is. Which is to say that's.
That's cool. Yeah, that is very cool that.
And so you've got these you mentioned there's the setup
where the agents will debate each other and they debate each
other just because you're saying, hey, look, give me a
reason not to do this, like try and.
Find. No, it's not even that.
No, no, no, it's it's actually better than that.
It's not like, because most of the time, at least with the
prompts I use or the context I'm using, it's not about doing or
not doing. No, you, you've known me.
Not unknown Demetrius, I'm always going to go do it right,
right. But my problem isn't am I going
to do it or not. My problem is like, am I going
to basically slam into a wall or how thick that wall is when I
slam into it right? That that's usually my problem.
So so you can tell like basic how I look at the tech is
actually you can still look at it from an optimistic
perspective, right? And so for me, it's not about
like, should I do it or not should I do it?
It's more like what's the right way to do it?
Like. And so in other words, as you
can probably hear or tell from the way I'm describing
everything, my obsession is high quality going back to the roots
of actual like tea making, right?
I'm not an expert in that at all, but I respect the quality.
I respect the art. I respect the cultural
significance of of it. That that I can say.
Yeah, I can vibe with that. Right.
And so because of that, then the idea is that what I'm asking the
agents is basically look, I'm giving the agents that context.
So then why they're debating with each other, like, oh, well
then what region of Taiwan am I looking at?
What's the elevation, what's the pH, what's the soil quality?
Am I dealing with lots of, for example, like the theory is if
we were to grow matcha in Taiwan, it would be more floral
based, especially in Nanto County.
Why? Because that's what Nanto's
famous for. Like their teas have a lot of
floral, lot of lychee like. So for example, like coffee,
that coffee block that I, I sent you, right?
That one, his coffee beans were notorious, were famous,
notorious, the wrong word famous partially because there was very
strong, very, very strong hints of lychee in the coffee beans
themselves. Right.
And the reason why is because literally the lychees were
growing wild in his coffee farm. So literally like no joke, we
when we got there to try out the coffee, the grandma was
literally just giving us lychees for free.
And just like, don't you want us to pay for them?
No, no, they grow everywhere. They're literally a weed at this
point. Like I'm like, I've never heard
lychees referred to as weeds, but OK, like right, but that's
literally the context. It's like 05 be able to find
cool things like that. And so that's the that's the
attempt to discover. That's the attempt to like try
to like somebody like myself who has the obsession of it or the
OCD Ness, but not necessarily the knowledge.
Letting the two systems debate with each other provide me the
additional information to say to know what other questions to
ask. Yeah, because when you read
through that debate, then you go, oh, I didn't even think
about that. That is such a great trick that
I recognize and you know what? I one of the actually toys that
I've been making, and I don't know why this isn't a common UX
for dealing with agents is I now have created a little widget
that you can highlight any piece of text, whether you're inside
of the chat window with the agents or you're on a website.
And it will pop up a little prompt box and it will read the
context of whatever the either the paragraph or the whole page
that you're on. And you can prompt whatever ask
a question to it just about. Because a lot of times what I
recognize with myself is that I don't want to continue this
conversation in the actual chat that I'm having.
I want to shoot off and do this on a different thread because
it's just like something that isn't necessarily needed to
pollute this conversation with the agent that I'm having.
But I still want to know like, hey, that you see, you reference
this thing here. What does that mean?
Go deeper. Give me more context on this
specific thing, but not in the main chat.
So it's like little sidebar conversations and I've got this.
I, I use this tool called clearance to read like markdown
files. And so now I created a little
right hand sidebar where you can have a side chat with an agent.
OK, I I know this is definitely wasn't done on purpose for
everybody that's listening to us right now, but you you wonderful
Segway to Omniton by the way, because OK, so for example, I
did this already. So I when I first started asking
the question about Machos and things like that, like what
regions, I learned a lot about more about the regions of Japan
for starters, and then the regions of Korea.
All right. But then where my end goal was
to figure out if I could do it in Nonto County in Taiwan.
All right. But of course, I, as I'm to your
point, I'm learning more about these reasons to Japan and these
reasons in Korea. I'm like oh this is really cool
so I want to know more but now. So I really wish there was a way
to fork my conversation. Yes, that's it.
So homage, it has the ability for me to fork.
So that's what I did. I basically had the
conversation, I forked it off that I allowed myself to
continue one conversation specifically about the Japan and
Korea areas, while the other one the the main line, I just
redirected the Nanto County. And so they both had the
original context by which we we had the conversation with.
But now I can have two very separate conversations so I can
learn more. Why?
Because I'm fortunate enough for it, because I live in Seattle.
No, I'm not being paid for this, but there's direct on Alaska
Airlines from Seattle to Tokyo, Seattle to Seoul now.
So I'm going. So yeah, so I hopefully maybe
they'll give me something for that.
But in that the reason I called that I was like.
Sponsored by Sponsored. Right.
But no, it's not I wish I'm not that we're we're both not big.
We're both not big enough for that right.
But. Exactly.
We need to become the true travel bloggers.
Exactly. Yeah, we're.
Just we're not we're not travel influencers.
So unfortunately that's not going to work.
But not yet. There you go.
True. Point taken.
Yeah, we we have goals, right? But nevertheless, the idea being
now I'll go directly to Korea, I'll go directly to Japan so I
can explore the regions myself. Yeah.
So that's why I started going down.
So I forked the conversation specifically so I could go do
that, yeah. But I don't understand why that
isn't a common UX pattern with these agents.
And specifically for me it's like I have this one sentence or
these 5 words that appear together in my context and I
want to fork them out and have a separate thread on that specific
piece. Like explain more but I haven't
seen any of the labs create this as a integral part of how we
interact with agents and so it it makes me think like maybe I'm
the only one who cares about. That no, no, no, no, no, no, no,
no, no, no, no. You're, you're look, when we
started all this, this entire AI community, right?
Think about how chat windows are still the even after all this
time, chat windows are still the way we work with things even our
sticking when we decide to use CLI and terminals, it's still
basically ACLI chat window That's basically all this for
all intensive purposes, right? Just that we might have some
Jason stuff shoved in there. Now that's that's the extent of
the difference, right? The reason why is because, and
you actually see harnesses like being created right now, like a
bunch of start-ups, a bunch, even the major lab, they all
create harnesses themselves, right?
And the reason why is because it's like look up to this point.
And this is like our definitely our learnings within database
itself, right? We all had like 5 terminal
windows open, 6 browser windows open.
Yeah, we're all going back and forth, back and forth.
You deep dive on a particular topic.
I'm coding something for Delta Rust, right?
Like so I can get off of T for a second, right.
So I'm, I'm coding something for Delta Rust.
I'm trying to understand the latest version of arrow or and
data fusion because they keep changing.
Hats off to the convenience. This is not a complaint.
This is just, but they do they do change things.
So we're trying to keep Delta Rust up to date, right?
And so when we're trying to make those changes, then I need to
run some terminal windows to I need to figure out.
What's the current version? Another terminal set is creating
the tests harnesses that I want to go create another browser
window is giving me oh, what's the latest security contacts?
What's the latest like, you know, security alerts associated
to each and everyone, the 20 different dependencies we have
that, that we have to go through, right?
And, and what are you doing? You're opening multiple windows
and you're losing context, no pun intended, right?
And you're, you're losing all of the connections because you've
got so many windows, so many terminal windows open, right?
So you know how Amazon made the idea of like the each team
should only have like 2 beats of boxes, right?
All right, the whole reason that was Crave was because it was the
idea that each team should only have five to seven people
reporting to it. OK, so each manager should only
have 5:00 to 7:00 because anything greater than that, they
basically start losing context and again, pun intended.
OK, well, this was determined, not Amazon might have made it
famous, but it was actually a military thing.
Like military, military command. In terms of units, teams
understood that each commander could only logistically handle
five to seven people reporting to them in order to be
effective. Right.
And so I'm not military, but I I am.
I used to be King County OPS, the search and rescue OPS.
Excuse me. I'm sorry.
I'm trying to remember SNR look, this is how tired I am.
OK, dude. Search and rescue operations.
So we learned that from so SNR OPS learned that from obviously
military. And then the idea is that this
is a a time value tradition, like an understanding and so so
in the tech world, everybody thought about two pizza boxes,
but basically this is a well understood logistical structure
within the military. OK.
And so cool. The reason I bring that up is
because, well, then in a lot of ways, if you consider an agent,
I'm almost like a junior developer that you're managing
once you go beyond 5. Yeah, yeah.
Once you go with seven, you're losing track of everything right
And so more than anything else, the things like to your point
exactly, but I want to go off and do that it's actually going
to make you lose context even more so right.
Part of the reason why we we've had basically built this idea of
forking was actually not originally that idea you talked
about, by the way, like to be very clear, like it was more
about literally we're coding. So you fork.
So that way you can literally go ahead and try out different
ideas, but had the initial same context.
But to your point exactly, coding and actually having a
real conversation isn't actually that much different.
Coding happens to be just around code, but it's still an
abstraction layer, right? And so conversations are just
another abstraction layer. Like everyone's like, well,
Cody, no, that's real code. I'm like, no, are you writing
this thing in binary or assembly?
No. OK, well then stop talking to
me. It's an abstraction layer, OK.
And so the idea is that if you've got all these different
things floating around right, you will lose context.
You will not remember why you asked in the first place.
So there's so there's a lot of research being done by the labs
and to an extent even a lot of the harnesses, which is like how
do you basically have now an agent fleet and the fleet manage
itself. Right.
And there's various like by the time this podcast comes out,
which is only a week or two, I'm sure everything will change.
So I'm not going to bother with the pretense of knowing what's
going to happen with fleets. But irrelevant, whether you you
call the fleet or you talk about an agent or there's a, some
other layer, the the whole reason we built what we built
was because we understood that this was going to be a problem
either way. And just as importantly, or if
not in many cases more importantly, the models keep
changing. The hardest keep changing.
I want to be able to go ahead and move all that context, all
the agentic memory, all that configuration.
I don't want to rebuild it so now I can use the latest open AI
model or the latest DeepSeek model or the latest anthropic
model or whatever it is I'm using.
I want this one layer that says whatever harness you're using,
whatever model you're using, it'll flow through irrelevant.
Where I get a little bit confused there is the harnesses
are optimized for certain models, right?
I guess if you're building a harness, you're building it for
a model and there's certain things that the labs are going
to be doing on their own that's like the inner harness and then
you have the outer harness and the inner harness.
They're RL ING that model so that it works really well with
their harness. 100%. How do you like envision these
hot swaps in a world where you know that?
Oh, because that that's actually why we value Omnigen as an open
source project to begin with, right?
Because the idea is that like for example, even just building
it, we notice for example, what deems to be simple but actually
changes people's workflows is like approvals, like the
approval process. I'm not going to get this quite
right. So I'm going to stay vague.
So that way I can, I can not make it sound like 1's better
than the other because that's not true at all.
OK, But the process is slightly different between harness A and
harness B and the process is slightly different between model
A and model B, just to be very vague.
So that way, again, I'm not calling anybody out, right?
And again, it's not even calling because like I said, it's not
like 1's better or one's worse. It's just purely they're
different. That's all this is OK.
So if if you look in the Omni code base, that's we actually
reflect that, right? And so then that way we're
actually leveraging what model A or harness A is really good at
because like the creation object isn't about going and saying
we're a better harness than everybody else.
We're not. No, that's why we call ourselves
a meta harness because we're saying no, no, we're on top of
this. So that way you, the individual,
the person that's using these things, all that configuration,
all that context, all that agent of memory that is portable, that
you have control over, that you can flow, right?
You can compose these different ideas where basically I want to
go ahead and run like just be a little bit more specific now,
forsake argument. I was using Opus 4.8 and
actually 4.7 of them implicate a real example to generate.
The code Owen better to be truthful.
Also, I'm glad I glad I have a kindred spirit here.
Right. Way more. 4.8 is not.
I'm not. Stoked.
Yeah, yeah, yeah, yeah, yeah. That's a whole other thing.
But let's not debate that, OK? I don't want to throw a kick in
my ass, OK, Right. So right.
But I'm using global force of it.
And then, but then I was using GBD 5.4 at the time to do the
test cases and fuck yeah, like it was wonderful because then I
got the 2 using Poly to debate each other in terms of high
quality coding. And then I and then I got one to
actually go ahead and and generate the test cases for me
as well, which is amazing, right?
And then maybe a week later I would have switched over to say,
you know what chat GVD 5.5. I like their code base a little
better. So I'm using that now.
I'll stick to Alpos 47 for the of the debate it I'll use maybe
sonnet to to go ahead and do the the test case because those are
relatively easier to once you put the original template.
I don't actually need something as powerful to generate the test
cases. I just need to sort of update
them. So this is perfectly fine.
And then when you know when Fable and all that stuff came
out that rewrote everything and then Fable got removed and then
we wrote everything right. The whole point is that I'm just
using Omni jets. I didn't care.
I just redirected it and like hell, I'm using π for a bunch of
my test cases now because it's cheap, like really dirt cheap.
So I'm like, why not? It's really good.
So just use that in some cases, right?
So that's most the point. Like it just gives you that
flexibility to say like, OK, you start off with everybody telling
you you're supposed to use the most advanced 1.
And I get it that you're tempted to do that.
And if you if you're allowed toka maxing, you will do that,
right? But the reality is this stuff
starting to get expensive. Right.
Prices are going up or discounts are going down or both, right.
And so my take on the whole thing's like, OK, well then try
to find the right model or the right harness for the job.
And so maybe even if you are initially using the more
advanced models to build it, a lot of the maintenance doesn't
require the most advanced model at all.
In fact, the most advanced models more times than not
actually make things literally more complicated.
Like you're going, I only needed simple test cases.
I didn't need this gargantuan thing that is impossible for me
to read. Right.
That's a feature, not a bug. That's why Anthropic's worth
it's whatever trillion dollars they are.
Very no, but see, that's what it's like in the end, you get my
trip, right? Like it's, it's about going
ahead. And then because after all, like
I'm definitely not mean to get in the debate just right now,
but like during, for example, DNA summit during the open lake
house contributors meeting, right?
One of the sessions that I actually end up helping to lead
was the deal with coding AI slump, right?
And one of the call outs we had, because the way we run open lake
house was very much in the thing of like, yeah, I might have led
the discussion, but it was meant for everybody to chat.
I'm not interested in like me telling people what to do.
That's not interesting at all. I'm we were getting all these
people in the, the open lake house and the AI community
specifically to go to debate. Like, like we're marking on
competing projects, whatever, I don't care.
But like, we all are suffering from same version of, of
insanity in one form or another. So can we help each other out on
that part at least, right. And so for example, there was a
whole debate like on the on various distribution lists,
including like Apache Iceberg on the delta list, whatever I was
like created by versus assisted by.
So created by is the person who did it assisted by is like it's
almost 47 or Sonnet or whatever it is that you were using at the
time, right? And the call out more than
anything else unequivocally was that you needed to.
In the end, everybody agrees with the idea that it is still
the individual, the user, that is the one that submitted the
code that there still is from a TLDR perspective, who's the neck
you're going to choke? It's not, I'm not choking the
neck of an agent. I'm not choking the neck of
Anthropic. I'm choking the neck of the
person that's contributing the code, right.
And so ultimately, like whatever we're phrasing as in terms of
creative biases by or anything else, the point is that me, the
developer, whether it's developing code or developing a
business project or whatever it is, you, it's still me, it's
still you, it's still the person that's making that decision that
how ultimately succeeds or fails.
So whether it's the the code for Delta rust or whether it's, you
know, figuring out which region of non telecom county that I'm I
want to grow macho leave, right. Ultimately it's like it's me,
right. The onus has to be on me alone,
right. And so knowing how to follow a
thread, knowing what these harness can do, knowing that you
can move your context, your thought processes across the
different models so that way you yourself can succeed.
Ultimately, that's it. And and I think you, Dimitra,
you and I have actually had this conversation on the side.
You've also did this as part of your own set of podcasts where
ultimately the main reason people fail when they're using
agents is because they they zero shot it and they have no context
applied at all. They there's no free thinking.
So people are like, well, of course it's garbage that came
out of it. Like, yeah, so that means these
ancients suck. I'm like, well, it's the old
garbage in, garbage out. Like if you, if you're not
putting any effort into this, it's it's going to be garbage
coming out. It doesn't matter how good the
agent is. It doesn't matter how good the
model is. Have you been playing with the
different? Because basically what I
understand is, hey, you bring your own model and you can bring
your own harness and then you have this meta harness above it
that is unopinionated. And it's like, cool, you want to
come with your open claw over here and then your Hermes.
You want to come with your Hermes?
You want to toss an integration with a llama?
Have fun at it by all means, yeah?
How have you and how have you been seeing that working in
like? Is there a world where you're
recognizing that now? What is important is
understanding what each one of these is best at.
We built homogene in about two weeks by the way, to give you
some context. All right, again, pun intended.
We literally used homogene to help build homogene, OK, like so
everything you see, we we basically used homogene itself
to basically build it, improve it the whole kin Kaboodle.
The reason I bring that up is because look we have a ton of
benchmarks right but before the two weeks we data bricks
engineering product we we were we were all suffering from the
same thing right We were suffering from like which wall
should we use this week? Should I use Kimi now?
Should I use right? Should I be looking more into
the open source ones right should like so if you like, if
you look at data bricks itself right, you'll notice that
there's a whole bunch of different models like that all
natively run within data bricks. They're all governed that's
great. I love that fact.
But then exactly to your point, I don't know which one's better.
And so, yeah, if you even go to the database website, we
actually have this thing where we update monthly, forget the
exact URL, but basically say like, here's the best ones based
on what we see this month, right?
And then we'll obviously for documentation, for debate, for
financial, like whatever, we just, we actually write down all
the steps because we're like, this is what we think in real
world scenarios. And you'll notice that the real
world scenarios often do not match benchmarks, right?
And so that's what got us to build Omni in the 1st place
because we were basically, we actually had a version of Omni
internally. We obviously can't share that
one. But the basically use that as
our starting point, mainly because we were recognizing that
every single engineer needed the ability to have that flexibility
to switch, right? Because they realized that maybe
that particular Co base that they were working on was better
with X while different team, same org, hell, same team, like
same division, everything realizing Oh no, but I'm doing
why? And it's better with this.
And so the idea of us trying to dictate you're supposed to only
use one like we were never, we're going to do that in the
1st place just because of the nature of data bricks.
But like it just the whole thing fell on his face almost
immediately. And So what we did.
The model provider could be down, which we see all the time.
Oh my gosh, yeah, yeah, yeah. Claude was just down yesterday.
So I was when I was just trying to run my stuff.
So fortunately, not a big deal. Flip it automatically flipped
over to codecs and all right, I'm back up and running.
Yippee, KA, barely, barely a blip of the whole thing, right.
But so ultimately it was more like, OK, do I have the
flexibility to change? Do I have the flexibility to, to
all, but also not to change but control my costs?
So like with the knowledge that we actually have policies put
right in there, would you say OK policies as simple as like, OK,
can I run this command? OK, that's one of them.
But like a like a shell command, right?
For example, you and I were joking about how just starting
off, you had run a skill that basically shut off your redirect
your browser, right? Well, guess what?
Like Aldrin has it built in where you have a policy that
says, no, you actually have to prove a such thing.
So it can't do that, right? But also even budgets, which is
like, OK, if this thing's going to cost you more than X dollars
or X cents or whatever it is, guess what you have to prove
before you're allowed running it, right?
So just things like that, which allow you have control over the
situation. And then the final little tidbit
is that inside of all the conversations you and I've just
had, right? The one thing is that it's a,
I'm still very like the OR a developer centric, right?
But in order to be successful more times than not, what we
have to do is we have to work with your team.
You want to share your learnings for your team.
And so that's the the coup de grace for on that, which is
like, OK, it's like a Google Doc type interface where basically
now, hey, I ran this agent or had this Polly do this
particular debate. Why don't you take a look, you
know, team leader X. And then not only can they see
what's being generated, they can then themselves run on that
session themselves because you shared it with them and they can
run the session, say, wait, can you run this particular test?
Did you take an account of X? Did you take an account of like
the new updated data fusion or whatever else that you need to
do? Right.
And then basically now that process continues on.
So you're actually sharing the workload together so the teams
can actually work together really well.
Right. Yeah, that is so cool.
And actually funny thing that you were mentioning, another one
of my little side projects, toy projects that I'm creating for
myself is a widget that goes up into the corner of your screen
and it just calculates how much money you've spent.
There you go. By day or by week.
And every time you hit another $100 increment, it goes cha
ching and confetti goes everywhere.
And this. Dude you should so do APR into
Omnagent to do that. Seriously because we already
have the basis for calculation. Do APR baby like I will use use
Omnagent to create the PR for you because that's all I've
done. Yeah, you know what else I did
though? I put a little GIF so all of a
sudden a gift pops up of like someone making it rain.
Make it so it's like 5 different random gifts that you can get
and it's like making it rain. And so you recognize, and I'm
created this with because I was at Fin OPS X which is this whole
conference around budgets and all they care about are
tokenomics. That's the whole thing now.
Nobody cares about dashboards for your clouds.
Oh God, yeah, yeah, 100. Percent, dude, we got that
figured out right. All they care about are
tokenomics and how people are spending their tokens.
And what I realized is that a lot of folks are creating
dashboards so that their CF OS or whoever in the finance
department can see how much the engineers themselves are
spending. But what I didn't see a lot of
is pushing that visibility onto the end users.
And I'm like, if I knew that I was spending 1000 bucks a day, I
might be a little bit more conscientious about the cost.
So wonderful call out like so I still remember because I'm a
long time Microsoft guy, OK, before all this.
And then when we switched to the cloud, the whole premise is that
like this idea of CapEx to OpEx, right, like we're changing cable
expenditures topics. In that process of OpEx, it was
the into DevOps, meaning that the developers actually had to
understand how much money they were spending on the
infrastructure that they're building.
Right. So if there weren't like
needlessly running a small query and spinning up a A50 node
Hadoop cluster when their laptop could have solved the problem,
right? So that's exactly what's going
on right now, which is basically that service based architecture,
SOA, you know where we've switched from this idea that we
had to go ahead and originally it was 1 central command to
understand what the budget was. And now that got pushed down to
developers because the developers could actually then
care, understand and help control that token.
Novice is the exact same thing 100.
Percent that's so true it's the same motion and you're just
we're reliving it, but it's a little bit different now because
what is funny and I think one of the hardest questions that I saw
people struggling with at the event was how do you know what
token spend is valuable versus wasteful?
Yeah and so there's all kinds of tricks that you can do on these
like these details and one of them being hey, choose a
different model. If it's not that big of or
complex of a problem, just use a better model, use an open source
model, use whatever. But still you have certain
engineers that are burning a ton of tokens and for some sub set
of those engineers that burn is worth it.
And then for another subset of those engineers that burn is not
worth it. And it's like what are they
getting done? No.
No, it's 100%. So this goes back to the idea
where it's actually just like with the switch from CapEx
topics, right? You actually need both.
You need to give the developers the tools to understand what
they're doing and what the spend is, but you also need to have
some central governance story so you can understand what's going
on right? Like so like you've, I've come
to you, we've actually had this conversations like why do we
have, you know, for example, uni catalog and why did we database
introduce uni a gateway. That's the whole point.
It's two sides of the same coin. Like homogen is there for the
developer so that they themselves understand they
themselves can apply policies. But then we have Uni a gateway
to to for centralized form, which says a look, yeah, you can
use Opus 4748, whichever one you're doing 47, right?
But wait, like for example, if you were to look at my
questions, especially in terms of when I was still beginning
with Ongen, when I was testing it out, there will be an absurd
number of which Bagel is better, Montreal versus New York, right.
And so if I was a central person, they'd be like, no, this
Denny guy, we we got. A token.
This is a ridiculous amount of tokens being spent on this
particular question. OK?
Especially because everybody knows Montreal bagels are
better. Yes, I see it.
All right, bye. But you get my drift.
Like, obviously there was a reason for what I did it, but if
I still do it today, oh, my goodness.
Yeah. They would just lock into Uniaga
anyway. No, we feel like no, you're not
allowed doing this. Shifting budget over.
Yeah, yeah. They're shifting budgets.
They're like, OK, like, no, daddy, you're allowed using pie.
You're allowed using like you're allowed.
Maybe if we're, if, if they're, if you're lucky, we'll let you
use Kimi just for the sake of doing it.
But not even a later version of Kimi, like an older version,
right? But like Opus, Oh hell no, we're
never going to let you use that, right?
So. You know what, on this idea of
choosing the models and choosing the harnesses and all of that,
where I see the biggest friction is when I have to manually make
the decision because I am a little bit lazy and I just kind
of go with whatever I've got on, you know?
And so I'm not going to, especially if it's my company's
money in a way, I'm going to be like, whatever, let's see if it.
Works. I can just blow the money and
ask the bagel. Question.
It's going to be like let's use the smallest model.
It doesn't work and then I'm going to go back and try and go
to the next smallest and cheapest and then it still
doesn't work. And then I've wasted time, which
is the whole reason I thought I was using these coding agents
was to save myself time. And so it's like I see that I
will overcompensate with bigger models just for the fear of a
smaller model. It's probably not going to be
able to get this, even though it totally will get it.
Like it's like, oh, you just changed some names in the code
base and it went through and you know, like control FT, you did
not need a 4.7 for that. Right.
So I don't think that's completely solved yet because
this is sort of like the same idea like the journey from CapEx
topics like we don't know when, when one should be applied where
like like in other words, preventing you from using it in
the first place is very limiting.
We don't want to do that, but we also don't want to get to a
point where you've blown 1,000,000 tokens for which color
of mauve is the best, OK, right. And so we're at a point where
especially with meta harnesses and especially with things like
in the AI Gateway, that now we at least have visibility and
some form of control. One of the things that we've
been experimenting on, and you're going to see some of it
show up actually in object itself because we've been
experimenting internally first because we want to see how many
engineers are going to kill us, right?
Is the idea of where do we auto choose for you based on the
questions that's being asked right?
But that's itself is hard and that itself may require us to
use an LM to make the choice. So do we just shift the token
usage that way? Right?
But this is where like a good old fashioned machine learning
may kick in for us. You know, like you know how.
RL is back in vogue again. Which is it?
Never should have not been invoked, by the way.
But you get my trip, right? RL is back in vogue.
It's like, oh, that's cheaper, That's faster, that's better.
Yeah, we've been saying this pretty much the entire time
guys. But yeah, so goes back to the
same idea. Machine learning is back in
vocal. I'm like, no, a lot of these
types of questions though, like in terms of the complexity of
the question, you can literally use good old fashioned machine
learning to make the choices. So that's what I'm saying.
We can. We're definitely seeing trends
like that to help with that process.
And ultimately, however it's done, the key thing is to your
point is like is that for example, with homogene there
isn't changing the code base to choose.
It's literally click and it automatically chooses for and
you have the the drop down list of which one to go with in the
1st place for starters. And So what we're hoping for by
the time we're done with everything is that once we get
into some good patterns around that, we actually tell you
upfront we're going to switch to Model X.
Here's the reason why and let it run.
As long as it if it fails it will retry with a better model
then I'm stoked because I don't have to spend my time being like
this. Doesn't really look that good.
Right, right, right, right. But then it goes back to like,
This is why are we like Polly so much?
And and Debbie? Because there's different
versions of what it means to say it doesn't work, if you get my
drift. Right.
So that's why we like the idea of the the agents debating each
other, because then you can walk away, let the two agents fight
each other out. And basically, yeah, because
they have enough context, they have enough memory, they can
actually figure out, OK, there's there's a reasonable assertion
here that actually because they're debating each other, you
will get a result that actually works.
Or at least even if it doesn't work, at least now you have the
basis of why it doesn't work. So it left because in the end,
especially with how powerful are these open source models are
these days? For me, a lot of time it's.
It's less about which model I'm using nowadays, and it's a lot
more about did I provide the data, the context, the
background, the configuration, the memory.
The big thing is, did I set up the correct loops?
Did I give it what it needed to just be able to run for an hour
so that I don't have to come back?
And that is one of them. I think that's a great point is
being able to debate and recognize what good looks like
so that you can trust, OK, it's a smaller, cheaper model, but I
know that the debate is going to happen and if it doesn't pass
the sniff test, then it will go back and it will loop again.
Right. And then this goes and why do we
see yet another resurgence of evals all over again for.
Precisely that we're still talking about evals.
We have to, because in the end there has to be some numerical,
There has to be a number that allows us to state if what was
created actually per evaluation actually is.
It's the old definition, what is good, it goes back to that
definition over and over again, right?
And so in the end, like I said, it's, it's becoming more and
more of an obvious that it's less about the models.
This is not a knock on any of the models.
They're all really good. Now at the point that's what I'm
just trying to get AT. And it's much more about, OK,
now that I know the models good, what else is missing?
And then the evals will actually help us figure that out.
Yeah, yeah, yeah, that. How can I make this harness top
notch exactly, 100%. I think the auto model choosing
is the way to go because if it is left on to the developer,
they're never going to choose a shitty model.
Or no, No, they're always going to choose the best model, yeah.
It's a psychological thing. You're like, I need to do this
because what if it encounter something in this run that it
needs the power? Let's just let's do it just in
case it's a few extra cents or dollars.
Depending on what, yeah, I mean, if a few extra cents, we're
probably OK with that. It's the few extra dollars $100.
And unequivocally, no, you're not wrong.
You're absolutely right. And I think like this is part of
that story for us. Like it's, that's part of the
story why we made it open source because we're going, look, we're
not pretending we have all the answers.
We're just pretending. We're just stating the fact that
this is the problem we faced internally.
We want to share this with the world.
So that way y'all can like we can all work together to get to
that solution that you're talking about, which is like,
how do we do a proper auto model, right?
How do we, how do we do a proper thing?
So that is absolutely 100% vendor agnostic.
How you know what I kind of question with because certain
harnesses have certain ways of expressing their harness, right?
And I'm thinking specifically like in Hermes and Open Claw,
you have the soul, but then in Cloud code you don't have that.
How does this meta harness of Omni agent act when you're
running these different shapes? Of harnesses.
So the the TLDR is the root you, you ruined the DS Π project
right? Yeah, right.
The root of a lot of our learnings is basic from DSPI in
the 1st place, which is the idea of how do you, how do I go ahead
and allow myself to functionally run these queries against all
these different models that actually instinct that actually
have different ways of running in the 1st place, right?
Different parameters, different weights, different everything,
right. And so with that, the TLDR is we
started looking at the systems that we're integrating with and
we tried our best to basically take an account of each one.
And so that way any integration we have basically takes an
account of like what each one is strong at at the code level,
right at the conflict level. But I would be remiss with if I
didn't also basically mention the facts.
Like that's actually why we're reaching out to a lot of these,
both the model makers and also the harness of makers, because
we're going to look, we're not pretending we got it right.
We're pretending we got a good start.
That's all we're saying, right? We'd love to work with you to
make sure we have configured it the best to represent, you know,
your particular model, your particular harness the best.
Because, yeah, we're, we we're not pretending we understand at
all. We're just simply saying that as
you can probably guess, we've used a lot of them.
Yeah. So we we've incorporated as much
of that knowledge as we possibly can.
But that still doesn't replace the fact that there are people
in these other who crave the harnesses themselves or crave
the models themselves. They would know better than us.
Fair. And again, why is an open source
project? So you get to look at the code,
tell us what we screwed up on. We want to go ahead and work
with you to figure out how to make it better.
Simple as that. You know what gives me this like
visual in my head of almost like you have your different
harnesses. And so the outer harness we
could consider like the MCP servers that I set up, maybe I
do memory in a certain way, etcetera, etcetera.
And then you've got the meta harness, which is the it's
outside of all these outer harnesses.
And that is where you have this way to almost like be an
overlord of all your agents. So like if you're looking at a a
circular thing like the inner harness and the outer.
Right. Basically the outside layer of
the onion. Yeah, yeah, yeah.
Yeah, exactly. It's it's the onion or it's the
earth with the different like rocks.
That core mantle shelf. Yeah, the magma, all of that.
So that's kind of how I I visualize it.
Is that the way that you would? Is that a good way of thinking
of it? Oh.
Absolutely. I mean, look, everything we do
business wise, development wise, whatever else, we're always
trying to figure out ways to abstract it, right?
The reason for the abstraction is because it gives us
flexibility. It gives U.S.
Open standards, it gives us the ability to go ahead and like
simplify said business logic, right?
So that way you're able to focus on the real problem, not on it
was a little problem. I mean, if if you and I were
having discussion five years ago when ChatGPT just came out
right, you and I would have definitely been debating on
things like what were the weights?
What were the hyperparameters used?
What's the? Prompt right, right.
Are we still having those debates now?
Some damn good AI researchers, of course they are, because
they're trying to improve the models, right.
But you and I, as users of these systems, we're not having that
debate anymore. We were barely qualified to have
that debate then. We're certainly not qualified to
have that debate now, right? And so, but what, what can we
do? We can talk about like how to
effectively use agents, right? Well, remember what I just said,
effectively use agents, not effectively use LMS, right?
Because we've already accepted the fact that like whether it
was called compound AI systems before or whatever else
afterwards, the reality is like the agent is basically not just
the LM, it's actually a lot of the things that are around it as
well, right? And then us saying harnesses,
yes, exactly to your point, like whether it's an MCP server, it's
ACLI, it's reading adjacent from somewhere and XML from
somewhere, hitting a database, whatever, whatever it's doing
doesn't matter, right? The point is that we're trying
to figure out how to make these things portable and stateful.
Well, guess what? That is an old fashioned
business database problem that we've been doing since the 60s,
right? Like, why are databases back?
Like why? Why do we talk about like
Serverless Postcrest? Why did Databricks talk about
Lake Base and Neon? Why?
Because literally we have to have cheap stateful memory for
these agents to run, right? As simple as that, right?
Well then if I have cheap stateful, then I'm like, OK,
what's stateful? The best thing is a database.
Cool. Why is poster sequel?
Why is there resurgence? I would argue there never was a
drop in the 1st place. That's what I would argue
actually. But irrelevant of like you want
to call resurgence or not, the premise is that I need something
that's trustworthy, that can keep state.
That's it, that, that. And so that's never left us.
And I mean, like, remember I was like former sequel server.
So like I'm like, yeah, yeah, like the database is still
there. It's no matter what we've done,
no matter how much we've talked about lake houses, which
obviously I'm a big fan of, and we're talking about agents,
which is the to the shore these days, we still need something
called state. Well, that state is databases.
And then guess what? That's if you can make it
cheaper, you're more successful. Well, that's how these agents
have run. That's how all these systems
work where we're just extracting the way and then storing what
needs to be stored at IE state. So that way you can be
effective. And so and everything when it
comes to like how we started with databases before B i.e., TL
pipelines, it's the same action plan, right?
With agents right now, which is like, how do I modularize this?
How do I make it portable? And then for a bunch of us, how
do I make it open so that way it's easy for you to move from
one to the other?