Something we were talking about a little bit earlier around Open
the Eyes, release of guaranteed capacity.
That just happened, right? I find it really interesting
because it's not advertised as a discount mechanism.
It's, it's advertised as a, you know, guaranteed capacity, you
know, and, and we're just. Saying like, if you need this,
you're always going to have it, so you're never going to be
without resources. We're not going to go down on
you. Exactly.
But from our standpoint, we haven't really seen capacity
issues. So it's almost like they're
they're solving a problem that doesn't exist yet.
That has to be very hard for you to look at and it probably makes
your job like you have a fixed price you can't go over.
I mean, there's lots of ways to address that.
At the same time you, you see the subscription model going
away and you know, usage base or a combination of that, those
things happening. And that's the reason why
because it's fixed, prices just don't work anymore.
All right. I'm here with Josh working at
Superhuman leading the fin OPS. I want to hear about your
journey into tokenomics. It's been about a three-year
journey overall, but but yeah, prior to Superhuman, and I'm at
the time when I joined them, they were Grammarly.
I was at AWS and, you know, had no exposure to AI when I was
really there outside of just managing some of the billing
aspects of it. Come to Grammar Lane, you know,
we have some hosted models that I was helping, you know,
distribute the cost and, and report on.
And then on early 2023, we started using this new thing
called Azure Open AI. So I started kind of poking
around. I'm like, I think this is going
to be kind of big. So I want to understand like if
there's a area where I can help. So I started kind of poking
around, asking questions and found all this is like a fixed
capacity that we're, you know, we want to make sure we're
utilizing enough of it. And, and, you know, also
understanding, oh, if we don't have enough is this will create
an outage for some of our products.
So we got it. So introduced some new concepts
like I've never been involved with making decisions that could
actually cause an outage on one of our products, right.
That wasn't like a thin up thing, but it kind of fell into
my scope. So, but as I started doing that,
I realized after going to some Fin OPS Foundation events that I
was one of the only ones doing that type of work back then.
So I certainly encourage anyone to, you know, even definitely in
this day and time with AI, just most practitioners, if you poke
around hard enough, you'll find a area where you can drop a lot
of impact for sure and help. But it's funny how you started
poking around and then you realized, wait a minute,
resource management is kind of seems like my job in this whole
new field. Yeah, resource management, a
very, very expensive resource, yeah.
Especially in those days, yeah, maybe talk to me about the
trends and how you saw the trends dropping and then now
reversing of the trend of the cost and.
You look at me a year or two ago, I'm like, oh, costs are
just falling off a Cliff and now it's just going to be the the
vendors fighting for market share and it's going to be raced
to the bottom. So yeah, in the first probably
year or two sold over 80% drop in cost per token.
And I think some of that was the vendors getting more mature and
the infrastructure behind the scenes and also them trying to
gain grab market share. And then over the, I'll say
starting in, you know, really last year, but over that
following year, from 2024 to 2025, the trend started to
change and, and it whether it started where it was not, you
know, maybe they were about the same to now with, you know, for
example, GBT 54IS like double the cost of five one.
I think Opus just came out today and it's like double the cost.
On top of that, you've got other skews like web search and and
reasoning and of these other aspects that are just
compounding the cost. So now we're the trends are is
not only is volume like a hockey stick right now, but costs are
rising too. No matter like what is pizza
elsewhere, rates are going up. For sure.
And how did the way that you were doing your job back when
you thought in the next 6 months, is it just trending to 0
versus now where you're kind of looking at it and you're saying
it may go up, but it definitely is not going down?
It's going to just, it's, it's going to exponentially go up.
It's that's your prediction you're calling in here.
It's only going up. It's only going volume is not
going to go down. If it's not going up then as a
business, then you know that's a bad for us, right.
So you know, and ideally we want more and more users using the
products, but we just want it to scale efficiently.
But the, you know, I would say with external LLM that's, that's
it's becoming in my opinion of larger and larger financial risk
because of the the rate trends that we're seeing here.
So volume and we know is going up.
We can't have pricing also doubling just about every time
too. And then that's where I'm
looking at exponential increases.
So fascinating. So that's making you nervous
looking at how it's been performing and recognizing that
the volumes going to continue to grow even if it stays at this
price that we're at right now. That's scary.
But there's a non 0 probability that we're looking at the prices
also increasing. And so that forces you now to
look at different options. How does their outlook change?
The bulk of of Grammarly's suggestions are based on open
source models. And that is where I really want
us to go just from a cost aspect, you know, but we're also
an AI company, so we have a special set of teams and skills
that can can make that happen. Most companies don't have that.
But I would say right now, I mean, we, we have two sides of
this AI coin. We have our production cost
which far outweighs our AI coding internal cost.
I would say right now most companies are dealing with the
AI coding side of it and that's probably what they should be
dealing with. But for us, it's like both
sides, we've got them really managed to make sure and that
that both are efficient and measuring the value in some way.
Yeah, because the product is powered by LMS, yes.
Again, mostly open source, but more and more as you want to
have very high velocity of releasing product and such.
I mean those Frontier models are great for that.
So that's where the growth is happening on the external alum
side. And you also mentioned to me
before we hit record on the hidden costs around the price
increases just with the models. Can you talk a bit more about?
That Oh yeah. So when you are looking at say,
you know, maybe using GBT 51 for whatever reason and, and five
four comes out and let's say you have data residency, U.S. data
residency requirements. So now you, you see the rates
and you start using it and you're like, you know what, we
do want to use this and we see that as two times the cost, but
we see the value here. But then you don't notice the
fact that there's a 10% data residency charge on top of that
that didn't exist with the five one model.
And that's happening with the four Opus 4, I think 46 or 47
and later. And basically Anthropic and Open
AI are both adding this premium to US based models.
You're already seeing this before this with the
hyperscalers, AWS and GCP and such.
But now it's that's another premium you're paying a lot of,
I believe a lot of companies are kind of overlooking.
Yeah, you don't realize that until you later need it.
And then you go, oh, we're getting charged for that.
Yeah. So whenever you're doing your
estimations and comparing these models, you just make sure you
look at the fine print about the this data residency piece
because at 10% adds up. Yeah.
And Speaking of estimations, I know that you've built a cool
cost calculator. You vibed it.
I vibed it. I mean it.
So I'm not an engineer. And I was telling him before
that, you know, before I started a grammar, I had really no
experience with AI at all. And which is my kind of poking
around in curiosity that kind of put me where I am today.
So I was took the initiative to say, OK, I'm going to space out
some time in my days to spend time building some agents to
help when I was practice. And I really, I knew I wanted to
build some type of calculator and didn't really know what it
was going to look like. And I built like a HTML version
and then a CLI version and it's using, We have the LLM proxy
that already has pricing in the back end of it.
So it's just grabbing that information and I did it in
about 15 minutes. And people use it.
It's just pretty new, but I've had our, some people, a part of
our evaluation team start playing around with it and they
were very happy with it. And what this is done, you know,
when you're evaluating and I'm just learning this myself too,
more about this process for the workflow at least is that they
are building these prototypes and up till this calculator was
here, they didn't really have a good way to go and estimate
cost. They would build the prototypes,
get a sense of the maybe what the request per second will be
and you know, different steps this agent's taking and the
token types and all of them and token volume.
And they would send it to me in a spreadsheet and then I'd run
the numbers and estimate it for them.
And now they can just hit the CLI and it's going to bring in
OK for this model, this type of agent would call system out and
it's safe to the calculator. And then they bring in another
model, do the same thing, save to the calculator.
And then they now they understand before they even
start doing an experiment that what the costs are going to be
and if it's even scalable. And she made the comment to me
that there have been times where they did a prototype, went to do
the experiment, was too expensive and had to start over
again. So not only is is this helping
us keep costs down, but also speeding up development velocity
because they don't have to go back to the board.
They know what the costs are going to be before they get past
the prototype typing stage. Yeah.
And that's just an example of how you recognized this is
something that's happening. Can I make it easier on the
folks who are closer to this problem?
How can we get these tooling? I don't want to be in a
spreadsheet doing estimates anymore.
That's not where my true value is.
My true value is is getting maybe building, thinking about
these tools and building them and then putting them in the
people's hands that are actually doing the work to make the tool,
the, the agents or what have you.
And then, you know, again, shift left, get that in early as
possible. I would love for the product
people to start maybe playing around with us if they have the
data to do it right. So I want to be as early in
development cycle as possible. And this really helps with that.
And honestly, this FP and a, my counterpart in FP and a
hopefully she sees this that, you know, kind of said, Hey,
could we book some kind of calculator?
And I'm like, you know, I think I can, I don't think it'd be too
bad. And and then, you know, about a
week later I had this thing. So, you know, it wasn't even
totally like me saying I need to go do this.
It was like, you know, collaboration with the
financing. Oh, I think we should we start
working toward this. And that that piqued my
curiosity and the luck I could. Do it.
And I like how the evaluation team gets to now understand this
and that's probably one of the core metrics that they're
looking at as they're evaluating if this is feasible.
Exactly. I know another piece of that is
support request to go to our channel, they can send in a
spreadsheet or what have you if they somebody wants.
And then my support bot is going to use my calculator I built to
calculate that. And so I that's one thing I
don't have to go and do anymore. OK.
So tell me more about where you're thinking of your ability
to add products or empower the engineering team, the rest of
the organization with this knowledge because they're closer
to those problems. So I know where the data is, I
know how to kind of structure like what things should properly
cost and and so that's kind of the magic I kind of have for
myself and I just need to take that.
And I think for all 5 thinnest practitioners probably have the
same knowledge, but they need to take that put it into some kind
of tooling. So it's closer to the
engineering in the development process.
And then shift their focus up in the organizational and be
focused more on strategy and and and actually just enabling the
teams to make better decisions moving forward on the products,
no matter, you know, even if it is going to be something
expensive that they have understanding all.
Yeah. Well, this is we have a
reasoning behind this because you know, product market fit or
what have you. So if everybody can move forward
from engineering up to in product to obviously the C-Suite
with making decisions on knowing what the costs are ahead of
time, we're going to be in a much better state as a business.
And if we looked back at again, going through that journey that
you've had of when you first were encountering AI on Azure
with Open AI and then versus now and the maturity that you feel
like you've been able to come into.
Can you explain some of the things that now you're like we
could not live without XYZ? Or say our telemetry streams.
Telemetry streams. From from my perspective, I
built some telemetry streams. So we have LLM proxy as home
build that that gives me the token volume by calling service.
And then a feature is another, you know, step below a calling
service. So like, for example, we have
our main AI chat and there's a bunch of connectors for that.
And that's a bunch of features, thousands of features, all those
connectors. So I take that information, bump
it into our, our, our cost reporting platform, which is a
third party and, and then allocate cost based on the open
AI bill. And I can do that for Bedrock
and, and any other vendor that we're, we're bringing into that,
that tool. And all it's doing is looking at
the token type. And then, you know, looking at
that cause versus the token volume of each service and all
these other metadata that's using it and then proportionally
allocating it by token type service and all those things.
So without that we would have, we would not have visibility to
actually what these agents. Cost.
And then are you looking at spikes and trying to figure out
what's going on? Why did we just spend a lot of
money here? Yes, I've got agents that are
starting to do that a little bit for me.
But I would say our we, we don't have a lot of spikes scales with
business hours. Just about all our production
stuff just scales with business hours.
Now we'll see a trend all of a sudden, you know, cash tokens
are going down and costs are going up.
Like we'll see things like that we need to pick up.
But it's very, very rare that we have like a big spike outside
of, you know, research and use cases or experiments that you
would expect that. But usually if it is a giant
spike, it's usually from my experiment that maybe, you know,
was a little higher than expected or, you know, research
that I knew about ahead of time. Speaking of research, you had
mentioned before we hit record about the research team and how
they're being empowered also by some of the stuff that you're
doing. Can you explain that?
Well, yeah, I'm, I'm trying to actually just today sent
something to them about sent them the calculator and said,
hey, this is going to be of use to you.
But no, they, I think what we're talking about is around
optimization where and it was touched on in the keynote today
too. And I thought it was a great,
great presentation and saying, you know, optimization in the
cloud was just, you know, optimizing resources and, you
know, buying savings plans in our eyes and things like that,
that fin OPS going to handle. Or it's just more
straightforward and it doesn't really impact the product at the
end of the day. When with LLMS, whenever you're
changing, OK, you know, cashing rates or I'm trying to reduce
output tokens and input tokens and all these things and trying
to use cheaper models that actually impacts the output and
the user experience. Drastically.
So it is so much more nuanced. So I have found much more
success with optimization when this research LED optimization
where our researchers are looking into saying, OK, I think
we could probably use maybe a mini model combined with a
standard model and not impact our latency, which latency is a
very important aspect there. You know, all our stuff needs to
be pretty much sub one second latency.
So you know, it's not going to impact latency and we're going
to still getting really good outputs and saving money.
So they, they, you know, maybe they'll come out with the
initial kind of launch of, of an agent and then from there they
look for ways to optimize it. We do the same thing with our
open source LLMS and we launched speculative decoding last year
and it saved us a bunch of money and on the on the open source
models and we're just trying to try to try to learn from those,
those teams and optimizations they did there.
And how can we apply this to these external LLMS?
And and again, it's all usually the big heavy optimizations are,
you know, really started from the research side.
And the product research is what fascinates me the most on that
because it's not like AI researchers.
It's not. It's, as you mentioned, the
product is so affected by any little changes in the AI side of
the house because you're in a very unique situation where your
product is driven by AI at the end of the day.
And so you have to make sure that that's the highest quality
product you can put out. But also, your main job is
making sure that it you're just not using and bringing a bazooka
every time you need to do the smallest little comma update or
whatever. Exactly.
And we have data behind the decisions we've made and we've
you know and I think most companies like this, but we just
like the data. You know, right now we have the
cost data and now we need to better tie that to revenue and
then other and experiments and you actually get experiment IDs
into the telemetry streams. I was talking about then seeing
what the cost of this experiment was and was it worth it or not.
So like it's, you know, while we've been in this and I've been
doing this work for three years now, we're still like getting to
that value phase is hard, especially as you acquire
companies and different products and different measurements, but.
Bring them into the fold. When you say experiments you
mean doing small little rollouts on?
Or do you mean like behind close?
We have a very mature experimentation platform where
we're actually, we'll be running live experiments on, you know,
they usually AB test to see, you know, how maybe a new new change
or new feature impacts. It could be in product
engagement type of rates. And and that's how we kind of
decide on or should this thing go full scale or do we need to
go back to the drawing board? And again, it's interesting to
me because it is product type of features, but then inside of the
core product, there's probably a lot of different experiments
that you're running with all the models or with the ways that
you're making those models useful.
It's the same product for me as the end user.
I'm still getting if it's we're talking about Grammarly.
I'm still getting suggestions on my grammar, but you behind the
scenes are running these experiments that I may or may
not know about. You may not know about and all
of a sudden you're engaging with it in a, you know, slightly more
volume and you may not even realize it, but then, you know,
our, our people are picking up on, OK, we got more, more, more
accepts here for this suggestion.
Or, or they actually, you know, the clicks on this specific
agent actually increased over time or the amount of time turns
they had with this agent increased.
Like little things like that that indicate, hey, this is
actually making a meaningful difference for our customers.
And you want to be able to, just to close the circle on this, as
you mentioned before, when you have those experiments,
recognize the cost of those experiments.
Exactly. Tie that to the cost
fluctuations that we're that I'm seeing on my side because right
now it I'll see these spikes and I do see spikes.
It's usually around experiments and I have to do a lot of
research or have you know, I'll let them go do it for me to
figure out what exactly is causing this.
And even then, the correlation is light.
It's really hard to to find a correlation.
So you know, we've got, we got a couple teams collaborating to
bring that data into our our cost reporting so we we can
actually measure the cost of those experiments.
Because I imagine the teams aren't saying like, hey, I'm
going to run this experiment now, like just in case.
Sometimes, sometimes they'll. Give you a heads up.
I've been very blessed with with a good relationship with a lot
of the the teams that do these experiments.
So the larger ones, they usually come to me and either to
estimate what this is going to cost or just kind of let me
know, hey, we're going to run this thing.
So I know about the big ones, but then they're still small
ones that are supposed to be small, but then they end up
being a little bigger than expected, right?
But overall, you know, there's really good communication
between us. So on those times where you see
the spikes or when you're getting a heads up, is that
something that now you're expecting folks to have a better
idea of because of the cost calculator that you created?
Yeah. I mean, ideally they would, they
wouldn't need me to do those estimates beforehand or at least
they'll have an idea of what the costs are going to be before
they run the experiment. And I haven't built this into it
yet, but I'd like to actually have that going into a doc, a
coded doc, you know, one of our company.
But I'd like for whatever submissions that they have in
this, this tool. And they can, I'm assuming they,
I'll probably have something where they can actually choose,
OK, this is something to submit. And then it goes to a dock where
we see all the estimations in aggregate and then can pick up
on any red flags or at least do better planning.
OK, we know we're running this thing during this date or within
the next few months. We usually don't have dates
until, you know, pretty late into the process and we can
better plan on, you know, what our finances or costs are going
to be down the line. And also compare it with the
real cost to make sure that hey is our calculations, are we
projecting, is my calculator. Broken or is, you know, request
rates are a little, you know, off.
So yeah, that's true. And what are the different
inputs that you ask folks to give you on this cost
calculator? So average request per second
more minute and then average request size.
So when I say request size, that's input tokens, output
tokens, cashed input. And these are all things that I
said they're kind of building the prototype and, and testing
out the, the outputs that's coming.
You know that they they get those token counts from as a
response body from the the open AI or Anthropic, whatever you're
using. And I wasn't clear when you were
talking about how you have the research, the product research
team, then they have this knowledge of we can do XYZ with
these models or we can cache this, or we can set up a
different architecture that will be a better product experience.
Does that then get propagated through the different teams so
that whatever next team knows oh I now have the ability to make
my experiment much cheaper? We could be better about that,
we could be better about advertising how teams are doing
this. Well, one other issue there
though is you know, when you look at our mail product, the
way they use LLMS is way different than than the way say
Grammarly uses LLMS or even our new Go product.
So that is one kind of issue. Now I think there should, could,
should be more sharing across the, the product or the business
units and, and those products because I'm sure they're still
learnings there. But again, you know, these
acquisitions happen not that long ago.
So it's, it's going to take time for those things to be sure.
And I think it's probably part of my job to make sure that
those are broad and broad kind of broadly told to the company
so that other people can't take advantage of.
But that's something I think we can improve.
Well, it does feel like what you're saying too around a lot
of times folks will try and let's use the easy example of
just caching certain results or certain tokens so that you can
spend less. But then it ends up having a
worse product experience some of the time.
And so you want to make sure that the teams aren't using that
type of architecture at the cost of like having less token cost
I. Guess Oh, exactly.
And I mean, I first and foremost is always the best product for
our customers. Like we, we are not going to
sacrifice that, especially in this very competitive area that
we're in. We're not going to sacrifice
that at all. But it's more about OK, be
dealing the best product. But what are some of the things
that that are we can still deliver the best product, but
also save money on the back end. So the team is just because
these researchers have much deeper understanding of these
LLMS, they're able to go in and tweak those certain things or
know the direction to go. And then they are in in the
position to go in and evaluate the outputs and kind of go
through that process to make sure it's all good.
There's no way a fin OPS person could could do that even if they
knew how to adjust the LLMS and make it cheaper.
Evaluating the outputs in a way a data scientist or a researcher
or a company with computational linguist as well.
They that's a whole another skill set that, you know, fin
OPS people aren't going to have. So that's what makes this a lot
harder to do. I mean, I've also tried to use
claw to optimize claw and our, our like clawed API products
and, and it gives good advice, but it's usually still just not
quite, quite there. We've already, you know, tried
it or the impact wasn't, you know, that large, but that could
be another Ave. is just use AI to optimize AI.
So tell me about how this reserving capacity kind of cost
benefit analysis goes in your head.
Do you get cheaper tokens if you reserve more capacity?
And is that something you even look at, or is it something that
has kind of gone out of fashion? So early, early on, that was
really the only reserving capacity as you're up in the eye
of the time, PT use was the only way to get low consistent
latency. And as I mentioned with with
Grammarly, you know, sub one second latency is just hugely
important for our product. So yeah, we need a low
consistent latency. So yeah, in the beginning I was
managing capacity, having to buy capacity for the peak.
And if we didn't, you know, we would scale beyond our our peak
and and have an outage. But then over time, we did see
token rates come down for that capacity, but it was a really
difficult thing to manage. One, the actual utilization of
the capacity you could get, you can get metrics to know what
your utilization was, but understanding it beforehand was
really hard because you know, everything's measured in these
units, abstracted units that Azure measures them in.
And depending on the request size or shape that the each
agent is sending, that utilization is going to be a
little different. So knowing that ahead of time
was tough. RL and proxy team actually built
a way they call it shadow traffic where it would mimic the
traffic of this thing that was going to be coming out to see
what would in load test the actual capacity and see what
the, you know, peak tokens per minute were.
And then that's how we would estimate, OK, we need to add
this much more capacity for this new thing that's about to launch
or you know, we can actually reduce capacity because things
have come down. So that was back in that day,
but it was just still very difficult to measure ahead of
time and it was a lot of this mathematical lean limbo and
exhausting. Then I want to say it was like
late 2024, early 2025, Open the Eye came out with priority
processing. So this has a late latency
SLA's, availability SLA's. It is double the cost as far as
from a rate perspective as the standard RAID.
But again, we needed a low latency.
What attracted me to it was I didn't have to manage capacity
anymore because this is completely on demand And so it
scales with our usage. Our usage scales with with
business hours. So then I did the numbers and
looked at, OK, if we had the even looked at if we had an
ideal amount of reserve PT use and then also with our proxy
sent some traffic to priority processing.
Like what you know, could that save us money in general, just
comparing PT US, you know, fully using PT us versus priority
processing. And really it came down to you
could see some savings with the minis and nanos of the world,
but you couldn't, it wasn't enough to, to need to manage
that capacity with the standard size models.
There was no savings even with using PT us.
And this is a, this is a year or so ago, it may have changed, but
I doubt it. So priority processing was an
absolute no brainer. Not on top of just making my
life easier. It also allowed it makes much
easier to allocate cost because again, it's just very difficult
to understand how much of a single service he's using of, of
this PTU. Because yeah, you could look at
token volume of raw, but tokens are not equal across agents.
And I took so it was just very complex of this and I improved
our visibility, saved us some money at that time at least and
and and it made my life a lot easier.
I can only imagine you doing this complex math like what's
the right blend of PTU or just straight token.
Pretty large people sheets out floating around.
And then you're like, wait for quality of life.
Let's just do it on this there. That should be one of the
factors too. Like you don't have to do this
crazy math anymore. You know what you're going to
get. And it also if it's the same
more or less the same price then it's a no brainer.
Exactly, exactly. And then something we were
talking about a little bit earlier around Open the Eyes,
release of guaranteed capacity. That just happened, right?
I find it really interesting because it's not advertised as a
discount mechanism. It's, it's advertised as a, you
know, guaranteed capacity, you know, and, and I'm just.
Saying like, if you need this, you're always going to have it,
so you're never going to be without resources.
We're not going to go down on you.
Exactly. But from our standpoint, we
haven't really seen capacity issues.
So it's almost like they're they're solving a problem that
doesn't exist yet. Or they're foreshadowing what's
coming. So I'm very interested to see
how this play out. It could also be first a first
step toward a savings planner RI kind of construct.
But but yeah, nonetheless, I, I see a lot of financial risk in
these external loan providers and, and prices are going to
continue to rise and or they're going to want multi year
commitments. Because with the guaranteed
capacity, you can't do that for a month, right?
Oh. No, no, they're multi year is is
what I believe that they're looking for there.
Yeah, yeah. Which I don't know 100% for
certain, but yeah, that's, that's my impression.
Yeah. That is something that now
again, you're thinking about it and you're starting to realize
these cost benefit analysis and saying, huh, if we're good
enough now we're never going to change.
But then later down the line, if it starts to get a little shaky,
now you have to recognize, is it worth it for us to get this
guaranteed capacity? Or as you mentioned earlier,
too, like they're slipping in a lot of new costs like the data
residency, but now there's this new cost on this.
And so they're almost upselling you in every way, shape and
form. Exactly.
And I think it's really hard from my perspective at least to
do any kind of multi year commitment against, you know, in
this AI world that we live in that is that's really hard to
look at all the changes in the market and say, OK, we're going
to be using, you know, said provider for three years.
So it's just it's too fast-paced.
So yeah. But if there are people that are
probably in position to, you know, be able to do that, our
proxy also gives us the ability to kind of use whomever we want
to use, which is it's nice having a Bender agnostic proxy
for that.