Hi everyone, My name is Patrick Akeel and joining me today is
Sura Hosseini, Co founder over at ORC, where they enable AI
product teams to put AI solutions in production.
And that's exactly what we talked about today, how to be in
control in production with AI solutions with regards to
observability, the specifics of Gen.
AI with regards to models and rate limits and much, much more.
Fascinating conversation in my opinion.
So enjoy from your perspective in this world of AI that we now
live in, things escalated I think a year or two ago,
specifically with regards to Gen.
AI and Chatti BT kind of coming out to the public.
We now have a lot of fluff, hype, potentially also really
good things in production running.
What for you is really valuable in this field of AI and AI
product development and what is still kind of unexplored or more
fluff and hype territory? So what I see a lot of times is
actually I, I will try to answer it in a different way than tell
me if I didn't answer your question right, happy then to
change. But it's a lot of laziness as
well, right? Whenever you're reading up on
stuff or new frameworks come out, new libraries and it always
turns into what this is out with our framework, how you build and
a lead generator or how you build an, a personalized e-mail
outreach. It's always those 2-3 same
examples, which is actually intellectual laziness, right?
And what I'm always trying to sift through is like a very
product specific AI features that teams build and ship that
is so specific to their product and not these boilerplate stuff
like, Oh yeah, sales outreach agent or you know, an e-mail
outreach agent. And everyone has that as an
example. So it's more about what is what
I'm always looking for is in product teams, what is the
strategic differentiator where we can infuse AI into our
product that makes it truly different, right?
Like because the type of data we have or the type of users we
have or the type of workflow we support, what is the creative
rethinking we can do around that to infuse that with AI?
That gets me going and excited. And whenever I read, oh, this is
how we do ICP outreach or, you know, lead generation outreach.
Yeah, then that's kind of a, a fluff for me.
And another one is, you know, all these people on LinkedIn
writing stuff and then you open their profile and you see like
they're now an AI strategist, but strategist.
But six months ago, they were still like 12 years of project
manager and some boring bank, and now they're an AI
strategist. That's also like, yeah, is
someone actually building and shipping or are they just
regurgitating stuff that they found or they have just checked
GPT, write some blogs for them to post on LinkedIn to become an
influencer. Yeah.
So it's either do you have the track record or are you doing
something really creative with AI, right?
Molding it in your favour instead of just the standard
right to personalized e-mail outreach.
Because my inbox is full of them, right?
Like my spam box at this at any given day is like around 150
called emails a day with yeah these AI generated crap that no
one reads. It's a, it's a sad reality that
we get there right. And I, I understand from let's
say the perspective of someone that sees this out on the market
that really wants to actually contribute, how do you
differentiate yourself? That's probably a question
they've asked themselves, but the answer is the same, right?
Yeah. Act like you're an expert, or
start creating content or start playing around with it and
actually see how you can deliver value.
The easiest part is to act the part and not actually do the
thing. Do the work, put in the effort
to try and understand deliver value.
Figure out use cases in your personal productivity, and then
see what use cases can be solved in in the outside world.
So maybe people take the easy Rd.
And maybe just on what's fluff and what's not fluff, right?
So are people actually building and shipping or just talking
about it, right? It's easy to filter those out.
But also important one is whoever thinks AI is this magic
pill you take and all of a sudden your life changes, right?
Whoever makes that promise in this current day and age, right?
I'm not saying five years down the road what AI will be able to
do, but now it's never a magic pill.
Like you just turn it on and you're all you start getting
leads and seals and you know, everything blows up.
So you see that also often right where it's being sold as just
take this magic pill and your whole life will be better after
this. Just turn our on our AI sales
agent and your whole funnel will be filled up with qualified
leads. Yeah.
So that's also important to see us fluff I.
Feel like. Same people saying I'll take
this pill and you'll lose weight.
Yeah, it's never the case, right?
It's a it's a grind to really do something.
I mean, it's always been a thing.
The the example that you give, take this pill and you'll lose
weight. Like that's what people want.
Yeah. And that now is kind of out
there with Ozempic. But we won't go in there like
that. That's what people are looking
for. And when they're there, they
will take it. That's the easiest way.
Well, even with an Ozempic, you need to change your lifestyle.
Yeah, you need to work out, you need to do strength training.
So it's not like, oh, I'll take a shot and my life changes,
right? So yeah, yeah, it's really
experimenting, testing, making mistakes, right?
It's almost like building a startup, like getting your
organization going with AI. It's almost as building with a
startup. You have idea, you test it, it
works, it doesn't work. You roll it back, you change it,
right. So yeah, it's never a magic pill
for anyone, I think. Have you seen any common use
cases that have been put into production with regards to AI?
Stuff that actually works, even though the data might be
specific to an industry or an organization by using the tools
in this way, like. That's a solid use case.
Yeah, already right. A couple have been proven, which
I also kind of believe so one is of course code, right, coding.
We at this moment, right, like all our engineers have to be AI
first. So we don't hire juniors
anymore. It's either matured Meteors or
definitely seniors because at any given time they're
orchestrating, you know, four or five coding agents that actually
output high quality work. And our platform is really,
really complex. So it's not like, oh, just, you
know, create a landing page, but it still does a really good job
and it improves our efficiency, right?
So coding is already clear that that works.
Customer support, right? That's clearly one that works.
It's also why those are always the examples that are being used
because those are more matured and have proven that they
actually work. Yeah.
So like in big buckets, those are the two buckets that already
prove value if implemented correctly.
Right. So with coding, your people need
to know what they're doing. The orchestrators of the agents
need to know the architecture, the vision of the product so
they can assess the quality of the output, right.
So you can't. Again, it's not a magic pill.
You. I cannot as a non-technical
founder, turn on five coding agents and then develop a
business critical scalable platform.
So the senior people orchestrating them need to know
what they're doing. It's just that they now all of a
sudden have these tireless team members that can output the, the
work and they are the quality assurance for the architecture
and scalability of the platform. Same with customer support.
It's not just, oh, we'll upload all our FAQs in a, in a chat bot
and then it's solved, right, Because you will have knowledge
gaps. The data is outdated.
So you are, you do need to change your processes and change
some roles of people and make other people responsible for
maintaining the knowledge gaps. But how do you have visibility
on the knowledge gaps of your AI chat bot?
And do you give the AI chat bot support bot access to different
APIs so they can do a refund or, you know, do some trigger some
specific processes? Yeah, all that you need to
solve, right. Again, there's no magic pill
that then does that for you. Yeah, I have seen people and
especially now that MCP came out as kind of this new integration
block that are afraid of doing that right, giving a model then
access to refunding. And like I, I get it from an
information gathering standpoint.
If I'm having a production issue and I have a multitude of
systems, I would love to have one interface that goes to all
the systems and it gives me a diagnosis.
Yeah. And I can go back and forth with
regards to one interface that just gets the information from
all systems, but I'm in the driver's seat.
Yeah. If I were to then be like, yeah,
I'm a customer support engineer, and I can see issues with
regards to an order and then an automatic refund kicks in.
And I was like, maybe that's not the right solution, but
something else decided to do that.
That's where people are a bit more fearful.
No. And that's it's all about
control, right? Do are you in control?
Control of the costs of the performance of the data, of the
actions that the AI is doing. So how do you give someone a
sense of control? Right.
The same way that a dashboard in a car has been invented over
time because you as a driver, it is proven.
OK, I need to have understand how fast am I going?
How's my engine doing? Do I have my lights on or not?
Right. There's some information you
need. And that's why a dashboard has
evolved into what it is. Yeah.
We're now in this change. Where people start putting AI in
production and then they realize, oh, I don't have
control, right? Or I'm not in control or this
use case sounds cool, right? Giving refunds, but I need to be
in control. We need to have auditability.
There needs to be maybe a human in the loop.
But if the human is in the loop, the human needs to have the
information to say yes or no, right?
Can I just say, hey, can I make a refund?
Because seven questions will come after that.
Like to for what? What's the reasoning, right?
So here's that. Here's my information why I
think this person needs a refund.
Do you think? Right.
So an entirely new what we call agent control tower, but it's
like human agent interface now needs to be invented, right?
How does that look like air traffic control tower in a, in
a, in a airport, right? Like, how does the air traffic
controller feel in power, in control, that they're not going
to make mistakes and create accidents?
Well, that is now organizations are going to need these agent
control towers where an individual can orchestrate a
swarm of agents. And it's different per use case,
right? But it's like the agents are
going to get stuck. They're going to throw
exceptions, they're going to ask for approvals.
But how do you feed that to the human?
So first of all, they're not flooded with the information
because they cannot read 20,000 logs per second.
And yeah, so you have the just in time and management by
exception principles, right. Everything has already been
invented in, in management and business administration is like,
what's the just in time flow you need?
What is the management by exception flows you need?
So an orchestrator can do their work.
Yeah, that's now the phase we're in where now the the next stage
is starting to become uncomfortable for people,
forcing us to invent new concepts.
Yeah, I want to get into like the maturity of organizations.
What maturity you want to have as an organization before you go
there. But before we do, let's touch on
a little bit of what you're doing with ORC, because I think
we already hit some of the surface there.
Oh sure. So at ORC we we provide an AI
engineering platform. So it's a SAS that we build it,
we maintain it for teams where engineering teams and product
teams who have a vision, right, who who want to infuse either
existing or new products with generative AI in a scalable way.
So they use our platform and middleware in there to develop
hypotheses, test and experiment with their hypotheses, evaluate
how their use cases are going to perform next, put it in
production, but in production to do granular rollouts, Canary
releases, AB testing. So that's you know what whatever
AI feature you have shipped in production acts and works the
way you intended it again, control and then observability
monitoring evaluations needed there.
And then finally the human in the loop, right.
So the product managers, domain experts, right?
They're not all Python engineers.
How can they be involved in this whole process to give feedback,
to make corrections, to identify hallucinations?
Now teams are patching this together, either with some
random open source libraries or they build their own solutions.
Well, we bring this as a fully managed, well integrated
platform. So the teams can just focus on
building their features instead of maintaining this whole
enablement layer underneath. Yeah.
Because one thing teams never have enough of is resources,
right? So the whoever they can get on
board and the right AI engineers, they can find, you
want them to be working on your features and not maintaining,
let's say, the pipelines under the ground.
So that's where we then come in. Yeah.
And then finally, because of our AI gauge where we're integrated
with all the hyper scalers, all the cloud providers, all the
model providers and we unify that into a single API.
So teams first of all are not locked into a single provider
and they can actually operate multi cloud out-of-the-box.
And and now with all the geopolitical stuff and data
residency concerns, they can even decide like which providers
do we use for which entity, in which region, etcetera.
Again, control, control, control that they need.
Yeah. I think that's mainly like if I
look at software development life cycle, we have strong
conventions and those conventions can still differ in
organizations, but the tooling is there to kind of adhere to
those conventions to put them in place or we don't reinvent the
wheel there necessarily. And from what I hear you're
saying, you're trying to be one of the tools in the tool belt
there. And a lot of people are trying
to figure out, do I build my own or do I indeed grab something
off the shelf, which I think validly.
So with Gen. AI in production, we've had I
think the most out of control situations in production, right?
Because I've had many demos where people tell me, this is
our product. We do XY and Z and then
something happens and they're like, well, it's still
generative AI. So there is a level of
hallucination. That's not what I want to hear,
especially if I'm buying a product like I want, I want
guarantees. I do want that level of control.
I don't want someone to say, well, it's a thing and it just
hallucinates, so I fully understand that people are
looking for that. Yeah, of course.
You know, you want to make the margin of error as small as
possible. Yeah, but if you had humans
doing that job, you would not be shocked if one of the employees
would make a mistake, right? You would not hold them
accountable for 100% perfection. Yeah, most of the time depends
on which field, but most fields you know, you're not going to
fire your people if they make, you know, one in 1000 errors,
right. But you, you do have that kind
of tendency to when it comes to AI or self driving cars as well,
right? Like how many people are getting
killed by normal cars versus self driving cars.
But still, right, 1 is already too many.
So that's kind of also a mind shift that needs to happen.
But indeed, in in software delivery, as you mentioned, a
lot of the concepts are, you know, decided on, right.
You need code control, you need pipelines, you need CICD, you
need a regression testing. No one questions that.
And then either even some of those you buy or build.
But now you're starting adding these little black boxes of
unpredictability in your software.
Yeah, right. That behind the scenes even get
updated without you knowing it, right.
Like open AI behind and they can just update their model behind
an API and you would not even know it.
And all of a sudden your use case starts acting differently.
So the life cycle management, which is now not being really
discussed becomes a problem. Everyone is already happy if
they get Apoc out the door that they can demo to management and
get some applauses. But no one has thought about,
OK, well, how do we put it in production and how do we get
into then the continuous delivery and improvement flow,
right. You also would as a with your
background in the product management, etcetera, you know,
like it's getting it out there the first time.
That's when the the work actually starts because then you
start getting feedback and learnings and edge cases and
weird behaviors. So how do you set yourself up
for the cycles of learning insights, changing your
hypothesis, testing them, putting it on my back in
production? And now what you see with teams
is then they get totally stuck, right?
Everyone start trembling over each other and tripping and
Excel sheets with the color-coded corrections getting
emailed back and forward, the engineers going crazy because
it's not, you know, solidified yet the process.
So that's also where we try to come in is giving you that end
to end process. So you can do this continuous
delivery almost like CICD for Gen.
AI right? Yeah.
Is that then what the a resilient production ready
solution looks like with regards to an in control state.
So I deploy something I have insights with regards to, is it
guard reels, is it boundaries, is it performance?
Yeah, everything with regards to.
Observability definitely. I mean again related back to
normal software, right? You will check how fast is it
working, what's the latency? Why is this specific call taking
so long? Oh, I don't know.
We have some, you know, we don't have an index in the database
for this one. That's why it's so slow.
We need to improve it. You will have the same analogous
challenges with Gen. AI, right?
Like why is this model getting slow over time?
Why are we hitting our rate limits?
Do we need to do and do orchestration, retries,
fallbacks? The models are not up 99.999%.
So how do you build in resiliency Cost is a compared to
normal traditional AP is it's it's very different than Gen.
AI, right? You can literally get yourself
into negative margins by building the wrong AI feature.
That was one of we were speaking to a prospect who they do it
like a billion revenue a year and they were considering to and
they build APOC of a feature that does some really cool
recommendations. But he was like, if we would
turn this on in production and half of our users would actually
use it, Yeah. Then we would spend 500 million
a year on AI tokens, which is half of your revenue.
Yeah, just on that one feature, which is not even a killer
feature that will change your life as a either user or as a
business. But it was recommendation half
of their revenue, right. So cost is, for example,
something that's really important and who is responsible
for cost? In a lot of matured
organizations, the product manager, right?
They own the PNL of their product or their features.
So they need to know, hey, how much is it costing me to ship
this feature versus what is it getting us?
So again, control right from multiple dimensions you need.
Interesting. Yeah, I feel like a lot of
people are seeing where they can deliver value indeed.
But then if it's worth it, that's a whole another question,
right? I've seen many proof of concepts
with regards to, I mean, the environment that I used to be
in, which is a bank because it's a really cool technology and we
don't want to be behind. So we're going to do proof of
concepts to see where we can deliver value.
And if it's a yes, then usually we should look into scaling
things. But then through that you also
can congest the system because only a few things can have
support from the right engineers and actually make it to
production. And if you just go wide with
regards to use cases, you don't actually see it.
OK, everything has value, but what is the cost and what is the
business case? In the end of the day, I feel
like there is a lot of value in reducing manual labour where you
just have documents, PDFs and people are going from that to
filling it in. In the systems I've seen
hundreds of pages. Manually analysed to be then put
into a system. I feel like there is like a very
big use case. The sooner you can have those
data points kind of automatically pre filled it
human in the loop with regards to kind of an assessment for I
principle as you will. If the first two is are AI and
then go through like you can increase productivity in that
way and that is more of a cost reduction exercise.
But for me, that's a very tangible use case.
Yes, and at the same time, but that's like my strategy
consultant had on immediately will be like, but was that
process needed in the 1st place, right.
So sometimes even just blank campus, blank slate, rethinking
the whole process of why, why, how did we end up in this
situation with this workflow and then see what we can do with AI
instead of just trying to patch it with AI Because yeah, you're
just trying to fix a shitty process to begin with, right?
Like, like we're now in going through some funding processes
at the moment and then I'm starting being confronted and
I'm super allergic by all these inefficient processes, right?
Like we literally, I have to drive to one note tree to pick
up a physical book, yeah, of the shareholders and then drive it
to a different note tree and they need that booklet.
We need to go everywhere with our IDs, my Co, Vander and me to
identify ourselves in person even though they have everything
about us already. Endless amount of KYC forms I
need to fill in. Even though the Chamber of
Commerce has all that information, like I'm registered
as the, you know, the director. But then I'd still need to
manually fill in some random PDF with all those.
So it's like, OK, I can automate that with AI, but isn't it a
stupid process to begin with? I like you as a no tree.
Why don't you just pull that from the Chamber of Commerce?
Why do you want me to download it from the Chamber of Commerce
and print it out and then fill it in by hand and then post it
to you? Right.
It's like the whole process sucks to begin with.
Yeah, I mean, if people come to ORC then specifically or to you
with regards to a use case, and we want to use ORC in production
because we think we can be in control, Do you then also
challenge those use cases? Because I think from your, if I
were your customer, for me that'd be incredibly valuable,
right? AII see this, my competitors are
doing this. I have the feeling I'm behind.
I want to do something resilient in production.
Think you're a great fit. This is our use case.
And if you then say, well, actually do you need to have
that in the 1st place, that would be incredibly valuable,
yeah. We do, although we have great
consultancy and strategy partners who do that in the more
complex projects already. So they go and do that.
So we come in with the technology after the second is
who are not our buyers are teams that are thinking about starting
with Gen. AI, right?
Because then we're trying to sell them a Formula One car when
they don't even know if they need a bike or, or, or a scooter
or a car. So the teams that knock on our
doors and you know, we start working with a lot of times have
done multiple projects, have had some bloody noses, have had some
hypotheses killed already. So it's also a different dynamic
versus we are thinking about AI. What about you guys?
That's not a fit. No, because they don't know even
what they're looking at. When then we show the platform,
they're like the buttons work and the guys are nice, but we
don't know what to do with this piece of technology, right?
Is that then also kind of the maturity progress an
organization goes through? They see AI, they start
experimenting small, and then they see kind of what it means
with regards to use cases before they put it into production in a
controlled manner. Yeah, and they see like the
projects getting slowed down when going into production or
not making production at all or in production getting some
bloody noses of things going wrong.
And then when they see other platform, they're like, oh,
shit, we needed this six months ago.
But when we would show them the six months before, they would
not recognize the the value of it, right.
So they understand the concepts that we then explain, Oh yeah,
we need the experimentation, we need observability.
But you know, once you've had a couple of those bloody noses,
then you're like, oh, I really need this.
So that's part of the maturity curve.
And we, yeah, then we always are like, hey, let's check back in
six months or let us help you. But yeah, you're you're not
going to get the value out of the platform.
Just are starting and thinking about AII.
Feel like that's like it's it's interesting that organizations
need to experiment, fail and learn in this kind of manner
because it's so accessible and people can try out and people
can kind of see in their own domain what would be valuable
and what would not be valuable and actually start doing that.
It's very different from let's say data-driven solutions.
With machine learning, you needed to have a full
infrastructure layer and histories of data beforehand.
And now you just generate stuff based on what you already have.
Or you tweak your content. If you're you have a marketing
tool and you personalize it based on demographic data that
you have. I feel like people can get up
and running faster and then this cycle of iteration of failing,
getting feedback and improving, they can do that on their own
nowadays. Yeah, because it's a lower a
hurdle of entry, right. Generative because you just use
natural language. Everyone can use natural
language to communicate what most people can, right?
Yeah. You didn't have that in ML, in
the, you know, in traditional ML because there you need like, you
know, researchers and PhDs and data scientists and labeled data
to get a feature out here. It's like, yeah, everyone uses
NACHA GPT. And if you yeah, if you just get
the concepts, a lot of people can already start working with
generative AI, which also brings his own challenges because it's
so simple. People underestimate what you
need for business critical features in production, right?
Someone that says, oh, I use Jet GPT, so we can also put this in
production. No, that's not how it works,
right? Then that's why also requires an
entirely different type of stack than traditional ML needs.
I mean that, I think for me explains why there's so many.
Like the opinions are not very coherent.
It goes all over the place. And I feel like what we see out
there, especially on LinkedIn, but definitely also on Twitter
with regards to different types of thinking and saying, OK, non
engineers can definitely build products.
And like it's, it's so out there and people are either trying to
sell something, they might be bots.
I'm naive as hell and I just see this happening.
I'm like, it's actually I, I don't think it's ever been this
messy with regards to different signals that someone can pick up
one individual and then that helps form their own opinion.
And every cycle you do have these snake oil people, right,
trying to sell snake oil. So I mean, crypto time, right
then when blockchain legit technology, but then you had all
these crypto crap being built or hustled on top of it.
Yeah, this is kind of the new era again, right, Where you have
these people leeching on the people who don't know and just
trying to, yeah, scam them in a way.
Yeah, if someone is. By my e-book.
Yeah, it'll a refund, but you don't get a refund.
If someone is listening and they they hear this and they are
maybe familiar with AI, what would you recommend to them to
actually start getting a fundamental knowledge with
regards to learning what it is, what is out there for their own
organization? Start experimenting.
How would they start? Well, the most important one, of
course, there's so many resources outside.
I mean, you can literally use Chet GPT to get your knowledge
up to speed, right? But to really understand in
which fields this technology can actually help and which ones it
cannot, right? Because whatever question you
ask it, it will answer yes. It doesn't mean yeah, right?
It doesn't mean it's correct. So people think, well, we'll
have AI fix it, right? So yeah, let's plug in AI.
So first of all, it's used as this blanket statement of well,
AI will fix it, which is not the case.
Is really you need to be in the engine room, you know, bloody
noses, dirty hands to really get a feature properly working.
But also understand the boundaries of, OK, you know,
processing natural language, classifying text, all that
stuff. It's great for.
But you know, you cannot just have it run your accounting in
Excel without, you know, with just a prompt.
So really understanding where, how are these models built?
So what are their strengths, but also what are their weaknesses?
So let's double down on the strengths, you know, and map it
to what we need as an organization.
But also let's not fool ourselves by having a use case.
What is actually a weakness of an LLM model, right?
Thinking that again, it's a magic pill that will solve our
accounting processes. You have AI start-ups building
stuff for accounting, but then they're building very custom
solutions for that. But you cannot just, you know,
send an Excel sheet with all your statement bank statements
to a model and then have it, you know, turn out your whole
accounting and and your reports and balance sheets properly.
Gotcha. Yeah.
So the boundaries of the capabilities is important.
Yeah, figuring those, I mean, because it's so accessible, you
can do a lot of experimentation. And I do think with a sceptical
mind you can find those boundaries, even though
everywhere on the Internet sometimes makes it seem that
everything is possible and it's just this magic.
It definitely feels magical, I must say.
I've read a little bit about what goes under the hood, but
from a user perspective, it's like, yeah, it's, it feels
accurate, it feels natural. I've used voice assistants.
They are incredible. Like it just keeps getting
better and better, but critical thinking will always be
necessary. And then you have all these non
functional things as well, right, like rate limits and
token usage and all that stuff. These are all non functionals.
But we have now clients who have very successful features, but
now they're getting crushed by all these rate limits.
Yeah, that currently is right. So that's a limitation of the
technology at this moment. You cannot, you know, service a
million of your users with one API key of one provider.
You will hit rate limits, right? It's the same, you know, I
always say history. Well, I didn't invent the same,
but right, History doesn't repeat, but it rhymes.
During the whole blockchain hype, I was like, yeah, this can
be our centralized database or we're going to, in our company,
use blockchain to store info. Well, and then you had the
problem of the speed of these databases that you cannot use it
as a database for software because it's just too slow,
right. That blockchain.
And why would we, we have a database in our company, right?
Why would we put up a blockchain?
So the the technology being misused for the wrong reasons is
is inherent also allows you to allows us as as mankind to find
new use cases for technology. But you also did not, should not
be afraid to write stuff off because it didn't yield to the
results you wanted. Linking back to the
experimentation mindset you need.
Yeah. One of the operational questions
I had was indeed with regards to rate limits because I've, I've
been in organizations and I've put Gen.
EI solutions to production, but rate limits due to our size,
internal application not reaching a certain level of
skill, not being an app that goes to consumers.
I feel like if you reach a certain skills, rate limits are
definitely going to be an issue. Do you then go and you switch to
a different model or how do you accommodate for rate limits?
Do you really just have to swipe the credit card and pay more?
No. So yes, even that is sometimes
not possible, right? So they're making reserved deals
with the providers, but then you really need to have big
commitments. So as a small startup, you're
not going to like commit to 200 KA year to Google to use to get
dedicated space. Also because you don't know if
you want to be on that model in in two years, maybe a better
one, cheaper one, faster one lock come out.
Yeah, right. So that is one of the reasons.
So what we then do is these fall back orchestration, right?
So in org, for example, people can have yeah, up to five now,
but it can be 10 if you want fall backs.
So if you so you're load balancing your rate limits
across multiple data centers and multiple vendors even, right?
So because a lot of the popular models are provided by multiple
providers, right? So Entropic models, you have
Entropic, but they also run on AWS and on GCP.
So if you put those set up right, like if you use Cloud 4
Opus and you have both, you know, all three providers, you
kind of have an aggregate of the of the rate limits.
It's it's the same model. So you don't have a quality
difference, right. So that's the different ways you
can orchestrate around it. Or you can even decide like as a
fourth fall back, we will electric fall back to open AI
then, which is maybe a different quality or not as good, but at
least we're not down, right. So as a product team, you're
constantly making these trade-offs.
How how good does it need to be? How fast does it need to be?
How cheap does it need to be? And per use case, right?
It's not even per clients, but per use case.
You make different trade-offs. If you're building a customer
facing chat bots, yeah, you, you cannot wait 25 seconds for every
answer. So then speed is of importance.
But if you're running some background process, you don't
care if it's three seconds or 30 seconds, as long as it's proper
processed properly. So then you're optimizing for
correctness, right? And as a product team, you're
always making trade-offs. You do that also in software,
right? Like, do we want it to be fast?
Do we want to be reliable? Do we need to have multiple zone
availabilities? Yeah, you're making all these
trade-offs constantly. Yeah, I like the saying.
There's no good or bad decisions, there's only
trade-offs, Yeah, and that's basically it.
But I love this solution of there's multiple cloud providers
hosting these same models and then having kind of availability
across that. Yeah, to kind of up your rate
limit. I feel like it's genius to be.
Honest. Yeah, well, for example with
Azure then, right, You, you can literally deploy a model in
different regions. Yeah.
So you actually have the regions as each other's fall backs.
So you have now all of a sudden like your whole European rate
limit of all the different Azure data centers combined?
Yeah, that's really smart. I was only thinking along one
kind of axis, which would be switching models.
But switching models, I think for content, and this is, I mean
conversion optimization always goes to the nitty gritty.
But for content, if you have a marketing tool and that does
personalized content based on user demographics, one model
might outperform another one, but it should be negligible
because we're talking about content here specifically.
So switching a model then is not going to be the end of the
world. But I like that you have the
option to keep the same models consistent and then still
manoeuvre around those rate limits that you have.
Yeah. And then content, I agree.
And even there, right, some people like Claude's creative
writing versus some other language differences, right.
So because we always think in English.
But yeah, if you have a friend we have literally with our
contextual router, you can do contextual prompting.
You can literally say, hey, the French users will send to
Mistral, the English will send to GPT, and the Dutch will send
to Anthropic because the Dutch localization is better in that
language. And Indian we send to an Indian
LLM. Well, Indian is not a language,
but Indian languages will send to this specific LLM for for
localization. Yeah, right.
You all of a sudden need that. Yeah.
I like, I like that a lot to be honest.
Like for me, I, I thought, OK, getting stuff in production,
that's one thing. Having kind of the, the
operations mindset with regards to observability guardrails,
seeing where things kind of go out of the boundaries that we
think we have is one thing. But I've never thought of this.
I mean I have thought of this but I I never thought you could
go that deep with regards to using and leveraging different
models in the same use case, right?
It makes sense to have one use case and then one model will
pick the best model. But your use case might vary so
much that you can have multi model solutions based on locale
like you mentioned with regards to different writing styles.
Even data residency, right? So we have clients that like
different business units, the data is not allowed to leave
their region. So they will literally route use
cases to an LLM deployed in a specific region, right?
And they need that orchestration all of a sudden so it can go
really crazy. That's because data residency in
AI becomes more and more of a criteria in every decision.
It's definitely a hot topic. Right.
Yeah, yeah, right. Where does the model run?
Is literally a thing. And then we now have Australian
healthcare companies. It cannot literally leave the
boundaries of Australia, right? And India has the same type of
laws and regulations. Well, GDPR, we all know, right?
It's all like, like physically it's not allowed to leave.
So how do you guarantee that and make sure that people have that
granular control? And of course, if you're just
summarizing some SEO marketing text, you don't really care
whether you send it to the US or not.
But if you like summarizing healthcare records, which
clients of US do in healthcare, yeah, it literally physically is
not allowed to leave the boundaries of the EU, right?
Yeah. So you get those type of a non
functional requirements. Yeah.
As a last thought, I wanted to ask you this question because I
see a lot of companies that are starting up and trying to do
something with regards to Gen. AI, right, Trying to find
product market fit, either B to BB to C, anything they can find.
What would be your advice with regards to the challenges
they're going to face or what to look out for when they're trying
to go on that path? Find the right people and what I
mean by that is try to stay away from all these snake oil sales
people, right? Because they will apply to jobs,
especially if you put up an AI job ad now the the top, top tier
people will not apply. But then you attract all these.
Well, not everyone applying, but you know, it's a higher risk
that you attract also some wannabes and because companies
are so desperate, especially if you have difficulty attracting
the right talent, you're not a tech first, AI first company,
then you're like, OK, I hope this version is the magic pill,
right? Then they sell themselves really
well. And then you would think, oh,
they're going to help us with our AI strategy and it won't.
You as a leader also need to be in the trenches and get your
hands dirty. So attracting the right people
and then just starting, right, just start developing hypothesis
and testing and set yourself up for that continuous testing as
well. But that comes maybe in Step 2.
Step 1 is just get going with the right people.
Gotcha. Now how, how do you do that?
Do you find all the people through your network?
You already mentioned juniors. In the phase that you're in is
not really what you're looking for.
Yeah, we have. I can remember last time we put
out a job ad, it's always so our best top tier performers we have
in the company, they just walked into the office like, guys, I
want to work here, I don't care what.
And as far as they also had yeah, yeah, yeah, or yeah, or
through network, they make sure they get an intro.
So it's always been people applying and not even like, hey,
here's my resume, but more like let's talk.
And then it matches. Also because we're not
recruiting 10s of hundreds of people, so we only need a couple
really good ones and we're in a sexy space at the moment.
So yeah, we people come to us, Yeah.
In that front, yeah. Even though you don't have a job
position open, would people still come and you would still
hire those people? Right, talent, we make it work
also because we're always looking for, you know, what we
call M profiles, people that have broad interests and
capabilities and actually can go deep on a couple of fronts.
Gotcha. Right.
On AI or on engineering, but they're also just mini CEOs on
their own. Yeah, right.
Most of our people are from day one have been saying one day I'm
going to have my own company. I'm here to learn and I'm
totally supporting that. I would love to, you know, be
their Angel investor one day. So yeah, that's the type we're
attracting and working with. It's all these mini future
founders themselves. That's awesome.
Thank you so much for coming on and sharing.
So this was amazing. I I love that it was a lot of
fun. I must say it was good.
All right then we're going to round it off here.
Thank you so much. The best way to support the show
is to leave a like. Whether you're on Spotify, apple
pie cast or on YouTube doesn't really matter.
Leave a like anywhere and we'll see you on the next one.