What is up everyone? Hope you were jamming to that
music as much as I was. I can see the chat is already
getting off to a good start. I'm excited because we've got a
packed house today to talk all about good old state of agentic
retrieval. There's so much that we're going
to get into and we've got so many incredible folks here
today. But I want to set the scene
before we bring out the guests of honor.
And I want to talk to you about why this session feels so
important. And it is probably why you
actually came. You know, there's so many
different challenges and changes that we've had with retrieval
over the past two years. I have seen such an evolution.
And so I thought, why don't we get together some people who
have been knee deep in this whole retrieval game to talk to
us about what it was like, what did we have back in the day and
what do we have now and what are ways that we can optimize it so
that we are finding success right now.
So without further ado, I'm going to bring on the team from
Quadrant. We've got the Devrel folks
coming out. First up is my man Dylan.
Where are you at it? Hey Dylan, how you doing?
We've got Jenny, we've got Eva, Neil and Andre.
I think this is the most people that we've ever had on a round
table. So let's just kick it off, start
strong with some hard hitting questions.
I'm going to ask what has changed in the last two years?
I I've seen so much. What do we need to focus on?
Yeah, sure. Yeah, I can take that.
Hi everyone, Neil here. Yeah, thanks for joining the
session and thanks for the intro.
Demetrios. There's so much happening in
Retrieval and Surge. I, before I joined Quadrant, I
didn't fully appreciate the, the field of search.
And we kind of just expect search to work.
And we definitely notice when search doesn't work well and we
notice it through just getting like terrible results and, or it
takes a long time to get results.
We can't find what we're looking for.
And the more data that we capture and from the real world,
the more we need to get that information, get, get the data,
the right data we need, get the information out of our, the data
we're storing. So what the back to the
question, what's changed? One of the biggest things has
changed is agents and the topic of today's round table agents, a
lot of them don't really search well, I would say, in my
opinion, there's a lot of they, they kind of search like a, like
a, a beginner would just search by just like kind of throwing a
lot of things into the, into the hat and just like trying to like
pull, pull, pull results. And they go through lots of
turns and they burn a lot of tokens and they try to find this
context, but they're not using all the tools available for
search and this. And if you don't have the right
context in your AI, that's going to really make your AI less
efficient, less effective. It's going to prevent you from
reaching your goals. So agents and, and one of the
things that agents are doing that's also interesting is like
a human might search like once or twice a minute or so.
Agents are searching thousands of times per minute.
So they're just going boom, boom, boom, boom, boom hitting
it. And they're not able to evaluate
those search results. So same way human can, because
humans have a lot of more judgement that they can apply.
So yeah, I'll stop there. But those are those are some
things that have really changed. Dude, you're preaching to the
choir on the different ways that agents search and then the ways
that they interpret that data. And so I, I have some questions
here about like, hey, now that we've got agents and these
agents are figuring things out, they're kind of, we've all seen
it. They will stumble through
different problem statements where they'll try and figure it
out and they don't necessarily go about it the most efficient
way possible. I wonder if you all have found
tricks on nudging the agents to be like, Oh, yeah, by the way,
like there's obviously the go look in that file trick, but
there potentially could be plenty more that you've seen.
Because this at the end of the day, is a search and retrieval
problem. And the faster that you can get
those agents, that knowledge, the faster they can hopefully do
what you've asked them to do. Yeah, I'm I'm actually going to
take jump in here again, but I'm going to pass the Dylan on this.
So Dylan, get ready. So just to set like a little
more context there is, you know, it's not just the efficiency,
but it's the effectiveness of search like finding the most
relevant response, relevant results.
And I think that it goes into evaluation.
How do you evaluate that the results are good enough to use?
Should you try again? Should you change your query?
And if you should try again, when do you stop?
When like how how long does the agent go before it stops?
What tools can it use available to it?
And Dylan's actually done self correcting agent loops that he
can talk a little bit about. So I'm going to hand to you to
talk a little bit about what you found.
Yeah, absolutely. Thank you.
So my first points that I'm going to talk about is that
Neil, you just mentioned evaluations and I feel like a
lot of people are familiar with like the general, you know,
evaluation frameworks and evals that you can run on an agent.
But also in the information retrieval space, we have a lot
of our own like evals and metrics like NDCJMR that are
like really relevant for, you know, search pipelines and that
that we use extensively. But going into your what, what
you mentioned about like self evaluating agents.
So, you know, there's a lot of techniques for for that.
You know, the most obvious one is to spin up like a small agent
and say, hey, was that result relevance?
Did it contain, you know, all the information that the user
was looking for? But the problem here is that,
you know, you add another like LLM roundtrip so that it is
extra latency, extra cost, Ian, and it makes your pipeline
slower, more complex. And so something that we've been
working on was computing statistical signals that, you
know, very cheap signals based on the, the results.
And so, so for example, you know, when when you retrieve,
you know, using like dense or sparse embeddings, basic
basically you get like a score as a result.
So for like dense embeddings, it is between, you know, like 0 and
one which one being like the most confidence.
And you know, you return like the top K results, so the top
five or the top ten. And basically you can do some
statistic based on like the spread of those scores or how
the difference between the the top one and the top second
results. And basically from here, try to
create a programmatic, you know, signals that will that that will
be able to tell you if the quality of your retrieval was
good or not without virtually any adding any compute to your
pipeline. And then once you you have that
that signals that signal that tells you, you know, the the
retrieval was good or not, then you can basically route your
retrieval agents to like different paths.
And you can choose, for example, to do like more expensive
retrieval method, like a cross encoder or late interaction
model. Or if the result was not good
enough or not like not good at all at all.
It's like, you know, there was like a huge divergent in in
between like all the scores that you retrieved.
This is when you can invoke an additional LLM.
So the in order testings, this was like really a good way to
tell apart like good from bad retrieval in a way that is
really cheap and allows you to spend more money only only when
it was necessary. And then we also have some
something on that topic on skills.
So I'll pass it over to Jenny. I think it's super funny.
We're working like, you know, actually a multi agent chain out
there because we're passing tasks to each other and
Demetrius, you're the evaluator of how the pipeline is going,
right? So we're just emulating search
and informational retrieval from our heads in the going.
And yeah, like the little addition as I think everybody
knows about the genetic skills also, right?
So they're also applicable in search.
The problem is agents need to know how to use search as humans
actually sometimes also do it bad because they also don't know
how to search properly for something, but they do it
differently. With agents.
There is a good solution of teach them how to approach it is
to show them skills. For example, we designed some
bucket of search engineer skills which we're trying to maintain
in quadrant which explained our agent which approaches our
vector search engine on what to use and how to combine it to get
different outcomes. For example, if you have a
problem that your search doesn't show for you as an agent,
relevant results at the top and you feel like there is something
more to it, then there is a specific markdown part of the
skill which tells, Hey, you should do this, this, this with
our, with our APIs, I think we can share the link later.
Or I could try to share the screen, but I'm so afraid to
share. You know, it's the hardest
thing. That's why like I'm senior
developer relations, but I never managed to share a screen
without sharing my private chats.
So let's see, worst case we're just dumping the link and
showing how the skill evolvement for the search pipelines is also
helping our agents to use our search infrastructure properly.
Classic. OK, I've got some follow up
questions and there are some awesome questions that are
coming through in the chat, but ever I wanted to give you some
time in case there's anything you wanted to tag on here.
Sure. So Jenny mentioned a little bit
of how we can teach agents how to know search.
So essentially when we added the agents, we moved from a very
static system of RAG to something much more dynamic.
But actually the search primitives that are underlying
remain the same. So one research paper that came
quite recently last month that I found really interesting on the
topic is called Sera, which is the Super intelligence retrieval
agent. And the whole idea there is to
achieving super intelligence, which is when the multi round
search is compressed into a single corpus retrieval.
So we've essentially avoided the latency that Dylan was talking
about the the latency lags. And what they've done that was
quite relevant to to our today's topic is they figured out how to
use the LLM to enrich the documents.
So have this iterative loop whenever the search vocabulary
was missing. So they looked at the queries
that were generated, evaluated what queries that were missing,
and essentially predicted omitted vocabulary and added to
it before they passed it on to more traditional search
paradigms like BM25. OK.
There's a lot to unpack with all three of you saying this stuff
because basically, let me just say what I caught real fast and
then you can correct me if I'm wrong there.
Dylan, You're talking about different ways that you can use
statistics to leverage and get better results with the
retrieval and with the agents recognizing what to grab and
what to retrieve and what is used.
Tell me what? Yeah, tell me what?
Sorry, yeah. So basically it was like a cheap
way to get early signs into bad or good retrieval without, you
know, adding any latency or cost to your pipeline and then routes
based on those signals. All right.
Then from there, Jenny, you mentioned using skills and how
you all at Quadrant have created a bunch of skills.
Are those open source? I didn't catch that one.
That was the quick. Everything that we do,
everything ish that we do is open source.
So, yeah, obviously. Drop that into the chat here and
I can relay it to the chat or put it in wherever, whatever
chat you want. But we definitely I want to see
all of those skills. And lastly, Eva, you were saying
that there is a paper on the Super intelligent retrieval
agents and the way that they're able to leverage and I didn't
catch now. Now my memory is starting to
fade trying to hold all these things in my hand head.
The Super intelligent retrieval agents were doing what now?
So it's just a framework of how do we know that we've build an
agent that essentially does successful retrieval?
Incredible. OK, cool.
So before I hit some of these questions from the chat and and
I'm just like looking at the chat about big question that's
coming up is like, hey, what about ground truth?
How do we figure that out? What is cause to know if the
retrieval is correct, right? We also need to know what the
the actual thing is, the ground truth that we're looking for.
Does anyone have opinions there? I see some things taken I.
Would say, I would say a proper bunch and each of us has like I
would say different sides of it. And also about what ground truth
is, I think our problem is that we got the agentic systems and
we got super excited and we're like, are they solving search
fully? But the problem is that
evaluations weren't fully solved before with the classical
search. And now with agentic search, it
gets even more tricky because you're like, is the ground truth
that the agent successfully fulfilled the whole task and
arrived to the right point? Or is there is a ground truth on
the age point of the iteration of what it does in the process
and there are several approaches to it.
There is ground truth in retrieval as probably people
know can be created from some golden data sets per SE.
So some from the data that you already have and you know that
this question should get this answer from your data set.
Usually in production it's not so easy to get it, but LLMS got
pretty good at helping to generate the synthetical data
sets with golden truth. The one tip and opinion that I
have as the person who previously worked in
crowdsourcing as a devil of crowdsourcing approach, so where
the humans were gathering these data sets is that don't
immediately expect dumping your protectional data to LLM that it
will create your ideal golden set evaluation.
And creating the golden truth still is the work where you need
to be in the dialogue like you're in the dialogue
developing a cold project. So the relevance notion, the
ground truth notion still comes from you as a domain expert and
that can be like LLM could be used as a tool to develop the
ground truth data set that you can then inject in your
evaluations. But when it comes, I know I have
too much to say 1 little thing, one little 11 little thing.
But when it comes to the whole thing as the whole pipeline to
be successful for you like solving the task, there are very
different approaches on how to teach model to do the correct
search trajectories which goes a little bit in more into
reinforcement learning domain. And I feel like there is a lot
of emerging there teaching agents to do the right decisions
in search. And there are the ground truth
is the result being correct given the task, given the search
task, the search input. Maybe your audience knows Ralph
root loops. So they're also to some extent
applicable to the search. But Dylan here is coming from a
rise and he was doing evaluations as as the bread and
butter so. So, Dylan, this dovetails nicely
into one of the questions that's in the chat.
Someone was asking about how teams are versioning and
evaluating retrieval pipelines in production.
Yeah, absolutely. So you know, something that I
want to preface for preface with first is that we are a vector
search engine. So you know, we only provide
only one of the cogs in the machine.
And you know, this is really by purpose.
We don't want to be selling like or providing like a retrieval
agent. We really want to stay very
close to the metal and provide like the, the best retrieval
engine possible. But when it comes to those kinds
of like evaluations and versioning, I will usually
suggest to use like an evaluation framework or
platform. Of course, you know, I come from
our IAIS, so I'm, I'm, I might, my opinion might be biased
there, but there's a bunch of free evaluation tooling that can
help you do that that versioning there.
Awesome. And I think Andre, you had some
other thoughts about this too. Where are you?
I got to change the view to see. There he is.
There we go. Yeah, Andre's been doing.
Space on the screen. So I, I would like to say that
from my experience, what can also help if we jump back for
the, for the golden set is when you kinda have some kind of
assumptions based on the human, on some human manual labor work.
What can be good for your golden set so that the LLM has some
kind of criterias that it can evaluate about.
And I would like to say that it's like an iterative process
in which you try to improve your LLM as a judge.
Yeah, time from time so that the golden set that it is creating
and the evaluation in general becomes better.
So it's a long process. It's iterative process that just
becomes better as you train it more, yeah.
Awesome. Well, we've got more questions
in the chat. I see the let me just let me
just grab this. So Eva missing search terms are
Grafton dynamically. How does that handle prompt
injection shenanigans, if at all?
This is a good question so in the paper and feel free to share
the link in the chat. Hopefully you have it.
I I haven't found the cybersecurity aspect of it, but
I think this would be like a really nice segue to agentic
harnesses. And as you design your system,
retrieval will not solve everything.
You still need to add tools onto your agent that would make it
more secure. But I think this is a really
important design principle to to consider because prompt
injection is definitely a very serious serious threats.
And I think especially recently we had open claw and or miss an
agent that came out. And with open claw there was a
lot of hype, but also we saw when we did not properly sandbox
it that it was able to just go completely wild and access all
of our data. So yeah, I think that is not
specifically mentioned in that paper, but consideration that
every single good agentic retrieval system should have,
yes. So let me continue with a few
more questions from the chat and then we will go on to the next
topic that I had in my notes. There is my biggest pain point
from John is he's saying biggest pain point I've had with rag
stacks is defining edge and entities dynamically with LLM
analysts. Can you speak to different
approaches to that problem? And I'll just throw that to the
crowd. Anybody have strong thoughts?
I agree because I've heard it a lot on the conferences, but I
believe it's more of the graph ontology construction problem.
And we as the vector search engine don't do the ontology in
construction. We usually combine because in
many, many domains it's a very good complementary thing.
For example, in memory of agents, it makes sense to mix
both graph and vector search. That's how many agentic memory
providers do, for example, Cogni.
But I've heard that list from practitioners on the conferences
and don't quote on me. I would ask Neo 4 J guys on
that, that if you atomize the task of ontology instruction in
the sense you don't try to build the whole ontology of the
domain, but you do it bit by bit in the more atomized setting
that actually performs much better, like the outcome is much
better. But I also know that Andrey
before worked with ontology. So maybe you have like a little
point of view on that. Yeah.
So can you repeat the question again so that I I have the
latest? The biggest pain point I've had
with Rag stacks is defining edge and entities dynamically with
LLM and analysis. Do you have any approaches on
that? Oh, I would say that, yeah, the
problem is that these are kind as, as Jane, you already said,
some kind of ontologies that you try to provide to the vector
search and to the semantics. So you you you kinda try to put
the explicit semantics into the vector semantics.
And I think these are laying down two different layers.
Yeah, 1 is application layer and another one is, is more of
implementation right of the technology.
So I would say right now what you can do is basically just
like the the current state of art we can say is that there are
lots of GitHub profiles and GitHub projects that are trying
to combine new 4G approach here with explicit entities and also
nodes with something that is vector based.
And there are some successes there because or what does the
graph tell you? It can tell you the
relationship, can tell you the dependencies and in some sense
it can give you additional context that semantics not
always can get, you know, because like you can have
something that is 2000 from 2001 and something from 2014 and you
want something specifically that is from 2014 to be more aligned
on top. You can you can fix it of
course, with some custom, some custom scoring, some custom
relevance metrics that you provide by yourself.
Maybe again, it depends on the domain, but I would say there
are these approaches that they basically try to combine vector
and graph in one stack. Yeah.
Or you can try to encode in some way information into the red.
But yeah, it's quite hard, yeah. Awesome.
Well, let's keep it cruising because I want to get into
memory and it feels like that is a perfect segue into memory and
how folks are approaching this, how you all have seen the best
in the business do it. And especially like I know there
is a lot of talk about long term memory and the architectures
that you have for those. And also if they are that
valuable because sometimes I will not necessarily want
something to be remembered, but it gets remembered.
And then for every single question that I'm asking my
agent, and I'll give you a concrete example of this, I
told, I told the chatbot that I was vegetarian.
And now for things that have nothing to do with food, it
says, well, given your vegan lifestyle.
And I'm like, I asked you a question about my taxes.
Get the fuck out of here. Why would you need to reference
my vegan lifestyle? So anyway, that's just a little
bit of a tangent. Maybe we can talk long term
memory or we can also. I'm going to throw different
things out there like episodic memory.
Do you want to go event based verse?
Semantic memory? How do you do the factual verse
conceptual? Who wants to take this one?
I'll throw it. Throw it up there and let
anybody rise to the challenge. Let me try to jump in because I
relate a lot to this part about the vegan taxes.
I think it's the classical, you know, LLMS, like please don't
think about elephants. I think Dylan taught me that.
And then immediately whatever you do, it's going to recall
this elephants. That's why it's also remember
when you do anything around memory is also to think about
forgetting as a conception. But it's a very hard one, an
interesting one. And there are like several tool
links on how to do that. So basically whatever I wanted
to say is very quick is that I think people out there probably
tried Hermes or open claw agents because who didn't?
It was my first Wow effect, to be honest.
And we recently recently been like day before yesterday,
actually, it's, you know, hype driven development a little bit
released a plug in which allows to store episodic and semantic
memory of a Hermes agent in quadrant.
And I don't have a super sick demo or anything, but I chat
with my RMS agent a little bit and some of the memories of me
trying to prepare for this envelopes meetups got actually
started in my quadrant classer. I can show it but before also
like a quick note in case somebody just this not very well
versed with the concept of different memories.
There is a working one which is the context window.
The chat you're in there is semantic or factual memory.
It's the one that something is true about you.
For example, that the Metris is vegan and likes to do taxes.
Do you? No, OK, doesn't like to do taxes
then happens to all of us. Then there is a procedure,
procedural one. It's actually more like skills.
So what to do in order to achieve something.
And the fourth one would be episodic one.
This perfect thing for the vector search.
Actually, it's all the recalls of what you have discussed in
the past and that it could help your agent to understand how to
deal with the stuff better considering all its private
previous knowledge. So I wired my Army's agent to my
cordon cluster specifically for that to show you.
Let me try to demonstrate it. It's going to be a very lame
demo, but we have a cooler 1 so don't get don't get discouraged.
This is nice because people were asking for demos in the chat,
and so I think I just asked in our background chat if anybody
else has one. Dylan, you have one too, huh?
That we can throw later on. All right, cool.
Yeah, I'll follow up with them as well.
OK, I'm I'm going to tell you, you're going to love Dylan's one
and you're going to maybe like mine.
What am I currently showing? Is it my Slack or is it like UI
of collections in quadrant? Hold on, I got to check it out.
I see this no us so yeah, quadrant you're.
Good. I'm good.
OK. So as you see, this is a very
impressive memory of 37 points approximately in quadrant.
But basically what happens, all of the facts that I have in the
conversation with my Hermes agent, which I'm not going to
obviously share because I ask embarrassing questions all the
time, that's why I wipe the memory.
So all the facts and all the turns in the conversations are
saved with the different metadata of the sessions that
they happened in. And that will help agent recall
some similar information. For example.
I'm a big fan of this GE Chepa approach which Jan Dikun is
recently saying that it's the next big thing for embeddings.
So agent can find similar memories.
You can see that with dense vector search for example about
this method you will be able to find some other memories which
are kinda about the same fact. And this plug in you can also
visualize kinda the memories. There is not so much to see yet
because there is not too much memories.
But we can see that for example with the semantical embeddings,
there are similar memories about me preparing some smart facts
about the Transformers attention window will being very important
for context windows being very important for vector search.
And that's why vector search now won't be dead as much as RAC can
become dead at some point. And if I would start chatting
now with my Hermes agent, for example, that I will add the
fact that well that Demetrius is vegan.
So now all of my inputs are also going to be polluted about.
This. No, it was the other one that
was true. I like doing taxes.
And Demetrius likes to do. I'm not showing you my telegram
where I'm chatting with Ernies, but that's what's happening.
So technically, if the double goals are nice to me, we should
see that the points are going to get updated at some point.
But if they are not, the demo gods are just not nice to me and
you will see the nicer demo. We can come back.
We can come back because I think my RMS agent obviously decided
to sleep in the moment I decided to demonstrate something.
But the general conception is that basically you have your
memory bank because information about you that you do now will
grow and grow and grow. And some point MD files won't be
enough to recall all of the information.
And then you need some, well index which will organize this
memories, let them forget and let them be surfaced in the
right moments. And I think this is very
important. Where are we going?
Because the information, as Neil said at the very beginning,
grows and grows and grows. Demo gods absolutely obliterated
me. But I know that Dylan has a very
cool 1 so everybody can forget this.
And oh, go ahead, sorry, we finished it.
I. Think we've got, well, I wanted
to talk for a minute about forgetting and that whole thing
because I know that can be very difficult.
And there's also a question in the chat that I want to bring up
about like vector databases still being the preferred choice
for long term memory. So maybe Eva, can you talk
forgetting real fast? And then if you have any
thoughts about vector databases, I can imagine I know which way
you're going to follow on that one.
But let's let's go. Hopefully you won't forget that
one. So yeah, I mean, Jenny showed
what's like the benefit of of memory and dealing of all of
that. But I think memory is also
polluting a lot our context window.
So forgetting is actually a massive and really interesting
topic of how do we even decide. Being vegetarian might not be
relevant when you fill your taxes, but on your next session
you might be trying to optimize your diet and that information
will be very important. So this is the world of decay
functions and also relevance feedback.
So agents have this nice thing that you can iteratively tell
them what exactly in that particular search query is
relevant and you can boost that. So this is the ability where I
think vector search actually really shines because you can
figure out for that particular session, what information should
I include in my decay function and just forget.
And that can be based on temporary time stamp or that
could be filtered by keywords, etc.
And then there's another thing, which is some memories actually
overtime that you're putting into the sections into our
sessions, my duplicates. So that's another really
important thing is in order not to completely jam packed your
context window and actually get out of your engine agent what
you want, you can use vector search to de duplicate those
memories. So we have very clean context
window and next time you're filing your taxes, it's just
that and you can boost it and decay based on certain filters
and scores the information that's not relevant for the
session. That's awesome, Neil.
I feel like you had some things to say.
Yeah, I haven't. Talked to you in a while.
Yeah, I don't have anything insightful.
I just wanted to hand over to Dylan with a little bit of
context. We're talking about memory and
forgetting and search. And we, we weren't planning on
showing this demo on, but since there were requests for demos,
we'll go ahead and show it. This is a demo demo.
Dylan's gonna show on actually on device search.
And so when we think about memory and forgetting an AI,
like physical AI is becoming a really interesting area and how
agents and AI on physical devices need to be able to
remember certain contexts, be able to use that context, and
then how humans can interact with that.
There's also something really cool that Dylan can can show.
So yeah, Dylan handing over to you.
Absolutely. Thank you.
And yeah, so you know, we're kind of like steering away from
like the agency topic. But this is the good thing about
vector search is that it's not only limited to to rag and
Agentic. And so we're today we're so I'm
going to show showcase your project that I built for a
robotic use case. So can you guys see my screen?
OK. Not yet Hold.
On Let me do my Job, there we go.
All right, perfect. So this is like just like the
the GitHub project. It is completely public and free
and open source. So if you guys want you to run
this yourself, so today we're going to run it on like a
prerecorded video, but it works with like any, any kind of
input. You can connect your your camera
and it will stop working live. Can you make it a little bigger?
Yes, absolutely. There we go.
Now we're getting there. All right, so basically we're
starting with zero that agents knows nothing about the world.
It doesn't have like a database of like labels of what items
looks like, of what is a chair, what is the floor lamp, what is
a coffee table. And so we have like three kinds
of models that run in parallel. So it's all all running on
device, it's all running locally.
You know, if I turn off my network right now, you would
lose me, but I would not lose that product.
And so basically what what's happening here is that we have
first a an image recognition, an object recognition model that is
called yellow, and then a second model that is an image to text
model. And this model basically creates
a label and also a description for each item that it sees.
And then what we can do is that basically we can do semantic
search on every item that the robot has seen before.
And so, you know, it started with like a completely clean
memory when I, you know, when I opened up that that app, that
robot has never seen anything before.
And now it's able to recognize every single objects and
memorize them and then recall when and where is so those
objects. So, you know, the applications
for robotics are pretty pretty much endless.
And here basically you can so you know, this graph is a 2D
representation of the actual embedding space.
And so you can see the, the robot, the robot's brain being
built in real time and you can see all the memory of basically
all the concepts that are close together.
So you see like all the hallways are grouped together here.
We can see like there's like the dining table and chairs that is
grouped together here. You know, we can see that we, we
went into the bathroom so that those memories are pretty far
away in the embedding space. And then you can really recall,
you know, everything that you has that you have seen before
and the the robot will be basically to able to recall
every single mirror that he has seen.
And yeah. And so you know, this is just
to, so you know, we're, we're not selling embedded models,
we're not selling vision models. We're really just that memory
layer. We're just.
We have a model that creates embeddings based on those image
and text, and we allow for very quick search and recall on those
memories. That's so awesome.
Whose house is that? Is that your house?
Oh, I wish. I was going to say that you
negotiated a pretty good salary if that is your house.
No, that that was just, you know, like on online tour from
like a leasing agent or a realtor.
Oh that's awesome. That is so cool.
The fact that it's on device too with these the small models is
really impressive. Yep.
So, so you know, this was like a way to demo like Quadrant Edge,
which is the embedded version of quadrants and this version can
run on a Raspberry Pi. I'm not even like the top of the
line US1. I have one form like five years
ago that runs it just fine. Awesome.
Now there was a question that I said in the chat I was going to
ask and I totally skipped over it.
I told folks so we're I want to get to it.
It's asking about vector databases still being the
preferred choice for long term memory or our knowledge graphs
and structured memory stores gaining traction.
I, I would say from what I'm saying, it's not an either orbit
of both. We're, we're they, they both
have their uses in different contexts.
But you know, like dense, dense vectors, dense embeddings are
going to give you a lot of like semantic meaning, but like
vector stores are not really the best for like relationships
between those different entities.
And so using like we, we have a great partnership with Neo 4J
and there's a video I put in the private chat.
Maybe you can share Demetrios, but like graph rag and using
knowledge graphs with vector stores together can make a
really good substrate for memory overall.
And just to kind of like shout out a couple other companies
doing this is like Cogney and Men 0.
Cogney has they use both like transactional data or well, they
use like transactional data, they use vector data stores,
then they use knowledge graphs all together.
So you can check those out too. Nice.
A question came through about the demo and it was mainly about
how detailed all of these captions are.
Like are you getting things down to the level of types of
material and style or is it just coffee table?
For Dylan. Sorry, can you repeat that
question? You were looking at the chat
huh? You were you were busy or you
were re watching the demo thinking damn this is a really
good product that I made How? How?
How do you know me so well? That's well played.
So, so basically how much detail do you get from those different
captions or whatever it's being labeled?
Is it going down to the like hey this is a postmodern coffee
table made out of glass or is it just like coffee table?
Yeah. So it really depends on like you
know, the image to text model that you you want to use.
So I use SIGLIP 2, if I remember correctly, which is a very like,
you know, basic like open source model that can that can run on
your on your device. And so that model will usual.
And I did not try to ask for much in depth definition.
You know, I was just looking for coffee table table here.
So maybe that model has the capability to have a more in
depth definition. But if not this one, I'm sure
that, you know, image to text models are very good these days.
And if you want a really like detailed explain the issue of or
like description of the object, I'm sure this is something that
can be done. Nice.
So we've got, I want to bring us back a little bit to the topic
du jour, which is a genetic retrieval.
We were going hard on that and the chat was loving it.
And then we also, the chat was asking for some demos and we got
some cool stuff. But there's a lot that we can
talk about still with agentic retrieval.
And so maybe we can veer in that direction.
Neil, I feel like you have some things you wanted to say.
Yeah, sure. I can.
You know, we got like roughly 10 minutes left.
And so as we center back on agentic retrieval, some general
things that I think the audience should, you know, some ground
setting, ground levelling things that I think the audience should
all be aware of with vector search is that even whether it's
pertaining to retrieve a genetic retrieval or just any retrieval,
it's not only semantic search. And this goes back to the Neo 4
J and graph Rag kind of question too.
But vector search is really vectors and embeddings are going
to be the highest density format in which you can store
information. You're taking in dense
embeddings, you're taking tons of unstructured data.
And yes, you're storing that into a fixed dimensional vector,
but there's also sparse embeddings.
There's also, there's different embedding models.
All vectors are is a format and you as you use different
embedding models for images and videos and memory conversations
and all this like embeddings are just a way, they're a vehicle
for this. And we can do keyword search and
lexical search and we can do all these different exotic things or
maybe exotic strong words, just advanced skilled ways to find
the needles in the haystack. And that becomes really
important for agentic retrieval because just centering back on
agentic retrieval, you want to have access to information in
the most efficient way. And we, we have looked at file
search and other approaches and there are places for file
search. There is not necessarily you
must use vector search for everything.
You must set up an embedding model and you must set up your
chunky. You must do that for everything.
It may not make sense in certain scenarios, but when you as we
start collecting more data in our, our agents become more
sophisticated. Are their search tools become
better? I think that vector search has a
really strong place there. And I also like intentionally
use that term vector search instead of vector database
because you want a system that optimizes for not just storing
and not just storing and having like a blind retrieval method to
get your vectors out. You want to actually have
something that is focused on the retrieval side of the equation,
the search side of the equation. So like that's why I quadrant we
consider ourselves a vector search engine.
But yeah, I'll stop kind of rambling there.
I don't know if anyone else wants to add thoughts to where
they see vector search fitting into agentic retrieval overall.
I can jump in on the scale. So Neil mentioned how like we're
operating at millions of billions of of vectors scale and
search will certainly get much, much bigger.
So early search primitives like the BM20 fives of of the world
are already 5 orders of magnitude smaller in terms of
memory that they consume on the machine with the latest one.
And another research that I wanted to share with you comes
from the group research group called said one.
They have a really interesting approach and they think that
reinforcement learning for identic retrieval is the way to
get to that next 5 order of magnitude efficient identic
retrieval. When the context window gets
bigger, when the number of vectors that we're have to parse
through, it gets much, much bigger.
As I'll share the link so you can read into their approach and
why reinforce reinforcement learning exactly.
Yeah, I know, Jenny, you had mentioned that earlier too
maybe, but can you talk a little bit more to that because I like
this idea. We were talking about RL 2 for
evals and being able to create these environments, which are
super trendy these days, I guess you could say.
And so you have these environments and then you can
eval them. But the RL for retrieval, what
is that even like? Break that down for me a little
more. Sure.
So what I got out of that paper is that the TLDR is basically
they're using a multi turn RL and it's a mixture of synthetic
and real questions. So Jenny mentioned before the
golden data set and the reinforcement learning approach
would be not that we have like human generated golden data set,
but it is the environment where you learn, you interact with
over time with some real questions that resemble that
golden data set. And some of them are completely
AI generated. And there needs to be a reward
system designed in place that rewards every single time that
our agentic system hits the correct answer and punishes it
whenever the learning step was not made correctly.
So the paper outlines how you can design those, but I, I won't
be diving too much in the detail there.
So that's the, that's the high level overview of what exactly
they're doing there. Yeah, that is so cool to see and
I feel like there's so much potential there.
The thing that I always wonder, and I also see that folks in the
chat are asking this, it's more along the lines of like, hey,
I've got this agent. How do I either set up a
framework or encourage it to do different kinds of searches or
search techniques at different times?
So I want to do some kind of semantic search when there's
XYZ, like is there a framework? Is there a way that is it a
skill that we were talking about earlier that you can say like,
hey, agent, here's whenever you need to retrieve something,
here's what you should go through.
And that feels like probably the easiest.
When I see Jenny, you're shaking your head.
You might have some thoughts. No, because I think it's
exactly, you answer the question, there are two ways,
right. There is like the way zero shot
way you teach it procedurally, which is like the set of skills,
the set of search skills and it has its pros because it's easy
enough and you can adopt it easily enough because it's just
writing. I mean it's not just, but it's
composing the set of instructions and we kind of know
how to approach that. And the second branch that we
are seeing now emerging is to actively teaching agents to
search through this reinforcement learning
environments. I don't see now easy ways of any
practitioner to do that, but I feel like that might be the next
thing in the future because everything becomes more
accessible now, right. So maybe we will get our Aral
gyms. Do you see my biceps?
Yeah, I'm saying the Aral Gym, but we will get them soon where
you can actually give the set of what you want to achieve and the
set of the search tools and your problem and it will converge to
the like the small search agent will be able to converge to a
path which actually is, well what you want to see in your
specific pipelines. We're going to stay and watch
that because we represent kind of the default layer that humans
or agents build upon. I wouldn't discard humans steal
from all of this picture, by the way, because I'm human.
So I still, I still sometimes want to search stuff myself.
So I think it's just important to have the tooling which will
be usable by all the categories of the search users.
I think it was a tautology, but you get me I guess.
And I think what we were also trying to say that vector search
is also so much more than just retrieving you a semantically
similar facts. I feel like we kind of fell into
the strap of the rag cage where you just think about it as the
machine that spits out you the text chunk based on your
question. But I think there are other
emerging interesting parts where it could be used for anomaly
detection, data analysis at scale, image to text search,
videos to whatever audios, so all of that stuff.
And at some point agents will be also able to use it 3D.
And I'm really looking forward to see how they're going to
approach this part of vector search.
Well said. I think that is a perfect way to
wrap it up. I know that there are still
folks in the chat that are stoked and asking questions.
I will just mentioned that everyone who is in the chat and
watching this right now, on your left hand sidebar there is a
button that you can press that says match.
If you go there and more than two people do it, you will
randomly get put together with somebody else that is watching
this. So you can meet someone that is
just about as crazy on retrieval as we are, and you can talk to
somebody that is watching this too.
So it's a great way to bond with the rest of the community if you
want to stick around for more time.
But for this session, I think we are going to wrap.
I know that there are some great places the Quadrant team hangs
out. You all have an awesome Discord,
so I'll drop a link for that in the chat.
And then of course, if you weren't already bought into
everything that quadrants do in and following these good folks
that are here with us today, you should definitely do that.
Go and give Quadrant a star if you haven't already on GitHub
and stay up to date with what all of their amazing Devrel is
doing. Follow them on LinkedIn and X
and all the places. So thanks everyone for this
excellent session. I will bid you farewell.
And for those that want to hang around, click the match button
on the left hand sidebar and I'll see you all later.
Thank you. Thanks everyone.
Thank you. Thanks to meet you.
Bye.