Hi, and welcome to the Neil Ashton Podcast.
In each episode, we explained some of the fascinating ways
that science and engineering are changing the world around us.
We talked to leading engineers from elite level sports like
cycling and Formula One to some of the world's top academics to
understand how fluid dynamics, machine learning and
supercomputing are bringing in a new era of discovery.
We also hear some of their life stories, their career advice,
and lessons they've learned on the way that I hope will be
helpful to you too. So sit back and enjoy this
episode. Hi, and welcome back to the Neil
Ashton Podcast. So today I wanted to talk about
the topic of foundational models for fluids or for
computational fluid dynamics. It's a topic that I brought up
and asked quite a few of the people that I interviewed
recently. But I, I wanted to give some
personal thoughts on this, partially because I had the
pleasure a couple of weeks ago of chairing a panel discussion
at the second data-driven fluid dynamics conference, which was
an ERCOFTAC Euromech thing that was done at Imperial
College London. I was involved in the first one
in Paris the previous year. And it's a great initiative that
has tried to bring the fluids community and I guess the AI
community together, particularly with a sort of European
flavour, I guess to it. But in this panel session, it
was really interesting because we had representatives from, you
know, Harvard, but also Caltech and, and some from industry
and, and really representing the experimental and the
computational side. And one of the things that I did
was to try and ask people about their opinion of foundational
models. But more interestingly or as
interesting was I tried to ask the audience some of the
questions and there was a couple of hundred people in the
audience. And so we use this Mentimeter
app that I have to confess I haven't really used before, but
it's great because you can essentially do real-time polling
of people. So I asked the question, do you
believe that we will have foundational models?
And in fact, I'm just looking on the on the screen now because I
have the results up. Well, I should tell you, the
first thing I did ask was what's your favorite British food?
Bit of a tongue-in-cheek really. And fish and chips won,
followed secondly by the vomit emoji, which I was a bit
offended by. And then a full English
breakfast. So there we go.
But more seriously, on the actual topic of foundational
models, I basically said, will you know, can we get
foundational model for fluids? How generalisable could it be?
And what I asked was, could it? Could this foundational model
provide instantaneous pressure, velocity for any boundary
condition or geometry? So as in you can say to it
here's a geometry with a certain boundary condition, can it
predict it and give you the instantaneous pressure and
velocity? A bit like a CFD solver.
Now you give it a geometry, you give it some boundary
conditions, and it would go and solve it.
Second option I gave people is exactly the same, but time
averaged. So not instantaneous, but time
averaged, and I'll get into the nuance of that in a little bit.
Then the third option was yes,
but only for a specific use case, as in maybe like a road,
car or a plane. And finally, option was
essentially no physics. Sorry fluids are just too
complex for an AI model to learn.
And interestingly, out of the scores it was 38 and this is out
of 100 and something. So the the the most popular
answer was only for a specific use case. 38 people close 2nd.
37 was provide time average to any boundary condition or
geometry followed by instantaneous pressure.
And finally 13 had fluids that are just too complex.
So out of that audience, which is biased because they are
researchers looking at fluids and AI, the sense was it really
is only possible for a specific use case.
And therefore the terminology of foundational model is perhaps
pushing it a little bit or needs redefining.
OK. So that's why essentially I
wanted to to talk about it and I wanted to share I guess some
opinions of my own, some stuff I've seen along the way.
I'm in a lucky position, I guess.
I'm working in the tech sector that I get to engage with a lot
of customers and academics from different groups.
So I like to go to conferences.
And I'd like to say I'd therefore have a reasonably good
view of what's going on in the community.
And I would actually say that things are looking very
interesting as in only a couple of years ago there was really
no work into this idea of a foundational model.
And another way of thinking of foundational model is simply a
pretrained model. So what most people were looking
at was simply can we come up with some sort of algorithm,
some architecture that could allow somebody to go and train
an AI model based on prior simulation data.
So we're not talking about accelerating an existing code,
we're talking about replacing the code with a surrogate model.
And that was certainly the main focus probably five years ago
or four years ago. And there was some seminal work
done with things like Mesh- GraphNets and others which took
a graph neural net approach where you could essentially
take the nodes of the mesh as the input.
And that was something or particles I guess because it was
all particle examples of this. And this rapidly became one of
the success stories in the weather and climate, but also
was showing potential for CFD. And this led to a number of
start-ups that that formed over the time, the sort of NAVASTO,
Neural Concept and others who took some of this work or
pioneered things themselves and, and released some of those
things that people could go and, and try for themselves.
And so you saw in the past, you know, several years, people
really wanting to test out those methods and with some success
for sure. But obviously that requires you
to have your own training data, your own ability to train a
model and to have, I guess, both the software and the knowledge
to do it. And I guess if you can compare
that against what you would typically see in the LLM space
where the majority of us are not training our own LLMs, but we
are simply doing a prompt and asking a question, doing an
inference, we're not training it.
The logical question was, well, could it be possible to train a
model and then just provide it to people as a pretrained
model? And it's a really interesting
discussion because it has pretty large it, it could have a
potentially very large impact on the CFD community.
Now, there are some of you out there who still have very
negative opinions of AI and sort of seeing it as a fad, and it
will come and go. But I would say those voices are
slowly starting to be overtaken by the realization
that actually these methods have a lot of potential.
Only, like I said, five years ago this was basically not
happening. Nobody was even talking about
it. I remember I got involved maybe
three years ago in this topic, so still quite late to the game
I guess. But even three years ago I
remember most people had no idea what this was all about.
They just thought of this as like reduced order modelling
ROMs, really no sense of what was going on, didn't know what
architectures would work. There was only maybe one or two
start-ups in the space, wasn't very prominent.
Fast forward to now and I would say that it is certainly rapidly
expanding and I would say companies can be categorised as
either trying to lead in this way, being very interested and
the number who were just set against it is now sort of
getting lower. But I would say that this is
still a little bit split between the industries.
And like with there's a bit of a parallel with high fidelity
methods, I see the automotive sector as being much faster to
adopt these new technologies or investigate these new
technologies. Automotive companies today use
hybrid RANS, scale- resolving methods on a day-to-day basis,
whereas the aerospace sector is still sort of proving out that
they think they can be used in in production.
And some of that is because Reynolds numbers and
complexities. Another is just a mindset, I
believe. And interestingly, I see the
same thing with AI. I see that the automotive
companies are far more motivated to try and find AI approaches
that can be faster potentially and are embracing it.
Whereas I see aerospace is more in the mindset of I don't think
this can work for me. And I guess part of me, the
reason doing this podcast episode is really just to put
across that, and this is a genuine, this is not influenced
by the company I work for. These are all sort of
independent opinions, personal thoughts, but I really do think
that these are looking incredibly interesting for the
obvious reason that, you know, now if you do an inference on a,
on a model, you're going to get an answer in seconds.
But the accuracy is debatable whether people have truly proved
out that these these AI methods are accurate enough.
They give a very good impression of, you know, approximately
what the flow field is or approximately what the lift or
drag is. But I would say that we are
still not at the case where they're widely accepted or there
are very clear accuracy. I would say it's a bit like CFD
in the early days where you could argue there was clear
points where it worked well, but there were many examples where
it didn't work. And hence why CFD took decades,
you know, to to progress and be trusted enough to be a dominant
design tool. I'd say AI is still in that
sense where it's probably not trusted as a key design tool
yet. But I would say that the number
of companies really investing in this is significantly
growing. The number of start-ups who are
emerging in this space is growing month on month.
And the interest from the big ISVs is also growing month on
month. But the challenge is the data.
So do you have available data? And this has always been the
challenge for the idea of a foundational model, as probably
people know, you know, the reason that these LLMs and
others can do so well is that they have a very large data
source to, to to, to get to. So typically scraping off the
Internet, but nowadays maybe doing commercial deals with a
certain repository of data. You know, if you want to do
something on journal papers, you might do a deal with Elsevier
or John Wiley. If you want to do something on
video, you might do a deal with a video, you know, distributor
or or a newspaper. If you want to do on text or
pictures, you might do with Adobe.
You know, there's a lot of commercial arrangements going on
now. And I think that's what's super
interesting on the CFD side because on one side you could
say the data is just not available.
If you want to do combustion modelling or multiphase or
hypersonic something. This is not publicly available
data. So how do you you know, how do
you solve that? And this is where I think
there's an interesting technical but also commercial angle to
this, which is, is it worth it for a company to independently
run lots and lots of simulations to the point that they can
pre- train a model and then supply the model and, and and
charge it at a certain rate. And we have seen some early
signs of this happening. You know, there's a start up
Luminary Cloud. Some people know that I spoke to
to Juan Alonso, the founder, and they recently released a sort
of foundational model for road cars where they'd, you know,
run a whole bunch of simulations themselves in collaboration
with an end customer of theirs and and then released that and
pretrained the model. Now that it's probably too early
to see, you know, how ultimately successful that that that will
be. But they've sort of fired the
starting gun, so to speak, on an actual company releasing a
sort of pretrained model. And I think it's really
interesting to observe how useful that is because on one
side, it removes a massive barrier for a company to to
collect all the data, find all the data to have the expertise
to use an AI tool. It's much easier just to do
inference on a pretrained model.
So I think that's the first thing I predict that we're going
to see many more of those come out.
Many more companies will will, will do that both I think from
the startup and ISV space. But how, how much data can a
company generate and how can they incentivize it?
Well, one interesting thing, when I spoke with Prith, who's
the CTO of Ansys, he said in one of the talks that I gave that,
you know, maybe the ISV needs to incentivize people to allow the
ISV to have the data. So I think that's a really
interesting proposition that, you know, what if you would tick
a box that was to say, well, you know, as you run your CFD
simulation, I give permission for the ISV to use that data and
perhaps in return they get, you know, some commercial
incentive to do it. And I think that's a really
interesting proposition. And I'm, I'm, I'm keen to see if
some of the ISVs go down that route.
And this actually links why I've often spoken about the cloud
being an important avenue and the cloud being an important
avenue links to this because of the SaaS bit.
If you give somebody a on-prem binary, even if they tick some
box to say we're happy for you to look at our data, what's the
mechanism to share that file? Well, it's pretty difficult
because it's running on their local network.
How are they going to, you know, send that over to you?
It's not practical where with a SaaS solution, it's much easier
because it's by definition running in, let's say someone's
cloud. And so if you say I want to
share some of my data with them, it's actually much easier to do
it. So I predict that part of the
motivation and I think we'll see an acceleration of the
SaaSification is to make this sort of AI and data collection
easier both. If you're an existing AI start-
up, you know, you could say, hey, use my AI tool and I'll
give you a discount if you share your training data with me and
over time will collect more data.
And also it links into one of the things I've said repeatedly
on this podcast about fast CFD solvers or CAE solvers.
Because now one of the big things is if you've got to
generate 5000 CFD cases, if your CFD solver is twice as fast as
another CFD solver, that's a big amount of money to save.
Or for the same budget, you could run twice as many cases.
So if we assume that, to build these foundational models will
have to run hundreds of thousands of cases, then
actually there will be a huge focus on on enabling the code to
be as efficient as possible. Now, one of the other things
that I haven't brought up, but I think is another interesting one
is everything I've spoken about now assumes that the CFD
code is independent from the AI code.
But as anybody knows anything about, for example, in-situ
visualisation, we'll know that there's a strong move now
towards this idea of doing it, you know, online rather than
saving everything to disk and then doing the visualisation.
Can you do it whilst it's running?
And this is obviously something that's not just me, you know,
saying this for the first time, it's known that I think
there'll be a big increase, and there already is, certainly in
the academic world of trying to build your CFD code in the
same framework as your AI code.
Some groups are doing this in PyTorch, you know, how do I
write a code in there? Or they're using some sort of
Python code that you can automatically, you know,
differentiate and move between to pass gradients along and to
the neural networks. But I predict this will be a
will be important because if you are trying to build some
pretrained model, some sort of foundational model, you don't
really want to be doing your CFD independently to your AI
training. You want them to be tightly
coupled and potentially adding more points as you need them and
not having to be, you know, bound by I/O issues, having to
keep the data. Which is particularly true if
you have the dream, and I think this is a much longer term dream
of being able to do instantaneous.
So time dependent, most training now is done on time average
solutions, even if it's a time accurate simulation, because
just the amount of data, I mean having created with colleagues,
you know the DrivAerML dataset, it's already 30 terabytes with
one time step essentially as in the time averaging, if you were
going to try and do it for all of them, 200,000 iterations,
well, you can do the maths, it's unbelievably large.
But if at every time we were running those simulations, we
were passing that data to a model to train, then, of course,
we don't need to save the time steps out.
We're just passing it in memory. So we actually could have done
it because we were running the simulations time- dependent
anyway. So I think this is where there's
a lot of interest in these. I feel this is where new
technologies are converging with AI as the sort of motivating
factor. So foundational models.
I, I really would encourage any academic who's listening to
this, I think this is the topic to propose as a, you know, a big
university project or European project or government project
or, you know, anything that is big.
Because if this could be made and there's so many debates on
whether it should be open source, should be closed source.
It could transform the way that
we are doing CFD today. If it is possible, it would
change the commercial landscape, it would change the technical
landscape. And if you imagine that it's
possible for a certain class of applications, you've got to
imagine the transfer learning, the fine tuning.
You know, at some point how different is a car than a plane
or a city if you are starting with, if the model is able to
learn some of these interactions.
I haven't really spoken about the idea of including physics
into it, but I I really do feel that just as five years ago AI
was an interesting thing for people to look at, I think the
whole concept of foundational models is becoming interesting.
I believe at the beginning it will be for specific use cases
as the audience voted, but I perceive that there will be a
bit of an arms race between the ISVs and startups to build
this, and it'll be interesting to see whether companies see
this as their secret sauce. Whether you know, an aerospace
manufacturer or an automotive manufacturer says, well, I don't
need the ISV, I'm going to do it myself.
And this will be an interesting balance between the software
suppliers, the companies, because on the other hand, the
automotive company can say, well, we're not a software,
we're a car designer. Why are we going to write our
own codes? And that's certainly been the
case even in the aerospace. There's a move towards
commercial codes. Well, you know, it used to be
the case, everybody would write their own codes and that was
their IP. And then they realised their IP
is making cars or planes, not writing software.
So it, I still suspect that most of this work, these foundation
models will still make their way into commercial sort of big
software companies, ISVs. But given that they don't have
the data, it's the sort of engineering companies who have
the data and they need to somehow get that data or produce
that data. There'll be an interesting
dynamic, I think in collaboration.
So these are just my personal thoughts.
I could be completely wrong, but I am quite bullish on the idea
of some of these foundational models.
And, and certainly, you know, in my day job, this is something
I'm actively pursuing. I'm I'm academically interested.
I'm interested in, you know,
NVIDIA doing its bit to help things along the way.
And yeah, I'll be really interested to see what happens
in a few years time. So maybe I I'll try and set a
reminder in two years to record another one and see how much of
this. I was right on and maybe it was
it'll all not happen and I was completely wrong.
But I'm, I'm going to make a bet that in a couple of years we
will have progressed quite a bit further than than than we have
done to date. So with that, thanks for thanks
for listening. I would really enjoy your
comments. Let me know what you think if
I'm completely wrong, if you have a different viewpoint on
it, please put your comments in the YouTube or, or send me a
message. And yeah, thanks for listening
and hope you enjoyed it.