Hi, and welcome to the Neil Ashton Podcast.
In each episode, we explain some of the fascinating ways
that science and engineering are changing the world around us.
We talk to leading engineers from elite level sports like
cycling and Formula One to some of the world's top academics to
understand how fluid dynamics, machine learning and
supercomputing are bringing in a new era of discovery.
We also hear some of their life stories, their career advice,
the lessons they've learned on the way that I hope will be
helpful to you too. So sit back and enjoy this
episode. Hi, and welcome back to the Neil
Ashton Podcast. So today's episode is with
somebody who I've actually really enjoyed getting to know
better and chatting at various conferences that we've attended
together and some mini symposia that we've done, which is
Johannes Brandstetter. So he's got an interesting
profile because not only is he a professor in Linz at JKU,
Johannes Kepler University in Austria, but he also founded or
Co founded a startup recently called Emmi AI.
His background is one that is very well positioned to help
advance the state of machine learning for, for for CFD and I
guess CAE more broadly. And yes, this is an episode on
machine learning again, I had sometimes feel bad that I keep
doing this, but I think this season there's a mixture of of
episodes and I still feel that there is value in going through
the machine learning topic. And it's selfishly something
that I I'm constantly interested in learning more about and
learning more about. It is one thing that I always do
when I speak to Johannes. And he has an interesting
background because whilst he has I guess a high energy physics
background that he he, he did and he was at CERN and he worked
in that sort of area. He then moved into the machine
learning side, I guess, having spent time with Max Welling at
the Amsterdam University in the machine learning lab.
And then I think he also, well, I don't think I know that he
was then at Microsoft Research where Max was also at.
And so the two of them collaborated a lot.
You know, if you don't know who Max Welling is, well, first
Google his name. He's one of The Pioneers of
machine learning. But I also did an episode with
him in the last season that I think was really interesting and
I think he must have been inspired by Max a little bit,
even though we didn't talk about it in this episode, because Max
also has been a serial start up founder.
And the reason I said that he has an interesting background is
because whilst he was at Microsoft, he also worked on the
Aurora machine learning for weather and climate project.
That I think is extremely useful when you have people who have
gone through these major projects in another field
because they bring with them lessons and learnings that are
important. And, and what Johannes has been
doing is first of all, bringing a very academic mindset to this
academic in the terms of publishing and transparency,
which I think is very welcome. So you'll find in the link in
the YouTube a couple of the papers that he's published and
and his startup has published a new sort of transformer based
model that I, at least from my reading, is unique and certainly
seems to be at the one of the most cutting edge in
state-of-the-art models out there today, both in terms of
conceptual but also the accuracy on the data sets that they've
shown. But I admire his vision and his
willingness to try and solve the problem rather than being
focused on the model, as in some people seem to, once they come
up with a model, fixate on that being it, rather than being
willing to consider that their model may be the right thing at
the time, but then there'll be other models that get better.
And he, his willingness to accept that, I think is a breath
of fresh air and definitely will help the community to, to
evolve. So that's what we talked about
today. Really we, we tried to go
through and discuss, you know, more the general topics around
machine learning, but really diving into this transformer
based approach that he has. And I'm trying to understand
some of his thoughts around the similarities to neural
operators, some of the slight differences, some of the links
to, I guess, graph neural networks and MeshGraphNets that
probably most people are aware of.
And then we talked a little bit about the process of forming
the startup and some discussions around future topics.
As with anything, I always feel bad because I finished the
episode and I think I really should have discussed more or,
or, or dived into certain topics, but I'm conscious it was
already over, you know, an hour 20 and people will, you know,
fall asleep listening to some of these episodes.
But hopefully it's enough to spark your interest and
encourage you to read more about the topics that we discussed.
And please leave any comments or let me know if you have any
suggestions for for other topics.
But for now, please sit back and enjoy what I found an
extremely interesting episode with Johannes Brandstetter.
Cool. All right, thanks for joining me
today. I wanted to have a chat with you
for a while. We've had some nice discussions
over wine and and coffee, but nothing public.
So this is the public version of our discussions, but maybe
it'd be good for people to learn a little bit more about you.
You know, where you're at today.
Yeah. Who?
Who is Johannes? Yeah, thanks first of all.
Thanks really Neil for for having me.
It's it's a great pleasure finally sitting here with you
in a one to one setting, so to say.
Yeah, my background is actually I'm I'm a learned physicist,
some say a failed physicist. But after my PhD, I switched to
machine learning, as many people also did.
I have spent time with Sepp Hochreiter and three years in
Amsterdam with Max Welling. I was in industry for two years
at Microsoft Research. And there at Microsoft Research
I really, really discovered my likings for large scale
simulation, especially for weather and climate modelling,
because these are like some of the biggest problems you can
have and you can tackle with machine learning.
I also saw the transformative impact on this, on the systems,
and that was then the point where I decided to to do my own
thing to, but to apply it not to known problems like weather,
but rather uncharted territories like engineering,
simulation and this kind of thing.
And I decided to get my own group at university to build up
my knowledge base back in Austria, where I'm coming from.
And along the road. It also happened that I founded
a start up Emmi AI where we do this simulations at scale and
where things are coming together with a bit more compute and a
bit more resources. Nice.
Well, we'll we need to dive into each of those, but maybe the
first one was the Aurora project.
I guess this is maybe it, it was simultaneously both a
motivation, I think certainly for me and others who saw what
was happening in weather and climate and was saying, you
know, surely we could do that in engineering.
But also it was an example of a, yeah, a real large scale
project. So what was it like working on
that Aurora? What was the key lessons you
learned both from maybe an ML point of view, but also a
project and a team point of view?
So the actual Aurora team was pretty small.
I mean, all the credits to to Cristian Bodnar, Wessel
Bruinsma, Megan Stanley and Ana Lucic who did that, all the
heavy lifting afterwards to finalize this project, to train
to build the data loader. The Aurora was the learnings I
think are threefold. First of all, with computer
vision tricks, you can get very, very far in engineering or
scientific application. In the end it was a Swin
Transformer trained in in a 3D Swin Transformer.
Secondly that the the data engineering is potentially the
hardest one and and thirdly that weather has a very unfair
advantage for machine learning because nobody knows how weather
is functioning. So you, you will always train a
machine learning model on like actual data, which basically has
the actual physical laws hidden.
Whereas the, the the numerical methods, they have to come up
with these laws by themselves. And that already brings these,
these key interests of mine, which is that that neural
surrogates will never replace numerics.
They will just be a different branch which depending on how
you play the card acts in your favour.
And that it's all about engineering and scaling these,
these things to this, to this basically to this problems where
they really have an impact and there really matters.
And the impact for Aurora, you could see if you look through
this, this small, through this downstream tasks that the larger
these challenges are, the more impact you can basically
generate. Yeah, it seemed to definitely
spur on. And now it seems that there's
been so many of the weather and climate models, it's almost got
not congested, but there's there's, there's quite a few
coming out. And so it's not obviously so
clear how each are progressing past each other and and how much
are those of just an individual groups need to have their own
model, if you know what I mean.
But did you, I guess one of the big things that I want to
discuss today with you in particular was to dive into, I
guess where machine learning, how can I phrase this machine
learning for engineering, for CFD, for CAE.
You know, it's something I've spoken to a few people about and
there's a lot going on, but it's hasn't always been so clear
where the big breakthroughs will be or how, how good are we
today and where we're going. It it's it's still seems a
congested space with different opinions, you know, with PINNs
on one side and then these sort of MeshGraphNets and other
things. So it's a bit of a, an
open-ended question to you, but I guess like where do you see
the state-of-the-art? What what's been your journey
from the sort of MeshGraphNets towards your current thinking?
Like how, how have you try to solve this problem, I guess.
And let's, let's say, let's take CFD, maybe is the the the
problem like automotive aerodynamics for example?
Yeah, that's a very good, my favorite topic actually making
the connections to weather. Let's let's say what what
basically was driving this weather modelling.
I would say that NVIDIA was the first with the FourCastNet
paper to to bring out a model which worked that was trained on
ERA5, which is a publicly available large scale data set.
Depending on how you sample you
get roughly a petabyte of data or a few 100 terabytes of data
from it. And then you do basically mean
squared error training that that was the the scene set and and
as soon as this problem was defined of this input output
relation of this metrics of of what to test, then people did
what, what they do best. They optimize on this sort of
problems. If you go to engineering, things
are very different in, in many, many aspects.
So first of all, we don't have this data set.
We don't have an ERA5 data set.
And even if you had an ERA5 data set, I mean, there are now at
least 2-3 publicly available CFD data sets.
Needless to say that my favorite one is the DrivAerML data set
which are industrial standard. But even if these data sets are
out there, people don't know how to train on them.
And everyone trains differently.
And this comes from the fact that we just don't know what you
really want to optimize. There is people who sub sample
parts of the surface and map to, to pressure values on the
surface, people who sub sample part of surface and volume and
so on and so forth. So similarly to whether we have
to figure out what's the right task, what's the right learning
task, which we which we have to to do or like the tasks in, in
order to really develop the right models in the right
frameworks for that. Secondly, I think what is also
very different to weather is that we have this input output
relation. So in weather, it's always clear
you take time T and you map to T + 1.
That's basically a segmentation task on steroids because in the
end every pixel on the Earth gets mapped to to a pixel on
the earth like a time time point later, which you can basically
you can take all the models from computer vision.
And obviously there is some tricks with resolution and and
if you take the sphere into account, yes or not.
But in the end it's it's a pixel segmentation task.
And this is definitely not true for for many CFD related tasks
because the task there is from a pure numerical perspective, you
have a geometry which you can characterize with couple of
parameters, probably less than 100 even.
And then you get this full-fledged flow fields around
and on the car so that the actual input to output ratio is
very, very different. And we don't have architectures,
we don't have frameworks, we don't have an understanding of
how to tackle that. But that makes it so exciting I
would say. And maybe thirdly, if you, if
you really look at these weather models and what they can do with
hurricane predictions and so on, obviously there's always
room for improvement and obviously you can go farther and
farther. But if you just look what
happened in the last two years and the resources and weather
are I would say it's tiny, tiny fractions to the resources in
LLM and and video generation. You see this tremendous progress
and what what hard problems and hard multi scale problems we
are already able to model and which we can model better than
numerics. So it's very clear that if done
rightly and if the right incentives are there, this
progress and this type of complexity we're able to do in
in CFD. Obviously then we have to play
the game numerics and ML together.
Yeah, which is then I think the hardest task to answer.
Yeah, that, that does seem, I guess the joke that the Earth is
always the Earth. And so the geometry is sort of,
you know, not so much the problem.
And yeah, I guess it all depends on what is the data.
You're right, in the CFD example, the geometries could be
hugely varying and that link between the geometry and the
volume is a Yeah, it it's so the geometry is geometry and the
boundary conditions is ultimately what affects the
flows. So where have you seen, I guess
a lot of people listening to this and myself included,
probably had their first introduction with, you know,
what the DeepMind team did with MeshGraphNets.
I mean, there was obviously stuff before that, but that felt
like the bit when everyone stood up and and noticed why?
Why do you not feel that that is the right approach?
It's always right or wrong. It's always hard to to to judge.
But I think that the, the MeshGraphNets and, and, and and
and graph neural network simulators from the Peter
Battaglia group were tremendously important
milestone in this field, mostly because they, they were writing
down this learning problem. And I myself went into this
field of simulations because of these papers trying to model
particle particle systems. What, what we noticed back in
the days, I mean, this is like 5-6 years ago, that it's
tremendously hard to model all particle particle interaction as
numerics is doing. So if you have this 10,000
system of particles and they interact and you have you have
something like a graph neural network and you have to make all
these interactions correctly because otherwise the temporal
integration will will just, yeah, just go, go, go crazy.
That's just a very hard learning problem.
And this is not what machine learning is really good at.
Machine learning is good at to understand the global dynamics,
to understand the behavior of the system, to understand where
things are going on a global scale, but not on a particle
particle scale. So I, I think from, from, from
setting up this learning problem, this was a tremendously
important step. Obviously new ways of, of, of of
models are coming the way for me personally, it was a big
breakthrough that we started to, to see things as fields.
So everything in my, in my head is a field.
And because you a field is something which evolves over
time. It can be an occupancy field,
which tells you where is mass, where is no mass.
It can be signed distance field.
It can be a velocity field, it can be a displacement field,
whatever field you want. But it's much easier for a, for
a model to, to tackle, which is actually obviously the, the next
step if you think of this whole scientific machine
learning and neural operator and, and, and so on and so
forth. And in the end, I mean, this is
also how you, you model the, the, the weather.
So I, I think conceptually from me personally, from me, I don't
know how it is for others. Understanding that you can model
large scale systems as a field was, was breaking a lot of
boundaries in, in terms of scalability.
Because if you model a system of 10 million, 100 million
particles as a field, it's computation and not the problem.
It's from a learning perspective, it's not the
problem because you just don't need all these particles.
You you'll learn the dynamics with the very heavily subsampled
representation and and and and obtaining the full field is also
not a problem because this this exists.
So I would say for me understanding what machine
learning is really good at and and and and and and what
numerics is really good at and and and and and playing those
two on, on on those two fronts was was was making the the
difference. Yeah.
Well, one thing that I've maybe would appreciate you explaining
to the audience and maybe let's first introduce some of the
work that you've done in this recent paper of yours.
So you developed this transformer-based approach, what
you call the Anchored-Branched Universal Physics Transformer.
What? What?
Well, maybe you explain what? Yeah, we.
Have actually a couple of works of 1 is which we call Universal
Physics Transformer. That's basically how we do
latent space modelling of physical systems.
Then we did something which is called NeuralDEM where we
applied those to to multiphase flows and and and multi.
Particle systems and then we, we, we recently applied this to
to large scale CFD similar framework, similar approach,
which we call Anchored-Branched UPT, because these are the two
ingredients which which which we need to do in order to get
this to work on large systems. So in that context, what I'm
hoping you could do is maybe explain a little bit, because
there's a lot of people who listen to this podcast who are
maybe not ML specialists, but are CFD primarily specialists
who are now, you know, more and more interested in ML and maybe
actually developing it and, and yeah, sort of progressing.
And you mentioned a really interesting point to me and I'm
hoping you can explain this in a way that is understandable by
others is that I, and I guess some of this is a maths
exercise, but you know, and I've been guilty of this.
Sometimes there's this hierarchical thought process of
being like, OK, you've got some sort of graph based thing where
it's more like point to point. Then you've got a neural
operator where it's like more to solutions or fields.
And then I initially thought of, OK, and then you have
Transformers. And yet you see some people who
say, well, everything is a neural operator if you do the
maths or design it. So can you explain and maybe
dive a little bit deeper into that concept of solution mapping
or field mapping and then how it links to Transformers?
Because I think it is a little bit confusing for for for
people. So if yeah, if we could maybe
get into that, that would be in in the context of let's say
this DrivAerML or car, however you want to explain it.
Yeah, happily to do so. So for me, in fact neural
operate and this is also how it is introduced is something
which maps between function spaces, which is very yeah,
which is not very informative if you if you, because what does
it mean? You map between function spaces.
But I always think of it as you
have two functions, one input function, 1 output function.
And no matter what you sample from the input function and what
you predict on the output function, this has to be
fulfilled. Meaning that if I sample 10
points from the input function, I should be able to predict as
many points as I want from the output function and those points
I predict are actually on the output function.
If I sample with the same network with the same operator 6
points from the input function, I should also get the same 10
points and if I want more points, I get more points on the
output function. Obviously there is a limit.
If I sample too few input points, then obviously the
approximation network is not able to really get the
information it needs. But if if this this this
sampling limit is is reached no matter how many points and
where those points are spaced, it should always give you some
sort of representation which allow you to construct the
output function. So in the latent space this
mapping is, then is then, so to say, resolution agnostic and
and that with some small tricks you can do with convolutions,
with Fourier neural operators, with Transformers and so on
and so forth. And obviously Transformers with
their. Can I if I've seen to it just
for one second, just to make it even more clearer, when you say
sampling the input function and the output function by input
function, you're meaning like the geometry surface in this
conceptual sense and the output function being let's say the
volume. Would that be 1 interpretation?
Yes, for example you can. You can think of it that the
input is, the is the geometry and the output is either the
field on the geometry or the field in the volume, or both
actually. And the idea of this so where
you know, this sounds like a bit of AI.
Remember when this first came out, I was I think it was billed
as a resolution independent. So it which always to a CFD
person feels like the no free lunch sort of theorem.
Like how can this possibly Yeah, you know, how can that be
true? You're basically saying I don't
need to map the entire geometry, I can just take a few
and still get the same answers.
So where's the catch? I I guess on this?
Yeah. I mean that that's a very good
point. And I think I can make my, my
point very clear with, with really now going to CFD and,
and, and before that, I really have to say, if people think of
resolution, we always think again of this segmentation as,
as we do with weather, right? You have a certain number of
longitude and latitude points. And if we double those points,
do we get higher resolution or do we just have an interpolation
and so on and so forth. So can we just run our whatever
unit and interpolate between this resolution or is there a
way to get this stuff resolved?
So there's always this input output mapping and obviously it
will end up with interpolation effect.
So either you interpolate somewhere in your network or you
interpolate on the output grid because as you said, no free
lunch. But if we go to CFD, things are
a bit different, especially that the input output
resolution, as I said before, is very different or input
output connection. And, and actually we show in
this recent paper that if you subsample a few of the points,
so both on geometry and surface and, and you do some, some
training, there is a certain amount of points, but the
performance stagnates. So if it's, it's for for DrivAer
ML, this is roughly 128K, I think or 256 I, I, I don't
know by heart. And if you then add more points,
the performance is not getting better because all the, the
information which in neural network needs is in those
points. And, and if you sample it, let's
say cleverly so, so that the distribution really covers the
whole surface and the critical parts.
And then also points at the closer to the surface and the
volume are represented correctly.
All this, this type of stuff. But there is a certain amount of
points where you really capture the whole phenomenon.
And that's the, the, the, the resolution invariant.
So to say. Obviously, if you go lower with
the point to sample, you lose a bit of representation.
So you don't, you don't resolve the full physics.
So you're never able to really recover the whole physics.
But there is a, a certain amount of points you need.
And this is for, for each type of physics problem that you have
the full information in the network.
And then obviously depends how good your network is.
And then obviously it makes much more sense to not have a
network which again spits out this 128 points, but which
conceptually is able to spit out as many points as you want
because that's the neural operator, right?
That you can really get every point on the output function if
if needed. And this is something which is
which is very, very interesting in CFD.
Yeah. So I guess, again, correct me if
I'm wrong, if I'm distilling this correctly, is you're saying
if the surface in reality has 8,000,000 points or whatever the
exact number is, you're saying that you should be able by
training and sampling progressively different points,
you shouldn't need more than 256,000 to actually do as good
job as 8 million. Whereas I guess we've the
analogy. I suppose what I'm trying to
pick out is with MeshGraphNets or with graph neural networks,
people could in theory take a million, but they would normally
like down sample, but they're down sampling at least.
Correct me if I'm wrong, you are then changing the problem
statement in this it it will give you a different answer and
and that I know was certain experience that we had and
others have had where you go well, how much do I downsample
and where do I pick the point to downsample?
And so a lot of people take an 8,000,000 cell can't fit in a
GPU's memory and take 500,000 but they're worried when they do
inference they then would have to give the exact same mesh
distribution or if they give a different mesh distribution.
Do do you know what I'm getting? At that's lovely.
I mean you, you say the most important point, which I forgot.
Obviously if you have to
calculate something with drag or lift coefficient for which you
you really need to get the correct value.
You need the full simulation mesh, the full 8 to 9 million
surface mesh. And obviously the network has to
give you this full mesh because otherwise you're never able to
calculate correct drag and lift coefficient.
But the network is giving you this full network idea.
Sorry, this full drag and lift coefficient on the full surface
mesh, no matter if you give it 64,128 thousand 256,000 input
points, it's just at some in number of input points the
output is stagnating or the performance is stagnating
because you have reached the maximum performance and the the
input and and obviously you can also give 8 million input
points. The the the self attention will
be terribly slow and I don't know what what parallelization
tricks you need to do, but nevertheless it will give you
the same answer in the output. So you can drastically reduce
what you give as input as long as the output still gives you
the correct answer. And that's, that's the magic we
we got from transformer, which we cannot do with with other
methods for. So for other methods we really
had to make the problem much easier, which is obviously
giving you the wrong physics. So if we down move on to and I
guess this is the the novelty that I saw in your approach and
and I thought it was good to double click on a bit.
It's why the Transformer, what specifically is that
architecture giving you versus alternatives because yeah, and
how and how that relate, if you could maybe go into a little
bit because Transformers, I think most people know
conceptually, you know, tokens, etcetera, LLMs, but maybe not
in the context of CFD. Yeah, so transformer are
tremendously flexible, tremendously optimised and and
and tremendously well understood powerhouse, workhorse in, in,
in deep learning, right. They are they the the engine
behind large language model computer vision and and all
these tricks are already done and they have a few certain
properties which are super nice. The 1st is the the invariance
with respect to sequence length. So no matter how long the the
sequence length is which corresponds how many points you
input to the network, it will do the same calculations, right?
Which is very important for the
scaling properties. And then they have this how to
say so I call it discretisation convergence.
So if you sample more points in a in an error in an area, it is
basically it will converge to some to some field.
So, so the more points to sample, the, the better the
resolution gets, which is obviously extremely valid,
important property for this neural operator paradigm.
They they are made. So the transformer paradigm is
made and, and stress tested again and again and again for
scaling, meaning that you can build larger models, that you
can ingest larger data sets. Yeah.
And, and, and all these things make them an ideal fit for us.
I mean, I'm very happy that not everyone is using Transformers.
That gives us some edge, but yeah.
But specifically, So how do you tokenize the problem then?
How could people conceptually understand, you know,
MeshGraphNets conceptually was: I have a node which
corresponds to my mesh, I have some edges and then I take
that like you're not taking a token per node clearly or that
wouldn't scale, right. So how could people conceptually
think of that sort of tokenizing the the surface or
the volume? I think that the way you have to
approach this is not from a physicist perspective, but from
a computer vision perspective. So what do people do in computer
vision? So I think when do, when you do
something like image generation where you type in a prompt, I
want a horse in the, in the in the woods with whatever.
So you have some text and then you mix that with some image
which is generated, right? So you have basically 2 streams
and, and the second important part is this concept of
patching. So the transformer for for text,
it's very clear each word gets a token.
So each word in a sentence or probably each each sign or
whatever is tokenized for for images, small patches of the
image will correspond to a token and a word.
So everything is a token which then can interact with each
other. But in in language, it's the
token's words in, in in vision, the tokens is small patches of
of an image. So if we do the same thing in in
CFD, what we actually have, we have basically three type of of
information. We have the information of the
geometry. So in general, what geometry we
have, we have the information of the volume and we have the
information of basically the physics on the on the geometry.
So it makes sense to treat this as either two or three
modalities as we call it, right?
So as as you have text and and and image and then you can
tokenize either small areas, which makes sense.
If you if you look, try to encode the geometry so small
neighboring areas get pulled into one token so that you have
this information which gets locally aggregated.
Or you can for, for actually doing physics, you just take
some samples of the mesh and see them as a token.
And then you have the same setup, you have different
tokens, they represent different physical objects or physical
parts of the physics. And then similar to what people
do in computer vision, you let those tokens speak to each
other. You link these volume tokens
between each other and the volume tokens to the surface
and so on and so forth. So it's really borrowing a lot
of concepts from computer vision because that's what works.
So if I understand it correctly, essentially what you're doing is
patching. So if you were to look at the
surface mesh and if you have, you know, visually, if you
looked at it and you had a clump of 5 by 5, then you, you would
have, you know, 25 cells and however many edges and nodes,
you would say, OK, well, that can just become one token and
then you go into the next, the next.
So if you, so essentially you're able to go from 8,000,000 points
and instead of taking 8 million tokens, you would have
you know, 10,000 or what whatever the number is, is that.
And the assumption is that the
differences within that patch should be small enough to be not
important. If you were to take too big a
patch, I assume you would lose some accuracy because you're
losing some local information. Would that be fair to say there
comes a cut off point? Yeah.
So if you if you talk about representing the geometry, what
you correctly said, you can do two things, right.
You can treat each point of the geometry individually.
It turns out this, this DrivAerML data set and so on
so forth. The information is so rich.
So in order to really represent the geometry, you need roughly,
I don't know, half a million points.
So it makes sense to pool them before, to aggregate them
locally, and then to use those as, as, as tokens.
That's just computationally much more efficient to have some sort
of pooling that the number of tokens is not exploding.
You could also obviously use representatives of the geometry
as your points, but it makes sense to pool them before.
And the same in the volume, right?
You're essentially, instead of taking 130 million nodes, you're
breaking them into patches which themselves take into.
Yeah, the The funny thing is so so there is this tool.
So we always have the the number of points which it takes to
represent physics correctly. This is 2 slightly different
things. So for the geometry we really
need a lot of tokens in order to really capture the geometry
because the geometry is influencing the physics and so
on and so forth. For the volume we don't need so
many points. So we we only need roughly
128,000 or or even less to really capture the physics.
So that's we are totally fine to pick those points individually
and actually performance is not degrading much if we go to 32 or
64,000 that will get more interesting if we get even
larger data sets. But then you can also use some
tricks. So there it's obviously fully
fine to use individual points. So in that context, is it fair?
A mental model is that you're
saying that the number of points that are needed for a
numerical solver should not be a one to one mapping.
That you shouldn't think just because I need half a billion
points to solve my PDE, that I should.
I should need half a billion points for my mission, my
machine learning model to learn the behaviour that we need to
break from this concept of almost one to one mapping.
It's that part of your argument, because that's not most people
that I think assumed in the GNN that I need to have this sort
of 1 to one mapping that I'm, I'm sort of taking my notes
and I'm putting that into my ML and that's the most logical
way of learning the problem. Yeah.
So I, I really like those questions, I have to say.
So I would my argument is, and I mean you tell me you're the
CFD expert, but I would say what a model has to be able to do
is it has to produce the simulation mesh.
So it has to give you predictions on the simulation
mesh both on the on the surface and in the volume.
And this is not only cars, it should hold for airplanes and so
on and so forth. And if we, if we think of meshes
which approach a billion of mesh points and where this is #1
condition, we cannot, we cannot use parallelisation here
because then we end up using thousand GPUs and this is not
scaling very well. I'm looking at the actual
information content. So it has to be the model has to
be able to produce this this rich outputs in order to be a
good tool to use. But on the other hand you don't
need this huge output for training.
You can train on much much smaller sub sampled version of
the same problem. However, you have to make sure
with neural operator learning with function approximation
that those models you trained are then in inference or in
test time. Able to give you the full answer
on the full simulation mesh. I think this is the true CFD
problem or the true learning problem similar to the ERA5
setup in weather modelling which we have to do in CFD.
So you yeah, so this is an interesting point that I it's
good to maybe briefly discuss which is at inference time.
So we discussed about training time, you know what you take,
how many of the points, how many do you agglomerate, you know,
etcetera, etcetera. But I guess what you're getting
to now is the inference time, which is something that maybe is
not so obvious to some people, but for me was always a bit of a
conceptual challenge, which is how do you generate the mesh to
do the inference on. So normally if you are running a
normal CFD problem or you're taking the DrivAerML data set,
you split it into a train and test, but both of them already
have a mesh generated. You know, we're giving it you.
So you already have the mesh. But the real life engineering
problem is I want to predict a new geometry that I've never
seen before. And the question is, do I have
to mesh it like I would mesh a CFD problem or can I just take
the CAD or can I just do some arbitrary case?
Am I understanding it right that this is sort of your argument
of the neural operator or the field sort of base approach
That if I take a new geometry of a new car that I haven't done
anything before on and I mesh it 5 different ways and then ask
your model to give me an A prediction, it should
essentially give pretty much the same answer for those five
different ways of meshing it. Yes, I think this is the true.
To an end, to a point. I mean, two things.
One, one thing I haven't said so, so I I was talking about
the, the difference in in training and inference mesh.
I also should say for for us it's very important that the
algorithm we are developing works for for CAD geometry.
So if you input inference only the CAD geometry, you're able to
obtain the full surface and full volume without giving any
information of the surface and volume mesh in inference.
The only thing the model gets the CAD input.
Obviously the performance slightly degrades, but I'm
amazed how well that works. And you can do that by using,
obviously in training that the simulation mesh and, and, and
using some tricks on the simulation mesh that because
obviously you need this information in some way in
training. But that you can tell your model
how to work with, with basically structured points in the in
the volume and, and certain points on the on the surface
such that inference it's the model is able to deal without
any of those meshings. And I think this is somehow the
true power of, of these deep learning models because they
cannot only scale. You can, you can think of that
the whole CFD compressed simulation is suddenly
compressed to, to neural network weights.
You just have to ask them the, the, the model, which region in
the, in the 3D you want to have the prediction.
But you can also do that from pure CAD inputs.
And so, so no storing, no whatever, right?
And yeah, this is truly exciting, I would say.
Yeah, I think that's one of the bits that I think is the true
test. Because if you have to generate
a mesh, the real time element of the prediction starts to go.
Because for many people, generating the mesh, the volume
of the surface is itself quite a challenge.
And I think most people would prefer to go straight from a CAD
geometry or, or at least be less sensitive, you know, to, to,
to, to the mesh. The, the only conceptual
challenge to this, and I, I don't know if we've even
discussed this before, but I'll just throw it out there, which
I think still is part of the challenge in people's heads is
if you do a CFD simulation, you would expect there to be a
difference with the mesh, right?
It it should give a difference if I run that DrivAerML with a a
grid that's half as coarse or twice as fine.
I want it to give a difference because by having a coarser mesh
I have higher numerical error which should affect the flow
field. And if I have a much finer mesh
I should see a difference because I'm reducing the
numerical error. I think where it gets a little
bit harder with the notion of sort of resolution independence,
is it sort of breaks from that mindset that, that I should be
able to give you a mesh that's twice as fine or twice as
coarse. And yet I get the same answer.
It it, it sort of messes with the CFD mind because you think
it should be different because I I'm expecting in my normal PDE
solver for it to be different but it's not remember.
Oh, it's, yeah. I mean, actually this was one of
the reasons how we developed this, this anchor tokens, we
call them anchor tokens approach because this very much is, I
think the neural analogy of what you're saying.
So what these anchor tokens are and basically they are the
reasons how we can scale. So we train on a selected set of
tokens on the surface and on the volume.
And those tokens, as we already said, they need to capture the
physics. So it's, it needs to be a
certain amount, but they are sufficiently well to train with
self attention everything on Transformers and inference side.
What we do, we, we, we pick these tokens.
Those are representative points in the whole 3D volume from
surface and the volume. And then basically those tokens
span the weights of the keys and values.
So in the in the Transformer, you have three different blocks
and those span two blocks. So you do a first pass of the
model with those tokens and then this is fixed.
So you basically have what you've built is you've built a
model which which is is a representation of of your full
flow and and then you can ask for specific points in the
volume or on the surface to get another value.
And these points can be arbitrary points, any point.
And this point will just run through the the model with all
these weights already fixed. So, so it only like a small part
of the model is adjusted and it will give you an output.
And obviously the more anchor points you have, the finer this
this resolution gets, the better this prediction are obviously
until a certain threshold, right.
So if you think of 16,000 points in the volume, you can still
reconstruct the full flow field, but it's it's not the full
accuracy. If you go to 32,000, 64,000 at
some point, if the training allowed it, you will have the
full field covered. And then with these, with these
points, you can get arbitrarily fine resolution and similar to
in CFD and you and you cover all the the points in space.
I think I see this anchor points really as as some, some some
sort of of of this computation cells in CFD.
Obviously the the discrepancy between number of mesh cells you
need in CFD to get the simulation to converge towards
anchor points you need in order to cover the full mesh.
Is is drastically different? But it's just because AI is
drastically different, different than the numerical simulation.
But there is this analogy. And obviously if you if you
don't use any of these points, it's very, very hard to
construct these dynamics. Yeah.
I think conceptually some of the interesting challenge of that
is, correct me if I'm wrong, but you, your ground truth is your
train data. So you feel that if you match
the train data, that's the best it can be.
Right. But the training data itself is
dependent on the mesh. And so one of the, I guess
research topics that people have discussed is, well, how about
multi fidelity? So you, you know, you have some
high fidelity data, some low fidelity data, but how?
I'm so going off a tangent on this a little bit, But one of
the things I always say to people is if you have a concept
of like a foundational model where you're saying, OK, I'm
going to have a model that predicts cars.
I'm more and unless my mind is just too static it it's a
foundational model, but just for the mesh design that you've
picked and the CFD method you've picked.
If I then go and change that CFD method and regenerate the
DrivAerML ML with like a RANS data set, I'm going to get a
different answer. So it's sort of hard to
conceptually imagine the ML being like a foundation model
because the, the input data itself is so dependent on the
settings you have of the CFD. So ideally it should really be
like experimental data should be the, the, the real ground
truth. But it's, it's sort of a little
bit of a philosophical thing where, yeah, the ML model is
only as good as the training data that it has.
And so if now you've sort of solved the ML problem, well, I'm
not saying you've solved it. It's almost the data is the most
important bit now. I couldn't agree more.
So, so I, I, I would say talking about foundation models for
engineering is very, very dangerous because you're kind of
data agnostic in a way. You in the end, what goes into
these models is a huge amount of knowledge in this data
generation. And as you said, data on
numerics is not the ground truth.
It's just one version of, of, of our reality, right.
And, and machine learning is not replacing whatever this, this
version of reality, it's giving a certain new access to this
reality. So I, I think the power of
machine learning comes when you, when you really pick certain
areas where it's tremendously important to have this
surrogates and to have this vast iteration to have whatever and
then really think what it takes to build those surrogates.
But I think from a modelling perspective we are especially in
this static systems, we are pretty far that we can give them
enough data and high fidelity data and so on and so forth.
We can build these systems with reasonable errors compared to
the data is trained on. The question is then always how
to interact with numerics. So if depends on what you want,
but I think that the power and the true transformation comes
when when numerics and CFD really go and deep networks
really go hand in hand and not as as two different parties.
Maybe I should also say to that we were doing this DrivAerML
data set now for I don't know 10 months like quite intensively
trying different things. We've reformulated the learning
problem ourselves a couple of times.
So what we actually want to train, what we actually want to
test and so on and so forth, because it's really, really hard
to because it's, it's just not a replacement of the CFD.
Your model is just doing something different and you have
to frame that there's a problem.
And and so that's why I think these two fields need to come
closer together that these questions are answered.
Yeah, no, I and I was just looking off to the side because
I was just double checking if I was correct that the ERA5 data
set is a combination of actual satellite data, right
observations with some sort of modelling.
So it is quite different in and I think that's kind of my point
that the weather stuff has literally got what the
satellites are seeing. Therefore you could argue is the
real thing because it is the Earth.
They saw what the weather's doing on the Earth, whereas in
the CFD it's, it is a, a just a numerical simulation.
And, and so that's the bit that I still think is a bit of the
limitation, but we are so sparse when it comes to data.
This is a bit of a segue into a next question to you, which
has long been the argument of the PINNs people, which is are
we ever going to have a vast quantity of data to train sort
of data-driven models? Is is that a reasonable
assumption or is it better to say, well, actually we have
pretty well defined equations and rules and models and is it
not just about using ML to solve them faster?
Like do we need, if we have such a a sparsity of data, do we
not need to include physics into the models?
Where do you sort of lie on that conundrum of just generating
more data and let the models learn off the data or do we need
to incorporate more physics into these models?
I'm very pragmatic in this. I mean this, This was the same
in weather modelling. Everyone was talking about
physics informed, not physics informed.
But people only talk for, for, for that amount of time until
they realise how heavy the data loading is and how hard of a
machine learning problem that is.
Because you don't want to talk about the physics-informed
neural network if you have to, to check it between loading
gigabytes of data for one data point on the GPU and what to
do. And in order to get this correct
and in order to get the decent learning signal and so on and so
forth. And I, I see it the same for
this engineering simulations. We first have to be able to
really run simulations. At scale.
And I'm not talking about 10,000 cells.
I'm not talking about ShapeNet, I'm not talking about even
even the DrivAerML. I'm talking about airplane
simulation and I'm talking about transient simulation and that.
And we are still far away from doing that both from how we
generate, how we store, how we train data.
And I think only if we, if we are able to really have a
workflow there and know what it takes and know where it breaks
and, and, and where all these problems are, we can then start
thinking about which type of physics to include and and which
not because, I mean, we, we also showed that in our paper,
it's very easy to build a divergence free model for, for
vorticity, because you can bake in the, the divergence free
constraint, their construction a hard constraint.
We try the PINN loss for the, for the mass conservation, which
is much harder to just take the degrades performance.
It in principle works. And you would need some tricks
and I would say getting this to work is not the amount of time
which needs to do all the scaling.
But I would say that first the problems we have to solve first
are the really how we interact with numerics, how we scale to
these problems and then how we actually get industry ready.
Well, you, you mentioned the beginning, the sort of T to T +
1 analogy when it came to weather.
And clearly the bit that we missed out maybe in our
introduction is that the engineering, we're not doing
that right. You're going from yeah, straight
to a time average or something at least, at least in, at least
in, well, in in the sort of way that most of these data sets for
cars and planes have been to date.
Given. How would you change the
learning problem or the ML architecture if now you had a a
full time history and would that help, hinder or not make a
difference? I mean same argument as as we
have in in the spatial. If you if you look at the the
neural network which has a few megabytes of weights and which
compresses, I don't know the, the GB CFD simulation or 10 or
20, I don't know how many GB actually CFD simulation, you can
also play the same game over time, right?
So neural network is just a very, very good way of of
storing this, this information. And if you somehow manage to to
store the temporal part in, in some sort of modelling, which
allows you to give you the the temporal answer to a certain
problem, this would be tremendously huge gain.
Because suddenly you can really simulate stuff in, in, in, in
real time, but also have the the ability to observe what's going
on and, and, and to really understand the temporal
dynamics. Whereas the training is just
somehow pumping this simulation into the neural network weights,
ideally without storing them. But that's that's very, very
hard. And then probably the next to,
next to next to step. But yeah, I think first step is
to get to get the decent neural network architectures for
temporal modelling, which we are actually working on to be
honest. OK.
And, and would you have the sense to be something similar
where you know, you may want to have a million time steps, but
that you'd expect that there would be some way of, I don't
know, is there, is there a sort of equivalency in terms of at
inference? Ideally you wouldn't need to go
1,000,000 steps, right? Do you have a sense of how much
this is more question for people who are generating data,
You know, do I need to generate a million time steps of
data so that you can learn the million?
And do you expect that then you would be able to predict to the
millionth but skip out every hundred?
Because you, you don't need to, you know, to do that?
Do you think you could learn the temporal problem without every
single time step from the traditional PDE?
Do you know what I'm trying to get a sense of?
Like the how the temporal challenge would link to the
spatial challenge? I think it's it's very much
relatable. So in in space we see this
phenomenon that you don't need all the points to to to capture
the physics and then depend. Then you have a huge advantage
in learning because you can always sub sample different sets
of these points which give you a huge data augmentation.
And you have the same in a temporal domain.
You don't need each time step in order to to present the full
physics to the neural network. But obviously the the fine
grain, the more the more fine graining you have in your in
your solution, the better you can do the data augmentation in
the temporal dimension. But obviously machine learning
models can do much, much bigger time steps than numerical
models. And yeah, this is something
which which is a true advantage.
They just don't work the same and they don't have stability
issues but also no stability guarantees well.
I was going to say, isn't that one of the problems, the roll
out problem that, you know, I guess if you do inference
inference, inference inference, you know, isn't, isn't these
sort of instabilities going to build up or like, you know, how
do you constrain it? Is this where some of the
physics needs to be added? Do you think to sort of if you
need to do 100,000 iterations, how much are you going to
guarantee that just a small difference in that first is not
going to, you know, bifurcate into different solutions?
Yeah, I mean, there, there is 2 to answer.
So that one is obviously you. Luckily people do video
generation nowadays. So we get a lot and lot of
tricks of video generation, which actually really, really
start to make this thing stable. And one of these, these
definitely modeling paradigms is this generative model that you
model the distribution over time and that you kind of make sure
the distribution always stays the same.
And, and this boils down to a bit of, of physics
understanding, right? If you, if you make sure the
distribution is the same, if you make sure that that future and
past they considered that there is attention across the right
axis and so on and so forth, we get stability.
It's always depends on how you define physics.
For me, it's the information the model needs and the way you,
you, you, you interact with future and past and so on.
I truly think that this generative modelling is there's
the breakthrough in this, this rollout stabilities.
I remember when I ran first into this rollout problems, it was I
think 4 years ago and, and, and basically I, I did this paper
together with Daniel Worrell and I texted him and saying, hey, I
don't know, we have to do something.
Our rollouts always exploding. And then I have no idea.
And, and he said, well, yeah, there has to be some, some
tricks in literature and then something smart and, and, and
definitely people have thought about it.
And I was like, yeah, I cannot find anything.
There is like, why is nobody having this problem?
And it's basically just a very different problem we have in in
in this engineering task that the the input itself each time
step is so huge from information content that every
autoregressive model, every every autoregressive trick you
you had before is just not working because they work with
much smaller input vectors. But but now that computer vision
is going to videos and that people are aware of this problem
and that frequency spectrum need to be conserved and so on
and so forth. This gets really a lot of lot
of progress. And I would say that that the
having the frequency spectrum stable is is is what made many
of these things stable. So it sounds like in a way there
is still a lot of potential to learn from the other advances.
Yeah, in ML, you know, the fact there's always an analogy
problem I guess is what you're saying.
There's the stuff you can learn in terms of video generation,
text generation, weather, climate week.
You can borrow ideas from other fields, but it suggests that you
therefore need to have people with a a broader ML background.
You know, if you just have a
fluids background, you may struggle to solve this problem
because you need to know what or you need to have an
understanding and an interest in looking at what's video
generation doing or what's this doing.
Would that be fair to say? This is why it's a
multidisciplinary problem. Almost all you need a team that
is multidisciplinary. I mean that that's happening in
biotech and in other areas as well, right?
We are just very in agnostic to that and language and vision
because you don't need a linguist to build the LLM and
you don't need basically, I don't know even what the
computer vision like. What's that analogy there?
But for, for, for, for all this AlphaFold and so on so forth.
The surely had domain experts on the team and surely this is
what, what's the exciting part about engineering you, you have
problems which are so hard to crack that you need the top
notch machine learning people that the top notch domain
people, they need to speak the same language.
And even if you can borrow a lot of concepts, they need to be
heavily adjusted. Because the problem with this
large mesh is with what's the input, what's the output?
There's this huge ratio difference and, and, and, and
turbulence and whatnot. This is just a, a totally
different piece to, to crack. So that's what's the exciting
part here. I, I wanted to maybe pivot it a
little bit to the process you went in creating a start up.
I, I think a lot of people would be interested to know and I'm
sure other people, you know, are in a pub or, you know, alcohol
doesn't need to be involved, of course, but you know, there's
somewhere and they've got ideas and, but not many people
actually go out secure funding and, you know, create a start
up. So obviously not giving away any
sensitive details, but what was that experience like for you?
Like how did it begin? How did you deal with venture
capital companies, how to like?
Yeah, I'd be interested if you could share some of your
experience in founding a start up, some of the lessons learnt
maybe. Yeah, so I mean, I, I was really
lucky. So, so first of all, a lot of
knowledge comes up from my university.
I, I have to know, I have to say that the people like Benedikt
Alkin and and Tobias Kronlachner who are really, really pushing
the efforts, they have been with me from, from day one actually.
Also people like Stefan Pirker
here in Linz University is a well known simulation expert who
who knows all these tricks. When I met him the first day,
basically after returning to Austria, we were 5 minutes and
we were both like super excited that, that we do things in this
area. And yeah, we, there was a lot of
momentum there. We were at a different company
and NXAI where at some point it was clear that the simulation
team, which was was really working well, should branch off.
I was also lucky to have met people like Dennis Just and Miks
Mikelsons, who did this before, created startups, very
successful startups and actually knew what to do.
They were the first to listen to me when I told them, hey, this
engineering like scaling these things up.
What we do to this billion of mesh problems is actually what
what industry needs. But just give us some time.
We will figure that out. And so I think it was all very
natural. It was not planned, but it's
obviously a huge excitement if you can, if you can do what
you're really burning for in, in, in also in a, in a
commercial set up where you you really can make an impact with
what you're doing. So yeah, I would say in that
sense I was very lucky. And I would also say that
especially in this area, we, we see a lot of momentum generated
and we're not the last start up and there is hopefully coming
more it, it, it there is a momentum especially in Europe
now in in this area. And yeah, just very exciting.
I would do it exactly the same way again.
And so how do you think you could have done this any other
way? Do you think you could have come
up with what you're coming up at a company or at the university
or is it is this is the start up really the only way of I asked
this question because I asked the same thing to Max Welling
and he was sort of saying, well, yeah, start-ups is the only
place where this can happen. The university's either too
small and doesn't have enough funding or companies too big and
too slow. And the start up is this sort
of, but it feels like in Europe, but often a bit risk averse.
We almost feel like start-ups is for somebody else.
But I'd be interested to know your thoughts on.
That I'm not so sure if what we're pulling off.
We just need the, the, the best people and, and, and, and some
compute. One thing which is really,
really, really, really hard is the problems.
So it's actually we are choosing our customer mostly by the,
the, the, the scale and size of the problems they have because
they're really interesting problems come from industry and
the more challenging and those problems are, the better they
are for us and, and for developing us.
So obviously we now have have a customer which have huge
temporal problems where you have, we have different
modalities interacting and this can bootstrap our temporal
modelling capabilities and and and and and and and and, and,
and this is something which you wouldn't get in the industry I
at university. So I would say it's a mixture.
I would say for me, it's, it's, it really helps to get hands on
to the, to the real problems and, and on, on the other side,
it's, it's just helps me to get into touch what industry really
needs and, and not sit in my ivory tower.
I wouldn't, I couldn't stand doing things where I know people
wouldn't use them. So this is a good, yeah, good
mix. But I also have to say there is
luckily some people like Neil Ashton who generate publicly
available data sets that that people can really start looking
at that We just have to give the community the urgency that
that they really look at the hard data sets and not a
ShapeNet car. Yeah, it, it seems like the
there is definitely a movement in the industry towards this
machine learning problems. You know, every time I go to a
conference, there's more and more people doing it, but it
still feels like we haven't reached that inflection point.
It still feels like you're either a start up or you're, I
don't know. It still feels like we're not
fully at that yeah, inflection point in terms of mass adoption
of this. I feel like we're getting
closer. I'm sure you feel the same.
You know, lots of industries interested, but maybe it is the
data that's the issue. Maybe it's the ERA5 was such a
large open data set that data wasn't the issue and therefore
people just got into the modelling.
Whereas I feel like now there's been quite a lot of advances in
let's say road-car external aerodynamics.
But road-car external aerodynamics, I don't know
percentage wise, but it's it's a small percentage of the entire
CFD domain. And yeah, yeah, it feels like we
still need to convince maybe the whole CFD community to do
more, to really advance it and make this a more systematic, I
guess. Would that be true to say that
you can only work on the problems that you updated for?
I mean, it sounds like an obvious statement to make, but I
assume if you had 10 times more data, you could potentially
go and hire more people and work on more problems.
But you're not going to go and hire more people and work on
other problems if there isn't the data or the willingness to
do this. And, and most people also don't
know what to, to optimise for, right.
So, so I think the, the, the industry or the, the, the real
problems need to need to set the, the, the, the, the, the
stage they need to set the, the machine learning problem.
Then we can start to think of how to solve it.
Yeah, I, I, but there is a momentum.
People are still reluctant because getting a NeurIPS paper
easier on a smaller data set than on a larger data set.
But but I think there is quite some movement and and and and
things are are really changing. And I also from for me, it's,
it's kind of our philosophy that to be a bit open source to show
our models to show what we're doing to publish, because I
think this is this is the way to go forward.
Closed, closed doors policy is is is not, not what what what
gets progress. I mean, we we see that the whole
progress in LLMs due to open source and then and so on and so
forth. And I think this is also what
will, not what should, but what will happen in in this space.
Yeah, I must say that I do commend that you're releasing
the data and the models. Open source is is a novelty.
And I think, yeah, it's good to see because I like particularly
your paper now. I mean, I think that has hurt a
little bit of trust in the CFD world so far.
Instead, as you know, the tradition and I'm don't sure
where the tradition came from. If you develop a turbulence
model or you develop a numerical method, whatever it
is, it's almost always published.
It's very rare actually for a model to make itself into a
large commercial, even, you know, ISV CFD code and not to
have a corresponding publication.
It's sort of seen as like a thing that you don't make money
on the method, you make money on the code.
That's sort of been the like thing like OpenFOAM or STAR-CCM+
or Ansys Fluent to others, typically the models themselves.
You sort of know it's more the the the little tricks that you
might do and but then more the support and the code and all
that. Whereas it felt like so far in
the ML4 CAE world, with the exception of a a couple of
individuals or or companies, most of the start-ups have not
published or made a thing of saying, here's our exact method,
but we've implemented it in such a way that, do you know what I
mean? And I feel like that's hurt a
little bit of trust because it's hard for someone to believe
results if they don't see a publication about it.
So yeah, I don't know if I'm alone on this, but I think your
strategy of publishing more, I think actually helps at least
CFD people to feel a little bit more like comfortable.
Which I, I, I think it's the, it's the only way around.
And I if I make a bold statement, if in the machine
learning world where things are moving so fast you're afraid
that others are are copying your stuff and overtaking you, you
should probably reconsider your your capabilities as a company.
Yeah. And the only thing I would say
though, which is why I've asked you sometimes to up level the
explanations, is I feel maybe even the biggest barrier is just
an understanding barrier that, you know, if you're a CFD
person, you're probably going to read an ML paper and be like,
I don't really understand this.
That's almost the challenge. You know, it's like an education
thing is that the people making decisions or higher up
may not even understand because it's such a different
field that that is the challenge.
So anything that you can do to make like, yeah, simpler
versions of papers or like summaries of papers or, or
education, which I guess comes back to why having a dual
affiliation between a university and a start up is probably a
good thing. Because I feel that you need to
be teaching and producing like open source teaching material to
educate people as much as also producing commercial solutions.
I don't know if you've noticed that when you speak to customers
that sometimes their knowledge of ML is part of the blocker.
Yeah, but it's also the other way around, right?
There's hardly any people who have the knowledge of the real
domain expertise and knowledge of ML because only if you have
the knowledge of domain expertise and you know what ML
can do, you really find the problems where ML can really
have an impact, right? And, and and this goes in both
ways. So education both ways is super
important and it's very, very hard.
I mean, I, I see that myself all the time that I can get lost in
details and, and, and think things are obvious.
And on the other hand, when I talk to real domain experts I
get get really lost and have to ask questions and yeah.
Yeah, yeah. Well, maybe This is why it still
will take a few years to reach a greater maturity because both
sides need to upskill themselves and, and, and sort of
learn. So great.
Well, really appreciate you chatting.
I mean, what I'm going to do is put some links in at least on
the YouTube to, to a couple of the papers and the models that
you that we've referred to, because I think people should
really spend a bit of time looking at the papers.
I have a feeling that, you know, we might need to chat again.
I think this field is moving so quickly that everything that
we, it'll be interesting to see if we listen to this
conversation, a year's time, are we going to be completely
wrong or in two years time? I very much hope so.
But what we won't be wrong is that some things are moving fast
and that we will have fast progress in two years.
So we will be, oh, we didn't expect that to happen, so.
Yeah. Yeah, that's going to happen.
That's a good thing though. Maybe that maybe that's OK.
But yeah, for now, then, until we speak again, thank you very
much and yeah, hope people that enjoy looking at your paper and
your work. Thank you very much, Neil for
having me.