Hi, and welcome to the Neil Ashton Podcast.
In each episode, we explained some of the fascinating ways
that science and engineering are changing the world around us.
We talked to leading engineers from elite level sports like
cycling and Formula One to some of the world's top academics to
understand how fluid dynamics, machine learning, supercomputing
are bringing in a new era of discovery.
We also hear some of their life stories, their career advice,
the lessons they've learned on the way that I hope will be
helpful to you too. So sit back and enjoy this
episode. Hi and welcome back to the Neil
Ashton podcast. So today's guest is Professor
Nils Thuerey. He's an associate professor at
the Technical University of Munich TUM and has been a
pioneer in physics-based deep learning.
We had a really interesting discussion today on first of
all, some of his background, Very interesting that he
actually won an, an Oscar working in the visual effects
industry for, for movies after his postdoc and PhD before he
then turned to academia full time, which is kind of
incredible actually, and is interesting because I've noticed
the lessons learned from the visual effects industry in terms
of photorealistic representations of, of, of the
world is actually something we, we then talked right the end of
the, the podcast. So make sure you listen to the
end about the link between foundation models and surrogates
to world models. These world models that are
being created essentially as a synthetic environment to train
autonomous vehicles and robots.
And we actually talked a lot about whether there needs to be
more physics in these world models, which I actually thought
was a very interesting discussion.
And we only got to it at the end.
But you know, he he's been one of these people who was doing
deep learning in the mid twenty 10's and early twenty 20s before
ChatGPT and the sort of big rise.
And it was interesting to him talk about how his work was
looked in those times compared to now and some of the barriers,
you know, and, and skepticism maybe people had.
He's also been someone who's really pioneered open source and
pushing the boundaries of differentiable physics and some
of the codes that him and his team has developed and now have
moved into both generating data sets like the SuperWing data
set, but also addressing the challenge of foundation models
also from a sort of PDE point of view in the latest Tadpole
paper. So we, we, we kind of go through
some of those topics around those different strategies of
sort of PDE based methods, you know, pre training on those
versus doing more of the traditional, I guess, building
out large data sets. We talk about the role of
startups and industry and academia.
Interesting to get his perspectives on that and how all
three can work together to solve some of these problems.
And we also talk about the open source, why he's so pro open
source and how we think that can help progress the, the, the
field along. He's, he's actually contributed
together with his team, many important papers covering not
just on fluids, but also on the, on the weather side, interesting
like WeatherBench. So I, I put a whole link list of
papers in the chat notes that you can in the show notes that
you can have a look at. But I, I really enjoyed this
conversation. He's someone who is very modest
in, in what he's done. But if you actually read his
papers and look through, you can see he's being quite influential
in steering these topics and always seems to be one step
ahead of where most of the field is at.
And so if you look at his 2025 and 2026 papers, you'll know
what I mean by that. So hope you enjoy this episode
with Professor Nils Thuerey. Well, yeah, thanks for thanks
for joining today. I, I've been very keen to speak
to you because as I've been getting up to speed myself over
the past years on machine learning and differentiable
side. Your name is a, a constant, you
know, top of the list in terms of influential papers, in terms
of doing things what seemed to be earlier than most others have
done, you know, setting the scene, which suggests to me that
you have quite a good outlook and, and, and thought on it.
So, yeah, thanks very much for for for joining.
And yeah, maybe as a starting question, you took a diversion.
Well, not diversion, but you
spent some time doing visual effects after your PhD and
postdoc maybe. What did you do your PhD?
What was the postdoc? And then I'm I'm intrigued on on
the movie stuff. Thanks for the invite.
First of all, Neil. Yeah, I'd be happy to tell about
this. My background, right or over my
career, I have switched fields a couple of times.
My PhD was in a in a computational numerics
computational physics lab, basically targeting multigrid
methods and and fluids. And then I switched to computer
graphics for a while. I guess that that actually was
one of my driving my motivations from the start.
I guess I'm a visual person. I'd like to see how things
evolve. And I think that's that's still
actually one of the main things that's fascinate me about
fluids, these swirling motions and the things that actually do
come out once it's working. So we also worked on computer
graphics, basically, for computer-based animation for
quite a while. My postdoc at ETH and then
afterwards also did actually work in industry for a while
because I was curious how things would actually work out in
industry in a practical environment.
And partially also because I couldn't really get a good
faculty position at the time. So right, not not completely
voluntary in a while in a way, but right, it was good to see in
the end. I noticed for me, research is
the more I think the better field.
But yes, that's why I did my postdoc, basically.
And then I think that if you 10 years ago, we started looking
at machine learning. So previously we did in this ETH
group what was called data-driven approaches in
computer graphics, essentially also trying to work with data
and right, somewhat aligned, but we didn't have the tools back
then. So it was, it didn't really
work, to be honest, but we tried.
And then when AlphaGo came along, I thought, oh, this has
to be something that works. I didn't understand it at the
time, but I thought this, there's got to be something
there. And then, yeah, since 10 years,
I think we've been working on, on basically finding these
applying these techniques to numerical simulations and fluid
mechanics specifically. And I think, in a way, it's a
good time at the moment. It's finally starting to work.
But right, it took a while. So you then joined TUM and
you've been there ever since. That was your your period.
And so where did you? What was it like?
I'm trying to put it into the picture because it's very hard
to the people who were working on these things before now, you
know, before ChatGPT, what was the field like?
What was the community like? What was the reception like when
you were bringing up ideas of sort of data-driven and machine
learning? Was there a lot of skepticism
and push back? Yes, actually there was.
So I mean, initially, especially in the early phases, all these
AI-based—back then called deep learning— techniques were
focusing on, on learning descriptors.
I remember our very first papers and graphics, they, they
basically couldn't synthesize anything.
You learned some reduced representation.
They told you about what happens in high dimensional data, but
right, it, it couldn't really generate anything.
Also, if you think about AlphaGo, the early success
stories, it was basically 8 by 8 with, with three states.
That's a tiny domain and tiny state space.
And back then that was challenging.
So in a way, it really couldn't do much.
And I, I remember quite well talking to colleagues and
telling them, look, we're interested in this deep learning
stuff. We, we're doing simulations.
And then some of my colleagues literally laugh me in the face
like, you're doing learning for PDEs.
What, what could be more ridiculous?
If you have a PDE and you can solve it, why, why, why on earth
would you do anything right learning based And I think so
there was a lot of scepticism for years.
I always had to defend in the very first slides of my talks
that this makes sense at all, at least makes sense to consider
this combination. And luckily, interestingly, I
think for the wide community, I think the picture really
changed with ChatGPT. I think in computer science
and and computational fields. People have been aware earlier,
but since ChatGPT, it's really on everybody's plate and in
every view but before. But you but you were
interestingly was looking through some of the papers and
you you had slightly earlier on probably some of the first ones
are sort of like in the AIAA, the deep learning of like some
RANS of airfoils, and so quite a few in the 20 twenties before
ChatGPT. So what was it like sort of pre
and post? Was it was the issue partly the
compute side? Was it the the architectures?
What? What was holding back?
Exactly the the architectures were a fundamental issue.
I think the whole infrastructure right in the beginning, CNNs
were were a big breakthrough, yes, before you had MLPs.
So CNNs definitely for field-like data worked quite
nicely, but also right the, the, the shift from basically
computer graphics to things like aerodynamics.
The a paper you mentioned, we're a bit motivated by the
observation that it's, it's really tough to synthesize stuff
in 3D plus time. You essentially have 4D fields.
Even if you simplify it and and
treat regular geometry as regular grids, this is really
difficult to do. It's still challenging these
days actually with, with neural networks.
So back then it was right bridge out of the question to do
this. And in graphics, you actually
have extremely good approximations that aim for the
visual aspect of, of things. So it, it was extremely tough to
compete with that. And it was interestingly in the
computer graphics field also, if you cannot directly apply it
on a scale and, and with a fidelity that's fit for movies,
then your papers typically get rejected.
It's really, it's computer graphics makes sense, but it
needs to look good in the end. So write this 2D proof of
concept simulations. Even if they were fairly
accurate, you you couldn't do much.
It had had to be right movie quality basically.
And and that just was extremely difficult.
Only also for niche applications like super resolution.
This was was one that worked very nicely.
But they can look at repeating structures and very local fields
for synthesis on on larger scales, it was basically not
possible. And that's why we also started
actually doing these proof-of- concept papers like the RANS
predictions by AI back then also these were tiny
resolutions, right? 128 by 128, basically.
So very small domains. And also I remember back then
actually, I'm glad it got into AIAA, but there were also
critical questions even back then.
Why would you even do this at all?
This is crude approximations. Does this make sense?
But yeah, basically we noticed in in aerodynamics and in
engineering applications, there's much more demand for
getting things right and really going forward converged and
solutions that are accurate and can be evaluated.
And there are many problems there that that are like really
important basically for for a variety of industries.
So we basically switched from from graphics trying to apply
these for engineering applications.
And where did the, you know, PhiFlow and the sort of
differentiable physics, where did that start to come into the
grid? Because that seems to be now
quite a hot topic. But you were publishing in 2020
on these, so where did that first come from?
I. Don't even remember, it seemed
like a very obvious thing. We were working on computational
and numerical methods, the the networks couldn't really
generate a whole simulation. So why not start with the
simulator, combining it with the network and then if you want
to get a gradient through it actually need differentiability
on on the solver side and right the the available ones couldn't
do this. So we just started experimenting
with our own solvers. PhiFlow became a good testing
ground for very basic methods across different APIs.
Back then, right, there was TensorFlow, PyTorch and JAX
and the whole bandwidth of of different methods.
So I think it was good for experimentation and flexibility.
And ultimately we notice here it's a certain trade off.
Flexibility and generality don't go too well together.
And so now we're specialising a bit more.
But PhiFlow was one of these early testing grounds.
But for us, it really seemed like in a way obvious, obvious
way to go. And then later on, we notice,
yeah, there's actually quite some work to do and also
interesting questions. How do you get gradients through
long chains of solver operations, which is still a
topic? So maybe shifting a little bit
towards the what's going on right now.
I mean, there's 22 fundamental questions that I see you've also
been trying to tackle and be interested to get your thoughts
on it. And that is around, I guess the
foundation model topic. The idea of I guess it's
logical, isn't it? You look at ChatGPT and you see
it as essentially a data-driven problem that you know, applying
up data, have a decent architecture and you solve it.
But you well, two questions. One was on the idea of the
ground truth and can you ever surpass the ground truth?
And you know, one of your recent papers looked at can the model
do better than essentially the quality of the training data?
And the second one is more just a general question on where you
see the likelihood of foundation models going.
How, how realistic is it? But on the first one, maybe you
could explain that paper because I that really intrigued me
actually, right? There was actually a great
observation by Felix, one of one of the PhD students from my
group, that he noticed in some situations.
He also looked at basic combinations of solvers and
networks and that's the networks really seem to give extremely
good situations in certain situations extremely good
results that seem to be even better than the the training
data we essentially used for these cases.
And looking into it a bit more, it's unfortunately not a topic
that I think can be generally applied and when analysing it in
terms of numerical errors and, and how these pieces fit
together. It's basically a combination of
the priors imposed by neural networks in terms of smoothness,
in terms of right, being able to over sharp more reproduce and
be trained for producing certain parts of the solutions that
benefits the the outcome. So in a way, if you can leverage
or if you understand your solutions enough to leverage
these particular behaviours of neural networks, then basically
the outputs can be better than what you're trained with.
But it's really this interplay of, of what neural networks do
as essentially also numerical methods to compute a solution
and what, what's in your data and what you like to get out.
So it also it raises some interesting questions on how to
evaluate it because if especially in the numerical
field, we're trusting the ground truth, there are also
typically approximation errors in there.
And it it can actually happen that solutions output by the
network might be even better if you consider other solves or or
other ways to approximate the same problems.
But I think it's we also for this paper thought about it.
It is very difficult to really do this in a targeted way.
So we had some new cases where it worked for some basic
advection-diffusion and Burgers. Burgers equation was a good
example actually because the the shock representation and
smoothness—Burgers equation is non-trivial, but seemed to be
a very good showcase for this. Directly applying it to other
scenarios is is challenging so we haven't unfortunately we
haven't gotten around to finding a good way to leverage this
for real-world Navier–Stokes problems or so.
But what about foundation models in general?
Like, well, how was your group tackling this?
What? What's your strategy in a sense,
you know where, where, where do you see this going?
Recently we switched pretty much from these differentiable
solvers to foundation models. I was quite skeptical of it
for quite a while. I think it's also there's also
just quite some skepticism talking to other people in the
field with foundation models. I think partly it's because of
the name, right? Foundation model sounds super
general and I'm sure you and and most people know these fields of
PDEs are extremely diverse. The thought of having one model
that just is everything out-of-the-box is yeah, but is
is a bit unbelievable and I think also still out of quite a
bit out of reach. Nonetheless, if you just right
foundation models also by now I think it's actually not so well
defined. People use it in in all various
forms and where I think it makes a lot of sense and in a
way it's also the direction we are typically tech typically
tackling is to see it as a model that's pre trained on on
something and then applicable to other tasks, other problems.
So there's just some disparity between training data and the
actual application we saw. We want models to generalise.
Also, generalisation is not a well defined topic, right?
It's in distribution is clear, but how far out of distribution
you need to go to have proper generalization.
It's completely open. But in the past that's been
really tricky. Anyway, it's a classical
transfer learning problem. Now, I think with these
foundation model tools borrowing a lot from large language
models, we can actually do this if we pre train on on one
part and then apply it to something else.
And how else how how different is is up for very specific to
problems and and application domains.
But that seems to work. And that's specifically what
what I'm quite excited about is that we notice you can actually
pre train on very cheaply generated data, offer us
training foundation models in in 3D plus time.
The amount of data needed seems very scary and also somewhat
impractical given given the current cost of of doing these
training runs. But we noticed that in a way you
don't really need these huge amounts of data.
And I think that that really changes the picture, that
changes how we can approach foundation model training.
And I think in a way it makes it much more attractive as as a
starting point. Could you maybe go into a little
bit more detail when you when you talk about not needing as as
much data or all the high- quality data?
Right. So you also thought about this
quite a bit, right, as you recent paper discussing the the
scaling of of of data and and computer requirements for fluids
is is a good starting point to think about it.
What we noticed in this Tadpole work, we we called it typically
was basically that if you have a good solver.
So it is actually also based in a way on on previous work on on
this APEBench benchmark with very synthetic data.
So if you and if you take the right solver, these spectral
solvers, then you can extremely efficiently compute super good
solutions for a class of basic PDEs.
For advection-diffusion, these ETDRK solvers—in a way,
they're really perfect. So also doing the learning on
this on this level does not really make sense because the
solvers are so good, the errors are so low, you can prove that
they are basically as accurate as it gets for for these basic
solutions. So we can basically generate
this data extremely quickly and very broadly if you vary the
parameters and we notice that right, you can then pre train a
model with this just generating a lot of different solutions for
for varying parameter ranges for these different PDEs, from
advection-diffusion to transport to some higher-order terms with
Kuramoto–Sivashinsky and some some polynomial chemical
reaction like constraints. And basically pre training this
was on this very broad class of very basic synthetic solutions.
Nonetheless, it's actually not not a very easy task and
benefits or seems to benefit in in quite a bit of downstream
tasks. And the nice thing then is
right, these solutions to these synthetic problems are
really not interested in or interesting in a way you can
compute them almost as quickly with the solver from from
scratch basically. So you can train with it, you
can run it through a model to get a gradient basically, but
then you can throw it away. So it's basically just running
the solver alongside. You typically have a node with
four GPUs. So one GPU does the data
generation, the other 3 train and just generates data on the
fly, throws it away right after it produces more data.
But the nice thing is it's produced on the fly and it's
really just a continuous stream of data to train with.
You don't, you don't have limitations on the bandwidth
side on the hardware disk space to manage 10s of terabytes of
or hundreds of terabytes to upload to HPC centres for for
anyone who's tried this, it's, it's very painful actually to
do this and it's not trivial. So right, having this online
data generation with the training is is really quite neat
and worked quite well in our experiments.
But where? Do you, how do you see that
progressing? I mean, I think this has often
been my challenges and you've been on both sides of it.
You know, one side of it is, let's take realistic in well, in
if we assume that the model should work for industrial scale
problems, then you know, the the idea is you generate data like
you did with the SuperWing or, or now this HiLiftAeroML or any
or company, you know, generating data.
And then you you train on that.
And you know that is one argument.
And the other one is you're starting from the far left side
of basic PDEs. But where do you see how can
they go in the middle? The basic PDEs are—can you go
to Navier–Stokes? How does that scale to more
complexity? Yeah.
So what we basically notice is that even if we pre trained also
one of the one of the tricks basically that I didn't mention
just now was also that we don't train for a time evolution.
It's really more of a of a latent space representation that
is learned for for single spatial fields.
So there's no time in the pre training and then the the basic
architecture is basically so so general that you can just put
time on top of that, right. So we learn in this latent
space, one step to the next seems to work nicely.
And also time in a way is also specific application already I
would argue, right. There are steady state problems
that that essentially go for an equilibrium already or an
average solution where time does not really the the evolution of
time doesn't really play a role.
It's more a mapping of some initial conditions in the
geometry or so to the steady state.
You might need time and other solutions, the occasions where
you really want to resolve what happens over time.
But in a way taking that out of pre training already makes
sense in in retrospect I think.
And then with the synthetic PDEs, probably you also might
not get the out-of-the-box best accuracy for right high
accuracy aerodynamic predictions.
But I think it, it seems to be a very good starting point also
for Navier–Stokes. So what I see as a practical
kind of approach in in the future would be to pre train
some very generic models as a starting point.
And then step by step was first further pre training stages
move towards more application specific models, aerodynamics,
maybe other one, maybe aero acoustics, other one, maybe
structural structural problems or branch off different
directions. But pre training doesn't need to
be restricted to these canonical PDEs, but could go in certain
stages and then at a company you might for a certain use case
then have 10 final models to fine tune the the last one for a
certain application. I just think just this this
outlook is basically not starting from scratch but pre
training and then downstream needing less data.
I think it's potentially a good.
So do you so because of that, do you feel that there can be, I
mean how general can these models go do you think?
I mean if you if you put a looking glass.
Admittedly, that's an open question and in in a way also we
were surprised that the models can learn something useful from
these admittedly relatively useless synthetic solutions.
But it's a really interesting question to analyse what they
actually learned. So there are we had we had some
hypothesis. Basically, do they really just
learn or less random abstract fields or physical structures
like the key modes? Or do they learn differential
operators? Also, these PDEs are built from
certain derivatives and it's difficult to disentangle.
But for example, the initial conditions are relatively easy
to test. So the PDEs are initialized.
If you only train with these random fields, the models
actually also learn something, but it clearly does less well on
Navier–Stokes than if you pre-train with the PDEs.
So basically a transformation of initial conditions into actual
states of a PDE seems to be beneficial.
I think this is just the first step.
It's it's actually not right right now difficult to
disentangle what they learned. And intuitively, I think the
network network currently don't has any reason to clearly
disentangled modes or differential operators or so
probably just mixes the most prominent structures that that
come up. But in a way also most of the
problems we're dealing with in in engineering and in real world
applications are typically built from certain like building
blocks to differential equations in a way have a have a
very limited vocabulary. So my guess is that the
structures that come from these actually in a way can be pre
trended and reused. Yeah.
And I guess the the other, I mean, so maybe to put it in a
nutshell, in terms of your research and your thinking, do
you think it is more better to go down this sort of PDE route
with an integrated solver in the loop rather than coming from the
other direction, which is just generating lots and lots of, you
know, if if we're trying to ultimately get to predict a
wing, practically speaking, do you think it's better just to
simulate a million wings in all different conditions or to sort
of build up from the ground from a sort of PDE based that
may be pre trained? And then you go to more complex,
more complex, because they still do feel like different
directions to go, right? The PDEs definitely aim for
generality in a way I think also based on previous work, if your
data set is large enough, if you have enough data to really
train the large scale model specifically for the task you're
interested in, I don't think you actually need this this much
more generic starting points. So if you can do this, I think
probably it's not not necessary. It's really more motivated by
the practical constraints. I think at the moment that's
especially in in 3D for large scale PDEs, it's actually
difficult to gather these really large collective, Yeah.
I, I'm, I'm, I'm kind of intrigued on that point just
because, you know, as that fluid intelligence paper tried to
point out, there is a certain economic challenge, which is
feels to be different than ChatGPT where, you know,
ChatGPT's cost does not really include the data—the token
cost per se. And therefore it's all on the
training side. And I, I still, I still
fundamentally wonder, is it massively inefficient?
Because someone said to me, well, if you run cases, maybe
I'll pose this question to you. So someone said, OK, so if you
need to generate 2,000,000 cases in order to fully scope out what
is a reasonable definition of problems within fluids, So all
possible planes and cars and data centers and all the rest of
it, and then you use all that to train a model.
Haven't you already got the solution from CFD for most of
these problems anyway? Because you probably run it
like, what? Where is it just a, you know,
you're almost so massively in distribution that you almost
could just map that solution to a new.
Yeah. But you're right that that that
might happen. So 2 million seems seems a
little bit scarce for for actual geometry aside.
But if if you could pretend enough of them, I guess you
could hope that any new one you run would actually just hit
close to one of the existing samples basically.
Then I think a practical problem would still be that essentially
you need to look up this data somehow, right?
So these two all right, millions of simulations would probably be
I don't know how how many petabytes or gigantic lots of
storage. So it could be beneficial to pre
train just to have a reduced compressed representation to be
able to efficiently look up these data points and maybe also
give give some smoothness in between, right.
That's even if your exact geometry is not in there, you
get in in between solutions. So it, it could actually still
be beneficial from a from a practical standpoint.
But yeah, given how difficult it is and, and the amount of data
involved with these online generated data sets, it seems
that you can actually get away with much less data in a way.
The the hope would be that you do this as a first step and then
right maybe you can get away with 1,000,000 off for CFD
simulations or significantly less at least to to get to the
same level of accuracy. Practically in your maybe one of
the topics that also interests people and be interested to see
how from a coding point of view you've seen this progress.
So you raise the point of training the data, sorry,
generating the data, throwing it away because you're ultimately
using it. So maybe you could talk a little
bit more what you saw at the challenges from an
implementation from a coding point of view, because that
that's I guess also part of the problem if everybody's
generating these hundreds of thousands of cases offline and
then trying to bring into a model that also seems quite
inefficient, so. The infrastructure work is quite
substantial and I think it's getting better also largely
thanks to all the methodology is converging on this LLM
transformer-style processing, which probably is is one of the
reasons why now finally we're we're in a stage where it starts
to work because also we have the the tools and outlooks for
how to scale things up to these larger resolutions.
It's nonetheless still quite tricky in practice.
So also in my group took quite a while to to figure out how to
put these pieces together. And there are numerous caveats
and, and things that can go wrong.
So actually, admittedly, one of the things we're still fighting
with is the non linear scaling. So even with Transformers, which
which are demonstrated to scale up to billions of
parameters, if you take one model and you just try to
increase the size of of layers and the overall capacity, it's
highly non linear. So it's not guaranteed to to
work if you rerun this, just longer with an increased size.
Most likely at least need to adjust the learning rate, the
additional hyper parameters like smoothing of of network states
over time. How to actually adjust how to
scale up the size of the network in terms of this embedding
space that the Transformers have or the the weights for the
additional components that unfortunately or it seems it's
necessary to revisit for, for every new case again and also
for the PDE case, unfortunately.
So it's still not not right, not not really trivial to.
Do this. And then right, also just
ability, we fought quite a bit just with with the data
management. So we rely on these official
compute infrastructures here from the very end.
And in Germany we now have supercomputers with a fair
number of GPUs. But your someone need to get the
data over there and then you can just store it on on temporary
drives. And if you don't pay attention
then suddenly your trainer data has gone if you don't train
often enough and issues issues like this basically.
But how did you solve that with some of the just wanted to
double click on the you know one GPU to generate the data, 3 to
train. How?
How were you getting around the passing the data between?
Yeah, I'm just trying to learn a little bit more how you
approach that that topic of the online training.
With the with the online training, so right, the classic
approach would be right there. You, you have your huge data
sets on disk and eventually if you don't want to overfit to
what fits into memory, you somehow need to get it from the
disk, get it to the GPUs with this large data set that that
becomes a bottleneck, at least for right, actually for even if
you've hundreds of millions of parameters on HPC systems, I
think we notice especially and the the hardware, the
interconnects are good, but not fast enough to really get
the data in quickly enough for training.
So the standard set up would be right.
You have your GPUs; they need to load the data from this, then
shuffle it through the network to get a gradient and they need
to load the next sample basically.
And for this online training, we are already basically forced
to deal with a separate process that does the generation.
So it's natural to put a buffer in between.
So you basically have a, a buffer that is filled up by the
simulator and then the training sets just pull the data from
there or you take a random sample that's available, then
train with it. And we try to replace it as
quickly as possible. So in practice this this doesn't
guarantee every sample is used only once, but you can measure
right the the throughput of your simulator versus the training
and it's typically at least below 2, so somewhere between
11.5. The South descendants get reused
before they get replaced. So did you buffering?
That's interesting. I mean, I need to look a little
bit more at the the paper and the code, but that was, that was
the bit that I guess I was assuming maybe for your PDE
problem is easier. But if we're doing what is,
let's say if we want to have something really accurate and
we're doing an LES simulation, the actual LES simulation might
need to run on 64 GPUs for like 8 hours, but the training is
obviously way faster than that.
So do you have any thoughts on that?
How you would balance from a time from a loading point of
view? So my guess is that that's why
this online training so far hasn't really taken off before
because like you mentioned for all classic simulations, exactly
you have the the warm up time until you get some equilibrium
that's physically valid and that that could be used and it might
take for a realistic real-world case, it might take hours until
you're in that regime and need fair number of CPUs at least,
or GPUs if you have a modern solver.
So that's, that's really unattractive because the, the
speed of generating the data is way below what you need for
training, which is why I think it's actually so interesting to
use these canonical PDEs with these spectral solvers because
it, it's really, the solver is basically as fast as, as a
network or typically we run actually the, the training we
run on little regions like 64 ^3 and the solver typically produce
large, produces larger ones like 256 or so.
And even those are pretty close to the, to the training speed.
And then you can basically cut out different pieces for data
augmentation. But it, it matches quite nicely.
So I think without such a solver
doing a large scale pre training is is infeasible.
I think there are some approaches doing this, but you
if you want to match the generation capacity with the
training capacity you would need for a supercomputer to
generate data on the fly just to feed a decent sized model.
Yeah, that though then my other thought was actually whilst it
takes 8 hours on 64 GPUs, let's say, to do it, that is the
entire simulation. But actually to generate one
time step is probably only a second or two seconds.
So part of my interest, and maybe some groups already doing
this is training per time step.
Now, the variation from one time step to another is pretty
minimal. So you could argue that you're
not really learning that much between, but you are.
That's the only way I could see that the time scales being
similar, You know, which I guess ultimately addresses one of the
challenges that I think many people have, which is almost all
of the standard data sets and standard machine learning
approaches do take some time averaged solution.
You know, which if you're trying to get to real problems and real
accuracy, turbulence is transient, you know, and giving
just a steady state answer or time average answer is, you
know, not, not really representing the ultimate goal
of like weather forecasting. I guess you would have the
temporal evolution of it. So I don't know how you how we
can get over that. Issue.
I think it's a great order to to really target time.
I think there are some probably what I see there are quite some
hurdles for for practitioners because we have all these
pipelines set up for for averaged quantities.
All right, potentially I think this would be would be neat to
have in place. Regarding your first point
though, with the kind of exploiting the, the fast solves
over time for data generation. Unfortunately, here with
Tadpole, with this online training, We have some, some,
some data that is pointing to to problems there.
So what we noticed even with this online generation if we
don't replace the data fast enough.
So if this reuse is too high of our pre-training buffer
basically the models do start to overfit.
So typically for foundation models you're dealing with
fairly large models. So ours are not even extreme,
but on the order of maybe 100 million parameters.
And if the samples are reused too often, there was already—we
saw some signs of of performance deteriorating if the
generator didn't catch up. And basically this was just out
of pure luck because of of scheduling on these high
performance systems of right. If there's some one of the GPUs
is is for some reason slower and suddenly on one training run
you reuse the data more often. We saw a drop in performance and
even there the the correlation between the samples was not
overly strong. So it still generated a fair
amount of data. But I would be worried if you
have these strongly correlated samples over time.
Even if you're able to swap them out and reload reload different
configurations for a solver, I think this would be would be
difficult to ensure that the variability in the data is large
enough to right not overfit a large model.
Yeah, that's a very good point actually, that that's, that's
the gang. Yeah.
The drawback that we, the temporal evolution of the PDE
needs to be small in for numerical reasons, you know, to,
to avoid blowing up, you know, for because of the CFL
constraints, etcetera. So you you're kind of forced to
slowly March in time, whereas I guess you're right, if you feed
the machine learning model that it's just going to keep seeing
essentially the same solution, very similar.
Ones that. And and yeah, that's, but that's
where I still see a bit of a fundamental.
I just don't see. That's why I see almost as
that's why I was interested in your differentiable physics,
your your code writing essentially because I feel some
of the breakthroughs in this probably will come through a
clever, clever use of the training being linked to the
data generation in a way that. I think right now our focus is
on foundation models. I'm also confident that at some
point it's going to make sense to bring the solvers back in the
our our tests back back then a few years ago basically did did
pretty short. If you have a solver, there's
anything decent, the learning task is just simpler.
So once once we have kind of reached the decent reasonable
capacity for this pre training, I think it's might make a lot of
sense to put a solver back into the loop to just make it
more accurate in the end. Also, they are it's it is
challenging at the moment at the at the scope given the current
infrastructures and and large scale models.
It's typically challenging to just train a single model and in
a decent amount of time and then trying to get a solver into
the picture and maybe going over multiple time steps.
Things like this are are tricky, but it's also going to change
next year's. Yeah, yeah.
No, I I would agree. And maybe the the other topic is
around what were some of your lessons learned from doing, you
know, WeatherBench and APEBench and, and some of these more
benchmarking efforts because I guess that seems to have helped
quite a bit on the weather and climate side.
So what? What was some of the genesis
behind those efforts? So the WeatherBench effort,
yeah, in retrospect, it's, it's great that it's, it's doing so
well. I, I think it's especially, I
guess I should think that so, right.
What I'm trying to say is basically the, these benchmarks
have a big impact in the field. I think by now that is widely
accepted across all the fields. Back then it was probably more
clear in the vision area where fair number of of benchmarks
have been around for quite a while.
But in other fields, like weather, there was very little,
basically having established benchmarks and reliable
evaluations is really important for for any discipline
in the field. I think that's a nice pointer
and right, it's civic. It's it's quite some work
collecting the data, right and thinking about what should be in
there and how to evaluate it. So the way those efforts are
really important for any subfield within any data-driven
discipline. And I think by now it's great
also in this scientific Yeah, I feel that this is noticed that
by now more and more benchmarks are coming out.
I think that's. Really important you you
recently generated to your group the SuperWing data set.
I mean, where do you see this going?
And it's a little bit of a philosophical debate around data
because on one hand you could argue that we the progress in
this field does seem limited by data to a certain extent.
And, you know, does it mean that there should be some
coordinated effort to generate data?
And if so, who should be doing that?
And what should be the license attached to it, given that there
is clearly also a commercial benefit to be had?
That's a good question. I mean coming from from a
university, I think it's great if it's all public and as open
as possible. But given the commercial
interest and also by now the the key outlook that this will be
useful, that would of course be great to have support from from
commercial partners in this. I think it's getting better.
Also more, more companies see the need for AI and then also
the need for or benchmarks and data sets on that front.
So I think it's improving, but definitely an issue where things
could be done and could improve a lot.
And unfortunately there's so much to quickly wrap this up.
But I think there's so much proprietary data with IP rights
and so on that are probably on some servers and companies that
they that cannot be used. So it probably needs an effort
to generate data that's free and and usable.
Yeah, that I keep jumping between that because on one hand
you could say that it is to the a bit like maybe open
science experiments, you know, with, with astronomy or, or, or
things where there's a sort of public good and a lot of it is
funded through taxpayers. Ultimately, you know, through
sort of science funding and the data's made available.
Part of it feels that that would help all companies, you know, to
be doing it. But at the same time, is it
really the responsibility of governments and science to do
this? It it's it's kind of.
Yeah, that's good to argue about the companies.
So pay for it if they benefit from it, that's.
Yeah, that's it. I just look at, I don't know
about you, but I look at all the supercomputers in Europe, just
as an example, you know, and you think how much data could be
generated given that these systems now are quite large
because they're being scaled up for the task of also, you know,
large language model training, etcetera.
You know, if you've got a cluster with 10,000 GPUs, you
think how much data could you generate, you know, from fluid
simulations to a 10,000 GPUs, even just, you know, for a few
weeks or, or you know, or a month, you know, that does feel
like it could be a, a huge way of doing it.
And if, if it's only a commercial company that does it,
they probably have no incentive to release that data.
And it then becomes hard to
progress the field if only one company has that data where it
feels like with large language models, the data has actually
had quite an open movement, right?
There's quite a lot of open data obviously around this.
And so, yes, you still could say you can only make models if
you've got lots of compute, but there was already a movement
now with open-source LLMs, you know, going on, whereas I feel
now how can there be a movement of open source surrogate
models if the data is not there in an open way, You know, it
it it's. It's a good point.
It would be a pity if that gets hidden away due to commercial
interests. So it's almost the.
Progression of science, I guess, and this is, I don't know if
you've had this and certainly now my myself being a commercial
company or the last two companies that you know that I'm
at, there is always this debate of open source versus
commercial, and I still find that a tricky one.
I wondered how you've seen it. You know what, what at what
point does it help everybody to be open source?
Yeah, OK, right. I have AI have a good.
Argument for open source from from my career at least.
We used to work in computer graphics, basically, with movie
studios. I remember, during my postdoc,
one of the first papers was at our conference, SIGGRAPH.
There, we basically put out some open- source code, and
DreamWorks at the time actually picked this up pretty
quickly and directly used it for for one of the shots.
And I said, oh, can't we get, or maybe they can put us in the
credit somewhere. They send us a poster in the end
and also quite a few of my friends and kind of laugh or
look all all you got for for this work.
You put up your code, you got a poster for it.
Why did you sell it or something?
A couple of years later, these, all these movies where it was
used were one of the key things to apply for one of these tech
Oscars. There's also application
procedure. But in the end, we could show,
look, it's been used in all these movies.
And this this Oscar actually did help a lot also for applying for
faculty positions and so on. So this is largely due to to
open source availability. It just might impact in the
field. So I've had very good experience
with this and I'm actually very happy that now that, yeah,
I feel this is so obvious. 15 years ago it was very, it was
actually the exception that papers came with code and it's
great. It's changed so much.
Yeah, it, it does seem an interesting 1 though, because
it's, given the large amount of money that's required to develop
models and develop training data, you know, there has to be
some route to that money coming back.
Which is why for a commercial company, if they spend $100
million on compute and they, you know, they have to think, well,
how am I going to make a business out of this?
And so it's, it's kind of an interesting one that is the
business that they, that the value to the company is not the
model, but how you sort of tweak the model, so to speak.
And therefore, just because there's a foundation model out
there which is open source, the real value is that every single
company needs to customize it, which I think is the value at
the moment for open-source LLMs that ultimately companies still
make money out of it because everybody realizes that it isn't
fully ready in its base form. And that actually you take it
and you tweak it and you optimize it and you know, you so
that so there's money still to be made.
And that's what I kind of wonder on the fluids surrogate
modelling side or even beyond that, it would it still be in
the interest of some company to make it because ultimately the
money will be made customizing it and therefore being open
helps the whole community to develop it faster and make it
more competitive against close models.
Right, that's definitely. But I hope that, that we can go
towards some foundation models or some generalizing models in
the field that can then be fine-tuned and adaptive to
adapted to different, different applications quite easily.
I think that's, that's been super useful in the LLM field.
And I, I do see potential there on the PDE front and fluids
front. So, and I think if, if at some
point it's clear that you can get a really good result, right,
if if you actually download this model and then right fine
tune it on your couple of wing data set cases or so for certain
regime you're interested in, then.
But there could be a clear commercial also benefit for a
company and the use case of supporting these for supporting
these open models. Yeah.
So that could imagine in an infrastructure environment
working quite well also in this area, but that definitely needs
a coordinated and a large scale effort to build it up right now.
Yeah, that that's seems that
there's lots of separate efforts, I guess going on, but
but not uncoordinated. I I mean, maybe moving to some
of the final questions for you is where well, look, I, I guess
a couple of things before we get to the future looking one.
I'm just interested, where's your side on the like startups
or academia industry? Because I've noticed there's a
lot of, in recent years, there's even more interest in startups
and, and, and the value of startups.
But then that also in some ways can conflict with the open
source academic, you know, mindset.
Where, where have you seen that in the world?
Have you been tempted with startups of, of industry?
Where do you see that role of academia, start-ups and and
industry? Yeah, it's a good question.
Exactly. Especially in last one or two
years we've seen a very nice rise I think in terms of funding
and and also just founding of all kinds of spin off companies
that are pivots of quite existing companies towards this
physics AI direction. So I think also in a way
confirmation that now it's really starting to work, right,
It's ready for practical applications.
I've definitely toyed with the idea so far or also on my side,
there are no immediate plans for this.
In a way. I, I had my experience with
industry for visual effects, and therefore I'm quite happy
with the Open University research side.
So I've, I've always been a big fan of open source and just
being able to put out things and for, for research.
I mean, it's also effectively a market with this openness and
impact, right? You, if you can generate this
later on, you can say, look, give me more money for research
to do more of, of this. So it's not the commercial
market, but also their own market in terms of research
money. And but I like the openness of
that. So for now, I think for me
that's personally just because I like this openness the the
better direction. But we also, by now we're
working with all kinds of companies due to the commercial
interests. Yeah.
But we were basically trying to do this on the more Open
University side and then work with individual companies for
maybe adopting or or adapting these these techniques.
Yeah, I was going to say because the decking jury if all top
academics create start-ups, then there will be no academics
left to do that. The sort of so that that I'm
sure that has been discussed a little bit in the broader LLM
space. If the only companies doing the
latest stuff because of the scale are the big tech
companies, then there is a risk to open science, I guess
because everyone's tempted to go to a commercial company that
maybe aren't, as you said, incentivized to be open.
I guess as academia, you are incentivized to be open because
the reward structure is based around it publishing grant
money. Like if everything's closed, I
guess it may help the university a little bit because of grant
capture or something, but it's not really the the core aim, is
it? It's difficult actually for for
universities. I think in the past quite a few
labs had their in house solvers and then basically earned money
and and funding. With direct we basically support
or consulting contracts with companies.
But right now I think especially in the fast moving AI field,
it was, it was closed solutions, you would see very little
impact at the at the moment, Stephanie.
Works nicely no I, I, I think that's why I'm always
championing the academic side because I think people look at
research coming out of tech companies or research coming out
of the but forget that actually most of the things
start in the university most most fundamental ideas and.
They need to be kept being promoted and that's where I feel
the data's the issue, because the they if you don't have open
data, then you can't actually universities can't progress and
can't publish and can't advance.
The state-of-the-art. If everything's done closed
door, it actually hurts progress, doesn't it?
Yeah. So LLMs will be interesting to
some extent, right. That's there have been
initiatives of training open LLMs, but some of them I guess
did fairly well. But the top models are not open.
And right now I think there are very few people who really can
or try to train their own LLMs. That's really become a very
tough field, at least to operate in.
Yes, yes. So where do you see if you have
your looking glass or your, you know, crystal ball?
If we were to talk again in five years time, where where do you
think the conversation will be going?
What what do you think will be the breakthrough moments?
Do you feel like we're going to be just incrementally over the
next five years or do you perceive some big breakthroughs
or leap? Just hope that we're going to
see this breakthrough in terms of adoption.
So now also these all these initiatives on commercial side I
think point towards the field really starting to work.
So I hope that five years we we really see widespread adoption
for for actual applications in a way sounds a bit boring,
right, just being it used in in practice.
But I think right also working on this for almost 10 years.
I think it's about time that we that we do see that it's got
it's really useful for for real world world things.
So I just hope that now we are at a stage where this really can
be can be pulled off, but in the field is really progressing
extremely quickly. And so one of the, one of the
outlooks that I've been intrigued about are these world
models, right? Like vision by now, they're
already going beyond foundation models or specific language
and, and just visual models towards ones that just capture
the whole world. And right, also vision now
notices actually physics is an important part.
You only see so much. A lot of the complexity comes
from things you don't directly see like air moving or so I
think that's a really interesting outlook to to really
simulate basically the the whole world around us on a larger
scale with the physics accurately.
I think that could also be super useful for engineering and and
real applications in a bunch of areas.
But getting that right seems like a whole different level on
top of what we currently dealing with just only doing the
physics right with the foundation models.
But I think it's a really interesting outlook.
Yeah, it's actually a good point there.
Yeah, I forgot to ask you about that.
That I still don't fully get. I mean, I understand the world
models are obviously one very big use case is robotics and
driving drop, you know, autonomous vehicles and sort of
giving that synthetic data to help them to operate in the
world, The bit that I guess I'm less sure on.
And I wonder whether economically and scientifically
is the physics side, you know, does it make a difference how
good the physics is like, as long as it looks right?
And this, This is why I'm interested, because your movie
background, like does it, does it matter to the robots and to
the autonomous vehicles? Actually, I do think that at
least for robots, the physics play a big role because
ultimately the, the action that you're generating is, is the
control right of your, of the actuators of the actual things
that the robot should do. And you need to estimate surface
properties, the weight of an object, weight distribution
stability and all these things to at least in a, in a variable
environment to interact with the, with the world.
So think for robots by now rigid body type or in some somewhat
deformable and physics are playing quite a role.
So I would I would guess that it does make a difference for
autonomous driving. We could argue at the end you
only need to take right? Is it a dangerous situation or
would something have gone wrong?
You can probably do a lot without having to go through the
whole crash or actually simulating how the car would
tumble or crash into a building or so.
That might not be so important, right, As long as you can detect
this would have gone wrong. But for robots actually having
to really interact with objects, I can imagine that plays a
larger role. And then again looking towards
fluids, I, I think the challenge also in, in the current
development pipelines is this interaction with the real world.
I mean, I think in the end, the
current validation and certification still goes through
a lot of testing status and then you have all kinds of real
world flights and and experience to make sure that the systems
really do what they should in the real world.
But if you could shift some of that into an earlier stage, I
think that that could also be definitely interesting.
If right, you could have different environment
conditions, different flow conditions, and you could have
an have an object interact dynamically in in such an
environment. Yeah, I guess maybe maybe you're
right to maybe the examples I've looked at, you know, if you
look at a robot walking around a factory floor or it has no the
the drag, the lift, the thermal properties are very second or
third order effects. They don't really influence how
it's performing. But I guess you're right.
If it was like a drone in the sky or a boat in the water, then
the physics actually do make quite a bit of a difference in
and then I guess it really is depending what's the use of
these world models, if they really are like a true, true
representation of the world for that drone or or helicopter or,
you know, I guess the moment the autonomous vehicle, you're
right, is it's more sensory. You know, like I see there's a
person walking. It's not to design the car,
right? It's not to say I see now that
in this real world, by changing the shape, the drag is lower
because it's actually I guess maybe that's where you're
thinking, right, that the world model could be used as a true
synthetic environment or cars, I guess if you could.
Simulate a rainy and stormy situation and then really get
feedback on how the rain would splash around a car, or how gust
of wind would influence driving stability in extreme
conditions. Or right, all kinds of of
landing and starting scenarios for planes.
If you could really get feedback on how a deforming plane with
changing conditions would would behave.
I think that's that's still beyond even regular simulation
capabilities—that fluid-structure interactions
with changing conditions are really challenging.
You could get some feedback on these and then ideally put an
optimization loop around them, right?
You you would want to actually optimize your, you know, your
wing profile or the shape of of a vehicle to behave in the
whole longer sequence. I feel like this is also the big
debate, well not big debate, but a debate around the classic
tool calling thing. You know, like isn't LLM good at
adding numbers? No, so just go and call a
calculator. That's like ultimately, I guess
if you call ChatGPT to add 2 numbers together, it's just
calling a tool of a calculator, adding the 2 numbers together.
I kind of wonder it's sometimes in this world of, you know,
world models or optimizing, you know, is it, is it really that
it's truly integrated into a world model?
Or is it more likely that there is a sort of agent workflow
where it's calling a surrogate model and maybe that's the where
they're two separate models, if you know what I mean, rather
than just a world model. It's more of an agentic way,
if you know what I mean. Or not that that would already
be a big step, right, If if you had any visual model that could
on demand call some physics sub models surrogates to to get
feedback on how these different pieces it recognises should
interact. For rigid bodies, actually
these days you would just call it a rigid-body simulator.
You would probably not need a surrogate.
Yeah, that's. That's kind of where I always
debate it and that and that's the traditional one that you
said if the solvers get so fast, like the spectral solver you
mentioned or now with, you know, advancements in GPUs, some
people are saying I can do a simulation in a minute.
And then you think, well, do I need a surrogate?
If if I can, if the solver can run so fast?
Right, I think for world models, an interesting challenge will
be to to quickly switch from approximate fidelity for rigid
bodies or deformable objects, or winds, or different feedbacks
needed in the environment. Sure, if I know I need perfect
rigid bodies for a certain case that might be ideal, but but
right I need some feedback on on wind or how right my plastic cup
should deform or should it break predictions across the all
these different physical phenomena.
I could imagine that a flexible surrogate that at least gives a
gives the first prediction of estimates of of what what might
happen there could be beneficial.
I think that would be tricky to do with classical simulators.
Yes, yeah, yeah, yeah, that. I think this is where it's the
classic what's the, what's the use case of real time?
And sometimes if you go to an engineering team and I've had
this where you tell them that a surrogate model prediction in
one second, sometimes they'll say, well, we don't need it to
do in one second. You know, it's actually fine If
it takes 5 minutes or even half an hour, it's OK.
It's not a bottleneck. Whereas I guess if it's
integrated in a world model or something, you need it to be
essentially real time or else the whole value breaks of some
simulation. You know, like testing a car
moving. If you have to sort of pause the
simulator for 30 minutes, then it's not that.
Feel right? Yeah, I mean real time is is
extremely challenging. Then you need I guess to to be a
realistic virtual environment for a person.
Yes, you need milliseconds. Yeah, very real.
Yeah. So maybe a final question to you
looking more if, if there's a student listening to this or
someone doing HD or early in their career, given, given all
these changes, what would you recommend someone was would
study as a PhD topic? You know what was going to
future proof them in this world.
Maybe I'm biased here, but I think these are really important
techniques. I, I do see a lot of potential
for AI based techniques. So I think it's important to
have an understanding to know the classic, the classic physics
numerics. But by now within these
numerical tools, I think AI is, is a super important component.
So I could, could highly
recommend looking at at least combinations of classic and AI
based methods. And my, my guess is also that in
the future or the next couple of of years, we were not going to
be completely replaced by I don't think ChatGPT is going to
take over fluid simulation too soon.
So having experts that we understand what's happening
there, I think it's it's going to be needed for quite a while.
So I think it's still a good field to to work on.
Yeah, I guess that that's the the truth, I guess the, the
short, medium, long term, I guess I see at least now the
short term, there's even more need for specialists because
frankly, it's almost become that simulation is more of a
popular topic now. The fact that all these startups
are getting funded, the fact that, you know, there's more
money in this space, there's more need to find out who's an
expert in crash simulations, who's an expert in acoustics,
who's an expert to help computer scientists, you know, to
develop these models. I guess it's only in the very
long term future that you could imagine.
Maybe you don't need as many specialists because so much of
that knowledge is now ingrained in these models.
But there's a, in the short term, it's almost even more
specialists to help. But deep, deep specialists, I
guess you really understand it that, that at least that's the
way I I'm seeing things at the moment.
And I think that's usually a great topic for a PhD, really
getting to the bottom of things.
So I just, it's a nice opportunity to work on one topic
for a couple of years and yeah, get as much understanding
as possible. So on.
Right now I think PhD is in this area.
Still a very good idea. Yeah, yeah.
No, no, I agree. Well, thank you so much for
taking the time to speak. I mean, I, I find all these
topics fascinating and one of the things I'll do is put a list
of the papers in the, in the show notes because I think
actually there's a lot more to gather from looking in the
details and, and looking at the codes that your, your, your
group did. So I know we didn't have time to
cover all in, you know, full detail, but I'll, I'll put a
list and hopefully people can find the time to read through
the papers. That's a good idea.
Oh, great discussion. Thanks for the invite.
Yeah, interesting. We'll speak again in two years
time and we'll see if the predictions are.
That's interesting. Thank you.
Thanks.