1
00:00:00,240 --> 00:00:02,360
Hi, and welcome to the Neil 
 
Ashton Podcast. 

2
00:00:03,000 --> 00:00:05,880
In each episode, we explain 
 
some of the fascinating ways 

3
00:00:05,880 --> 00:00:09,080
that science and engineering are

 changing the world around us. 

4
00:00:09,760 --> 00:00:12,720
We talk to leading engineers 
 
from elite level sports like 

5
00:00:12,800 --> 00:00:16,840
cycling and Formula One to some 
 of the world's top academics to

6
00:00:16,840 --> 00:00:20,445
understand how fluid dynamics, 

machine learning and 

7
00:00:20,445 --> 00:00:22,960
supercomputing are bringing in a
new era of discovery. 

8
00:00:23,920 --> 00:00:27,040
We also hear some of their life 
 stories, their career advice, 

9
00:00:27,560 --> 00:00:30,040
the lessons they've learned on 

the way that I hope will be 

10
00:00:30,040 --> 00:00:33,800
helpful to you too. 
 
So sit back and enjoy this 

11
00:00:33,800 --> 00:00:41,120
episode. 
 
Hi, and welcome back to the Neil

12
00:00:41,120 --> 00:00:45,080
Ashton Podcast. 
 
So today's episode is with 

13
00:00:45,080 --> 00:00:47,840
somebody who I've actually 
 
really enjoyed getting to know 

14
00:00:47,840 --> 00:00:52,520
better and chatting at various 

conferences that we've attended 

15
00:00:52,520 --> 00:00:55,920
together and some mini symposia 
 that we've done, which is 

16
00:00:56,160 --> 00:00:59,120
Johannes Brandstetter. 
 
So he's got an interesting 

17
00:00:59,120 --> 00:01:04,760
profile because not only is he a

 professor in Linz at JKU, 

18
00:01:05,040 --> 00:01:09,960
Johannes Kepler University in 
 
Austria, but he also founded or 

19
00:01:10,040 --> 00:01:13,000
Co founded a startup recently 
 
called Emmi AI. 

20
00:01:14,520 --> 00:01:20,600
His background is one that is 
 
very well positioned to help 

21
00:01:20,600 --> 00:01:24,480
advance the state of machine 
 
learning for, for for CFD and I 

22
00:01:24,480 --> 00:01:27,400
guess CAE more broadly. 
 
And yes, this is an episode on 

23
00:01:27,400 --> 00:01:29,720
machine learning again, I had 
 
sometimes feel bad that I keep 

24
00:01:29,720 --> 00:01:35,240
doing this, but I think this 
 
season there's a mixture of of 

25
00:01:35,240 --> 00:01:38,920
episodes and I still feel that 

there is value in going through 

26
00:01:38,920 --> 00:01:41,640
the machine learning topic. 
 
And it's selfishly something 

27
00:01:41,640 --> 00:01:44,560
that I I'm constantly interested

 in learning more about and 

28
00:01:44,560 --> 00:01:46,880
learning more about. 
 
It is one thing that I always do

29
00:01:46,880 --> 00:01:50,200
when I speak to Johannes. 
 
And he has an interesting 

30
00:01:50,200 --> 00:01:55,760
background because whilst he has

 I guess a high energy physics 

31
00:01:55,800 --> 00:02:01,640
background that he he, he did 
 
and he was at CERN and he worked

32
00:02:01,640 --> 00:02:04,840
in that sort of area. 
 
He then moved into the machine 

33
00:02:04,840 --> 00:02:10,400
learning side, I guess, having 

spent time with Max Welling at 

34
00:02:10,400 --> 00:02:13,040
the Amsterdam University in the 
 machine learning lab. 

35
00:02:13,320 --> 00:02:17,091
And then I think he also, well, 
 I don't think I know that he 

36
00:02:17,091 --> 00:02:19,720
was then at Microsoft Research 
where 
 Max was also at. 

37
00:02:19,720 --> 00:02:21,720
And so the two of them 
 
collaborated a lot. 

38
00:02:22,400 --> 00:02:25,280
You know, if you don't know who 
 Max Welling is, well, first 

39
00:02:25,280 --> 00:02:27,114
Google his name. 
He's one of The 
 Pioneers of 

40
00:02:27,114 --> 00:02:29,022
machine learning. 
But I also did an episode with 


41
00:02:29,030 --> 00:02:32,430
him in the last season that I 
think was really interesting and

42
00:02:32,430 --> 00:02:35,782

 I think he must have been 
inspired by Max a little bit, 
 

43
00:02:35,790 --> 00:02:38,430
even though we didn't talk about
it in this episode, because Max 

44
00:02:38,430 --> 00:02:41,230
 also has been a serial start up
founder. 
 

45
00:02:41,238 --> 00:02:45,920
And the reason I said that he 
has an interesting background is

46
00:02:45,920 --> 00:02:49,506

 because whilst he was at 
Microsoft, he also worked on the

47
00:02:49,506 --> 00:02:52,988

 Aurora machine learning for 
weather and climate project. 
 

48
00:02:52,996 --> 00:02:57,256
That I think is extremely useful
when you have people who have 
 

49
00:02:57,264 --> 00:03:00,260
gone through these major 
projects in another field 
 

50
00:03:00,268 --> 00:03:03,714
because they bring with them 
lessons and learnings that are 


51
00:03:03,722 --> 00:03:06,922
important. 
And, and what Johannes has been 

52
00:03:06,922 --> 00:03:11,495
 doing is first of all, bringing
a very academic mindset to this 

53
00:03:11,495 --> 00:03:14,428
 academic in the terms of 
publishing and transparency, 
 

54
00:03:14,436 --> 00:03:17,812
which I think is very welcome. 
So you'll find in the link in 
 

55
00:03:17,820 --> 00:03:21,300
the YouTube a couple of the 
papers that he's published and 


56
00:03:21,308 --> 00:03:25,569
and his startup has published a 
new sort of transformer based 
 

57
00:03:25,577 --> 00:03:29,512
model that I, at least from my 
reading, is unique and certainly

58
00:03:29,512 --> 00:03:33,880

 seems to be at the one of the 
most cutting edge in 
 

59
00:03:33,888 --> 00:03:36,598
state-of-the-art models out 
there today, both in terms of 
 

60
00:03:36,606 --> 00:03:39,909
conceptual but also the accuracy
on the data sets that they've 
 

61
00:03:39,917 --> 00:03:43,252
shown. 
But I admire his vision and his 

62
00:03:43,252 --> 00:03:46,576
 willingness to try and solve 
the problem rather than being 
 

63
00:03:46,584 --> 00:03:50,798
focused on the model, as in some
people seem to, once they come 


64
00:03:50,806 --> 00:03:55,080
up with a model, fixate on that 
being it, rather than being 
 

65
00:03:55,088 --> 00:03:58,612
willing to consider that their 
model may be the right thing at 

66
00:03:58,612 --> 00:04:01,360
 the time, but then there'll be 
other models that get better. 
 

67
00:04:01,368 --> 00:04:04,956
And he, his willingness to 
accept that, I think is a breath

68
00:04:04,956 --> 00:04:09,367

 of fresh air and definitely 
will help the community to, to 


69
00:04:09,375 --> 00:04:12,035
evolve. 
So that's what we talked about 


70
00:04:12,043 --> 00:04:14,052
today. 
Really we, we tried to go 
 

71
00:04:14,060 --> 00:04:17,370
through and discuss, you know, 
more the general topics around 


72
00:04:17,378 --> 00:04:20,500
machine learning, but really 
diving into this transformer 
 

73
00:04:20,507 --> 00:04:23,904
based approach that he has. 
And I'm trying to understand 
 

74
00:04:23,912 --> 00:04:26,307
some of his thoughts around the 
similarities to neural 

75
00:04:26,307 --> 00:04:30,517
operators, 
 some of the slight 
differences, some of the links 

76
00:04:30,517 --> 00:04:32,920
to, I guess, 
 graph neural 
networks and MeshGraphNets that 

77
00:04:32,920 --> 00:04:34,480
probably most people 
 are aware
of. 

78
00:04:35,080 --> 00:04:37,924
And then we talked a little bit 
 about the process of forming 

79
00:04:37,924 --> 00:04:41,760
the startup and some discussions

 around future topics. 

80
00:04:43,160 --> 00:04:45,760
As with anything, I always feel 
 bad because I finished the 

81
00:04:45,760 --> 00:04:49,000
episode and I think I really 
 
should have discussed more or, 

82
00:04:49,040 --> 00:04:52,120
or, or dived into certain 
 
topics, but I'm conscious it was

83
00:04:52,120 --> 00:04:56,280
already over, you know, an hour 
 20 and people will, you know, 

84
00:04:56,280 --> 00:04:58,440
fall asleep listening to some of

 these episodes. 

85
00:04:58,840 --> 00:05:02,400
But hopefully it's enough to 
 
spark your interest and 

86
00:05:02,400 --> 00:05:05,200
encourage you to read more about

 the topics that we discussed. 

87
00:05:05,720 --> 00:05:08,400
And please leave any comments or

 let me know if you have any 

88
00:05:08,400 --> 00:05:10,720
suggestions for for other 
 
topics. 

89
00:05:11,160 --> 00:05:14,705
But for now, please sit back and

 enjoy what I found an 

90
00:05:14,705 --> 00:05:17,480
extremely interesting episode 
with 
 Johannes Brandstetter. 

91
00:05:18,720 --> 00:05:22,440
Cool. 
 
All right, thanks for joining me

92
00:05:22,440 --> 00:05:24,760
today. 
 
I wanted to have a chat with you

93
00:05:24,760 --> 00:05:27,960
for a while. 
 
We've had some nice discussions 

94
00:05:27,960 --> 00:05:30,680
over wine and and coffee, but 
 
nothing public. 

95
00:05:30,680 --> 00:05:35,850
So this is the public version of

 our discussions, but maybe 

96
00:05:35,850 --> 00:05:39,800
it'd be good for people to learn
a 
 little bit more about you. 

97
00:05:39,800 --> 00:05:42,440
You know, where you're at today.

 

98
00:05:42,448 --> 00:05:43,730
Yeah. 
Who? 
 

99
00:05:43,738 --> 00:05:46,740
Who is Johannes? 
Yeah, thanks first of all. 
 

100
00:05:46,748 --> 00:05:48,406
Thanks really Neil for for 
having me. 
 

101
00:05:48,414 --> 00:05:51,120
It's it's a great pleasure 
finally sitting here with you 
 

102
00:05:51,128 --> 00:05:54,200
in a one to one setting, so to 
say. 
 

103
00:05:54,208 --> 00:05:58,680
Yeah, my background is actually 
I'm I'm a learned physicist, 
 

104
00:05:58,688 --> 00:06:03,380
some say a failed physicist. 
But after my PhD, I switched to 

105
00:06:03,380 --> 00:06:05,932
 machine learning, as many 
people also did. 
 

106
00:06:05,940 --> 00:06:09,816
I have spent time with Sepp 
Hochreiter and three years in 
 

107
00:06:09,824 --> 00:06:13,112
Amsterdam with Max Welling. 
I was in industry for two years 

108
00:06:13,112 --> 00:06:17,220
 at Microsoft Research. 
And there at Microsoft Research 

109
00:06:17,220 --> 00:06:20,840
 I really, really discovered my 
likings for large scale 
 

110
00:06:20,848 --> 00:06:22,880
simulation, especially for 
weather and climate modelling, 


111
00:06:22,888 --> 00:06:25,304
because these are like some of 
the biggest problems you can 
 

112
00:06:25,312 --> 00:06:27,270
have and you can tackle with 
machine learning. 
 

113
00:06:27,278 --> 00:06:32,600
I also saw the transformative 
impact on this, on the systems, 

114
00:06:32,600 --> 00:06:38,184
 and that was then the point 
where I decided to to do my own 

115
00:06:38,184 --> 00:06:42,860
 thing to, but to apply it not 
to known problems like weather, 

116
00:06:42,860 --> 00:06:45,600
but 
 rather uncharted 
territories like engineering, 

117
00:06:45,600 --> 00:06:47,520
simulation and 
 this kind of 
thing. 

118
00:06:48,600 --> 00:06:52,320
And I decided to get my own 
 
group at university to build up 

119
00:06:52,320 --> 00:06:56,360
my knowledge base back in 
 
Austria, where I'm coming from. 

120
00:06:56,880 --> 00:06:59,640
And along the road. 
 
It also happened that I founded 

121
00:06:59,640 --> 00:07:03,058
a start up Emmi AI where we do 
this 
 simulations at scale and 

122
00:07:03,058 --> 00:07:06,254
where things are coming together
with 
 a bit more compute and a 

123
00:07:06,254 --> 00:07:10,360
bit more resources. 
 
Nice. 

124
00:07:10,360 --> 00:07:17,160
Well, we'll we need to dive into

 each of those, but maybe the 

125
00:07:17,160 --> 00:07:21,440
first one was the Aurora 
 
project. 

126
00:07:21,440 --> 00:07:26,537
I guess this is maybe it, it was

 simultaneously both a 

127
00:07:26,537 --> 00:07:29,291
motivation, I think certainly 
for me and 
 others who saw what

128
00:07:29,291 --> 00:07:31,582
was happening in weather and 
climate 
 and was saying, you 

129
00:07:31,582 --> 00:07:33,672
know, surely we could do that in
engineering. 
 

130
00:07:33,680 --> 00:07:38,954
But also it was an example of a,
yeah, a real large scale 
 

131
00:07:38,962 --> 00:07:42,480
project. 
So what was it like working on 


132
00:07:42,488 --> 00:07:44,640
that Aurora? 
What was the key lessons you 
 

133
00:07:44,648 --> 00:07:49,182
learned both from maybe an ML 
point of view, but also a 
 

134
00:07:49,190 --> 00:07:52,515
project and a team point of 
view? 
 

135
00:07:52,523 --> 00:07:55,858
So the actual Aurora team was 
pretty small. 
 

136
00:07:55,866 --> 00:07:59,056
I mean, all the credits to to 
Cristian Bodnar, Wessel 

137
00:07:59,056 --> 00:08:03,752
Bruinsma, 
 Megan Stanley and 
Ana Lucic who did that, all the 

138
00:08:03,752 --> 00:08:06,545
heavy lifting 
 afterwards to 
finalize this project, to train 

139
00:08:06,545 --> 00:08:12,121
to build the 
 data loader. 
The Aurora was the learnings I 


140
00:08:12,129 --> 00:08:15,132
think are threefold. 
First of all, with computer 
 

141
00:08:15,140 --> 00:08:20,018
vision tricks, you can get very,
very far in engineering or 
 

142
00:08:20,026 --> 00:08:23,164
scientific application. 
In the end it was a Swin 

143
00:08:23,164 --> 00:08:26,046
Transformer trained in in a 3D 
Swin Transformer. 
 

144
00:08:26,054 --> 00:08:31,224
Secondly that the the data 
engineering is potentially the 


145
00:08:31,232 --> 00:08:37,263
hardest one and and thirdly that
weather has a very unfair 
 

146
00:08:37,270 --> 00:08:40,784
advantage for machine learning 
because nobody knows how weather

147
00:08:40,784 --> 00:08:45,149

 is functioning. 
So you, you will always train a 

148
00:08:45,149 --> 00:08:49,500
 machine learning model on like 
actual data, which basically has

149
00:08:49,500 --> 00:08:51,520

 the actual physical laws 
hidden. 

150
00:08:51,880 --> 00:08:57,320
Whereas the, the the numerical 

methods, they have to come up 

151
00:08:57,320 --> 00:09:02,120
with these laws by themselves. 

And that already brings these, 

152
00:09:02,120 --> 00:09:07,600
these key interests of mine, 
 
which is that that neural 

153
00:09:07,600 --> 00:09:09,960
surrogates will never replace 
 
numerics. 

154
00:09:09,960 --> 00:09:13,440
They will just be a different 
 
branch which depending on how 

155
00:09:13,440 --> 00:09:15,840
you play the card acts in your 

favour. 

156
00:09:16,320 --> 00:09:19,200
And that it's all about 
 
engineering and scaling these, 

157
00:09:19,200 --> 00:09:24,000
these things to this, to this 
 
basically to this problems where

158
00:09:24,000 --> 00:09:26,480
they really have an impact and 

there really matters. 

159
00:09:28,440 --> 00:09:31,360
And the impact for Aurora, you 

could see if you look through 

160
00:09:31,360 --> 00:09:35,640
this, this small, through this 

downstream tasks that the larger

161
00:09:35,640 --> 00:09:38,160
these challenges are, the more 

impact you can basically 

162
00:09:38,160 --> 00:09:42,480
generate. 
 
Yeah, it seemed to definitely 

163
00:09:43,240 --> 00:09:44,920
spur on. 
 
And now it seems that there's 

164
00:09:44,920 --> 00:09:49,040
been so many of the weather and 
 climate models, it's almost got

165
00:09:49,320 --> 00:09:52,240
not congested, but there's 
 
there's, there's quite a few 

166
00:09:52,280 --> 00:09:56,760
coming out. 
 
And so it's not obviously so 

167
00:09:56,760 --> 00:10:01,640
clear how each are progressing 

past each other and and how much

168
00:10:01,640 --> 00:10:06,280
are those of just an individual 
 groups need to have their own 

169
00:10:06,280 --> 00:10:08,632
model, if you know what I mean. 
 

170
00:10:08,640 --> 00:10:12,826
But did you, I guess one of the 
big things that I want to 
 

171
00:10:12,834 --> 00:10:18,928
discuss today with you in 
particular was to dive into, I 


172
00:10:18,936 --> 00:10:26,744
guess where machine learning, 
how can I phrase this machine 
 

173
00:10:26,752 --> 00:10:29,230
learning for engineering, for 
CFD, for CAE. 
 

174
00:10:29,238 --> 00:10:33,803
You know, it's something I've 
spoken to a few people about and

175
00:10:33,803 --> 00:10:39,420

 there's a lot going on, but 
it's hasn't always been so clear

176
00:10:39,420 --> 00:10:44,685

 where the big breakthroughs 
will be or how, how good are we 

177
00:10:44,685 --> 00:10:49,704
today 
 and where we're going. 
It it's it's still seems a 
 

178
00:10:49,712 --> 00:10:52,904
congested space with different 
opinions, you know, with PINNs 

179
00:10:52,904 --> 00:10:56,210
on 
 one side and then these 
sort of MeshGraphNets and other 

180
00:10:56,210 --> 00:10:59,020
 things. 
So it's a bit of a, an 
 

181
00:10:59,028 --> 00:11:03,894
open-ended question to you, but 
I guess like where do you see 
 

182
00:11:03,902 --> 00:11:07,100
the state-of-the-art? 
What what's been your journey 
 

183
00:11:07,108 --> 00:11:11,736
from the sort of MeshGraphNets 
towards your current thinking? 


184
00:11:11,744 --> 00:11:16,430
Like how, how have you try to 
solve this problem, I guess. 
 

185
00:11:16,438 --> 00:11:20,827
And let's, let's say, let's take
CFD, maybe is the the the 
 

186
00:11:20,835 --> 00:11:23,548
problem like automotive 
aerodynamics for example? 
 

187
00:11:23,556 --> 00:11:27,600
Yeah, that's a very good, my 
favorite topic actually making 


188
00:11:27,608 --> 00:11:30,948
the connections to weather. 
Let's let's say what what 
 

189
00:11:30,956 --> 00:11:33,072
basically was driving this 
weather modelling. 
 

190
00:11:33,080 --> 00:11:37,368
I would say that NVIDIA was the 
first with the FourCastNet 
 

191
00:11:37,376 --> 00:11:41,765
paper to to bring out a model 
which worked that was trained on

192
00:11:41,765 --> 00:11:44,984

 ERA5, which is a publicly 
available large scale data set. 

193
00:11:44,984 --> 00:11:47,120
 
Depending on how you sample you 

194
00:11:47,120 --> 00:11:51,520
get roughly a petabyte of data 

or a few 100 terabytes of data 

195
00:11:51,520 --> 00:11:54,720
from it. 
 
And then you do basically mean 

196
00:11:54,720 --> 00:11:59,299
squared error training that that

 was the the scene set and and 

197
00:11:59,299 --> 00:12:03,514
as soon as this problem was 
defined 
 of this input output 

198
00:12:03,514 --> 00:12:07,359
relation of this metrics of of 
what to test, 
 then people did 

199
00:12:07,359 --> 00:12:11,760
what, what they do best. 
 
They optimize on this sort of 

200
00:12:11,760 --> 00:12:15,120
problems. 
 
If you go to engineering, things

201
00:12:15,120 --> 00:12:17,440
are very different in, in many, 
 many aspects. 

202
00:12:17,440 --> 00:12:19,920
So first of all, we don't have 

this data set. 

203
00:12:19,920 --> 00:12:21,760
We don't have an ERA5 data 
 
set. 

204
00:12:22,200 --> 00:12:26,348
And even if you had an ERA5 data
set, I mean, there are now at 

205
00:12:26,348 --> 00:12:30,920
least 2-3 publicly 
 available 
CFD data sets. 

206
00:12:30,920 --> 00:12:35,240
Needless to say that my favorite

 one is the DrivAerML data set 

207
00:12:35,240 --> 00:12:40,120
which are industrial standard. 

But even if these data sets are 

208
00:12:40,120 --> 00:12:43,040
out there, people don't know how

 to train on them. 

209
00:12:43,040 --> 00:12:44,640
And everyone trains differently.

 

210
00:12:44,648 --> 00:12:48,848
And this comes from the fact 
that we just don't know what you

211
00:12:48,848 --> 00:12:51,830

 really want to optimize. 
There is people who sub sample 


212
00:12:51,838 --> 00:12:54,920
parts of the surface and map to,
to pressure values on the 
 

213
00:12:54,928 --> 00:12:57,624
surface, people who sub sample 
part of surface and volume and 


214
00:12:57,632 --> 00:13:00,982
so on and so forth. 
So similarly to whether we have 

215
00:13:00,982 --> 00:13:05,055
 to figure out what's the right 
task, what's the right learning 

216
00:13:05,055 --> 00:13:10,720
 task, which we which we have to
to do or like the tasks in, in 


217
00:13:10,728 --> 00:13:14,420
order to really develop the 
right models in the right 
 

218
00:13:14,428 --> 00:13:21,030
frameworks for that. 
Secondly, I think what is also 


219
00:13:21,038 --> 00:13:25,572
very different to weather is 
that we have this input output 


220
00:13:25,580 --> 00:13:27,704
relation. 
So in weather, it's always clear

221
00:13:27,704 --> 00:13:31,454

 you take time T and you map to
T + 1. 
 

222
00:13:31,462 --> 00:13:35,722
That's basically a segmentation 
task on steroids because in the 

223
00:13:35,722 --> 00:13:40,543
 end every pixel on the Earth 
gets mapped to to a pixel on 
 

224
00:13:40,551 --> 00:13:45,189
the earth like a time time point
later, which you can basically 


225
00:13:45,197 --> 00:13:48,938
you can take all the models from
computer vision. 
 

226
00:13:48,946 --> 00:13:52,480
And obviously there is some 
tricks with resolution and and 


227
00:13:52,488 --> 00:13:55,308
if you take the sphere into 
account, yes or not. 
 

228
00:13:55,316 --> 00:13:59,060
But in the end it's it's a pixel
segmentation task. 
 

229
00:13:59,068 --> 00:14:04,604
And this is definitely not true 
for for many CFD related tasks 


230
00:14:04,612 --> 00:14:09,960
because the task there is from a
pure numerical perspective, you 

231
00:14:09,960 --> 00:14:14,160
 have a geometry which you can 
characterize with couple of 
 

232
00:14:14,168 --> 00:14:16,258
parameters, probably less than 
100 even. 
 

233
00:14:16,266 --> 00:14:18,852
And then you get this 
full-fledged flow fields around 

234
00:14:18,852 --> 00:14:23,308
 and on the car so that the 
actual input to output ratio is 

235
00:14:23,308 --> 00:14:28,160
 very, very different. 
And we don't have architectures,

236
00:14:28,160 --> 00:14:31,984

 we don't have frameworks, we 
don't have an understanding of 


237
00:14:31,992 --> 00:14:36,072
how to tackle that. 
But that makes it so exciting I 

238
00:14:36,072 --> 00:14:39,522
 would say. 
And maybe thirdly, if you, if 
 

239
00:14:39,530 --> 00:14:43,729
you really look at these weather
models and what they can do with

240
00:14:43,729 --> 00:14:46,130

 hurricane predictions and so 
on, obviously there's always 

241
00:14:46,130 --> 00:14:48,899
room 
 for improvement and 
obviously you can go farther and

242
00:14:48,899 --> 00:14:50,040
farther. 
 
But if you just look what 

243
00:14:50,040 --> 00:14:53,160
happened in the last two years 

and the resources and weather 

244
00:14:53,160 --> 00:14:56,840
are I would say it's tiny, tiny 
 fractions to the resources in 

245
00:14:56,840 --> 00:15:00,600
LLM and and video generation. 
 
You see this tremendous progress

246
00:15:00,600 --> 00:15:03,607
and what what hard problems and 
 hard multi scale problems we 

247
00:15:03,607 --> 00:15:06,520
are already able to model and 
which we can model better than 

248
00:15:06,520 --> 00:15:09,080
numerics. 
 
So it's very clear that if done 

249
00:15:09,080 --> 00:15:11,880
rightly and if the right 
 
incentives are there, this 

250
00:15:12,000 --> 00:15:15,680
progress and this type of 
 
complexity we're able to do in 

251
00:15:15,680 --> 00:15:18,480
in CFD. 
 
Obviously then we have to play 

252
00:15:18,480 --> 00:15:21,080
the game numerics and ML 
 
together. 

253
00:15:22,120 --> 00:15:24,920
Yeah, which is then I think the 
 hardest task to answer. 

254
00:15:27,200 --> 00:15:30,480
Yeah, that, that does seem, I 
 
guess the joke that the Earth is

255
00:15:30,480 --> 00:15:33,000
always the Earth. 
 
And so the geometry is sort of, 

256
00:15:33,520 --> 00:15:35,320
you know, not so much the 
 
problem. 

257
00:15:36,880 --> 00:15:42,440
And yeah, I guess it all depends

 on what is the data. 

258
00:15:44,320 --> 00:15:48,400
You're right, in the CFD 
 
example, the geometries could be

259
00:15:48,400 --> 00:15:52,160
hugely varying and that link 
 
between the geometry and the 

260
00:15:52,160 --> 00:15:56,960
volume is a Yeah, it it's so the

 geometry is geometry and the 

261
00:15:56,960 --> 00:15:59,160
boundary conditions is 
 
ultimately what affects the 

262
00:15:59,160 --> 00:16:03,320
flows. 
 
So where have you seen, I guess 

263
00:16:03,320 --> 00:16:06,280
a lot of people listening to 
 
this and myself included, 

264
00:16:06,880 --> 00:16:10,840
probably had their first 
 
introduction with, you know, 

265
00:16:10,840 --> 00:16:14,440
what the DeepMind team did with 
 MeshGraphNets. 

266
00:16:14,600 --> 00:16:17,480
I mean, there was obviously 
 
stuff before that, but that felt

267
00:16:17,480 --> 00:16:24,120
like the bit when everyone stood

 up and and noticed why? 

268
00:16:24,920 --> 00:16:28,720
Why do you not feel that that is

 the right approach? 

269
00:16:30,800 --> 00:16:34,440
It's always right or wrong. 
 
It's always hard to to to judge.

270
00:16:34,440 --> 00:16:39,060
But I think that the, the 
MeshGraphNets and, and, and and 

271
00:16:39,060 --> 00:16:42,000
and graph neural network 
simulators 
 from the Peter 

272
00:16:42,000 --> 00:16:44,385
Battaglia group were 
tremendously important 
 

273
00:16:44,393 --> 00:16:48,605
milestone in this field, mostly 
because they, they were writing 

274
00:16:48,605 --> 00:16:53,395
 down this learning problem. 
And I myself went into this 
 

275
00:16:53,403 --> 00:16:57,545
field of simulations because of 
these papers trying to model 
 

276
00:16:57,553 --> 00:17:01,737
particle particle systems. 
What, what we noticed back in 
 

277
00:17:01,745 --> 00:17:05,079
the days, I mean, this is like 
5-6 years ago, that it's 
 

278
00:17:05,088 --> 00:17:08,188
tremendously hard to model all 
particle particle interaction as

279
00:17:08,188 --> 00:17:11,569

 numerics is doing. 
So if you have this 10,000 
 

280
00:17:11,577 --> 00:17:14,575
system of particles and they 
interact and you have you have 


281
00:17:14,583 --> 00:17:17,508
something like a graph neural 
network and you have to make all

282
00:17:17,508 --> 00:17:20,660

 these interactions correctly 
because otherwise the temporal 


283
00:17:20,667 --> 00:17:26,862
integration will will just, 
yeah, just go, go, go crazy. 
 

284
00:17:26,869 --> 00:17:29,692
That's just a very hard learning
problem. 
 

285
00:17:29,700 --> 00:17:31,595
And this is not what machine 
learning is really good at. 
 

286
00:17:31,603 --> 00:17:34,100
Machine learning is good at to 
understand the global dynamics, 

287
00:17:34,100 --> 00:17:37,304
 to understand the behavior of 
the system, to understand where 

288
00:17:37,304 --> 00:17:40,850
 things are going on a global 
scale, but not on a particle 
 

289
00:17:40,858 --> 00:17:45,336
particle scale. 
So I, I think from, from, from 


290
00:17:45,344 --> 00:17:48,446
setting up this learning 
problem, this was a tremendously

291
00:17:48,446 --> 00:17:52,062

 important step. 
Obviously new ways of, of, of of

292
00:17:52,062 --> 00:17:56,120

 models are coming the way for 
me personally, it was a big 
 

293
00:17:56,128 --> 00:17:59,080
breakthrough that we started to,
to see things as fields. 
 

294
00:17:59,088 --> 00:18:02,400
So everything in my, in my head 
is a field. 
 

295
00:18:02,408 --> 00:18:05,808
And because you a field is 
something which evolves over 
 

296
00:18:05,816 --> 00:18:07,802
time. 
It can be an occupancy field, 
 

297
00:18:07,810 --> 00:18:10,192
which tells you where is mass, 
where is no mass. 
 

298
00:18:10,200 --> 00:18:11,730
It can be signed distance field.

 

299
00:18:11,738 --> 00:18:14,660
It can be a velocity field, it 
can be a displacement field, 
 

300
00:18:14,668 --> 00:18:18,983
whatever field you want. 
But it's much easier for a, for 

301
00:18:18,983 --> 00:18:23,020
 a model to, to tackle, which is
actually obviously the, the next

302
00:18:23,020 --> 00:18:25,474

 step if you think of this 
whole scientific machine 

303
00:18:25,474 --> 00:18:28,859
learning and 
 neural operator 
and, and, and so on and so 

304
00:18:28,859 --> 00:18:31,280
forth. 
 
And in the end, I mean, this is 

305
00:18:31,280 --> 00:18:33,480
also how you, you model the, 
 
the, the weather. 

306
00:18:33,480 --> 00:18:36,440
So I, I think conceptually from 
 me personally, from me, I don't

307
00:18:36,440 --> 00:18:39,760
know how it is for others. 
 
Understanding that you can model

308
00:18:39,760 --> 00:18:42,920
large scale systems as a field 

was, was breaking a lot of 

309
00:18:42,920 --> 00:18:45,640
boundaries in, in terms of 
 
scalability. 

310
00:18:45,640 --> 00:18:50,200
Because if you model a system of

 10 million, 100 million 

311
00:18:50,200 --> 00:18:53,520
particles as a field, it's 
 
computation and not the problem.

312
00:18:53,520 --> 00:18:55,280
It's from a learning 
 
perspective, it's not the 

313
00:18:55,280 --> 00:18:57,600
problem because you just don't 

need all these particles. 

314
00:18:57,600 --> 00:19:01,520
You you'll learn the dynamics 
 
with the very heavily subsampled

315
00:19:03,360 --> 00:19:08,400
representation and and and and 

obtaining the full field is also

316
00:19:08,400 --> 00:19:11,040
not a problem because this this 
 exists. 

317
00:19:11,440 --> 00:19:15,280
So I would say for me 
 
understanding what machine 

318
00:19:15,280 --> 00:19:18,480
learning is really good at and 

and and and and and what 

319
00:19:18,680 --> 00:19:22,800
numerics is really good at and 

and and and and playing those 

320
00:19:22,800 --> 00:19:27,800
two on, on on those two fronts 

was was was making the the 

321
00:19:27,800 --> 00:19:29,440
difference. 
 
Yeah. 

322
00:19:30,800 --> 00:19:36,280
Well, one thing that I've maybe 
 would appreciate you explaining

323
00:19:36,560 --> 00:19:41,364
to the audience and maybe let's 
 first introduce some of the 

324
00:19:41,364 --> 00:19:45,160
work that you've done in this 
recent 
 paper of yours. 

325
00:19:46,600 --> 00:19:51,160
So you developed this 
 
transformer-based approach, what

326
00:19:51,160 --> 00:19:54,214
you call the Anchored-Branched 
Universal Physics Transformer. 

327
00:19:54,214 --> 00:19:54,800
What? 
 
What? 

328
00:19:54,880 --> 00:19:58,800
Well, maybe you explain what? 
 
Yeah, we. 

329
00:19:59,440 --> 00:20:03,680
Have actually a couple of works 
 of 1 is which we call Universal

330
00:20:03,680 --> 00:20:05,960
Physics Transformer. 
 
That's basically how we do 

331
00:20:05,960 --> 00:20:08,200
latent space modelling of 
 
physical systems. 

332
00:20:08,600 --> 00:20:11,080
Then we did something which is 

called NeuralDEM where we 

333
00:20:11,080 --> 00:20:14,640
applied those to to multiphase 
flows 
 and and and multi. 

334
00:20:15,200 --> 00:20:19,960
Particle systems and then we, 
 
we, we recently applied this to 

335
00:20:19,960 --> 00:20:23,880
to large scale CFD similar 
 
framework, similar approach, 

336
00:20:24,680 --> 00:20:27,520
which we call Anchored-Branched 
UPT, 
 because these are the two

337
00:20:27,520 --> 00:20:30,880
ingredients which which which we

 need to do in order to get 

338
00:20:30,880 --> 00:20:37,360
this to work on large systems. 

So in that context, what I'm 

339
00:20:37,360 --> 00:20:40,960
hoping you could do is maybe 
 
explain a little bit, because 

340
00:20:40,960 --> 00:20:42,920
there's a lot of people who 
 
listen to this podcast who are 

341
00:20:42,920 --> 00:20:48,440
maybe not ML specialists, but 
 
are CFD primarily specialists 

342
00:20:48,440 --> 00:20:52,280
who are now, you know, more and 
 more interested in ML and maybe

343
00:20:52,280 --> 00:20:55,960
actually developing it and, and 
 yeah, sort of progressing. 

344
00:20:56,320 --> 00:21:03,520
And you mentioned a really 
 
interesting point to me and I'm 

345
00:21:03,520 --> 00:21:06,600
hoping you can explain this in a

 way that is understandable by 

346
00:21:06,600 --> 00:21:11,080
others is that I, and I guess 
 
some of this is a maths 

347
00:21:11,080 --> 00:21:14,200
exercise, but you know, and I've

 been guilty of this. 

348
00:21:14,200 --> 00:21:17,640
Sometimes there's this 
 
hierarchical thought process of 

349
00:21:17,640 --> 00:21:21,360
being like, OK, you've got some 
 sort of graph based thing where

350
00:21:21,360 --> 00:21:24,074
it's more like point to point. 

Then you've got a neural 

351
00:21:24,074 --> 00:21:27,600
operator where it's like more to

 solutions or fields. 

352
00:21:28,240 --> 00:21:31,480
And then I initially thought of,

 OK, and then you have 

353
00:21:31,480 --> 00:21:34,440
Transformers. 
 
And yet you see some people who 

354
00:21:34,440 --> 00:21:36,708
say, well, everything is a 
neural operator if you do the 

355
00:21:36,708 --> 00:21:41,720
maths or design it. 
 
So can you explain and maybe 

356
00:21:41,720 --> 00:21:45,280
dive a little bit deeper into 
 
that concept of solution mapping

357
00:21:45,280 --> 00:21:47,880
or field mapping and then how it
links to Transformers? 

358
00:21:47,880 --> 00:21:50,960
Because I think it is a little 

bit confusing for for for 

359
00:21:50,960 --> 00:21:53,160
people. 
 
So if yeah, if we could maybe 

360
00:21:53,160 --> 00:21:57,729
get into that, that would be in 
 in the context of let's say 

361
00:21:57,729 --> 00:22:00,680
this DrivAerML or car, however 

you want to explain it. 

362
00:22:01,280 --> 00:22:04,200
Yeah, happily to do so. 
 
So for me, in fact neural 

363
00:22:04,200 --> 00:22:06,547
operate and this is also how it 
 is introduced is something 

364
00:22:06,547 --> 00:22:10,572
which maps between function 
spaces, 
 which is very yeah, 

365
00:22:10,572 --> 00:22:15,044
which is not very informative if
you if you, 
 because what does 

366
00:22:15,044 --> 00:22:17,352
it mean? 
You map between function spaces.

367
00:22:17,352 --> 00:22:19,960

 
But I always think of it as you 

368
00:22:19,960 --> 00:22:22,760
have two functions, one input 
 
function, 1 output function. 

369
00:22:23,000 --> 00:22:28,040
And no matter what you sample 
 
from the input function and what

370
00:22:28,040 --> 00:22:30,240
you predict on the output 
 
function, this has to be 

371
00:22:30,240 --> 00:22:32,960
fulfilled. 
 
Meaning that if I sample 10 

372
00:22:32,960 --> 00:22:36,760
points from the input function, 
 I should be able to predict as 

373
00:22:36,760 --> 00:22:39,680
many points as I want from the 

output function and those points

374
00:22:39,680 --> 00:22:41,600
I predict are actually on the 
 
output function. 

375
00:22:41,960 --> 00:22:46,760
If I sample with the same 
 
network with the same operator 6

376
00:22:46,760 --> 00:22:50,160
points from the input function, 
 I should also get the same 10 

377
00:22:50,160 --> 00:22:53,640
points and if I want more 
 
points, I get more points on the

378
00:22:53,640 --> 00:22:56,400
output function. 
 
Obviously there is a limit. 

379
00:22:56,400 --> 00:22:59,800
If I sample too few input 
 
points, then obviously the 

380
00:22:59,800 --> 00:23:02,600
approximation network is not 
 
able to really get the 

381
00:23:02,600 --> 00:23:06,920
information it needs. 
 
But if if this this this 

382
00:23:06,920 --> 00:23:10,074
sampling limit is is reached no 
 matter how many points and 

383
00:23:10,074 --> 00:23:12,797
where those points are spaced, 
it 
 should always give you some

384
00:23:12,797 --> 00:23:15,814
sort of representation which 
allow 
 you to construct the 

385
00:23:15,814 --> 00:23:17,720
output function. 
 
So in the latent space this 

386
00:23:17,720 --> 00:23:22,890
mapping is, then is then, so to 
 say, resolution agnostic and 

387
00:23:22,890 --> 00:23:27,688
and that with some small tricks 
you 
 can do with convolutions, 

388
00:23:27,688 --> 00:23:30,456
with Fourier neural operators, 
with 
 Transformers and so on 

389
00:23:30,456 --> 00:23:33,280
and so forth. 
 
And obviously Transformers with 

390
00:23:33,280 --> 00:23:36,880
their. 
 
Can I if I've seen to it just 

391
00:23:36,880 --> 00:23:40,480
for one second, just to make it 
 even more clearer, when you say

392
00:23:41,160 --> 00:23:44,680
sampling the input function and 
 the output function by input 

393
00:23:44,680 --> 00:23:49,480
function, you're meaning like 
 
the geometry surface in this 

394
00:23:49,480 --> 00:23:54,000
conceptual sense and the output 
 function being let's say the 

395
00:23:54,000 --> 00:23:56,600
volume. 
 
Would that be 1 interpretation? 

396
00:23:56,840 --> 00:23:59,600
Yes, for example you can. 
 
You can think of it that the 

397
00:23:59,600 --> 00:24:03,040
input is, the is the geometry 
 
and the output is either the 

398
00:24:03,040 --> 00:24:07,120
field on the geometry or the 
 
field in the volume, or both 

399
00:24:07,440 --> 00:24:11,400
actually. 
 
And the idea of this so where 

400
00:24:11,480 --> 00:24:12,600
you know, this sounds like a bit

 of AI. 

401
00:24:12,600 --> 00:24:16,560
Remember when this first came 
 
out, I was I think it was billed

402
00:24:16,600 --> 00:24:22,240
as a resolution independent. 
 
So it which always to a CFD 

403
00:24:22,240 --> 00:24:27,160
person feels like the no free 
 
lunch sort of theorem. 

404
00:24:27,160 --> 00:24:29,967
Like how can this possibly Yeah,

 you know, how can that be 

405
00:24:29,967 --> 00:24:31,838
true? 
You're basically saying I don't 

406
00:24:31,838 --> 00:24:35,042
 need to map the entire 
geometry, I can just take a few 

407
00:24:35,042 --> 00:24:36,280
and still 
 get the same 
answers. 

408
00:24:36,280 --> 00:24:41,880
So where's the catch? 
 
I I guess on this? 

409
00:24:42,760 --> 00:24:44,120
Yeah. 
 
I mean that that's a very good 

410
00:24:44,120 --> 00:24:45,960
point. 
 
And I think I can make my, my 

411
00:24:45,960 --> 00:24:49,880
point very clear with, with 
 
really now going to CFD and, 

412
00:24:49,880 --> 00:24:53,080
and, and before that, I really 

have to say, if people think of 

413
00:24:53,080 --> 00:24:57,720
resolution, we always think 
 
again of this segmentation as, 

414
00:24:57,760 --> 00:25:00,920
as we do with weather, right? 
 
You have a certain number of 

415
00:25:00,920 --> 00:25:04,120
longitude and latitude points. 

And if we double those points, 

416
00:25:04,560 --> 00:25:08,120
do we get higher resolution or 

do we just have an interpolation

417
00:25:08,120 --> 00:25:11,560
and so on and so forth. 
 
So can we just run our whatever 

418
00:25:11,560 --> 00:25:14,480
unit and interpolate between 
 
this resolution or is there a 

419
00:25:14,480 --> 00:25:16,270
way to get this stuff resolved? 
 

420
00:25:16,278 --> 00:25:20,500
So there's always this input 
output mapping and obviously it 

421
00:25:20,500 --> 00:25:23,496
 will end up with interpolation 
effect. 
 

422
00:25:23,504 --> 00:25:26,120
So either you interpolate 
somewhere in your network or you

423
00:25:26,120 --> 00:25:29,652

 interpolate on the output grid
because as you said, no free 
 

424
00:25:29,660 --> 00:25:33,195
lunch. 
But if we go to CFD, things are 

425
00:25:33,195 --> 00:25:35,593
 a bit different, especially 
that the input output 

426
00:25:35,593 --> 00:25:38,552
resolution, as 
 I said before, 
is very different or input 

427
00:25:38,552 --> 00:25:42,360
output connection. 
 
And, and actually we show in 

428
00:25:42,360 --> 00:25:47,120
this recent paper that if you 
 
subsample a few of the points, 

429
00:25:47,120 --> 00:25:50,600
so both on geometry and surface 
 and, and you do some, some 

430
00:25:50,600 --> 00:25:53,280
training, there is a certain 
 
amount of points, but the 

431
00:25:53,280 --> 00:25:56,760
performance stagnates. 
 
So if it's, it's for for DrivAer

432
00:25:56,760 --> 00:26:03,452
ML, this is roughly 128K, I 
think 
 or 256 I, I, I don't 

433
00:26:03,452 --> 00:26:07,560
know by heart. 
 
And if you then add more points,

434
00:26:07,560 --> 00:26:10,320
the performance is not getting 

better because all the, the 

435
00:26:10,320 --> 00:26:13,200
information which in neural 
 
network needs is in those 

436
00:26:13,200 --> 00:26:16,680
points. 
 
And, and if you sample it, let's

437
00:26:16,680 --> 00:26:20,600
say cleverly so, so that the 
 
distribution really covers the 

438
00:26:20,600 --> 00:26:22,360
whole surface and the critical 

parts. 

439
00:26:22,360 --> 00:26:25,800
And then also points at the 
 
closer to the surface and the 

440
00:26:25,800 --> 00:26:27,720
volume are represented 
 
correctly. 

441
00:26:27,800 --> 00:26:30,760
All this, this type of stuff. 
 
But there is a certain amount of

442
00:26:30,760 --> 00:26:33,160
points where you really capture 
 the whole phenomenon. 

443
00:26:33,680 --> 00:26:37,240
And that's the, the, the, the 
 
resolution invariant. 

444
00:26:37,240 --> 00:26:39,640
So to say. 
 
Obviously, if you go lower with 

445
00:26:39,640 --> 00:26:42,920
the point to sample, you lose a 
 bit of representation. 

446
00:26:42,920 --> 00:26:46,640
So you don't, you don't resolve 
 the full physics. 

447
00:26:46,640 --> 00:26:50,160
So you're never able to really 

recover the whole physics. 

448
00:26:50,160 --> 00:26:53,280
But there is a, a certain amount

 of points you need. 

449
00:26:53,280 --> 00:26:56,720
And this is for, for each type 

of physics problem that you have

450
00:26:56,720 --> 00:26:58,800
the full information in the 
 
network. 

451
00:26:59,240 --> 00:27:01,640
And then obviously depends how 

good your network is. 

452
00:27:02,080 --> 00:27:05,367
And then obviously it makes much

 more sense to not have a 

453
00:27:05,367 --> 00:27:09,456
network which again spits out 
this 128 
 points, but which 

454
00:27:09,456 --> 00:27:12,777
conceptually is able to spit out
as many 
 points as you want 

455
00:27:12,777 --> 00:27:14,770
because that's the neural 
operator, right? 
 

456
00:27:14,778 --> 00:27:18,490
That you can really get every 
point on the output function if 

457
00:27:18,490 --> 00:27:21,845
 if needed. 
And this is something which is 


458
00:27:21,853 --> 00:27:24,934
which is very, very interesting 
in CFD. 
 

459
00:27:24,942 --> 00:27:28,165
Yeah. 
So I guess, again, correct me if

460
00:27:28,165 --> 00:27:31,518

 I'm wrong, if I'm distilling 
this correctly, is you're saying

461
00:27:31,518 --> 00:27:35,980

 if the surface in reality has 
8,000,000 points or whatever the

462
00:27:35,980 --> 00:27:41,800

 exact number is, you're saying
that you should be able by 
 

463
00:27:41,808 --> 00:27:45,505
training and sampling 
progressively different points, 

464
00:27:45,505 --> 00:27:52,072
 you shouldn't need more than 
256,000 to actually do as good 


465
00:27:52,080 --> 00:27:57,620
job as 8 million. 
Whereas I guess we've the 
 

466
00:27:57,628 --> 00:28:00,140
analogy. 
I suppose what I'm trying to 
 

467
00:28:00,148 --> 00:28:03,430
pick out is with MeshGraphNets 
or with graph neural networks, 


468
00:28:03,438 --> 00:28:07,320
people could in theory take a 
million, but they would normally

469
00:28:07,320 --> 00:28:10,914

 like down sample, but they're 
down sampling at least. 
 

470
00:28:10,922 --> 00:28:15,824
Correct me if I'm wrong, you are
then changing the problem 
 

471
00:28:15,832 --> 00:28:20,042
statement in this it it will 
give you a different answer and 

472
00:28:20,042 --> 00:28:23,030
 and that I know was certain 
experience that we had and 
 

473
00:28:23,038 --> 00:28:25,974
others have had where you go 
well, how much do I downsample 


474
00:28:25,982 --> 00:28:28,712
and where do I pick the point to
downsample? 
 

475
00:28:28,720 --> 00:28:33,202
And so a lot of people take an 
8,000,000 cell can't fit in a 

476
00:28:33,202 --> 00:28:37,920
GPU's memory and take 500,000 
but they're worried when they do

477
00:28:37,920 --> 00:28:42,180

 inference they then would have
to give the exact same mesh 
 

478
00:28:42,188 --> 00:28:44,952
distribution or if they give a 
different mesh distribution. 
 

479
00:28:44,960 --> 00:28:47,644
Do do you know what I'm getting?
At that's lovely. 
 

480
00:28:47,652 --> 00:28:50,360
I mean you, you say the most 
important point, which I forgot.

481
00:28:50,360 --> 00:28:51,760

 
Obviously if you have to 

482
00:28:51,760 --> 00:28:54,880
calculate something with drag or

 lift coefficient for which you

483
00:28:55,240 --> 00:28:57,320
you really need to get the 
 
correct value. 

484
00:28:57,320 --> 00:29:00,600
You need the full simulation 
 
mesh, the full 8 to 9 million 

485
00:29:00,960 --> 00:29:03,480
surface mesh. 
 
And obviously the network has to

486
00:29:03,480 --> 00:29:07,040
give you this full mesh because 
 otherwise you're never able to 

487
00:29:07,040 --> 00:29:09,440
calculate correct drag and lift 
 coefficient. 

488
00:29:09,800 --> 00:29:12,920
But the network is giving you 
 
this full network idea. 

489
00:29:12,920 --> 00:29:17,000
Sorry, this full drag and lift 

coefficient on the full surface 

490
00:29:17,000 --> 00:29:23,440
mesh, no matter if you give it 

64,128 thousand 256,000 input 

491
00:29:23,440 --> 00:29:27,440
points, it's just at some in 
 
number of input points the 

492
00:29:27,440 --> 00:29:30,240
output is stagnating or the 
 
performance is stagnating 

493
00:29:30,240 --> 00:29:34,200
because you have reached the 
 
maximum performance and the the 

494
00:29:34,200 --> 00:29:38,440
input and and obviously you can 
 also give 8 million input 

495
00:29:38,440 --> 00:29:40,720
points. 
 
The the the self attention will 

496
00:29:40,720 --> 00:29:43,720
be terribly slow and I don't 
 
know what what parallelization 

497
00:29:43,720 --> 00:29:47,120
tricks you need to do, but 
 
nevertheless it will give you 

498
00:29:47,120 --> 00:29:50,840
the same answer in the output. 

So you can drastically reduce 

499
00:29:50,840 --> 00:29:53,960
what you give as input as long 

as the output still gives you 

500
00:29:53,960 --> 00:29:56,280
the correct answer. 
 
And that's, that's the magic we 

501
00:29:56,360 --> 00:29:59,560
we got from transformer, which 

we cannot do with with other 

502
00:29:59,560 --> 00:30:01,760
methods for. 
 
So for other methods we really 

503
00:30:01,760 --> 00:30:04,800
had to make the problem much 
 
easier, which is obviously 

504
00:30:04,800 --> 00:30:12,520
giving you the wrong physics. 
 
So if we down move on to and I 

505
00:30:12,520 --> 00:30:16,360
guess this is the the novelty 
 
that I saw in your approach and 

506
00:30:16,400 --> 00:30:18,240
and I thought it was good to 
 
double click on a bit. 

507
00:30:18,240 --> 00:30:25,320
It's why the Transformer, what 

specifically is that 

508
00:30:25,320 --> 00:30:31,120
architecture giving you versus 

alternatives because yeah, and 

509
00:30:31,120 --> 00:30:33,460
how and how that relate, if you 
 could maybe go into a little 

510
00:30:33,460 --> 00:30:36,265
bit because Transformers, I 
think 
 most people know 

511
00:30:36,265 --> 00:30:39,712
conceptually, you know, tokens, 
etcetera, 
 LLMs, but maybe not 

512
00:30:39,712 --> 00:30:44,960
in the context of CFD. 
 
Yeah, so transformer are 

513
00:30:45,400 --> 00:30:50,840
tremendously flexible, 
 
tremendously optimised and and 

514
00:30:50,840 --> 00:30:56,094
and tremendously well understood

 powerhouse, workhorse in, in, 

515
00:30:56,094 --> 00:30:59,048
in deep learning, 
 right. 
They are they the the engine 
 

516
00:30:59,056 --> 00:31:01,400
behind large language model 
computer vision and and all 
 

517
00:31:01,408 --> 00:31:07,050
these tricks are already done 
and they have a few certain 
 

518
00:31:07,058 --> 00:31:11,015
properties which are super nice.
The 1st is the the invariance 
 

519
00:31:11,023 --> 00:31:15,784
with respect to sequence length.
So no matter how long the the 
 

520
00:31:15,792 --> 00:31:18,715
sequence length is which 
corresponds how many points you 

521
00:31:18,715 --> 00:31:21,968
 input to the network, it will 
do the same calculations, right?

522
00:31:21,968 --> 00:31:23,720

 
Which is very important for the 

523
00:31:23,720 --> 00:31:31,840
scaling properties. 
 
And then they have this how to 

524
00:31:31,840 --> 00:31:35,920
say so I call it discretisation 
convergence. 

525
00:31:35,920 --> 00:31:41,280
So if you sample more points in 
 a in an error in an area, it is

526
00:31:41,960 --> 00:31:45,320
basically it will converge to 
 
some to some field. 

527
00:31:46,080 --> 00:31:48,440
So, so the more points to 
 
sample, the, the better the 

528
00:31:48,440 --> 00:31:52,240
resolution gets, which is 
 
obviously extremely valid, 

529
00:31:52,400 --> 00:31:55,800
important property for this 
neural operator paradigm. 

530
00:31:57,400 --> 00:32:00,880
They they are made. 
 
So the transformer paradigm is 

531
00:32:00,880 --> 00:32:03,640
made and, and stress tested 
 
again and again and again for 

532
00:32:03,640 --> 00:32:06,800
scaling, meaning that you can 
 
build larger models, that you 

533
00:32:06,800 --> 00:32:11,440
can ingest larger data sets. 
 
Yeah. 

534
00:32:11,440 --> 00:32:14,280
And, and, and all these things 

make them an ideal fit for us. 

535
00:32:15,320 --> 00:32:17,880
I mean, I'm very happy that not 
 everyone is using Transformers.

536
00:32:17,880 --> 00:32:20,120
That gives us some edge, but 
 
yeah. 

537
00:32:21,200 --> 00:32:25,840
But specifically, So how do you 
 tokenize the problem then? 

538
00:32:25,840 --> 00:32:28,745
How could people conceptually 
 
understand, you know, 

539
00:32:28,745 --> 00:32:31,820
MeshGraphNets conceptually was: 
I have a 
 node which 

540
00:32:31,820 --> 00:32:35,592
corresponds to my mesh, I have 
some edges and then 
 I take 

541
00:32:35,592 --> 00:32:40,136
that like you're not taking a 
token per node clearly 
 or that

542
00:32:40,136 --> 00:32:43,970
wouldn't scale, right. 
So how could people conceptually

543
00:32:43,970 --> 00:32:48,248

 think of that sort of 
tokenizing the the surface or 

544
00:32:48,248 --> 00:32:51,360
the volume? 
 
I think that the way you have to

545
00:32:51,360 --> 00:32:54,760
approach this is not from a 
 
physicist perspective, but from 

546
00:32:54,760 --> 00:32:59,480
a computer vision perspective. 

So what do people do in computer

547
00:32:59,480 --> 00:33:01,120
vision? 
 
So I think when do, when you do 

548
00:33:01,320 --> 00:33:05,800
something like image generation 
 where you type in a prompt, I 

549
00:33:05,800 --> 00:33:09,960
want a horse in the, in the in 

the woods with whatever. 

550
00:33:10,720 --> 00:33:14,000
So you have some text and then 

you mix that with some image 

551
00:33:14,000 --> 00:33:17,000
which is generated, right? 
 
So you have basically 2 streams 

552
00:33:17,680 --> 00:33:22,040
and, and the second important 
 
part is this concept of 

553
00:33:22,040 --> 00:33:27,040
patching. 
 
So the transformer for for text,

554
00:33:27,040 --> 00:33:29,800
it's very clear each word gets a

 token. 

555
00:33:30,360 --> 00:33:36,600
So each word in a sentence or 
 
probably each each sign or 

556
00:33:36,600 --> 00:33:41,240
whatever is tokenized for for 
 
images, small patches of the 

557
00:33:41,240 --> 00:33:43,800
image will correspond to a token

 and a word. 

558
00:33:44,720 --> 00:33:47,400
So everything is a token which 

then can interact with each 

559
00:33:47,400 --> 00:33:50,480
other. 
 
But in in language, it's the 

560
00:33:50,480 --> 00:33:55,080
token's words in, in in vision, 
 the tokens is small patches of 

561
00:33:55,080 --> 00:33:58,320
of an image. 
 
So if we do the same thing in in

562
00:33:58,320 --> 00:34:03,200
CFD, what we actually have, we 

have basically three type of of 

563
00:34:03,200 --> 00:34:05,320
information. 
 
We have the information of the 

564
00:34:05,320 --> 00:34:07,840
geometry. 
 
So in general, what geometry we 

565
00:34:07,840 --> 00:34:11,960
have, we have the information of

 the volume and we have the 

566
00:34:11,960 --> 00:34:18,159
information of basically the 
 
physics on the on the geometry. 

567
00:34:18,440 --> 00:34:21,880
So it makes sense to treat this 
 as either two or three 

568
00:34:22,280 --> 00:34:24,080
modalities as we call it, right?

 

569
00:34:24,088 --> 00:34:30,056
So as as you have text and and 
and image and then you can 
 

570
00:34:30,063 --> 00:34:33,672
tokenize either small areas, 
which makes sense. 
 

571
00:34:33,679 --> 00:34:37,876
If you if you look, try to 
encode the geometry so small 
 

572
00:34:37,884 --> 00:34:41,922
neighboring areas get pulled 
into one token so that you have 

573
00:34:41,922 --> 00:34:44,329
 this information which gets 
locally aggregated. 
 

574
00:34:44,337 --> 00:34:49,404
Or you can for, for actually 
doing physics, you just take 
 

575
00:34:49,411 --> 00:34:52,735
some samples of the mesh and see
them as a token. 
 

576
00:34:52,744 --> 00:34:55,520
And then you have the same 
setup, you have different 
 

577
00:34:55,527 --> 00:34:58,320
tokens, they represent different
physical objects or physical 

578
00:34:58,320 --> 00:35:01,640
parts 
 of the physics. 
And then similar to what people 

579
00:35:01,640 --> 00:35:04,367
 do in computer vision, you let 
those tokens speak to each 
 

580
00:35:04,375 --> 00:35:07,400
other. 
You link these volume 
 tokens 

581
00:35:07,400 --> 00:35:10,780
between each other and the 
volume tokens to the surface 
 

582
00:35:10,788 --> 00:35:14,746
and so on and so forth. 
So it's really borrowing a lot 


583
00:35:14,754 --> 00:35:19,224
of concepts from computer vision
because that's what works. 
 

584
00:35:19,232 --> 00:35:29,154
So if I understand it correctly,
essentially what you're doing is

585
00:35:29,154 --> 00:35:32,614

 patching. 
So if you were to look at the 
 

586
00:35:32,622 --> 00:35:35,685
surface mesh and if you have, 
you know, visually, if you 
 

587
00:35:35,693 --> 00:35:41,458
looked at it and you had a clump
of 5 by 5, then you, you would 


588
00:35:41,466 --> 00:35:46,190
have, you know, 25 cells and 
however many edges and nodes, 
 

589
00:35:46,198 --> 00:35:49,518
you would say, OK, well, that 
can just become one token and 
 

590
00:35:49,526 --> 00:35:51,275
then you go into the next, the 
next. 
 

591
00:35:51,283 --> 00:35:56,022
So if you, so essentially you're
able to go from 8,000,000 points

592
00:35:56,022 --> 00:36:00,191

 and instead of taking 8 
million tokens, you would have 

593
00:36:00,191 --> 00:36:04,144
you know, 
 10,000 or what 
whatever the number is, is that.

594
00:36:04,144 --> 00:36:06,640

 
And the assumption is that the 

595
00:36:06,640 --> 00:36:13,760
differences within that patch 
 
should be small enough to be not

596
00:36:13,760 --> 00:36:16,440
important. 
 
If you were to take too big a 

597
00:36:16,440 --> 00:36:20,760
patch, I assume you would lose 

some accuracy because you're 

598
00:36:20,760 --> 00:36:23,760
losing some local information. 

Would that be fair to say there 

599
00:36:23,760 --> 00:36:25,840
comes a cut off point? 
 
Yeah. 

600
00:36:25,840 --> 00:36:28,560
So if you if you talk about 
 
representing the geometry, what 

601
00:36:28,560 --> 00:36:30,600
you correctly said, you can do 

two things, right. 

602
00:36:30,600 --> 00:36:33,120
You can treat each point of the 
 geometry individually. 

603
00:36:33,440 --> 00:36:37,942
It turns out this, this 
DrivAerML 
 data set and so on 

604
00:36:37,942 --> 00:36:40,290
so forth. 
The information is so rich. 
 

605
00:36:40,298 --> 00:36:43,520
So in order to really represent 
the geometry, you need roughly, 

606
00:36:43,520 --> 00:36:45,495
 I don't know, half a million 
points. 
 

607
00:36:45,503 --> 00:36:49,080
So it makes sense to pool them 
before, to aggregate them 
 

608
00:36:49,088 --> 00:36:52,980
locally, and then to use those 
as, as, as tokens. 
 

609
00:36:52,988 --> 00:36:56,502
That's just computationally much
more efficient to have some sort

610
00:36:56,502 --> 00:36:59,788

 of pooling that the number of 
tokens is not exploding. 
 

611
00:36:59,796 --> 00:37:04,392
You could also obviously use 
representatives of the geometry 

612
00:37:04,392 --> 00:37:09,720
 as your points, but it makes 
sense to pool them before. 
 

613
00:37:09,728 --> 00:37:12,272
And the same in the volume, 
right? 
 

614
00:37:12,280 --> 00:37:17,300
You're essentially, instead of 
taking 130 million nodes, you're

615
00:37:17,300 --> 00:37:21,900

 breaking them into patches 
which themselves take into. 
 

616
00:37:21,908 --> 00:37:25,760
Yeah, the The funny thing is so 
so there is this tool. 
 

617
00:37:25,768 --> 00:37:29,388
So we always have the the number
of points which it takes to 
 

618
00:37:29,396 --> 00:37:33,654
represent physics correctly. 
This is 2 slightly different 
 

619
00:37:33,662 --> 00:37:35,288
things. 
So for the geometry we really 
 

620
00:37:35,296 --> 00:37:38,300
need a lot of tokens in order to
really capture the geometry 
 

621
00:37:38,308 --> 00:37:40,535
because the geometry is 
influencing the physics and so 


622
00:37:40,543 --> 00:37:43,640
on and so forth. 
For the volume we don't need so 

623
00:37:43,640 --> 00:37:46,562
 many points. 
So we we only need roughly 
 

624
00:37:46,570 --> 00:37:50,348
128,000 or or even less to 
really capture the physics. 
 

625
00:37:50,356 --> 00:37:54,436
So that's we are totally fine to
pick those points individually 


626
00:37:54,444 --> 00:37:59,035
and actually performance is not 
degrading much if we go to 32 or

627
00:37:59,035 --> 00:38:02,255

 64,000 that will get more 
interesting if we get even 
 

628
00:38:02,263 --> 00:38:04,855
larger data sets. 
But then you can also use some 


629
00:38:04,863 --> 00:38:06,720
tricks. 
So there it's obviously fully 
 

630
00:38:06,728 --> 00:38:15,048
fine to use individual points. 
So in that context, is it fair? 

631
00:38:15,048 --> 00:38:19,440
 
A mental model is that you're 

632
00:38:19,440 --> 00:38:23,107
saying that the number of points

 that are needed for a 

633
00:38:23,107 --> 00:38:27,800
numerical solver should not be a
one to 
 one mapping. 

634
00:38:28,720 --> 00:38:32,560
That you shouldn't think just 
 
because I need half a billion 

635
00:38:32,560 --> 00:38:36,240
points to solve my PDE, that I 

should. 

636
00:38:36,680 --> 00:38:40,480
I should need half a billion 
 
points for my mission, my 

637
00:38:40,480 --> 00:38:43,240
machine learning model to learn 
 the behaviour that we need to 

638
00:38:43,240 --> 00:38:49,240
break from this concept of 
 
almost one to one mapping. 

639
00:38:49,240 --> 00:38:53,680
It's that part of your argument,

 because that's not most people

640
00:38:53,680 --> 00:38:57,152
that I think assumed in the GNN 
 that I need to have this sort 

641
00:38:57,152 --> 00:39:01,036
of 1 to one mapping that I'm, 
I'm 
 sort of taking my notes 

642
00:39:01,036 --> 00:39:04,308
and I'm putting that into my ML 
and 
 that's the most logical 

643
00:39:04,308 --> 00:39:07,360
way of learning the problem. 
 
Yeah. 

644
00:39:07,360 --> 00:39:10,040
So I, I really like those 
 
questions, I have to say. 

645
00:39:10,040 --> 00:39:13,438
So I would my argument is, and I

 mean you tell me you're the 

646
00:39:13,438 --> 00:39:18,370
CFD expert, but I would say what
a 
 model has to be able to do 

647
00:39:18,370 --> 00:39:20,800
is it has to produce the 
simulation 
 mesh. 

648
00:39:20,800 --> 00:39:23,840
So it has to give you 
 
predictions on the simulation 

649
00:39:24,000 --> 00:39:27,080
mesh both on the on the surface 
 and in the volume. 

650
00:39:27,400 --> 00:39:30,647
And this is not only cars, it 
 
should hold for airplanes and so

651
00:39:30,647 --> 00:39:33,120
on and so forth. 
 
And if we, if we think of meshes

652
00:39:33,120 --> 00:39:37,720
which approach a billion of mesh

 points and where this is #1 

653
00:39:38,600 --> 00:39:41,694
condition, we cannot, we cannot 
 use parallelisation here 

654
00:39:41,694 --> 00:39:45,197
because then we end up using 
thousand 
 GPUs and this is not 

655
00:39:45,197 --> 00:39:48,080
scaling very well. 
 
I'm looking at the actual 

656
00:39:48,080 --> 00:39:52,200
information content. 
 
So it has to be the model has to

657
00:39:52,200 --> 00:39:56,560
be able to produce this this 
 
rich outputs in order to be a 

658
00:39:56,880 --> 00:40:02,640
good tool to use. 
 
But on the other hand you don't 

659
00:40:02,640 --> 00:40:04,840
need this huge output for 
 
training. 

660
00:40:04,840 --> 00:40:07,720
You can train on much much 
 
smaller sub sampled version of 

661
00:40:07,720 --> 00:40:10,240
the same problem. 
 
However, you have to make sure 

662
00:40:10,360 --> 00:40:13,825
with neural operator learning 
with 
 function approximation 

663
00:40:13,825 --> 00:40:17,411
that those models you trained 
are 
 then in inference or in 

664
00:40:17,411 --> 00:40:20,160
test time. 
 
Able to give you the full answer

665
00:40:20,160 --> 00:40:22,960
on the full simulation mesh. 
 
I think this is the true CFD 

666
00:40:22,960 --> 00:40:27,400
problem or the true learning 
 
problem similar to the ERA5 

667
00:40:27,400 --> 00:40:30,960
setup in weather modelling which

 we have to do in CFD. 

668
00:40:32,800 --> 00:40:37,720
So you yeah, so this is an 
 
interesting point that I it's 

669
00:40:37,720 --> 00:40:40,680
good to maybe briefly discuss 
 
which is at inference time. 

670
00:40:41,000 --> 00:40:45,000
So we discussed about training 

time, you know what you take, 

671
00:40:45,000 --> 00:40:47,960
how many of the points, how many

 do you agglomerate, you know, 

672
00:40:47,960 --> 00:40:50,360
etcetera, etcetera. 
 
But I guess what you're getting 

673
00:40:50,360 --> 00:40:57,240
to now is the inference time, 
 
which is something that maybe is

674
00:40:57,240 --> 00:41:00,040
not so obvious to some people, 

but for me was always a bit of a

675
00:41:00,040 --> 00:41:06,200
conceptual challenge, which is 

how do you generate the mesh to 

676
00:41:06,200 --> 00:41:12,480
do the inference on. 
 
So normally if you are running a

677
00:41:12,480 --> 00:41:15,480
normal CFD problem or you're 
 
taking the DrivAerML data set, 

678
00:41:15,480 --> 00:41:20,680
you split it into a train and 
 
test, but both of them already 

679
00:41:20,680 --> 00:41:23,560
have a mesh generated. 
 
You know, we're giving it you. 

680
00:41:24,200 --> 00:41:27,080
So you already have the mesh. 
 
But the real life engineering 

681
00:41:27,080 --> 00:41:30,720
problem is I want to predict a 

new geometry that I've never 

682
00:41:30,720 --> 00:41:33,600
seen before. 
 
And the question is, do I have 

683
00:41:33,600 --> 00:41:37,880
to mesh it like I would mesh a 
CFD problem or can I just take 

684
00:41:37,880 --> 00:41:42,240
the CAD or can I just do some 
 
arbitrary case? 

685
00:41:43,600 --> 00:41:46,850
Am I understanding it right that

 this is sort of your argument 

686
00:41:46,850 --> 00:41:52,106
of the neural operator or the 
field 
 sort of base approach 

687
00:41:52,106 --> 00:41:58,770
That if I take a new geometry of
a new car 
 that I haven't done 

688
00:41:58,770 --> 00:42:06,767
anything before on and I mesh it
5 
 different ways and then ask 

689
00:42:06,767 --> 00:42:11,280
your model to give me an A 
 
prediction, it should 

690
00:42:11,280 --> 00:42:16,200
essentially give pretty much the

 same answer for those five 

691
00:42:16,200 --> 00:42:21,240
different ways of meshing it. 
 
Yes, I think this is the true. 

692
00:42:22,320 --> 00:42:25,520
To an end, to a point. 
 
I mean, two things. 

693
00:42:26,160 --> 00:42:30,000
One, one thing I haven't said 
 
so, so I I was talking about 

694
00:42:30,520 --> 00:42:32,800
the, the difference in in 
 
training and inference mesh. 

695
00:42:33,200 --> 00:42:36,840
I also should say for for us 
 
it's very important that the 

696
00:42:36,840 --> 00:42:40,640
algorithm we are developing 
 
works for for CAD geometry. 

697
00:42:40,640 --> 00:42:45,462
So if you input inference 
 only
the CAD geometry, you're able to

698
00:42:45,462 --> 00:42:49,920
obtain the full surface 
 and 
full volume without giving any 

699
00:42:49,920 --> 00:42:53,160
information of the surface 
 and
volume mesh in inference. 

700
00:42:53,320 --> 00:42:56,360
The only thing the model gets 
 
the CAD input. 

701
00:42:57,320 --> 00:43:01,240
Obviously the performance 
 
slightly degrades, but I'm 

702
00:43:01,240 --> 00:43:05,560
amazed how well that works. 
 
And you can do that by using, 

703
00:43:05,560 --> 00:43:08,160
obviously in training that the 

simulation mesh and, and, and 

704
00:43:08,160 --> 00:43:11,440
using some tricks on the 
 
simulation mesh that because 

705
00:43:11,440 --> 00:43:13,480
obviously you need this 
 
information in some way in 

706
00:43:13,480 --> 00:43:16,720
training. 
 
But that you can tell your model

707
00:43:17,360 --> 00:43:21,577
how to work with, with basically

 structured points in the in 

708
00:43:21,577 --> 00:43:24,728
the volume and, and certain 
points 
 on the on the surface 

709
00:43:24,728 --> 00:43:27,702
such that inference it's the 
model is 
 able to deal without 

710
00:43:27,702 --> 00:43:31,640
any of those meshings. 
 
And I think this is somehow the 

711
00:43:31,640 --> 00:43:35,280
true power of, of these deep 
 
learning models because they 

712
00:43:35,280 --> 00:43:39,160
cannot only scale. 
 
You can, you can think of that 

713
00:43:39,160 --> 00:43:41,560
the whole CFD compressed 
 
simulation is suddenly 

714
00:43:41,560 --> 00:43:44,080
compressed to, to neural network

 weights. 

715
00:43:44,160 --> 00:43:48,840
You just have to ask them the, 

the, the model, which region in 

716
00:43:48,840 --> 00:43:52,000
the, in the 3D you want to have 
 the prediction. 

717
00:43:52,080 --> 00:43:56,280
But you can also do that from 
 
pure CAD inputs. 

718
00:43:56,720 --> 00:44:00,560
And so, so no storing, no 
 
whatever, right? 

719
00:44:01,040 --> 00:44:03,600
And yeah, this is truly 
 
exciting, I would say. 

720
00:44:04,920 --> 00:44:08,440
Yeah, I think that's one of the 
 bits that I think is the true 

721
00:44:08,440 --> 00:44:10,520
test. 
 
Because if you have to generate 

722
00:44:10,520 --> 00:44:16,080
a mesh, the real time element of

 the prediction starts to go. 

723
00:44:16,080 --> 00:44:19,240
Because for many people, 
 
generating the mesh, the volume 

724
00:44:19,240 --> 00:44:21,640
of the surface is itself quite a

 challenge. 

725
00:44:22,680 --> 00:44:26,720
And I think most people would 
 
prefer to go straight from a CAD

726
00:44:26,720 --> 00:44:33,588
geometry or, or at least be less

 sensitive, you know, to, to, 

727
00:44:33,588 --> 00:44:36,280
to, to the mesh. 
 
The, the only conceptual 

728
00:44:36,280 --> 00:44:38,000
challenge to this, and I, I 
 
don't know if we've even 

729
00:44:38,000 --> 00:44:41,447
discussed this before, but I'll 
 just throw it out there, which 

730
00:44:41,447 --> 00:44:46,320
I think still is part of the 
 
challenge in people's heads is 

731
00:44:46,840 --> 00:44:50,800
if you do a CFD simulation, you 
 would expect there to be a 

732
00:44:50,800 --> 00:44:52,546
difference with the mesh, right?

 

733
00:44:52,554 --> 00:44:57,440
It it should give a difference 
if I run that DrivAerML with a a

734
00:44:57,440 --> 00:45:00,128

 grid that's half as coarse or 
twice as fine. 
 

735
00:45:00,136 --> 00:45:04,584
I want it to give a difference 
because by having a coarser mesh

736
00:45:04,584 --> 00:45:08,155
I have higher numerical error 
which should affect the 
 flow 

737
00:45:08,155 --> 00:45:12,600
field. 
And if I have a much finer mesh 

738
00:45:12,600 --> 00:45:16,400
 I should see a difference 
because I'm reducing the 
 

739
00:45:16,408 --> 00:45:18,790
numerical error. 
I think where it gets a little 


740
00:45:18,798 --> 00:45:21,324
bit harder with the notion of 
sort of resolution independence,

741
00:45:21,324 --> 00:45:25,474

 is it sort of breaks from that
mindset that, that I should be 


742
00:45:25,482 --> 00:45:28,789
able to give you a mesh that's 
twice as fine or twice as 
 

743
00:45:28,797 --> 00:45:30,682
coarse. 
And yet I get the same answer. 


744
00:45:30,690 --> 00:45:33,682
It it, it sort of messes with 
the CFD mind because you think 


745
00:45:33,690 --> 00:45:37,660
it should be different because I
I'm expecting in my normal PDE 

746
00:45:37,660 --> 00:45:42,069
solver for it to be different 
but it's not remember. 
 

747
00:45:42,077 --> 00:45:44,889
Oh, it's, yeah. 
I mean, actually this was one of

748
00:45:44,889 --> 00:45:48,180

 the reasons how we developed 
this, this anchor tokens, we 
 

749
00:45:48,188 --> 00:45:52,240
call them anchor tokens approach
because this very much is, I 
 

750
00:45:52,248 --> 00:45:54,680
think the neural analogy of what
you're saying. 
 

751
00:45:54,688 --> 00:45:57,700
So what these anchor tokens are 
and basically they are the 
 

752
00:45:57,708 --> 00:46:01,477
reasons how we can scale. 
So we train on a selected set of

753
00:46:01,477 --> 00:46:04,063

 tokens on the surface and on 
the volume. 
 

754
00:46:04,071 --> 00:46:07,972
And those tokens, as we already 
said, they need to capture the 


755
00:46:07,980 --> 00:46:10,935
physics. 
So it's, it needs to be a 
 

756
00:46:10,943 --> 00:46:13,980
certain amount, but they are 
sufficiently well to train with 

757
00:46:13,980 --> 00:46:16,840
 self attention everything on 
Transformers and inference side.

758
00:46:17,520 --> 00:46:19,520
What we do, we, we, we pick 
 
these tokens. 

759
00:46:19,520 --> 00:46:23,240
Those are representative points 
 in the whole 3D volume from 

760
00:46:23,240 --> 00:46:26,320
surface and the volume. 
 
And then basically those tokens 

761
00:46:26,320 --> 00:46:28,440
span the weights of the keys and

 values. 

762
00:46:28,440 --> 00:46:31,160
So in the in the Transformer, 
 
you have three different blocks 

763
00:46:31,160 --> 00:46:35,640
and those span two blocks. 
 
So you do a first pass of the 

764
00:46:35,640 --> 00:46:38,320
model with those tokens and then

 this is fixed. 

765
00:46:38,320 --> 00:46:41,440
So you basically have what 
 
you've built is you've built a 

766
00:46:41,440 --> 00:46:46,440
model which which is is a 
 
representation of of your full 

767
00:46:46,440 --> 00:46:51,280
flow and and then you can ask 
 
for specific points in the 

768
00:46:51,280 --> 00:46:55,080
volume or on the surface to get 
 another value. 

769
00:46:55,080 --> 00:46:57,800
And these points can be 
 
arbitrary points, any point. 

770
00:46:58,080 --> 00:47:01,920
And this point will just run 
 
through the the model with all 

771
00:47:01,920 --> 00:47:06,080
these weights already fixed. 
 
So, so it only like a small part

772
00:47:06,080 --> 00:47:09,080
of the model is adjusted and it 
 will give you an output. 

773
00:47:09,720 --> 00:47:13,680
And obviously the more anchor 
 
points you have, the finer this 

774
00:47:14,120 --> 00:47:17,000
this resolution gets, the better

 this prediction are obviously 

775
00:47:17,000 --> 00:47:18,360
until a certain threshold, 
 
right. 

776
00:47:18,840 --> 00:47:22,480
So if you think of 16,000 points

 in the volume, you can still 

777
00:47:22,480 --> 00:47:26,520
reconstruct the full flow field,

 but it's it's not the full 

778
00:47:26,520 --> 00:47:30,600
accuracy. 
 
If you go to 32,000, 64,000 at 

779
00:47:30,600 --> 00:47:33,560
some point, if the training 
 
allowed it, you will have the 

780
00:47:33,560 --> 00:47:36,800
full field covered. 
 
And then with these, with these 

781
00:47:36,800 --> 00:47:41,160
points, you can get arbitrarily 
fine resolution and similar to 

782
00:47:41,160 --> 00:47:44,480
in CFD and you and you cover all

 the the points in space. 

783
00:47:44,760 --> 00:47:49,400
I think I see this anchor points

 really as as some, some some 

784
00:47:49,400 --> 00:47:52,480
sort of of of this computation 

cells in CFD. 

785
00:47:52,840 --> 00:47:56,040
Obviously the the discrepancy 
 
between number of mesh cells you

786
00:47:56,040 --> 00:47:59,560
need in CFD to get the 
 
simulation to converge towards 

787
00:47:59,640 --> 00:48:02,120
anchor points you need in order 
 to cover the full mesh. 

788
00:48:02,120 --> 00:48:05,480
Is is drastically different? 
 
But it's just because AI is 

789
00:48:05,480 --> 00:48:07,800
drastically different, different

 than the numerical simulation.

790
00:48:09,000 --> 00:48:11,680
But there is this analogy. 
 
And obviously if you if you 

791
00:48:11,680 --> 00:48:14,280
don't use any of these points, 

it's very, very hard to 

792
00:48:14,280 --> 00:48:17,000
construct these dynamics. 
 
Yeah. 

793
00:48:17,640 --> 00:48:20,720
I think conceptually some of the

 interesting challenge of that 

794
00:48:20,720 --> 00:48:29,000
is, correct me if I'm wrong, but

 you, your ground truth is your

795
00:48:29,000 --> 00:48:32,400
train data. 
 
So you feel that if you match 

796
00:48:32,400 --> 00:48:35,440
the train data, that's the best 
 it can be. 

797
00:48:36,360 --> 00:48:40,960
Right. 
 
But the training data itself is 

798
00:48:40,960 --> 00:48:46,760
dependent on the mesh. 
 
And so one of the, I guess 

799
00:48:46,760 --> 00:48:52,760
research topics that people have

 discussed is, well, how about 

800
00:48:52,760 --> 00:48:55,920
multi fidelity? 
 
So you, you know, you have some 

801
00:48:55,920 --> 00:49:02,560
high fidelity data, some low 
 
fidelity data, but how? 

802
00:49:05,640 --> 00:49:08,360
I'm so going off a tangent on 
 
this a little bit, But one of 

803
00:49:08,360 --> 00:49:12,200
the things I always say to 
 
people is if you have a concept 

804
00:49:12,200 --> 00:49:14,520
of like a foundational model 
 
where you're saying, OK, I'm 

805
00:49:14,520 --> 00:49:17,560
going to have a model that 
 
predicts cars. 

806
00:49:18,560 --> 00:49:24,200
I'm more and unless my mind is 

just too static it it's a 

807
00:49:24,200 --> 00:49:29,600
foundational model, but just for

 the mesh design that you've 

808
00:49:29,600 --> 00:49:32,360
picked and the CFD method you've

 picked. 

809
00:49:32,720 --> 00:49:36,007
If I then go and change that CFD

 method and regenerate the 

810
00:49:36,007 --> 00:49:39,493
DrivAerML ML with like a RANS 
data set, 
 I'm going to get a 

811
00:49:39,493 --> 00:49:43,880
different answer. 
 
So it's sort of hard to 

812
00:49:43,880 --> 00:49:49,400
conceptually imagine the ML 
 
being like a foundation model 

813
00:49:49,400 --> 00:49:58,000
because the, the input data 
 
itself is so dependent on the 

814
00:49:58,000 --> 00:50:02,880
settings you have of the CFD. 
 
So ideally it should really be 

815
00:50:02,880 --> 00:50:06,080
like experimental data should be

 the, the, the real ground 

816
00:50:06,080 --> 00:50:08,765
truth. 
But it's, it's sort of a little 

817
00:50:08,765 --> 00:50:11,980
 bit of a philosophical thing 
where, yeah, the ML model is 
 

818
00:50:11,988 --> 00:50:15,464
only as good as the training 
data that it has. 
 

819
00:50:15,472 --> 00:50:19,975
And so if now you've sort of 
solved the ML problem, well, I'm

820
00:50:19,975 --> 00:50:25,572

 not saying you've solved it. 
It's almost the data is the most

821
00:50:25,572 --> 00:50:30,200

 important bit now. 
I couldn't agree more. 
 

822
00:50:30,208 --> 00:50:35,160
So, so I, I, I would say talking
about foundation models for 
 

823
00:50:35,168 --> 00:50:38,700
engineering is very, very 
dangerous because you're kind of

824
00:50:38,700 --> 00:50:42,660

 data agnostic in a way. 
You in the end, what goes into 


825
00:50:42,668 --> 00:50:46,040
these models is a huge amount of
knowledge in this data 
 

826
00:50:46,048 --> 00:50:48,760
generation. 
And as you said, data on 
 

827
00:50:48,768 --> 00:50:51,105
numerics is not the ground 
truth. 
 

828
00:50:51,113 --> 00:50:54,321
It's just one version of, of, of
our reality, right. 
 

829
00:50:54,329 --> 00:50:57,860
And, and machine learning is not
replacing whatever this, this 
 

830
00:50:57,868 --> 00:51:02,715
version of reality, it's giving 
a certain new access to this 
 

831
00:51:02,723 --> 00:51:05,979
reality. 
So I, I think the power of 
 

832
00:51:05,987 --> 00:51:10,215
machine learning comes when you,
when you really pick certain 
 

833
00:51:10,223 --> 00:51:13,104
areas where it's tremendously 
important to have this 
 

834
00:51:13,112 --> 00:51:16,972
surrogates and to have this vast
iteration to have whatever and 


835
00:51:16,980 --> 00:51:20,696
then really think what it takes 
to build those surrogates. 
 

836
00:51:20,704 --> 00:51:25,100
But I think from a modelling 
perspective we are especially in

837
00:51:25,100 --> 00:51:30,155

 this static systems, we are 
pretty far that we can give them

838
00:51:30,155 --> 00:51:33,537

 enough data and high fidelity 
data and so on and so forth. 
 

839
00:51:33,545 --> 00:51:36,756
We can build these systems with 
reasonable errors compared to 
 

840
00:51:36,764 --> 00:51:42,080
the data is trained on. 
The question is then always how 

841
00:51:42,080 --> 00:51:46,042
 to interact with numerics. 
So if depends on what you want, 

842
00:51:46,042 --> 00:51:50,010
 but I think that the power and 
the true transformation comes 
 

843
00:51:50,018 --> 00:51:53,620
when when numerics and CFD 
really go and deep networks 
 

844
00:51:53,628 --> 00:51:57,155
really go hand in hand and not 
as as two different parties. 
 

845
00:51:57,163 --> 00:52:00,855
Maybe I should also say to that 
we were doing this DrivAerML 

846
00:52:00,855 --> 00:52:05,092
data 
 set now for I don't know 
10 months like quite intensively

847
00:52:05,092 --> 00:52:08,130

 trying different things. 
We've reformulated the learning 

848
00:52:08,130 --> 00:52:10,120
 problem ourselves a couple of 
times. 
 

849
00:52:10,128 --> 00:52:12,836
So what we actually want to 
train, what we actually want to 

850
00:52:12,836 --> 00:52:15,659
 test and so on and so forth, 
because it's really, really hard

851
00:52:15,659 --> 00:52:18,880

 to because it's, it's just not
a replacement of the CFD. 
 

852
00:52:18,888 --> 00:52:21,480
Your model is just doing 
something different and you have

853
00:52:21,480 --> 00:52:23,160

 to frame that there's a 
problem. 

854
00:52:23,760 --> 00:52:27,400
And and so that's why I think 
 
these two fields need to come 

855
00:52:27,400 --> 00:52:30,920
closer together that these 
 
questions are answered. 

856
00:52:31,760 --> 00:52:35,160
Yeah, no, I and I was just 
 
looking off to the side because 

857
00:52:35,160 --> 00:52:40,132
I was just double checking if I 
 was correct that the ERA5 data 

858
00:52:40,132 --> 00:52:45,280
set is a combination of 
 actual
satellite data, right 

859
00:52:45,280 --> 00:52:49,080
observations with some sort of 

modelling. 

860
00:52:49,080 --> 00:52:53,280
So it is quite different in and 
 I think that's kind of my point

861
00:52:53,280 --> 00:52:55,760
that the weather stuff has 
 
literally got what the 

862
00:52:55,760 --> 00:52:59,120
satellites are seeing. 
 
Therefore you could argue is the

863
00:52:59,120 --> 00:53:01,720
real thing because it is the 
 
Earth. 

864
00:53:01,720 --> 00:53:06,280
They saw what the weather's 
 
doing on the Earth, whereas in 

865
00:53:06,280 --> 00:53:11,680
the CFD it's, it is a, a just a 
 numerical simulation. 

866
00:53:11,760 --> 00:53:16,720
And, and so that's the bit that 
 I still think is a bit of the 

867
00:53:16,720 --> 00:53:23,680
limitation, but we are so sparse

 when it comes to data. 

868
00:53:24,800 --> 00:53:28,280
This is a bit of a segue into 
 
a next question to you, which 

869
00:53:28,280 --> 00:53:35,591
has long been the argument of 
 
the PINNs people, which is are 

870
00:53:35,591 --> 00:53:40,400
we ever going to have a vast 
 
quantity of data to train sort 

871
00:53:40,400 --> 00:53:44,360
of data-driven models? 
 
Is is that a reasonable 

872
00:53:44,360 --> 00:53:48,880
assumption or is it better to 
 
say, well, actually we have 

873
00:53:48,880 --> 00:53:54,120
pretty well defined equations 
 
and rules and models and is it 

874
00:53:54,120 --> 00:53:58,960
not just about using ML to solve

 them faster? 

875
00:53:59,040 --> 00:54:03,240
Like do we need, if we have such

 a a sparsity of data, do we 

876
00:54:03,240 --> 00:54:08,120
not need to include physics into
the 
 models? 

877
00:54:08,560 --> 00:54:13,160
Where do you sort of lie on that

 conundrum of just generating 

878
00:54:13,160 --> 00:54:17,267
more data and let the models 
learn off the data or do we need

879
00:54:17,267 --> 00:54:21,120
to incorporate more physics into

 these models? 

880
00:54:21,360 --> 00:54:24,040
I'm very pragmatic in this. 
 
I mean this, This was the same 

881
00:54:24,040 --> 00:54:26,000
in weather modelling. 
 
Everyone was talking about 

882
00:54:26,520 --> 00:54:28,400
physics informed, not physics 
 
informed. 

883
00:54:28,600 --> 00:54:32,920
But people only talk for, for, 

for that amount of time until 

884
00:54:32,920 --> 00:54:35,560
they realise how heavy the data 
 loading is and how hard of a 

885
00:54:35,560 --> 00:54:36,920
machine learning problem that 
 
is. 

886
00:54:37,720 --> 00:54:40,744
Because you don't want to talk 

about the physics-informed 

887
00:54:40,744 --> 00:54:43,614
neural network if you have to, 
to check 
 it between loading 

888
00:54:43,614 --> 00:54:46,504
gigabytes of data for one data 
point on the 
 GPU and what to 

889
00:54:46,504 --> 00:54:48,463
do. 
And in order to get this correct

890
00:54:48,463 --> 00:54:52,055

 and in order to get the decent
learning signal and so on and so

891
00:54:52,055 --> 00:54:56,288

 forth. 
And I, I see it the same for 
 

892
00:54:56,296 --> 00:55:00,264
this engineering simulations. 
We first have to be able to 
 

893
00:55:00,272 --> 00:55:02,280
really run simulations. 
At scale. 
 

894
00:55:02,288 --> 00:55:04,747
And I'm not talking about 10,000
cells. 
 

895
00:55:04,755 --> 00:55:08,380
I'm not talking about ShapeNet, 
I'm not talking about even 
 

896
00:55:08,388 --> 00:55:11,455
even the DrivAerML. 
I'm talking about airplane 
 

897
00:55:11,463 --> 00:55:15,484
simulation and I'm talking about
transient simulation and that. 


898
00:55:15,492 --> 00:55:19,240
And we are still far away from 
doing that both from how we 
 

899
00:55:19,248 --> 00:55:21,376
generate, how we store, how we 
train data. 
 

900
00:55:21,384 --> 00:55:25,600
And I think only if we, if we 
are able to really have a 
 

901
00:55:25,608 --> 00:55:28,322
workflow there and know what it 
takes and know where it breaks 


902
00:55:28,330 --> 00:55:31,695
and, and, and where all these 
problems are, we can then start 

903
00:55:31,695 --> 00:55:35,786
 thinking about which type of 
physics to include and and which

904
00:55:35,786 --> 00:55:41,239

 not because, I mean, we, we 
also showed that in our paper, 

905
00:55:41,239 --> 00:55:46,388
it's 
 very easy to build a 
divergence free model for, for 

906
00:55:46,388 --> 00:55:49,710
vorticity, 
 because you can 
bake in the, the divergence free

907
00:55:49,710 --> 00:55:51,654
constraint, 
 their construction
a hard constraint. 
 

908
00:55:51,662 --> 00:55:56,052
We try the PINN loss for the, 
for the mass conservation, which

909
00:55:56,052 --> 00:55:59,360
is 
 much harder to just take 
the degrades performance. 
 

910
00:55:59,368 --> 00:56:02,684
It in principle works. 
And you would need some tricks 


911
00:56:02,692 --> 00:56:07,702
and I would say getting this to 
work is not the amount of time 


912
00:56:07,710 --> 00:56:10,755
which needs to do all the 
scaling. 
 

913
00:56:10,763 --> 00:56:16,040
But I would say that first the 
problems we have to solve first 

914
00:56:16,040 --> 00:56:19,940
 are the really how we interact 
with numerics, how we scale to 


915
00:56:19,948 --> 00:56:23,045
these problems and then how we 
actually get industry ready. 
 

916
00:56:23,053 --> 00:56:27,832
Well, you, you mentioned the 
beginning, the sort of T to T + 

917
00:56:27,832 --> 00:56:30,138
 1 analogy when it came to 
weather. 
 

918
00:56:30,146 --> 00:56:34,085
And clearly the bit that we 
missed out maybe in our 
 

919
00:56:34,093 --> 00:56:36,200
introduction is that the 
engineering, we're not doing 
 

920
00:56:36,208 --> 00:56:39,506
that right. 
You're going from yeah, straight

921
00:56:39,506 --> 00:56:44,245

 to a time average or something
at least, at least in, at least 

922
00:56:44,245 --> 00:56:48,657
 in, well, in in the sort of way
that most of these data sets for

923
00:56:48,657 --> 00:56:50,860

 cars and planes have been to 
date. 
 

924
00:56:50,868 --> 00:56:54,220
Given. 
How would you change the 
 

925
00:56:54,228 --> 00:56:58,857
learning problem or the ML 
architecture if now you had a a 

926
00:56:58,857 --> 00:57:03,258
 full time history and would 
that help, hinder or not make a 

927
00:57:03,258 --> 00:57:06,630
 difference? 
I mean same argument as as we 
 

928
00:57:06,638 --> 00:57:09,658
have in in the spatial. 
If you if you look at the the 
 

929
00:57:09,666 --> 00:57:13,800
neural network which has a few 
megabytes of weights and which 


930
00:57:13,808 --> 00:57:19,222
compresses, I don't know the, 
the GB CFD simulation or 10 or 


931
00:57:19,230 --> 00:57:24,055
20, I don't know how many GB 
actually CFD simulation, you can

932
00:57:24,055 --> 00:57:26,930

 also play the same game over 
time, right? 
 

933
00:57:26,938 --> 00:57:31,688
So neural network is just a 
very, very good way of of 
 

934
00:57:31,696 --> 00:57:36,710
storing this, this information. 
And if you somehow manage to to 

935
00:57:36,710 --> 00:57:40,319
 store the temporal part in, in 
some sort of modelling, which 
 

936
00:57:40,327 --> 00:57:43,716
allows you to give you the the 
temporal answer to a certain 
 

937
00:57:43,724 --> 00:57:46,530
problem, this would be 
tremendously huge gain. 
 

938
00:57:46,538 --> 00:57:50,518
Because suddenly you can really 
simulate stuff in, in, in, in 
 

939
00:57:50,526 --> 00:57:54,390
real time, but also have the the
ability to observe what's going 

940
00:57:54,390 --> 00:57:57,160
 on and, and, and to really 
understand the temporal 
 

941
00:57:57,168 --> 00:58:00,672
dynamics. 
Whereas the training is just 
 

942
00:58:00,680 --> 00:58:04,730
somehow pumping this simulation 
into the neural network weights,

943
00:58:04,730 --> 00:58:07,507

 ideally without storing them. 
But that's that's very, very 
 

944
00:58:07,515 --> 00:58:08,852
hard. 
And then probably the next to, 


945
00:58:08,860 --> 00:58:14,982
next to next to step. 
But yeah, I think first step is 

946
00:58:14,982 --> 00:58:19,315
 to get to get the decent neural
network architectures for 
 

947
00:58:19,323 --> 00:58:22,374
temporal modelling, which we are
actually working on to be 
 

948
00:58:22,382 --> 00:58:23,520
honest. 
OK. 
 

949
00:58:23,528 --> 00:58:27,475
And, and would you have the 
sense to be something similar 
 

950
00:58:27,483 --> 00:58:32,700
where you know, you may want to 
have a million time steps, but 


951
00:58:32,708 --> 00:58:37,221
that you'd expect that there 
would be some way of, I don't 
 

952
00:58:37,229 --> 00:58:46,172
know, is there, is there a sort 
of equivalency in terms of at 

953
00:58:46,172 --> 00:58:51,174
inference? 
Ideally you wouldn't need to go 

954
00:58:51,174 --> 00:58:56,778
 1,000,000 steps, right? 
Do you have a sense of how much 

955
00:58:56,778 --> 00:58:59,988
 this is more question for 
people who are generating data, 

956
00:58:59,988 --> 00:59:02,816
You 
 know, do I need to 
generate a million time steps of

957
00:59:02,816 --> 00:59:04,680
data so 
 that you can learn the
million? 

958
00:59:04,920 --> 00:59:08,720
And do you expect that then you 
 would be able to predict to the

959
00:59:08,720 --> 00:59:11,400
millionth but skip out every 
 
hundred? 

960
00:59:11,560 --> 00:59:14,680
Because you, you don't need to, 
 you know, to do that? 

961
00:59:16,360 --> 00:59:20,000
Do you think you could learn the

 temporal problem without every

962
00:59:20,000 --> 00:59:23,080
single time step from the 
 
traditional PDE? 

963
00:59:23,080 --> 00:59:24,640
Do you know what I'm trying to 

get a sense of? 

964
00:59:24,640 --> 00:59:27,720
Like the how the temporal 
 
challenge would link to the 

965
00:59:27,720 --> 00:59:31,760
spatial challenge? 
 
I think it's it's very much 

966
00:59:31,760 --> 00:59:33,640
relatable. 
 
So in in space we see this 

967
00:59:33,640 --> 00:59:37,880
phenomenon that you don't need 

all the points to to to capture 

968
00:59:37,880 --> 00:59:42,000
the physics and then depend. 
 
Then you have a huge advantage 

969
00:59:42,000 --> 00:59:45,520
in learning because you can 
 
always sub sample different sets

970
00:59:45,520 --> 00:59:47,840
of these points which give you a

 huge data augmentation. 

971
00:59:48,160 --> 00:59:49,760
And you have the same in a 
 
temporal domain. 

972
00:59:49,760 --> 00:59:53,840
You don't need each time step in

 order to to present the full 

973
00:59:53,840 --> 00:59:57,360
physics to the neural network. 

But obviously the the fine 

974
00:59:57,360 --> 01:00:00,720
grain, the more the more fine 
 
graining you have in your in 

975
01:00:00,720 --> 01:00:04,520
your solution, the better you 
 
can do the data augmentation in 

976
01:00:04,520 --> 01:00:07,840
the temporal dimension. 
 
But obviously machine learning 

977
01:00:07,840 --> 01:00:12,200
models can do much, much bigger 
 time steps than numerical 

978
01:00:12,200 --> 01:00:15,120
models. 
 
And yeah, this is something 

979
01:00:15,120 --> 01:00:16,500
which which is a true advantage.

 

980
01:00:16,508 --> 01:00:19,520
They just don't work the same 
and they don't have stability 
 

981
01:00:19,528 --> 01:00:21,908
issues but also no stability 
guarantees well. 
 

982
01:00:21,916 --> 01:00:27,042
I was going to say, isn't that 
one of the problems, the roll 
 

983
01:00:27,050 --> 01:00:32,480
out problem that, you know, I 
guess if you do inference 
 

984
01:00:32,488 --> 01:00:37,100
inference, inference inference, 
you know, isn't, isn't these 
 

985
01:00:37,108 --> 01:00:40,548
sort of instabilities going to 
build up or like, you know, how 

986
01:00:40,548 --> 01:00:43,744
 do you constrain it? 
Is this where some of the 
 

987
01:00:43,752 --> 01:00:47,815
physics needs to be added? 
Do you think to sort of if you 


988
01:00:47,823 --> 01:00:52,240
need to do 100,000 iterations, 
how much are you going to 
 

989
01:00:52,248 --> 01:00:55,584
guarantee that just a small 
difference in that first is not 

990
01:00:55,584 --> 01:00:58,388
 going to, you know, bifurcate 
into different solutions? 
 

991
01:00:58,396 --> 01:01:01,640
Yeah, I mean, there, there is 2 
to answer. 
 

992
01:01:01,648 --> 01:01:04,974
So that one is obviously you. 
Luckily people do video 
 

993
01:01:04,982 --> 01:01:07,336
generation nowadays. 
So we get a lot and lot of 
 

994
01:01:07,344 --> 01:01:09,648
tricks of video generation, 
which actually really, really 
 

995
01:01:09,656 --> 01:01:13,904
start to make this thing stable.
And one of these, these 
 

996
01:01:13,912 --> 01:01:16,580
definitely modeling paradigms is
this generative model that you 


997
01:01:16,588 --> 01:01:19,564
model the distribution over time
and that you kind of make sure 


998
01:01:19,572 --> 01:01:21,200
the distribution always stays 
the same. 
 

999
01:01:21,208 --> 01:01:24,920
And, and this boils down to a 
bit of, of physics 
 

1000
01:01:24,928 --> 01:01:26,899
understanding, right? 
If you, if you make sure the 
 

1001
01:01:26,907 --> 01:01:30,958
distribution is the same, if you
make sure that that future and 


1002
01:01:30,966 --> 01:01:34,136
past they considered that there 
is attention across the right 
 

1003
01:01:34,144 --> 01:01:36,670
axis and so on and so forth, we 
get stability. 
 

1004
01:01:36,678 --> 01:01:39,930
It's always depends on how you 
define physics. 
 

1005
01:01:39,938 --> 01:01:43,135
For me, it's the information the
model needs and the way you, 
 

1006
01:01:43,143 --> 01:01:46,772
you, you, you interact with 
future and past and so on. 
 

1007
01:01:46,780 --> 01:01:49,960
I truly think that this 
generative modelling is there's 

1008
01:01:49,960 --> 01:01:52,200
 the breakthrough in this, this 
rollout stabilities. 
 

1009
01:01:52,208 --> 01:01:56,546
I remember when I ran first into
this rollout problems, it was I 

1010
01:01:56,546 --> 01:02:00,660
 think 4 years ago and, and, and
basically I, I did this paper 
 

1011
01:02:00,668 --> 01:02:03,594
together with Daniel Worrell and
I texted him and saying, hey, I 

1012
01:02:03,594 --> 01:02:05,312
 don't know, we have to do 
something. 
 

1013
01:02:05,320 --> 01:02:07,942
Our rollouts always exploding. 
And then I have no idea. 
 

1014
01:02:07,950 --> 01:02:10,812
And, and he said, well, yeah, 
there has to be some, some 
 

1015
01:02:10,820 --> 01:02:13,765
tricks in literature and then 
something smart and, and, and 
 

1016
01:02:13,773 --> 01:02:15,330
definitely people have thought 
about it. 
 

1017
01:02:15,338 --> 01:02:17,360
And I was like, yeah, I cannot 
find anything. 
 

1018
01:02:17,368 --> 01:02:20,454
There is like, why is nobody 
having this problem? 
 

1019
01:02:20,462 --> 01:02:24,700
And it's basically just a very 
different problem we have in in 

1020
01:02:24,700 --> 01:02:27,982
 in this engineering task that 
the the input itself each time 


1021
01:02:27,990 --> 01:02:31,380
step is so huge from information
content that every 
 

1022
01:02:31,388 --> 01:02:34,672
autoregressive model, every 
every autoregressive trick you 


1023
01:02:34,680 --> 01:02:39,072
you had before is just not 
working because they work with 


1024
01:02:39,080 --> 01:02:44,082
much smaller input vectors. 
But but now that computer vision

1025
01:02:44,082 --> 01:02:48,165

 is going to videos and that 
people are aware of this problem

1026
01:02:48,165 --> 01:02:51,068

 and that frequency spectrum 
need to be conserved and so on 

1027
01:02:51,068 --> 01:02:55,490
and so 
 forth. 
This gets really a lot of lot 
 

1028
01:02:55,498 --> 01:02:58,460
of progress. 
And I would say that that the 
 

1029
01:02:58,468 --> 01:03:01,200
having the frequency spectrum 
stable is is is what made many 


1030
01:03:01,208 --> 01:03:05,568
of these things stable. 
So it sounds like in a way there

1031
01:03:05,568 --> 01:03:09,829

 is still a lot of potential to
learn from the other advances. 


1032
01:03:09,837 --> 01:03:14,448
Yeah, in ML, you know, the fact 
there's always an analogy 
 

1033
01:03:14,456 --> 01:03:16,263
problem I guess is what you're 
saying. 
 

1034
01:03:16,271 --> 01:03:20,280
There's the stuff you can learn 
in terms of video generation, 
 

1035
01:03:20,288 --> 01:03:22,500
text generation, weather, 
climate week. 
 

1036
01:03:22,508 --> 01:03:26,795
You can borrow ideas from other 
fields, but it suggests that you

1037
01:03:26,795 --> 01:03:31,076

 therefore need to have people 
with a a broader ML background. 

1038
01:03:31,076 --> 01:03:34,520
 
You know, if you just have a 

1039
01:03:34,520 --> 01:03:39,520
fluids background, you may 
 
struggle to solve this problem 

1040
01:03:39,520 --> 01:03:41,880
because you need to know what or

 you need to have an 

1041
01:03:41,880 --> 01:03:44,400
understanding and an interest in

 looking at what's video 

1042
01:03:44,520 --> 01:03:46,320
generation doing or what's this 
 doing. 

1043
01:03:46,640 --> 01:03:48,360
Would that be fair to say? 
 
This is why it's a 

1044
01:03:48,360 --> 01:03:52,160
multidisciplinary problem. 
 
Almost all you need a team that 

1045
01:03:52,160 --> 01:03:54,880
is multidisciplinary. 
 
I mean that that's happening in 

1046
01:03:54,880 --> 01:03:57,240
biotech and in other areas as 
 
well, right? 

1047
01:03:58,240 --> 01:04:02,200
We are just very in agnostic to 
 that and language and vision 

1048
01:04:02,240 --> 01:04:06,160
because you don't need a 
 
linguist to build the LLM and 

1049
01:04:06,160 --> 01:04:10,120
you don't need basically, I 
 
don't know even what the 

1050
01:04:10,160 --> 01:04:13,440
computer vision like. 
 
What's that analogy there? 

1051
01:04:13,800 --> 01:04:18,400
But for, for, for, for all this 
 AlphaFold and so on so forth. 

1052
01:04:18,400 --> 01:04:22,920
The surely had domain experts on

 the team and surely this is 

1053
01:04:22,920 --> 01:04:25,880
what, what's the exciting part 

about engineering you, you have 

1054
01:04:25,880 --> 01:04:28,680
problems which are so hard to 
crack that you need the top 

1055
01:04:28,680 --> 01:04:31,160
notch machine learning people 
 
that the top notch domain 

1056
01:04:31,240 --> 01:04:33,400
people, they need to speak the 

same language. 

1057
01:04:33,840 --> 01:04:37,200
And even if you can borrow a lot

 of concepts, they need to be 

1058
01:04:37,200 --> 01:04:39,640
heavily adjusted. 
 
Because the problem with this 

1059
01:04:39,640 --> 01:04:42,840
large mesh is with what's the 
 
input, what's the output? 

1060
01:04:42,840 --> 01:04:45,720
There's this huge ratio 
 
difference and, and, and, and 

1061
01:04:45,720 --> 01:04:49,160
turbulence and whatnot. 
 
This is just a, a totally 

1062
01:04:49,160 --> 01:04:52,360
different piece to, to crack. 
 
So that's what's the exciting 

1063
01:04:52,360 --> 01:04:55,600
part here. 
 
I, I wanted to maybe pivot it a 

1064
01:04:55,600 --> 01:05:01,480
little bit to the process you 
 
went in creating a start up. 

1065
01:05:01,920 --> 01:05:05,320
I, I think a lot of people would

 be interested to know and I'm 

1066
01:05:05,320 --> 01:05:09,400
sure other people, you know, are

 in a pub or, you know, alcohol

1067
01:05:09,400 --> 01:05:11,200
doesn't need to be involved, of 
 course, but you know, there's 

1068
01:05:11,200 --> 01:05:15,440
somewhere and they've got ideas 
 and, but not many people 

1069
01:05:15,440 --> 01:05:19,760
actually go out secure funding 

and, you know, create a start 

1070
01:05:19,760 --> 01:05:22,960
up. 
 
So obviously not giving away any

1071
01:05:22,960 --> 01:05:25,840
sensitive details, but what was 
 that experience like for you? 

1072
01:05:25,840 --> 01:05:29,440
Like how did it begin? 
 
How did you deal with venture 

1073
01:05:29,440 --> 01:05:31,115
capital companies, how to like? 
 

1074
01:05:31,123 --> 01:05:35,260
Yeah, I'd be interested if you 
could share some of your 
 

1075
01:05:35,268 --> 01:05:39,115
experience in founding a start 
up, some of the lessons learnt 


1076
01:05:39,123 --> 01:05:41,454
maybe. 
Yeah, so I mean, I, I was really

1077
01:05:41,454 --> 01:05:43,650

 lucky. 
So, so first of all, a lot of 
 

1078
01:05:43,658 --> 01:05:45,148
knowledge comes up from my 
university. 
 

1079
01:05:45,156 --> 01:05:49,120
I, I have to know, I have to say
that the people like Benedikt 

1080
01:05:49,120 --> 01:05:52,026
Alkin and and Tobias Kronlachner
who are really, really pushing 


1081
01:05:52,034 --> 01:05:56,150
the efforts, they have been with
me from, from day one actually. 

1082
01:05:56,150 --> 01:05:58,240
 
Also people like Stefan Pirker 

1083
01:05:58,240 --> 01:06:02,400
here in Linz University is a 
 
well known simulation expert who

1084
01:06:02,400 --> 01:06:06,360
who knows all these tricks. 
 
When I met him the first day, 

1085
01:06:06,360 --> 01:06:09,040
basically after returning to 
 
Austria, we were 5 minutes and 

1086
01:06:09,040 --> 01:06:11,720
we were both like super excited 
 that, that we do things in this

1087
01:06:11,720 --> 01:06:16,240
area. 
 
And yeah, we, there was a lot of

1088
01:06:16,240 --> 01:06:18,840
momentum there. 
 
We were at a different company 

1089
01:06:18,840 --> 01:06:22,720
and NXAI where at some point it 
 was clear that the simulation 

1090
01:06:22,720 --> 01:06:26,880
team, which was was really 
 
working well, should branch off.

1091
01:06:27,720 --> 01:06:31,640
I was also lucky to have met 
 
people like Dennis Just and Miks

1092
01:06:31,640 --> 01:06:35,400
Mikelsons, who did this 
 
before, created startups, very 

1093
01:06:35,400 --> 01:06:37,600
successful startups and actually

 knew what to do. 

1094
01:06:38,040 --> 01:06:41,480
They were the first to listen to

 me when I told them, hey, this

1095
01:06:41,480 --> 01:06:43,600
engineering like scaling these 

things up. 

1096
01:06:43,600 --> 01:06:46,680
What we do to this billion of 
 
mesh problems is actually what 

1097
01:06:47,080 --> 01:06:50,560
what industry needs. 
 
But just give us some time. 

1098
01:06:50,560 --> 01:06:54,320
We will figure that out. 
 
And so I think it was all very 

1099
01:06:54,320 --> 01:06:56,800
natural. 
 
It was not planned, but it's 

1100
01:06:56,800 --> 01:07:00,920
obviously a huge excitement if 

you can, if you can do what 

1101
01:07:00,920 --> 01:07:03,600
you're really burning for in, 
 
in, in also in a, in a 

1102
01:07:03,600 --> 01:07:06,640
commercial set up where you you 
 really can make an impact with 

1103
01:07:06,640 --> 01:07:10,280
what you're doing. 
 
So yeah, I would say in that 

1104
01:07:10,280 --> 01:07:13,200
sense I was very lucky. 
 
And I would also say that 

1105
01:07:13,240 --> 01:07:16,680
especially in this area, we, we 
 see a lot of momentum generated

1106
01:07:16,680 --> 01:07:21,520
and we're not the last start up 
 and there is hopefully coming 

1107
01:07:21,520 --> 01:07:25,080
more it, it, it there is a 
 
momentum especially in Europe 

1108
01:07:25,080 --> 01:07:28,080
now in in this area. 
 
And yeah, just very exciting. 

1109
01:07:28,080 --> 01:07:30,120
I would do it exactly the same 

way again. 

1110
01:07:31,400 --> 01:07:36,480
And so how do you think you 
 
could have done this any other 

1111
01:07:36,480 --> 01:07:38,680
way? 
 
Do you think you could have come

1112
01:07:38,680 --> 01:07:42,440
up with what you're coming up at

 a company or at the university

1113
01:07:42,800 --> 01:07:49,680
or is it is this is the start up

 really the only way of I asked

1114
01:07:49,680 --> 01:07:52,240
this question because I asked 
 
the same thing to Max Welling 

1115
01:07:52,640 --> 01:07:57,160
and he was sort of saying, well,

 yeah, start-ups is the only 

1116
01:07:57,160 --> 01:08:00,640
place where this can happen. 
 
The university's either too 

1117
01:08:00,640 --> 01:08:03,240
small and doesn't have enough 
 
funding or companies too big and

1118
01:08:03,240 --> 01:08:05,080
too slow. 
 
And the start up is this sort 

1119
01:08:05,080 --> 01:08:08,440
of, but it feels like in Europe,

 but often a bit risk averse. 

1120
01:08:09,600 --> 01:08:12,320
We almost feel like start-ups is

 for somebody else. 

1121
01:08:12,520 --> 01:08:15,320
But I'd be interested to know 
 
your thoughts on. 

1122
01:08:15,520 --> 01:08:18,920
That I'm not so sure if what 
 
we're pulling off. 

1123
01:08:19,240 --> 01:08:22,640
We just need the, the, the best 
 people and, and, and, and some 

1124
01:08:22,640 --> 01:08:24,920
compute. 
 
One thing which is really, 

1125
01:08:25,080 --> 01:08:27,760
really, really, really hard is 

the problems. 

1126
01:08:28,160 --> 01:08:32,524
So it's actually we are choosing

 our customer mostly by the, 

1127
01:08:32,524 --> 01:08:37,240
the, the, the scale and size of 
the 
 problems they have because

1128
01:08:37,399 --> 01:08:40,760
they're really interesting 
 
problems come from industry and 

1129
01:08:40,760 --> 01:08:45,720
the more challenging and those 

problems are, the better they 

1130
01:08:45,720 --> 01:08:47,840
are for us and, and for 
 
developing us. 

1131
01:08:47,840 --> 01:08:51,560
So obviously we now have have a 
 customer which have huge 

1132
01:08:51,560 --> 01:08:54,319
temporal problems where you 
 
have, we have different 

1133
01:08:54,319 --> 01:08:58,040
modalities interacting and this 
 can bootstrap our temporal 

1134
01:08:58,040 --> 01:09:02,880
modelling capabilities and and 

and and and and and and, and, 

1135
01:09:02,920 --> 01:09:06,920
and this is something which you 
 wouldn't get in the industry I 

1136
01:09:06,920 --> 01:09:10,080
at university. 
 
So I would say it's a mixture. 

1137
01:09:10,080 --> 01:09:13,920
I would say for me, it's, it's, 
 it really helps to get hands on

1138
01:09:13,920 --> 01:09:19,960
to the, to the real problems 
 
and, and on, on the other side, 

1139
01:09:19,960 --> 01:09:23,359
it's, it's just helps me to get 
 into touch what industry really

1140
01:09:23,359 --> 01:09:25,800
needs and, and not sit in my 
 
ivory tower. 

1141
01:09:26,200 --> 01:09:29,240
I wouldn't, I couldn't stand 
 
doing things where I know people

1142
01:09:29,240 --> 01:09:34,120
wouldn't use them. 
 
So this is a good, yeah, good 

1143
01:09:34,120 --> 01:09:36,240
mix. 
 
But I also have to say there is 

1144
01:09:36,240 --> 01:09:39,760
luckily some people like Neil 
 
Ashton who generate publicly 

1145
01:09:39,760 --> 01:09:43,279
available data sets that that 
 
people can really start looking 

1146
01:09:43,279 --> 01:09:47,009
at that We just have to give the

 community the urgency that 

1147
01:09:47,009 --> 01:09:50,437
that they really look at the 
hard 
 data sets and not a 

1148
01:09:50,437 --> 01:09:53,720
ShapeNet car. 
 
Yeah, it, it seems like the 

1149
01:09:55,240 --> 01:09:59,880
there is definitely a movement 

in the industry towards this 

1150
01:09:59,880 --> 01:10:02,240
machine learning problems. 
 
You know, every time I go to a 

1151
01:10:02,240 --> 01:10:04,920
conference, there's more and 
 
more people doing it, but it 

1152
01:10:04,920 --> 01:10:10,320
still feels like we haven't 
 
reached that inflection point. 

1153
01:10:10,320 --> 01:10:16,800
It still feels like you're 
 
either a start up or you're, I 

1154
01:10:16,800 --> 01:10:19,120
don't know. 
 
It still feels like we're not 

1155
01:10:19,120 --> 01:10:25,640
fully at that yeah, inflection 

point in terms of mass adoption 

1156
01:10:25,640 --> 01:10:27,200
of this. 
 
I feel like we're getting 

1157
01:10:27,200 --> 01:10:28,920
closer. 
 
I'm sure you feel the same. 

1158
01:10:28,920 --> 01:10:35,120
You know, lots of industries 
 
interested, but maybe it is the 

1159
01:10:35,120 --> 01:10:39,240
data that's the issue. 
 
Maybe it's the ERA5 was such a 

1160
01:10:39,240 --> 01:10:43,440
large open data set that data 
 
wasn't the issue and therefore 

1161
01:10:43,440 --> 01:10:44,880
people just got into the 
 
modelling. 

1162
01:10:45,240 --> 01:10:50,320
Whereas I feel like now there's 
 been quite a lot of advances in

1163
01:10:50,320 --> 01:10:52,720
let's say road-car external 
 
aerodynamics. 

1164
01:10:53,600 --> 01:10:56,480
But road-car external 
 
aerodynamics, I don't know 

1165
01:10:56,600 --> 01:11:00,040
percentage wise, but it's it's a

 small percentage of the entire

1166
01:11:00,040 --> 01:11:05,520
CFD domain. 
 
And yeah, yeah, it feels like we

1167
01:11:05,520 --> 01:11:09,480
still need to convince maybe the

 whole CFD community to do 

1168
01:11:09,480 --> 01:11:15,414
more, to really advance it and 
make 
 this a more systematic, I

1169
01:11:15,414 --> 01:11:17,688
guess. 
Would that be true to say that 


1170
01:11:17,696 --> 01:11:20,092
you can only work on the 
problems that you updated for? 


1171
01:11:20,100 --> 01:11:22,582
I mean, it sounds like an 
obvious statement to make, but I

1172
01:11:22,582 --> 01:11:25,376

 assume if you had 10 times 
more data, you could potentially

1173
01:11:25,376 --> 01:11:28,040
go 
 and hire more people and 
work on more problems. 
 

1174
01:11:28,048 --> 01:11:31,666
But you're not going to go and 
hire more people and work on 
 

1175
01:11:31,674 --> 01:11:36,250
other problems if there isn't 
the data or the willingness to 


1176
01:11:36,258 --> 01:11:38,682
do this. 
And, and most people also don't 

1177
01:11:38,682 --> 01:11:40,450
 know what to, to optimise for, 
right. 
 

1178
01:11:40,458 --> 01:11:44,022
So, so I think the, the, the 
industry or the, the, the real 


1179
01:11:44,030 --> 01:11:47,057
problems need to need to set 
the, the, the, the, the, the 
 

1180
01:11:47,065 --> 01:11:49,460
stage they need to set the, the 
machine learning problem. 
 

1181
01:11:49,468 --> 01:11:52,572
Then we can start to think of 
how to solve it. 
 

1182
01:11:52,580 --> 01:11:55,824
Yeah, I, I, but there is a 
momentum. 
 

1183
01:11:55,832 --> 01:11:59,170
People are still reluctant 
because getting a NeurIPS paper 

1184
01:11:59,170 --> 01:12:02,920
 easier on a smaller data set 
than on a larger data set. 
 

1185
01:12:02,928 --> 01:12:07,295
But but I think there is quite 
some movement and and and and 
 

1186
01:12:07,303 --> 01:12:11,834
things are are really changing. 
And I also from for me, it's, 
 

1187
01:12:11,842 --> 01:12:15,458
it's kind of our philosophy that
to be a bit open source to show 

1188
01:12:15,458 --> 01:12:18,550
 our models to show what we're 
doing to publish, because I 
 

1189
01:12:18,558 --> 01:12:22,984
think this is this is the way to
go forward. 
 

1190
01:12:22,992 --> 01:12:29,023
Closed, closed doors policy is 
is is not, not what what what 
 

1191
01:12:29,031 --> 01:12:32,458
gets progress. 
I mean, we we see that the whole

1192
01:12:32,458 --> 01:12:35,240

 progress in LLMs due to open 
source and then and so on and so

1193
01:12:35,240 --> 01:12:38,092

 forth. 
And I think this is also what 
 

1194
01:12:38,100 --> 01:12:42,634
will, not what should, but what 
will happen in in this space. 
 

1195
01:12:42,642 --> 01:12:46,828
Yeah, I must say that I do 
commend that you're releasing 
 

1196
01:12:46,836 --> 01:12:51,754
the data and the models. 
Open source is is a novelty. 
 

1197
01:12:51,762 --> 01:12:57,165
And I think, yeah, it's good to 
see because I like particularly 

1198
01:12:57,165 --> 01:13:00,656
 your paper now. 
I mean, I think that has hurt a 

1199
01:13:00,656 --> 01:13:03,925
 little bit of trust in the CFD 
world so far. 
 

1200
01:13:03,933 --> 01:13:07,560
Instead, as you know, the 
tradition and I'm don't sure 
 

1201
01:13:07,568 --> 01:13:11,304
where the tradition came from. 
If you develop a turbulence 

1202
01:13:11,304 --> 01:13:15,060
model 
 or you develop a 
numerical method, whatever it 

1203
01:13:15,060 --> 01:13:17,760
is, it's 
 almost always 
published. 

1204
01:13:18,560 --> 01:13:23,040
It's very rare actually for a 
 
model to make itself into a 

1205
01:13:23,040 --> 01:13:26,920
large commercial, even, you 
 
know, ISV CFD code and not to 

1206
01:13:26,920 --> 01:13:28,280
have a corresponding 
 
publication. 

1207
01:13:28,800 --> 01:13:34,400
It's sort of seen as like a 
 
thing that you don't make money 

1208
01:13:34,440 --> 01:13:37,360
on the method, you make money on

 the code. 

1209
01:13:38,320 --> 01:13:43,840
That's sort of been the like 
 
thing like OpenFOAM or STAR-CCM+

1210
01:13:43,840 --> 01:13:47,480
or Ansys Fluent to others, 
 
typically the models themselves.

1211
01:13:48,080 --> 01:13:51,840
You sort of know it's more the 

the the little tricks that you 

1212
01:13:51,840 --> 01:13:55,120
might do and but then more the 

support and the code and all 

1213
01:13:55,120 --> 01:13:58,640
that. 
 
Whereas it felt like so far in 

1214
01:13:58,640 --> 01:14:04,880
the ML4 CAE world, with the 
 
exception of a a couple of 

1215
01:14:04,880 --> 01:14:11,000
individuals or or companies, 
 
most of the start-ups have not 

1216
01:14:11,000 --> 01:14:14,760
published or made a thing of 
 
saying, here's our exact method,

1217
01:14:15,600 --> 01:14:18,440
but we've implemented it in such

 a way that, do you know what I

1218
01:14:18,440 --> 01:14:20,000
mean? 
 
And I feel like that's hurt a 

1219
01:14:20,000 --> 01:14:22,680
little bit of trust because it's

 hard for someone to believe 

1220
01:14:23,360 --> 01:14:27,040
results if they don't see a 
 
publication about it. 

1221
01:14:27,040 --> 01:14:30,680
So yeah, I don't know if I'm 
 
alone on this, but I think your 

1222
01:14:30,680 --> 01:14:36,360
strategy of publishing more, I 

think actually helps at least 

1223
01:14:36,360 --> 01:14:40,080
CFD people to feel a little bit 
 more like comfortable. 

1224
01:14:40,600 --> 01:14:43,200
Which I, I, I think it's the, 
 
it's the only way around. 

1225
01:14:43,200 --> 01:14:46,440
And I if I make a bold 
 
statement, if in the machine 

1226
01:14:46,440 --> 01:14:49,320
learning world where things are 
 moving so fast you're afraid 

1227
01:14:49,320 --> 01:14:52,360
that others are are copying your

 stuff and overtaking you, you 

1228
01:14:52,360 --> 01:14:55,840
should probably reconsider your 
 your capabilities as a company.

1229
01:14:57,640 --> 01:14:59,640
Yeah. 
 
And the only thing I would say 

1230
01:14:59,640 --> 01:15:02,240
though, which is why I've asked 
 you sometimes to up level the 

1231
01:15:02,240 --> 01:15:06,360
explanations, is I feel maybe 
 
even the biggest barrier is just

1232
01:15:06,360 --> 01:15:10,542
an understanding barrier that, 

you know, if you're a CFD 

1233
01:15:10,542 --> 01:15:13,894
person, you're probably going to
read an 
 ML paper and be like, 

1234
01:15:13,894 --> 01:15:15,687
I don't really understand this. 
 

1235
01:15:15,695 --> 01:15:19,402
That's almost the challenge. 
You know, it's like an education

1236
01:15:19,402 --> 01:15:22,216

 thing is that the people 
making decisions or higher up 

1237
01:15:22,216 --> 01:15:25,267
may not 
 even understand 
because it's such a different 

1238
01:15:25,267 --> 01:15:28,440
field that that 
 is the 
challenge. 

1239
01:15:28,440 --> 01:15:32,440
So anything that you can do to 

make like, yeah, simpler 

1240
01:15:32,440 --> 01:15:35,720
versions of papers or like 
 
summaries of papers or, or 

1241
01:15:35,720 --> 01:15:39,920
education, which I guess comes 

back to why having a dual 

1242
01:15:39,920 --> 01:15:43,440
affiliation between a university

 and a start up is probably a 

1243
01:15:43,440 --> 01:15:46,920
good thing. 
 
Because I feel that you need to 

1244
01:15:46,920 --> 01:15:51,320
be teaching and producing like 

open source teaching material to

1245
01:15:51,320 --> 01:15:54,960
educate people as much as also 

producing commercial solutions. 

1246
01:15:56,400 --> 01:15:58,640
I don't know if you've noticed 

that when you speak to customers

1247
01:15:58,640 --> 01:16:03,035
that sometimes their knowledge 
of ML is part of the blocker. 
 

1248
01:16:03,043 --> 01:16:06,330
Yeah, but it's also the other 
way around, right? 
 

1249
01:16:06,338 --> 01:16:09,584
There's hardly any people who 
have the knowledge of the real 


1250
01:16:09,592 --> 01:16:13,088
domain expertise and knowledge 
of ML because only if you have 


1251
01:16:13,096 --> 01:16:16,442
the knowledge of domain 
expertise and you know what ML 


1252
01:16:16,450 --> 01:16:19,885
can do, you really find the 
problems where ML can really 
 

1253
01:16:19,893 --> 01:16:23,655
have an impact, right? 
And, and and this goes in both 


1254
01:16:23,663 --> 01:16:25,795
ways. 
So education both ways is super 

1255
01:16:25,795 --> 01:16:27,609
 important and it's very, very 
hard. 
 

1256
01:16:27,617 --> 01:16:31,720
I mean, I, I see that myself all
the time that I can get lost in 

1257
01:16:31,720 --> 01:16:33,648
 details and, and, and think 
things are obvious. 
 

1258
01:16:33,656 --> 01:16:36,415
And on the other hand, when I 
talk to real domain experts I 
 

1259
01:16:36,423 --> 01:16:40,548
get get really lost and have to 
ask questions and yeah. 
 

1260
01:16:40,556 --> 01:16:43,680
Yeah, yeah. 
Well, maybe This is why it still

1261
01:16:43,680 --> 01:16:47,628

 will take a few years to reach
a greater maturity because both 

1262
01:16:47,628 --> 01:16:52,370
 sides need to upskill 
themselves and, and, and sort of

1263
01:16:52,370 --> 01:16:53,720
learn. 
 
So great. 

1264
01:16:53,720 --> 01:16:55,640
Well, really appreciate you 
 
chatting. 

1265
01:16:55,640 --> 01:16:59,880
I mean, what I'm going to do is 
 put some links in at least on 

1266
01:16:59,880 --> 01:17:02,760
the YouTube to, to a couple of 

the papers and the models that 

1267
01:17:02,760 --> 01:17:05,280
you that we've referred to, 
 
because I think people should 

1268
01:17:05,280 --> 01:17:07,680
really spend a bit of time 
 
looking at the papers. 

1269
01:17:07,960 --> 01:17:10,400
I have a feeling that, you know,

 we might need to chat again. 

1270
01:17:10,640 --> 01:17:14,344
I think this field is moving so 
 quickly that everything that 

1271
01:17:14,344 --> 01:17:16,684
we, it'll be interesting to see 
if 
 we listen to this 

1272
01:17:16,684 --> 01:17:19,679
conversation, a year's time, are
we going to 
 be completely 

1273
01:17:19,679 --> 01:17:22,840
wrong or in two years time? 
 
I very much hope so. 

1274
01:17:22,840 --> 01:17:26,720
But what we won't be wrong is 
 
that some things are moving fast

1275
01:17:26,840 --> 01:17:29,480
and that we will have fast 
 
progress in two years. 

1276
01:17:29,480 --> 01:17:32,800
So we will be, oh, we didn't 
 
expect that to happen, so. 

1277
01:17:32,800 --> 01:17:34,560
Yeah. 
 
Yeah, that's going to happen. 

1278
01:17:36,200 --> 01:17:38,400
That's a good thing though. 
 
Maybe that maybe that's OK. 

1279
01:17:38,400 --> 01:17:41,720
But yeah, for now, then, until 

we speak again, thank you very 

1280
01:17:41,720 --> 01:17:45,360
much and yeah, hope people that 
 enjoy looking at your paper and

1281
01:17:45,360 --> 01:17:47,120
your work. 
 
Thank you very much, Neil for 

1282
01:17:47,120 --> 01:17:47,480
having me.