Hi, and welcome to the Neil Ashton Podcast.
In each episode, we explained some of the fascinating ways
that science and engineering are changing the world around us.
We talked to leading engineers from elite level sports like
cycling and Formula One to some of the world's top academics to
understand how fluid dynamics, machine learning and
supercomputing are bringing in a new era of discovery.
We also hear some of their life stories, their career advice,
the lessons they've learned on the way that I hope will be
helpful to you too. So sit back and enjoy this
episode. Hi, and welcome back to the Neil
Ashton Podcast. So on today's episode, we have
Professor Michael Mahoney, who is one of the world's leading
experts on machine learning, mathematics, and computer
science. He's also somebody who I've got
to know over the past year and has been a great help in
understanding this space. He's among other things, an
Amazon scholar. So we've managed to work
together a little bit. And hopefully, as you can tell
from this episode, he's a really nice guy and has a great
sense of humor and such an intelligent person.
Where do I start to talk about what he's done?
Well, he's a professor at UC Berkeley in the Department of
Statistics, but he's also at Lawrence Berkeley National Lab.
And as I mentioned, he's also an Amazon scholar and, and has a
few more hats as well. He, if you look at his Google
Scholar, which for a lot of academics is a way of, you know,
getting a look at what they've done, he has some pretty
impressive statistics. And one that sort of comes to
mind, not only does he have more than 36,000 citations, which,
which is a lot, his h-index is 84, which is very high.
But what's more impressive is, and again, this is probably
going into the details, but it makes a difference.
If you go on Google Scholar and you look at the number of
citations, his is exponentially growing.
So not only does he have more than, you know, 30,000
citations, h-index of 84, which is already in the very high,
sort of top 1% or probably even higher, it is increasing like
this, which basically shows that his research is becoming ever
more relevant every single year.
It's not plateauing or going down.
That's actually very impressive and that shows why he is so
highly regarded because his work is really cutting edge.
There are many, I think it's fair to say that because he has
that mathematics background and that computer science
background, he's put his hand to many different things.
But one thing maybe relevant for the listeners or viewers of this
podcast is a paper he did with some colleagues on
characterizing possible failure modes in physics-informed neural
networks. And this is, you hear many
people talk about PINNs: physics- informed neural
networks. And so they did a great paper a
couple of years ago looking at some places where it may not do
so well. We talk actually about that in
the podcast. We go through some of the themes
I've discussed with other leading ML experts.
And we talk about, for example, his opinion on, you know, do
you really need to include the physics in this AI for science
regime or is it good enough just to use data?
We discussed this at length. And we also get into the topic
of foundational models for science, something that he's
actually really being a leading voice on.
He's given numerous keynotes, important seminars, and
published papers in this area. So we have a good debate about
that. But we actually start off the
conversation talking about something he's also very well
known for, which is randomized numerical linear algebra, which
is quite a complex topic, to be, to be completely honest.
And I don't think we got to the bottom of it in this short
talk, but it really shows that some of his work from the pure
math side or pure math side is coming through and being
relevant as we increasingly look towards lower precision
methods and ways of doing mathematical tricks to try and
improve the speed of many linear solvers, which are common, of
course in many, many fields of science, including CFD.
We also talk about his general opinions or advice for people
looking to move between academia and industry.
Just general career advice, something he's well placed given
that he has an incredible academic record, but he's also
very close to industry and sort of industrial applications as
well. An hour or an hour and a bit is
not really enough to go through all of this, but I, I certainly
learnt a lot and I really enjoyed talking to him like I do
every time I speak to him. And I hope you'll, you'll find
the same thing. I will add some links if you're
watching this on YouTube because again, he's published so many
papers and done so many great talks that I really want you
after watching this, listening to this to go to his website
and and look through all of those.
And obviously if you are watching this on YouTube, you
may prefer to listen to it. Some people don't realise that
these podcasts are in video form and YouTube, but they're also on
Spotify and Apple for audio and vice versa.
If you normally listen to this and you didn't realise there's a
full video version, you can go to YouTube.
So yeah, I hope you enjoy this conversation with Professor
Michael Mahoney. One topic that you mentioned to
me when we were chatting in the past that I was fascinated by,
but if I'm being completely honest, I didn't fully
understand. So it's good for my purpose and
for everybody else was around the randomized linear algebra.
What exactly? For people who don't know what,
what is it and why is this becoming even mentioned on a
Netflix show? So that yeah.
So that's, this was a sign of success when it was finally
mentioned on Netflix—no, the Lincoln Lawyer a couple of years
ago, someone sent it to me and it was mentioned in in one of
the courtroom scenes. So randomized linear algebra,
randomized numerical linear algebra is basically an area
that uses randomness as an algorithmic resource to solve
linear algebra problems. A lot, a lot of linear algebra
problems are under the hood, whether you're doing machine
learning or scientific computing, they manifest
themselves in different ways. So the exact questions and
numerical issues and so on are, are different in those areas.
And that's the source of maybe tension in the area, but also
synergy and, and, and, and, you know, part of the reason a lot
of people are interested in it because it, it holds the
potential to solve a lot of problems people look at, But
it's, it's solving core linear algebra problems and, and linear
algebra ones appear in a lot of places.
If you're solving partial differential equations, it's,
you know, linear operators or iterated forms of linear
operators appear. If you're solving machine
learning, you may be interested in support vector machines or
ensembling methods or these days deep neural networks.
And so matrix multiplications are at the core of a lot of that
stuff. So very core linear algebra
problems. I think historically the way
people thought about the relationship between linear
algebra and and randomness or noise was that there's
randomness and noise in the world, meaning the data you
measure, think of least squares and it's your job to clean it
up. And then you call a linear
algebra problem least squares or low rank approximation.
And you get more or less a deterministic answer and you get
more or less the exact answer. I mean, and I say exact in
scare quotes because there's numerical issues and you can't
represent square root of two on a computer, but you know, in so
far as machine precision is exact and you get an exact
deterministic answer. And for a lot of things that's
just overkill. And so you can use randomness as
an algorithmic resource, meaning inside the algorithm to
speed up computation. And this may be most notable
historically, like in Monte Carlo or Markov Chain Monte
Carlo, where you run simulations of of fluid dynamics.
I mean, the Metropolis algorithm was developed in that context,
but this is for core numerical linear algebra problems.
And so most people, if they're running computations, sit on top
of BLAS and LAPACK and, and, and related software.
And if you're calling Python, you're calling something else,
you're calling something that's calling something that's calling
them typically. And so these are core libraries.
And so the question is, can you,
you know, use good theory from randomness and measure
concentration and high dimensional probability to
improve those algorithms or improve, you know, variance of
those algorithms? The short answer is that you
can. So how?
But so how does it work in practice then?
So if I'm solving a PDE and I'm normally using some sort of
linear algebra library and it takes me this time or this
amount of flops or this compute, how is it that the randomness
reduces that? Yeah, I mean, when you're
solving a PDE, you're typically solving it for a particular
application and you're calling certain core linear algebraic
primitives in in one way or another.
So for example, if you're solving it with a splitting
method or predictor corrector method, or you're solving a
finite element or finite volume or you're solving a second
order optimisation, too, as a piece of that PDE solver.
These have least squares, you know, linear solves things like
this under the hood. So typically you're improving
those. You could, you could improve the
modeling setup for the PDE also.
But but you know, you could, you could try and solve the, you
know, the improve the core primitive.
So take least squares as an example, a low-rank
approximation. A common motif in in a lot of
these algorithms is you have the data and you want to solve the
least squares of the low-rank problem.
And you could solve it exactly machine precision or whatever.
But you might want to on the other hand, get what they call a
sketch of the of the data, which is roughly a small number
of data points. It could be a small number of
actual data points, or it could be what they call a random
projection. And if you're familiar with like
if you're familiar with signal processing and electrical
engineering and, and physics, think of this as like a
randomized version of a, of a Fourier transform.
So it takes some signal that that might be localized in space
and, and spreads it out everywhere.
But you don't need this to be a physical space.
This is just a linear algebraic problem.
And so you have, you know, you have columns and rows, so you
could select the important columns or rows, or you could do
this random projection, which is essentially a, a basically a
random type rotation that spreads the information out and
then sample uniformly in that rotated space.
And so you take this sketch and you can do one of a couple
things that so some communities, the more theoretically
inclined communities want to take that sketch and solve the
subproblem exactly. And there's a range of
theoretical work that says the solution to the exact, the exact
solution to that subproblem computed any which way.
A traditional solver or something else is epsilon-close
to the exact solution to the original problem if you set
things up right. Now, in a lot of cases that is
sort of coarse because you can't.
It's hard to get machine precision just by drawing a
sample and solving the sub problem unless you take a huge
number of samples, just because Monte Carlo methods tend to
converge slowly as a function of the error parameter.
So you could take that, you know, low quality, low
precision, but not trivially bad solution and ask yourself what
is a preconditioner? And by preconditioner I just
mean a preconditioner for a PDE solver or or least squares or a
linear solver. And a preconditioner is
basically a low quality solution that you refine.
So you can actually take this preconditioner that the that
this is a sketch. The theoretically inclined
people just say good, done; I'm epsilon-good.
And you can say, now I'm going to take that epsilon-good where
epsilon is 0.1 and just use any of a range of traditional
iterative solvers and drive that epsilon down to 10⁻⁸ or 10⁻¹⁶.
And, and just like most
preconditioners, if, if the cost to construct it is less than
the cost to, you know, the cost to construct it plus iterate
is less than you know, the, the algorithm you're competing
with, you win. And it's a good preconditioner.
So there's a range of ways people solve it that way.
So those are called sketch-and- solve and sketch-and-
precondition, respectively. Increasingly for machine
learning applications that want medium precision, also relevant
for scientific computing when you're interested in low
precision data representations, you know, going to half- or
quarter precision, which is, is increasingly seen in hardware.
There's a more subtle interplay where you can get the solution
and then you can iterate it and get a second sketch and toggle
back and forth. And if you set the parameters
differently, you can get intermediate solutions 10 to the
10⁻², 10⁻⁴ and 10⁻⁶ quality. So there's a range of ways you
can use the sketches, but roughly the idea is you get most
of the information in the sketch and and solve it or do
something with it. And is there any, you know,
orders of magnitude of the savings that you could get?
You know, if I'm doing a Navier– Stokes solver or solving some
neural net using these randomized, you know, numerical
linear algebra versus BLAS or LINPACK, are we talking, you
know, a 10% saving? Is it an order of magnitude or
is it still under research? And so it's not delivering the
full yet. Yeah.
I mean, I think the question of of how well you'd improve upon
something depends strongly on what your baseline is.
And if you feed this into a big PDE solver, there's a lot of
moving parts and you're competing with very mature code
and they're doing a little bit better or even a lot better on a
solver may or may not matter for the downstream use case that
that the scientist that is looking at the PDE solver is
interested in. Or go to the other extreme.
And you want to say, I want to compete with BLAS or LAPACK.
These are extremely optimized pieces of code and it's, you
know, it's hard to beat them. So one of the big successes in
the area was with Blendenpik. And then soon after that was
LSRN and Blendenpik, which said we wanted to ask whether
these randomized sketches, you know, not in theory, not Big O
notation, not whatever can they beat LAPACK Because this is not
boutique code you have or I have or a solver that you don't
share with the community. So well, this is something
that's been stress-tested for decades.
And the answer is yes. I mean, basically on any tall
dense matrix, you can beat LAPACK with these techniques.
You got to set parameters right and be a little bit careful.
But, but the short answer is yes, how much you improve it by
depends on the aspect ratio and depends on properties of of the
matrix. But think of it as ballpark 2 to
10. Now this was back in 2010 and so
a lot's changed since then, in particular in the hardware
landscape with respect to GPUs and hardware heterogeneity and
the sunsetting of Moore's law and Dennard scaling and these
sorts of things. And so it is the case that you
know, you can be a factor of 10 better.
Similarly in in in low rank type approximations.
I think the the the way scientists versus machine
learners use low rank approximations is very
different. Scientists tend to say low rank
means you know 99% of the Frobenius norm, meaning 99.9% of
the Frobenius norm and 99.99 I mean basically the whole matrix.
When machine learners say low
rank, they mean sorry, sorry, the spectral norm.
When machine learners say low rank, they mean, you know, 50%
of the Frobenius norm, in which case, you know, you lose a lot
of information, but you might iterate and and do something
better. So think of sort of scaling laws
and the neural scaling context with, with neural networks.
And so the way you'd use those low rank algorithms and feed
them into other solvers would be very different.
And then that would lead to very different levels of
improvement. And so I think that's largely
open. And then with the memory wall
that you're increasingly seeing in, in general, but most
egregiously in the machine learning applications, one of
the big wins and it's just starting to be explored for the
randomized techniques is basically improving the memory
properties. And so this could be just
reordering algorithms in the sense of communication-avoiding
linear algebra could be much broader because the randomness
sort of decouples things, makes it easier to parallelise certain
sorts of computations. And so you see much bigger than
factors of 2 or 10 improvement there.
But then again the baseline's changing year to year as as as
people explore different representations and and low
precision and intermediate precision and stuff.
Interesting, so how was it mentioned in Netflix then going
back to the former? Well, the particular I, I think
so, I'm not a movie star or a movie producer, but I, I have a
sense that they, they want to, I, I mentioned this to someone
and they said, you know what that means?
They tried to take the most exotic thing that wouldn't make
sense to anyone and, and, and, and cited it.
So the context was I, I think it was early on and the, the, the
main character, who was this, the attorney needed to represent
someone and the person had to get out of, of, of whatever they
were charged with, which was not a, a, you know, relatively
minor thing, but it was the 10th time they did it.
And so they, the, the, the Lincoln lawyer said, why do you
need to get out? And they said, I have a defense
on Thursday. I need to get out.
And they asked what's the, what's the topic?
And they said randomization and numerical linear algebra.
So that was the context of how it was mentioned there.
I wonder how they found that out.
I wouldn't know how they actually.
Yeah, I don't know. They they, they, they didn't
call me up. So they must have had someone in
movie producer land or something that did did a Google
search on on something exotic and came across it.
That's fine. Yeah.
That reminds me of the Did you ever like Star Trek?
Yeah, I used to see if there's a bifurcation which maybe would
get us in into a separate discussion whether you like the
original version or the the second version.
But then there was this explosion of different,
different versions of it. Yeah, I was more the next
generation person, but I the reason I mention it is it was
often whenever they wanted to explain how they were breaking
the laws of physics, some stupidly complicated phrase was
used. And I'm sure it exists somewhere
and it just reminds me of that, you know, explanation of
how they're exceeding the warp drive.
Oh, it's because the something something.
Yeah. So I think it was a little bit
like that. I mean, I, I think it's a sign
of the times that what they're citing is not warp drive and
quantum gravity, but randomized numerical linear algebra.
This is a core statement about how areas are progressing and so
on. Exactly.
Yeah, yeah.
So maybe one of the topics would be good to chat with you about,
and it's definitely a hot topic at the moment, is the the
foundational models for science. I've seen quite different
viewpoints. There's one argument that is,
well, large language models take all this data from the Internet.
They train and they're
remarkably good at doing what they do.
What about if we could do it for scientific disciplines?
Wouldn't we be able to just, you know, ask a prompt to go and
simulate me a plane or, or, or do something?
And then there's the the other avenue, which seems to be more
the replacing simulation tools or augmenting simulation tools
like GraphCast or FourCastNet or, you know, these sort of
seminal pieces of work that have come out, but they were for
very, they had to be trained, you know, for something.
Where do you see this? Where do you see this going?
Are foundational models possible?
Is it we just don't have enough data?
This is a big topic. So yeah, I'm not sure how we
want to start it. I mean, we're in the middle of
this fray and I guess that's why we were asking about this.
And I should start off with saying I don't know what a
foundation model is. Actually.
I know what different people say a foundation model is and, and
it means very different things to to different communities.
And so I think articulating that a little bit helps articulate
some of the possible directions and some of the questions
you're asking. I mean, I think one way to think
about a lot of these machine learning methods and it's
highlighted particularly cleanly with these foundation models
is machine learning methods. And in particular, these
foundation models are, they're an infrastructure, right?
The, the, the, the Stanford report said, call it foundation
models, not foundational models, because it's a foundation in
which you build things and as and as opposed to a half dozen
other terms they could have used.
Now, whether that's right or not, I mean, but that, that's
what they said. And so since then people have
said, well, it has to be this big and that big and, and
whatever. And, and then people want to get
a foundation model, not for all of science, but for areas A, B
and C and subareas D, E and F and get finer and finer
granularity. So in a sense it's, it's a it's
a term. And, and if you ask people, most
people using it what it means, they sort of will acknowledge
that, that they don't know. And it's not a standard
definition. I think probably the best way to
think about it is it's, it's, it's infrastructure, it's a
foundation on which you build things.
And so the computer's infrastructure, I mean, you can
do different things with the computer than you can do with
pencil and paper. And so it wasn't obvious the way
computer science, even before it existed, would evolve.
And at one point, post World War 2, the US thought they'd need
five computers and that would be it.
And then it would solve all the nation's needs, right?
So it evolved in ways different than expected.
And it evolved in particular because it, it could complement
what people did. I mean, you could ask different
scientific questions than you could with pencil and paper and
you could ask different engineering questions than you
could. I mean, now you can simulate
whole things and, and the very, very last step is building it,
right. But in addition, you could do
lots of other things that were driven by industry.
And, and so there's a bifurcation in the area whether
you're doing more numerical things that are continuous or
whether you're doing discrete things that, that, that are not
continuous that were driven largely by business needs.
So I think the, the right way to think about this is it's that
sort of infrastructure and it's tied to the data more closely
than than, you know, database systems because the foundation
models have information cooked into them.
And so it wasn't obvious some number of years ago that just
with language you, you could do what you know, has caught the
popular attention in the last couple of years.
So I think so to anyone who says that confidently, you, you can
or you can't with science, I mean, probably you can't say
that they're sort of reliably saying what what will hold for
the future 'cause I think it's not at all.
I think it's not at all clear for two reasons.
One, there's certain technical things that really do matter.
I mean that that are different about scientific data than, you
know, language and, and vision data.
And then there's just very different cultural, there's very
different cultural things. I think every scientist thinks
that their data is special and unique just like they think that
they're special and unique. If ML's taught you anything.
These algorithms can predict what movie you'll watch and what
stuff you'll buy better than you can.
And so you're maybe slightly less unique, you know, than than
you thought in some sense. And so I think there's a
question, you know, what does foundation model mean and how
can it be used so narrowly? You know, I have a big model I
can train it in in domain A domain A could be weather and
climate have gotten attention, but it could be fluid dynamics.
It could be something else about stellar formation.
It could be properties of materials and doing density
functional theory and so on. And then there's a question
about whether you could maybe learn broad based models that
cut across domains, which would of course be the more
interesting thing. So there's something, Chronos
said, I was involved with the AWS people that sort of says, I,
I don't want to be the best at at the most extreme things.
I don't want to be the 1% that's predicting the most extreme
things I want to be. I want to do as well as as the
majority of users for for time series analysis and oversimplify
the story. I mean, time series is a
complicated area clearly of interest in a lot of cases, but
it's it's a little bit, you know, you got to be careful
about off-by-one errors and a range of other technical things.
And what that says basically is take a language model, language
model structure with a bunch of time series and just
mean-centre it and variance-normalise it.
So just do the simplest possible things you could and you got to
do some data augmentation stuff and boom, you know, you do sort
of comparably and, and better than a a wide range of public
benchmarks. Now, the public benchmarks are
not extraordinarily high because a lot of time series data is
valuable and so companies tend not to release it.
But but the fact that you can do that just by sort of variance-
normalising and and mean- centring is is, you know, not
obvious and sort of interesting. It turns out sort of on a
separate thread that you could say, could I do well compared to
the 1%, you know, the most extreme things and, and the the
answers that you can there too. You got to use different
techniques than you do there. And so that leads to questions
as to whether you could have a foundation model for sort of a
broad swath of, of time series and forecasting analysis.
And so, as you know, I said, I'm an Amazon Scholar working with
the Scott team and we're looking at that there.
And one of the interesting things I think about there, if
you look at the details of the model, not the Chronos, but some
other things, why would you expect that articles scraped
from Wikipedia, you know, public, publicly available
language models? What would, why would that allow
you to do better job predicting dog food demand?
You know, I mean, it's not obvious they have anything to do
with anything. You know, it turns out, I
suspect the hypothesis is that the text, the linguistic
structure that you're learning from the Wikipedia articles is
strongly related to sequence to sequence modeling, which, you
know, if you think of 1 dimension as a metric space,
it's a very, very special metric space, a very specially
structured metric space. And so you can learn not just
recent information and not just low frequency information over
the past, but maybe in information at different scales.
Because, you know, sometimes
articles in text refer back 10 words or 100 words or 1000 words
or 10,000 words. So you can learn these sort of
heavy-tailed sort of structures and that gives you a better set
of basis functions to learn dog food demand or whatever else.
So in a sense, these, the language models give you better
embeddings, you know, Fourier analysis and Laplace transforms
and, and these sort of things are great for pencil and paper.
They're all developed in the 1800s, right?
These are data-driven embeddings that are good if you have
computers and and data not necessary for pencil and paper,
but they're good for that. And so that leads you to the
question about if, if I'm really careful about doing foundation
model work in science, what do I need to worry about?
It's not the details of this PDE or that PDE.
It might be that I want to reproduce what happens in the
natural language processing (NLP) models, which is roughly
scale model and data and compute.
So none of them saturate. That's very different than the
usual strong scaling and weak scaling and high performance
computing. I want to change the amount of
data I'm putting in. I want to change the size of the
model. I want to change the size as it
computes. So none of them saturate and,
and the conjecture would be that if if any of them saturate,
it's going to be harder to do transfer learning.
So you really need the non saturation and you do that one
way with NLP (natural language processing) and CV (computer
vision). But in NLP and CV, there's much
weaker control you have and that you need on the spatiotemporal
geometry than PDEs, right? Is, is, you know, PDEs, if
you're below a Mach transition or some other physical
transition, things are sort of smooth.
If you're above it, things are very messy.
Dealing with that transition is hard and people spend their
whole careers dealing with that.
So can you come up with data generation or tokenisation
mechanisms that that respect the spatiotemporal properties?
So I think if you're going to have a foundation model that
applies across a broad range of sciences that's trained on
weather and, and, and climate and, and simulations of
different grid structures from the machine learning
perspective, it's not so different than satellite data
from satellites looking down, you know, it's a different grid
and it's an image. And then you say, how does that
couple with the spatiotemporal properties?
So I think if, if you want to deliver on the promise in the
same way as computer science had to work out a range of
numerical methods to really match and beat state-of-the-art,
you're going to have to do a similar thing here.
And so partly depends on these technical issues, but partly
depends on cultural issues. But how much do you think?
I mean you raised the 1% versus the 50%?
I guess that is one argument as well, which is how close does it
need to be to machine precision to be useful.
You could argue that any of these large language models, the
way that most people use them, there is still a correction you
need to add. You don't typically ask it to
write a document and literally take it word for word.
You usually go in and go. That's, that's remarkably close,
but I'm still going to go and fix it.
Or it writes you Python code. It's unusual, isn't it, that
it's perfect. There's usually something you
have to correct. So I guess with the science
side, maybe the argument is how close does it need to be to be
useful And the cost of getting the incremental increase in
accuracy, you know, is it, is there a like a trade off where
you need so much more? Data, I mean, again, I think the
best analogy is look at the history of computer science and
how that evolved, I think, I think framing the question to
say how close does it need to be to be useful?
You're already framing it in a way that that makes certain one
group comfortable, the numerical analysts and the PDE
people that frame a question a certain way.
That's not how people would have framed the question before.
I mean that, you know, before you had, you know, represented
continuous numbers on, on a computer discretely take a step
back and say, I want to solve a certain problem.
And, and the question is now that I have very different
trade- off in terms of compute versus data and I have this
new infrastructure that's language models.
Can I ask a different question and, and, and push the science
for it? I mean, so in, for example, in,
in chemistry historically, but also nuclear physics and, and in
fluid mechanics, there's this notion of a semi-empirical
theory, which is a theory sort of derived, you know, it's not
just curve fitting, it's derived from an underlying more
fundamental theory, maybe phenomenologically with
parameters that are then fit empirically or semi-empirically.
And So what you need there is not the theory to be right, but
sort of right enough at the level of in the, in the
chemistry is chemical accuracy, which is, is however many
kilovolts or whatever depends on the reaction you're interested
in so on. And so certain methods that
maybe were more principled were a bit too coarse for that.
And other methods, when you combined it with other
techniques achieved chemical accuracy.
And so none of them were low enough in the stack that you
they were better QR codes for QR from, from linear algebra.
But they solved the downstream problem at that level of
actually, probably what you'd see here is that right?
So it's, it's probably not going to be so successful to say I
want to get 10⁻¹⁶ or 10⁻² of my large language model, but but
a foundation model for science need not be an LLM.
It, it, it, you could be literally learning the
embeddings from a range of PDEs.
You could, you could train to PDEs of different types.
And you could say, I want to port if I, if I know the basics
of hyperbolic and parabolic. And I know that in the transport
equation, this can enter in a certain way.
And you have nonlinearities and certain types of forcing
functions, you know, model each of those and and solve each of
the components separately. So I think the question is,
what's the right level of abstraction to do that?
And, and one of the modalities you could use is language, but
of course you could use image or PDEs or simulations, any of a
range of things. So I think in the context of, of
scientific problems that the foundation model need not be an
LLM based. I mean, there's some people are
pushing that, I think, but but it certainly need not be LLM
based. We just query the model and say,
you know, tell me what to look for, the top quark or whatever.
Well that that's the bit maybe
you brought up would be interesting to explore.
There is, I would say, a relatively fierce or contested
argument around physics-informed or physics in it versus
data-driven. Where do you sit on that, on the
argument? Do you, do you?
Does the model need to implicitly have some of the
boundary conditions, some awareness of continuity of
energy? Or is it good enough just to
give it so much data that it essentially learns those laws by
itself? Yeah, I mean, I, I think you
could ask the same question about in natural language
processing, computer vision did you just, I mean, there's a,
there's a common story people tell which is just get more data
and everything works. But that ignores the fact as you
know, that a lot of companies and universities, a lot of
people have put a lot of resources into NLP and CV to try
and figure out how to make it work.
And you know, convolutions convolve things, they spread
stuff out. You know, if you have an image
in 2D that may make sense, you might want to average over
nearby things. But if there's, if there's a
sharp corner like, you know, the shirt against the background
wall in this picture of me, you may smudge stuff out.
And so there's a range of ways to try and sort of deal with
that. That's core structure.
The 2D structure that you're trying to average over is very
different than sequence to sequence learning.
That's also much more discrete in, in natural language
processing. So it's not like computer vision
or natural language processing, just learn stuff.
You gave it very particular architectures that had very
particular inductive biases and then you're careful about the
data and, and trying to get gobs of it and, and then then it
worked. So I think if you, you know,
you'd have to follow the same path here.
You can't just take an existing architecture and press a button
and hope it works. I mean, time series, forecasting
space—temporal modelling, these sorts of things will have
to be, you have to figure that out, how to do that.
So I think the question, I mean, some people, you know, I think
are, and, and maybe sometimes people that have a vested
interest in, in moving forward or not.
I mean, some people say, you know, just ignore everything,
let the data do it all. Some people say, Oh, you'll
never match what I can do with my careful PDE solver.
I mean, I think that's neither of those questions is the right
one. I mean, the question is, given
the fact that there's an infrastructure of code and
experience numerically, but but you're also at a very different
place. You've been sitting on top of
certain essentially hardware and, and, and linear algebraic
advances. I mean, for years, for decades.
People working on PDEs call certain linear algebra libraries
and have essentially had not to pay the technical debt
associated with the fairly high level of of complexity with
developing new linear algebra because just wait a year or two
and the machines are faster. And so if, if, if that's not the
case, you're going to have to start thinking a little bit more
carefully about the underlying linear algebraic computations.
Maybe there's different trade-off points in the space
and maybe bring something else to bear.
You know, a model that has learned certain coarse-type
functions of one dimension, say from the language model that I
alluded to before that aren't Fourier modes or aren't Laplace
modes or aren't something else like that that they're more
familiar with, learn a different grid type discretization.
So I think the real challenge in delivering on the promise of
all this stuff is figuring out what the right way to combine
those two. So you can imagine rather than
just writing down a physics- informed loss and pressing a
button saying, well, the way people actually would solve this
is to have some sort of splitting method and deal with
the two types of physical things and the diffusion and
advection in two slightly different ways.
And machine-learn one and take the other as numerical.
And then you couple them together, like maybe in a
differentiable end-to-end. Yeah.
The, the reason I ask this is because it does feel certainly
at least in the CFD community, that there's a, there's a real
sort of split within the community.
And in a sense that on one hand you have people saying, well, we
have all these PDE solvers, you know, developed for decades.
And you know what convince me
how your machine learning or AI method is going to be as good
and faster and cheaper versus on the other hand, you have quite
sensational claims of, you know, 10,000 times faster, you know,
sort of real-time. Well, of course, when you dig
in, you know, you start asking the questions, well, how much
data did you have? What was the time to create the
data? And but there is a bigger
question, I think more when you look to the future, which is,
is it just a matter of time? Is it a bit like in the 1980s,
they could only simulate a plane to a reasonably low accuracy
because the computers just weren't powerful enough.
And as you said, you just almost keep the same effort almost.
And you just literally add 20 years of compute on top and now
you can simulate. Or is it a we'll never get there
because we'll it's like an it's like impossible to have that
much data? This is the bit I think a lot of
VCs and start-ups are also asking that question if we pump
100 million into this or a billion, will we fix it?
Or is it a trillion dollar problem and it's not worth
doing? Yeah, Yeah.
I mean, OK. So I think there's a there's a
bunch of things in the question there.
Sorry. We'll we'll CFD people to BC and
and you know that they have two very different utility.
You. Know, I think, OK, so it when
someone says to me do this, convince me of that I have these
metrics that have been I've worked on for decades, I just
say thank you, good to talk to you.
I mean, because, because there's no way you're going to win
that battle because they're very fine-tuned metrics.
So they're very one little use case.
Even if you do win that battle, they're not going to admit it
because they'll, they'll tweak their method and, and you know,
be slightly better than you. And there's greener pastures few
everywhere. And this is not a hypothetical
statement. This is a very practical thing.
You're asking about this in the context of PDEs.
We saw this before in randomized linear algebra.
I mean, so literally, I remember it was very clear that the
techniques could potentially be useful, not just in theory, but
in practice, and that the reason the numerical people were
saying, oh, they won't work, You know, really.
I mean, those are those number of methods that clearly wouldn't
work. But but then there's the next
generation of methods. And that's sort of when I
entered the the area and it was clear that they could
potentially work and the objections people had before
wouldn't work. And so this was right around the
time of the Blendenpik paper that I mentioned that that
basically said, can you beat LAPACK?
And the short answer was yes. So now, for example, if you look
at the SIAM Linear Algebra meeting, half the talks are on
this topic, but there's been this process of creative
destruction where this they're still asking the same questions,
but putting randomness there. And so that's one group of
community. I mean, a very different
community is the is the people that say, I'm going to not do
that. I'm going to go and apply it to
machine learning and 10 other problems.
And, and, and so there's that tension I was alluding to
before. So I think if you, if you try
and, and satisfy an old school metric, it's, it's good to have
a few examples of that as a proof of principle.
And, and this is 15 years later, and I mentioned it twice here,
right? So this was a clear proof of
principle of the area. The area sort of accelerated
after that. And there's been a lot of theory
and empirical development. So I think there's been a few
things that, if not that are starting to look almost like
that in this general scientific ML area.
I I think. History is moving in that
direction. So it's clear then that, you
know, randomness would be important for linear algebra.
Now it's even more so for all the historical and hardware
trends you were talking about. I think likely the same thing's
going to happen here, right? And so a question you could ask
is, are the people who know the linear algebra, are the people
who know the computational fluid dynamics?
Are they at the table developing the methods?
Are they just going to pooh- pooh it and say, well, they're
not going to work because it's not going to satisfy my
particular measure as opposed to here's a broad class of
techniques and it's hard to imagine that there's nothing in
my area that will be improved by them.
And so I think that's the that that latter question is the
question to ask. And that latter is the question
the questions VCs and and other people will ask.
And I think the, an important maybe determinant of how this
will evolve is, is how the different players interact with
this in, in the sense that a lot of the machine learning, more
broadly, scientific machine learning, not just for the
foundation model, but more broadly machine learning really
is, really is a horizontal. I mean, it's, it's designed to
say, what can I do with lots of data and be relatively ignorant
about your application area? Because who knows why people
click on ads or click on links on their social media account,
right? I mean, you can tell the story,
but really who knows why? And so a lot of machine learners
implicitly or explicitly will say, I'm not interested in
solving your scientific problem if it's just a one off problem.
I mean, if you're a scientist using machine learning, you
might be interested in that, but that's because you're
interested in the domain. But if you're a machine learner,
you might say, I mean, I see some commonalities between your
fluid dynamics and this other solid-mechanics problem.
And I see some similarities there between that and something
in in 3D in the atmosphere as opposed to, you know, something
very different. And so identifying those
commonalities I think is important.
What I see in, in in some cases, and this is hampered by
universities and it's hampered by industry, and it's hampered
in a couple government labs in, in a couple different ways, is
a lot of people want to solve a scientific problem.
And so they're going to say, I'm only going to invest, invest
literally or figuratively in machine learning in that one
area. And that's just not how machine
learning works. You don't see the, I mean, think
of it as horizontal business with horizontals and verticals.
If machine learning is a horizontal, like high
performance computing that will solve a wide range of problems,
you got to apply it to a vertical to justify it.
That's that's where that that's where you know, the rubber hits
the road typically. And so you don't see the
successes in machine learning, scientific machine learning at
vertical companies, you know, companies working in genetics or
or oil exploration or any of a range of other particular domain
verticals. You see it in the horizontal
companies. The companies have built out the
technology infrastructure and they get a team of people that
know this is a vertical and applies it there.
And then, you know, a team that knows a different vertical and
applies it there. And so I think the question is,
how does that play out? And that's going to be a non-
trivial evolution in terms of. And your.
That day. Your comment was interesting as
well about the difference between sort of Transformers
versus CNNs and you know, if you just throw the whole world's
Internet at CNN, that isn't you needed to have the right
architecture to be able to throw a lot of data at it.
I guess the question is more for the science.
Is there a common architecture across the sciences that then
would allow you to sort of build out that big which could be
applied to verticals? Or is it the case that the needs
for weather or genetics or chemistry are so different that
you would need a different architecture for for each
different science? And therefore the, the, the
dream of a true foundational thing.
Besides, it will never happen because you know.
Yeah. So that's the big question.
And I, I think there's a technical component to that and
then non-technical component in terms of the areas, right?
And so technically it's open-ended.
I don't know, right? I mean, we're, we're in the
middle of the fray and we're, we're voting with our feet and
placing a bet there that the, that the answer is or could be
yes. To try and address that, you're
going to have to deal with the issues we were talking about
before. You know, one dimension is very,
very special. two dimensions, the surface area to volume
properties are very different. Three dimensions is very
different. By the time you get four in
scientific computing, that's sort of infinite, right?
Machine learners, by the time you get to a million, you
know, it still seems pretty small.
So mapping the methods to one versus two versus three
dimensions, open question, maybe there'll be no trade-off
point where the surface area to volume and the isoperimetry
and other things like that wins.
I suspect not. I suspect it could, but I mean,
open question. And then non-technically, if you
look at how computer science evolved, numerical analysis and
scientific computing was core to computer science.
Look at how numerical analysis and computer science evolved.
Yeah, these areas gave birth to computer science and then
computer science banished them because it was easier to build
the structure, the theory around, around Turing machines
and, and, and the Lambda calculus and, and discrete
notions that talked about complexity independent of the
data. And if you're talking about
complexity independent of the data, you don't need
preconditioners because it's independent of the data.
And so there's a cultural aspect that that says, you know, does
the vertical or the horizontal culture win here?
And so even if you stipulate that the answer is technically
yes, I think there's an open question on how it'll look,
it'll evolve and whether that'll win the day and solve the
problems that you're asking. But I think it's not, it's not
so much that I solved PDE type A and applied it to PDE type B.
It would be certain coarse things.
I mean, as you know, there's a class of things for which finite
element methods work. There's a different class of
things for which finite volume methods work.
And you know, you, you take a course, you learn a 101 method
and then boom, they bifurcate after that.
So what's the analogous taxonomy here?
And it's probably neither of those.
It's probably something that's more data-driven, but what's the
right way to taxonomize that? And even if there's a
technically a correct way, I mean, you know, computer science
is not computational science. Computer science is a coherent
area. Computational science is spread
over many departments and many divisions.
And so I think it's an open question about how this area
evolves. Yeah, it is certainly.
It is certainly interesting. I think one of the sort of key
topics to all of this though is the data availability and
licensing and financial reward.
So one thing that I've seen in I guess the early days is nobody
knew what was going on. So I guess the companies were
scraping half the Internet. There were a few CAPTCHAs or
there was a few sort of, you know, firewalls to stop things.
But in general, it was a sort of
free-for-all it seemed, Whereas now people have, you know,
cottoned on to that and some of the companies having to do
deals with publishing companies or do deals with, you know, to
try and legally get access or or permissively get access to
things. I saw the example on the weather
side that, you know, some of those databases were released
openly because they were released openly enabled, you
know, these various teams to go and do things like, you know,
FourCastNet and GraphCast. But those data may not be open
in the future. So how much you know, and you
work in academia, you see people want to have their own data.
Some people want to release later.
Do you see that that will ultimately be the blocker?
I mean, how do you incentivize people to want to make their
data available and should it even be done that way, you know,
that the whole open source data versus closed?
You know, this is this, I think, is the elephant in the room in
terms of this area because a lot of government grants require
that you make the data publicly available in some sense of the
word. Now, what does in some sense of
the word mean for a lot of scientists?
If the data is more than a year or two old, it's worthless
because they're trying to push the cutting edge of the science
and then they don't maintain the data for more than a couple
years ago. And so no one maintains and it's
hard to access. So there's a lot of public data
that's not de facto public or accessible, you know, in a lot
in those cases and maybe in, in, but, but certainly in, in
machine learning sort of applications more generally in,
in Internet and social media and so on.
Having older data is good, but you can't get more than a couple
decades older, right? And so oftentimes the the most
recent data is the most valuable and you just design a method
around that. And so it's not obvious to me
that the way the data are generated and, and collected,
distributed now will be the one that is going to be developed
going forward. I mean, if you get better
telescopes, if you get better, you know, molecular sort of
robotic systems to, to to make molecules, I can easily imagine
a situation where data that exists now or prior to a few
years ago is basically obsolete and irrelevant.
And so all of that data sits at whoever generated it.
And so then the question is, is does a government lab generate
it? And if so, do they make it easy
de facto to access? Is it done at universities
where, you know, you have a big infrastructure capital
investment in in particular, essentially in particular
science and verticals to generate it?
Is it done at companies that are in different verticals or is
there some sort of collaboration or interaction with, you know,
horizontal infrastructure companies, hyperscalers and then
they get their fingers in that and have access to it?
So I think these are all open questions in in terms of how
this evolves. Yeah, it it seems to be very
much, you know, traditionally and I guess if this is for me,
the difference between the pretrained, quote- unquote
foundation models versus just letting someone go and train
them. Although I mean, I guess most
people you and I would not go and train our own LLM.
We're just going to go and use someone who's audit and we're
just doing an inference on it. That's and I could see if I'm
BMW and I have data from previous car designs or
simulations or whatever, how they could train a model using
their data. But I'm sort of wondering in my
head, how could it ever become the point where some person has
access to not only BMW's data, but Fords and Audi's and Volvos
and Boeings and Airbuses and you know, ExxonMobil and Shell
and everyone else. Like how how could they collect
the data? What would be the incentive for
that to happen to or that's what I mean.
Or is it the case that that's why there will never be a
foundation and it will just be that individual people train
their own models because the data is just so sensitive and
difficult to collect in one? Yeah.
I mean, this is, I think this is the big elephant in the room.
I mean, it's not clear to me that one.
OK. So I think if you were to try
and generate a meaningful scientific foundation model,
you'd need data from a pretty broad range of use cases.
And so you gave, you know, several that in particular are
with a strong sort of financial backing.
I'm always reminded of the story.
Jim Gray was, I guess, the most prominent person in the
Microsoft Research division years ago, such that, you know,
he would have to have regular meetings with Bill Gates, the
CEO at the time. And Bill Gates would say, why is
the most prominent person in my research division applying
database techniques to astronomy?
This is when he started working with Alex Szalay and some others
on, on databases for, for astronomy and, and cosmology.
Why don't you work on something more useful like, you know, any
of our products? And, and Gray's answer was
basically, I'm working on it because it's worthless, because
there's no financial sort of incentive behind it.
I don't have to talk to the lawyers and we can work out IP
and all these things. It's public data and, and I can
work on database techniques and figure out latency and bandwidth
and all these trade-offs. So I don't know, but it would be
interesting to see. Will you have an analogous thing
here? So there's there, there is
public, there's, there's various places where public data is
available. And I think if you can
illustrate that you would have a proof-of-principle foundation-
like model. Then the question is, will
people come to it? Now clearly the incentives for
scientists will be different than than for companies there.
But I could imagine a situation where you build that out and
then companies could take it and fine-tune to their particular
data, right. It's not obvious to me that any
of the companies that you mentioned have data that that's,
that is that special in a scientific sense?
I mean, clearly the particular data they have is particular to
their particular situation and then the particular, you know,
product they have. But I, I can imagine that, you
know, the PDEs from early solar system formation and, and
hydrogen bomb explosions and hydrodynamics inside stars, you
know, is, has some similarities and some differences with
weather and climate and some similarities and some
differences with crack propagation in the Earth.
And that'll have some similarities and differences
with, with various sorts of sensing modalities that you
would, that have a geospatial component.
And so can you stress-test that and, and get even even a proof
of principle that, that if you would attack those, then, then
the answer would be yes, that, that you could have a path to
have a more foundation model. If the answer is yes, then the
question is you talk to various stakeholders.
Some of them are government entities or government sponsors,
some of them are corporate and try and figure that out.
And I could, I could imagine that that basically it's too big
a lift in the area stalls. So I think this is something to
be determined. But I think having a nucleus of
something is going to be important because right now I at
least what I've seen, I mean, in most of companies like that,
there's, there's, there may be some people a little bit
intrigued, but there's not a sufficiently heavy lift that
they go screaming at their CEO to say we need to do this,
probably because it's not extremely strong baselines that
it can happen. And their competitors by talking
to these public models and fine- tuning on their
proprietary data will do better.
I think at that point they'll be you can see a change.
And I think this is, I mean, I guess this is affecting the
current generative AI, which is how do you make money out of
this? You know, like you can do it for
the sake of science. That's nice.
I developed this thing. But I'm always interested to see
like the money that you put in to create the data to train the
model, can you ultimately make a profitable thing out of it
that justifies its cost? Now, with standard, I guess,
simulation software, there's an argument, well, if I simulate
it, I don't have to build it. If I can predict the weather, I
can help people figure out if it's there's going to be a
hurricane. And I can say, you know, there's
a, there's a clear thing. But because we can already
simulate things, we can already predict the weather.
We can always do it. For me, the machine learning bit
is just the argument, well, it's faster or, or, or it's
cheaper. That's where the amount of data
do. Do you know what I mean?
Is there a commercial? But I, I think, I think there's
a faster question and I think it's tempting to say, can I be
faster? Because faster means there's a
number you're comparing to. So am I bigger?
Am I more on some axis? And back to the, the, the
randomized linear algebra example, I mean, I, I was
talking to various people and saying, don't worry about the
randomness. You know, if, if you do this and
that, that'll work. And the people would, you know,
basically when you have a new idea, they'll beat you up.
And so they beat us up and so on.
And a few years later, one of those people in particular had
some methods and showed it worked, you know, not the blend
people, you know, something around the same time and their
students came back to me and didn't know who I was.
was lecturing me and saying, oh, don't worry about the
randomness, it'll all be fine. And so on.
There's just a cultural shift. And so there was a cultural
shift that really was not, is as faster as this is as better.
And so I, I, I don't think you're going to see the win here
just because something's faster and better.
You might need that to point to, but the win will happen because
you can solve other things that you couldn't solve before.
So just as an example, say you're, you know, you want a
better battery and you want to understand the way crack
propagates better, you know, so that I mean, you want, you want,
you want a longer lifetime, you know, warranty.
So you, you're working on developing batteries.
One of the ways this fits in, not in a scientific, but in an
engineering or commercial context is you want to write a
warranty to say that that the batteries are going to last so
and so long. And, and otherwise you'll, you
know, you'll eat the cost. And so can you predict battery
lifetimes. And maybe if you have a certain
crack propagation heterogeneity structure, you know that it
correlates better with how long the batteries will last.
So now there's a fair variability and a particular run
on the extreme tails. So that sounds sort of very
different than I want to do oil exploration and better fracking
or something. I send sound waves down into the
earth and I look at how they bounce back and I try and learn
something about it. But I could imagine that the
embeddings that you get from a core model that cuts across a
couple different domains learn spatiotemporal properties in
particular of complex systems that have sort of long-range,
non-trivial, heavy-tailed correlations, which is what both
of those examples have. And so say that you can come up
with a model that captures those embeddings.
Now, a company here may tweak it to their data, do some transfer
learning in the battery space. A company over here can do a
better job adjusting it to their data, which is, you know,
spatiotemporally very different in terms of, you know, better
fracking. And, you know, for all I know,
the same embeddings could be useful for, you know, you had a
very different use case, right? And certainly in the United
States, there's various places where insurance markets don't
exist for floods or for earthquakes or for a range of
things like that. I could imagine that you get
these coarse embeddings. These are coarse functions.
They're not sinusoids and Fourier modes.
They're something that's data-driven embeddings.
And you say on top of that, I want to do transfer learning to
the relatively small amount of data that I have.
That's very high touch. You know, you have 10,000
stations across the country that measure something in chemicals,
whatever I want to do transfer learning to that and I model,
you know, model the system, the way information flows in the
Mississippi Delta or something. And, and if I do that, I can do
a better job, you know, predicting the tails of flood
events. Why?
Because as a model, I don't know how things propagate, but you
know, subtle effects in terms of the density of the soil or, or
the moisture of the soil coupled with, you know, 10,000 other
things that the neural network learns in a complicated way.
What's hard to reason about scientifically means that now I
can create that market. And that's, that's something
that any of those companies I could imagine try and you know,
buy out a startup company that does that.
So I think you're going to see a win.
Not just I'm running faster, something faster than you, but
but in a space like that. Yeah, no, I, I, I think what
you're getting at is you, what we've seen from current machine
learning that it seems to do things that we don't fully
understand, but are like you, like you gave the original
example of the time series that an LLM that wasn't
explicitly trained on it actually ended up being quite
useful. So I guess we shouldn't
underestimate the potential of how some of these things may do
something that we're, it's not just about replicating what an
existing simulation tool could do, but maybe it could help to
merge different disciplines together and see the links and
the, and the, and the things between them.
I did want to pivot to a different topic, which is when
some of the other people I've spoken to, I'm always really
fascinated by the curiosity or the uniqueness or the
peculiarity of academia and, and the sort of different people's
careers and, and, and progression through.
And I think you have quite an interesting one, because you are
a very pure. Mathematician, academic, you
know, you, you are very much in that, but you also have, you
know, the Amazon Scholar position; you're at, you know,
Lawrence Berkeley. Was this a a strategic thing in
the sense that you wanted to work on real problems?
Was this just you find more interesting, like is there, How
has it benefited you? And would you advise it to
others? You know?
Yeah. Yeah.
I mean, it's there's a lot of, again, a lot of questions.
I mean, at least the narrow questions, universities versus
not, yeah. Yeah, there you go.
I'll start. That but, but, but yeah, you
tapped enough stuff into the question to make it forward-
compatible with lots of answers as it sounds like.
I mean, so my, my, you know, OK, so my PhD's in physics, I
never actually studied computer science or statistics.
And it was in computational statistical mechanics.
And at the end of the dissertation I switched areas
basically to theoretical computer science to work on
originally Markov chain algorithms, some of the stuff
I've been working on computationally, Markov chains
and molecular dynamics. So it was a technical
connection, but it was it was fairly loose.
And then the randomization inside the Markov chains that
the people I was working with were also working on these very
early versions of these randomized linear algebra
algorithms. So I got involved with that and
I'd done enough coding that it was clear certain ones are
going to be useful and certain ones weren't.
And so at that point I decided to switch areas partly because
of what I've been working on. Well, two reasons there was a
forcing function. This is I just mentioned this
because if you have younger people listening, it's good to
know that sometimes older people have had chequered past, let's
say. So Long story short, I had
career ending problems with the dissertation advisor and and
the supply-demand structure in the natural sciences and
engineering is very different than in computer science.
And so that was was motivation to, you know, jump off the
Cliff. I also realized that that the
particular work I've been doing in computational statistical
mechanics, this is a great area.
I loved it and and it's informed a lot of the stuff that I've
done since then. But you know, it was a great
area to be in 1970 because you know, forward-compatible with a
huge expansion. There's going to be Nobel prizes
given out. I mean, just lots of good stuff
going on. Not so much in 2000.
The area, it's sort of sad, but it was clear that that the
future is going to be data analysis and, and I joke that I
never knew more about algorithms or data than the day
I switched. And then I've just been becoming
more ignorant since then. So I don't know quite what I
meant by data analysis then, but but but clearly I was right.
I mean, it was a fruitful area for the next couple decades.
So I'd done work in theory of algorithms and, and with respect
to the academic question, if you're desperately wed to an
academic career, you shouldn't do this because the academic
hiring is fairly siloed and, and it's, it's not a good idea to
switch areas after your dissertation and, and so on.
If, if you, if there's a forcing function like I had, or you're
interested in sort of problems more broadly, then you can think
a little bit more broadly. So I've had a lot of students in
postdocs and I sort of give them this advice and you try and
carve out various projects that they're interested in.
Some want an academic route, so they should work on slightly
more conservative things in the general space.
Some definitely want industry and some, you know, could go
with either and, and the ones that could go with either,
I've seen some go one way and some go the other.
As a general rule, this is going to be a marketable space and
stuff I do. And so maybe it's a little bit
less of an issue, but but, but that's something that they
figure out. So I had done work in in
theoretical computer science and it was clear that these
algorithms would be useful more broadly because the randomness
entered in a very different way than classical algorithms.
And so then the question is, how does it evolve?
And so it evolved. I was at Yale in the mathematics
department as in one of the junior faculty positions.
I spent time at Yahoo and Stanford and moved to Berkeley.
I'm in the statistics department there and at the International
Computer Science Institute and Lawrence Berkeley National Lab.
And as you mentioned, a couple years ago started with as an
Amazon Scholar working in the supply chain optimization
technology group, where, you know, we work on better
forecasting, demand forecasting algorithms and supply chain
decisions. So I think this is, this is
something that's good for people to think about.
And you know, there's at least, I guess 4 hats there.
And each hat has pros and cons and pluses and minuses.
And so figuring out how to navigate that space is something
I try and encourage students and postdocs to think about sooner
rather than later. How?
How much should people expect to have to move?
You know, I always think this is one of the sad in a way,
things about academia, at least from what I've observed, that
the fact that you do your PhD and let's say you do a
postdoc, how realistic is it that that person in the US or
the UK should assume that they can stay at that same
university and just move up a chain?
Or how much should they expect that they're basically going to
have to move somewhere else in the country to find the
position? Yeah.
I mean, the short answer is that when you do a postdoc, it's
usually more targeted because you're working with a particular
person, a particular group and and so you could be at the same
place or not. It's usually a good idea to go
elsewhere to get experience with other people, the real filter's
at the assistant professor level.
And you got to move there because, you know, except in
rare cases, you don't get a job at the same place.
And that's not a statement about whether the place where you are
at is good or bad or whether you're good or bad is just just
run the numbers, right. If there's 10, 20, 30 or 40
places hiring and that you're looking at and and the chances
are one and whatever that that you get an offer, it's just
unlikely to happen at that same place even if you
ultimately end up there. Sometimes people go and leave
and come back. So moving, for better or worse,
is part of the part of the equation, yeah.
But did you, some of the questions I get is the should
they go into industry or, or just let's put this two ways,
does academia value industry? So if you've been an assistant
professor or you've been a postdoc and you then say, you
know what, I'm going to go into industry, do they value then?
And is the ability to come back or have you been out of the
game too long? You haven't sort of, you know,
got the publications or the teaching experience, You know,
is that a good piece of advice or would you say no, no, no, If
you really want to be in academia, you basically need to
stay in in academia. Yeah, I mean it, it's, it's more
textured than the following. But I, I think the short answer
is that that that you need to stay.
The slightly longer answer is it depends on the area, like
computer science versus statistics versus engineering
are rather different. In some cases having a startup,
in some cases having a postdoc in industry is good and you can
go back to universities and that tends to correlate with
computer science. I don't know as much in
engineering that may be changing over time, but less so.
And maybe statistics and applied math.
I think in a sense, on the one hand, people value the
experience you might have, but not really.
And by that I mean, you know, if, if you, if you check all, if
if you check all my boxes. And in addition, you have this
other stuff, good, but you got to check all my boxes and the
boxes are the things you alluded to.
And so there's sometimes there's postdocs in industry that are
basically academic post docs, you write papers.
So I'm not counting that because that's, that's effectively the
same sort of thing. But if you're out more than a
couple years and you have fewer papers, it just gets harder to
publish. There are certainly cases where
people do that, especially if they have some high profile ones
and, and can work their way back one way or the other.
But it's, you know, you should know going in that you're, it's
like a salmon swimming upstream.
And this is going to be a hard one.
So how does he, this is maybe people in the US are maybe a
little bit more familiar, but particularly people who are not.
So how does it work with the
national labs? So you have a position at the
national labs. How, how does that work between
is this a, a quite common thing in a way for these things to
happen? I I can only imagine how you
juggle the e-mail inboxes and the sort of meetings, but I
guess it's valuable. And interestingly, after an hour
today and I haven't been hit by e-mail.
That's why I've turned it off. Except, e-mail is a little
overwhelming. I mean, it's not so common.
So, so I, I with the Berkeley hat, because I'm in the
Statistics Department there, I'm at Lawrence Berkeley lab.
It's literally just up the hill.
And so I can, you can walk there.
And so they were interested in scientific machine learning and,
and I had done work and was well known in machine learning
and, and algorithms and statistics, large scale
statistics. And I had various projects over
the years with people there and elsewhere on scientific ML
problems. And so started about two years
ago. This is what we renamed, but the
scientific machine learning group, basically MLA: Machine
learning and analytics. And so it was partly facilitated
by the fact that there there is a moderate amount of interaction
between UC Berkeley campus and LBNL Lawrence Berkeley National
Lab, Oak Ridge, Argonne. Some of the other labs have
things like that, but it's not so, so, so common.
Most people are there, meaning just there and 100% of their
FTEs there and and there for long term but so it's not.
I wouldn't say it's very common. OK, which makes your experience
even more unique then to have done it.
Yeah, what maybe is a sort of finishing off comment or, or
topic is if you have somebody now who was wanting to get into
scientific machine learning, because this is I guess kind of
what we've been largely talking about, what would you get them
to focus on? They're doing a PhD or they're
doing a postdoc. But is there any, doesn't have
to be a very, you know, specific, specific, but what
sort of area we'd say, you know, this is something you
should look at. This is an area that is ripe
for, you know, progress. Yeah, that's a good question.
There's certain things, I mean, that are less ripe for progress
in scientific ML, even if they're important scientific
problems. So what I, what I, what I would
try and say is work. You should work on a problem
that's of interest to both sides.
Otherwise you're coming at it from one side or another.
And this is true whether you're coming from CS or stats
learning scientific problems or coming from one scientific
area. You should work on trying to
figure out how to frame what you're doing as something of
interest to both sides. So I have projects that, you
know, we, we, we construct projects, you know, a student or
postdoc owns a piece of it. And, and a goal typically is we
want a publication in the particular scientific area, but
also in an ML venue. And it needn't be the same one.
Sometimes it is, it's a long
version, a short version. Sometimes it's just two
different things, but work on a method that's broad enough that
machine learning people are interested in it and that by the
definition of machine learning, people are interested.
You convince 3 reviewers to say yes.
And so it's an imperfect process, you know the review
process, etcetera, but boom, you get it in a top machine
learning venue. And similarly, on the scientific
side, you know, a statement that it's of value to the scientist
is you get three reviewers to say accept and then it's
accepted on the scientific side.
And oftentimes the way we try and scope our projects is to
say, you know, we don't on the machine learning side, we don't
want to work on your problem if it's only of interest to you.
If I can't apply it to some other area, someone else, some
other domain, Because if that's the case, I probably need to
know so much about your area that I become, you know, an
expert in your particular area. And similarly, on the scientific
side, you know, if you have a particular method that is only
that's so heavily tailored to your domain that it's not going
to be useful more broadly to machine learning people, that's
a much harder sell. Then you're not doing something
that satisfies both sides. So something that I knew about
from working on years ago was like sequence alignment in in
genetics, right? Clearly an important problem,
not something that's portable the particular algorithm, not
something that's portable to lots of other scientific areas,
but better machine learning methods for solving
spatiotemporal forecasting problems with non-trivial
boundary constraints. That's clearly of interest to a
pretty wide range of applications.
And also to solve it, you're going to have to introduce
technical solutions that are probably, you know, go, can you
come up with a differentiable optimizer for an end-to-end
differentiable system where you have hard constraints?
I mean, that's clearly something of interest to machine learning
and optimization. So I guess I'd try and focus on
something there. And that is probably the way
that you're going to be. And unless you know exactly what
you want to be doing 30 years from now, that's probably the
way to make yourself most forward-compatible with with
however. That's a really, that's a really
good point. I guess what you're saying is,
yeah, it's true. Like later on in your career
you've become established in a certain area, but most people
during their PhD or the postdoc at that time have no real
idea, necessarily. So you're saying if you do
something broad enough that can get you exposed to different
groups and and have a foundation to your earlier comment, you can
more easily go into verticals because you've built that
foundation. Whereas I guess if you jump
directly into a vertical right from the beginning for you to
sort of reverse out that it's kind of harder, isn't it?
So I I think that's what you're alluding to.
It sets you up better. Yeah, yeah, because it's not at
all. I mean, all the questions you're
asking are fair ones. And it's not at all clear how
the whole area will evolve. And so depending on how it
evolves, having experience in one versus another.
And, and I think I mean, having worked in a bunch of areas, you
know, it's, it's, but, but speaking substantially only
English, but a but a little bit of other things I can imagine.
And I see, you know, you learn a second language, it's hard.
You learn a third language, the, the relative cost is a lot less.
You learn a fourth.
I mean, so there's a diminishing returns in terms of the amount
of extra effort you need to do.
So you work in one area, switch to the second.
It's very hard switch to the third.
You know, you can, you know, you can, you can do that.
And after that, one thing I've sort of been always struck by is
it's amazing how little you, you need to know about a
particular area, especially if you're working with good people
who'll complement what you're doing.
You can learn from them, right?
And so it's amazing what you don't need to know in order to,
to get interesting results in an area.
And so, and so learning new things.
By the time you know two or three things, it's easy to learn
the fourth. Yeah, yeah, I know that that
makes sense. Well, thank you so much.
I it's the day after Labour Day, which means that probably
there's a whole bunch of emails and stuff that's started to come
through on essentially your first day back.
And I know we probably could have carried on for for another
couple of hours. But yeah, thank you.
Really, really appreciate it and look forward to catching up in
person at some conference in the future.
Sounds great, thanks for having me, this has been fun.
All right. Cheers, Neil.