This is what happens when coders mistake being good at solving riddles for being good in physics: confidence goes up, expertise stays put, and the exaggeration begins.
How would you have treated the result if it had been entirely human generated? To me it seems that the standard in much of math has not been whether or not something is practically useful or not. If the NS proof stands the test of time, it seems a bit like moving the goal post to say it's not interesting/special. It is totally reasonable to ask a question and find out the answer isn't that exciting in hindsight, but people did think it was interesting enough to make it one of the millennium prizes.
How would you update your thoughts if one of the big frontier labs came out with a proof or resolution for another physics heavy MP question?
I think there's a separate issue, one that Terry Tao expressed. It's the idea that those considering a career in mathematics may think twice. Imagine setting out to tackle a problem and spending many years working on it, only to find that some else used an advanced model to prove or disprove the conjecture you were working on. It is unfortunate for Buckmaster and Alpöge that OpenAI's attention was directed to their project because of rumors that N-S had been solved. I read somewhere that OpenAI may have used an unreleased advanced model to take the problem to completion. How can you compete with that? How are mathematicians going to have confidence using Codex and other commercially available tools, when unreleased frontier models may steal their thunder? It seems that every week there is a story of some unsolved problem that amateurs solve. Some long mysterious cipher that gets decoded. This is different from Hinton's prediction about radiologists. Their existential crisis is being played out right now. NASA scientists may not care, but mathematicians do. That's why they signed the Leiden Declaration.
I’m just not sure whether some of the people building “smart prompts” to tackle these long-standing unsolved problems are doing it for a moment of fame, or whether they fully appreciate the importance and practical context of the problems they’re trying to solve.
This episode with OpenAI left me with the impression that, in some cases, we have very talented coders running models, code, and prompts without necessarily understanding the underlying science or the real-world applications well enough.
Thank you for this well articulated takedown of benchmark maxxing.
The research frontier of fluid dynamics has indeed long moved past the NS approximation, to handle applications where it’s not sufficient.
For example, there is a whole research program around using Renormalization Group methods to properly understand fluids from the particles up.
💯
How would you have treated the result if it had been entirely human generated? To me it seems that the standard in much of math has not been whether or not something is practically useful or not. If the NS proof stands the test of time, it seems a bit like moving the goal post to say it's not interesting/special. It is totally reasonable to ask a question and find out the answer isn't that exciting in hindsight, but people did think it was interesting enough to make it one of the millennium prizes.
How would you update your thoughts if one of the big frontier labs came out with a proof or resolution for another physics heavy MP question?
I think there's a separate issue, one that Terry Tao expressed. It's the idea that those considering a career in mathematics may think twice. Imagine setting out to tackle a problem and spending many years working on it, only to find that some else used an advanced model to prove or disprove the conjecture you were working on. It is unfortunate for Buckmaster and Alpöge that OpenAI's attention was directed to their project because of rumors that N-S had been solved. I read somewhere that OpenAI may have used an unreleased advanced model to take the problem to completion. How can you compete with that? How are mathematicians going to have confidence using Codex and other commercially available tools, when unreleased frontier models may steal their thunder? It seems that every week there is a story of some unsolved problem that amateurs solve. Some long mysterious cipher that gets decoded. This is different from Hinton's prediction about radiologists. Their existential crisis is being played out right now. NASA scientists may not care, but mathematicians do. That's why they signed the Leiden Declaration.
I’m just not sure whether some of the people building “smart prompts” to tackle these long-standing unsolved problems are doing it for a moment of fame, or whether they fully appreciate the importance and practical context of the problems they’re trying to solve.
This episode with OpenAI left me with the impression that, in some cases, we have very talented coders running models, code, and prompts without necessarily understanding the underlying science or the real-world applications well enough.
Sometimes the students have a mentor to guide them (https://www.htx.com/en-in/news/internet-shocked-unsolved-problem-by-fields-medalist-breache-IBDgZG2j/) which facilitates understanding. If they truly did the work, then their love of math will probably be elevated.
Agreed. It’s so important to do the work and get your hands dirty, despite all the distractions from AI.
Tons of people at OpenAI (and probably Anthropic too) have math and physics PhDs.
OpenAI solved a math riddle that engineers solved practically decades ago and mistook that for physics.