In a sense, how do you know whether the problem your AI has just solved is important? A simple proxy is to just check whether humans have thought it's important.
That's also why famous open problems are a good benchmark or proxy: you don't need to convince the rest of the world that the problem your lab's new AI just solved is actually useful or hard.
Not all of the 'famous' open problems 'mathematicians have failed to solve for decades' are actually that famous though. In some cases they've remained open because no one cared or had even heard of them. Those outside the mathematics community seem to think that all mathematicians can, and do, work on essentially all problems (for example, most mathematicians have in mind the millennium problems as a goal), but this is very far from the case. Most serious problem statements are not even particularly understandable to most mathematicians, let alone workable on.
There's also a difference between important and hard. There are important problems that turn out to be easy, and hard problems that turn out to be useless.
As usual, I think everyone would agree that something like "curing cancer" would be both hard and important!
I think the AI labs' current obsession with showing off mathematical results to uninformed outsiders is a cheap trick. If they were really interested in 'enabling human flourishing' (rather than just wowing, by any means possible, investors with more money than sense), they'd be showing off cures to diseases rather than solving obscure problems in combinatorics only previously considered by three Russians fifty years ago and then declaring that The Singularity is here.
In a sense, how do you know whether the problem your AI has just solved is important? A simple proxy is to just check whether humans have thought it's important.
That's also why famous open problems are a good benchmark or proxy: you don't need to convince the rest of the world that the problem your lab's new AI just solved is actually useful or hard.