> their existence is contrary to everything we thought we knew about neural networks prior to building ones that worked so well

Why is this true? Because their existence implies some sort of preferred basis that aligns with the dims of the neural network, which is surprising?

It's not obvious why their existence is so contrary to what we knew.