* They're better at graphics than that other stuff,
* Other auxiliary processors exist which beat GPUs on most kinds of that other stuff (though they aren't as accessible as GPUs),
* Most importantly, what sells GPUs is first and foremost their performance on graphics workloads. A GPU with stellar OpenCL performance but sucky OpenGL performance will be outsold by a huge margin by a GPU with the reverse characteristics.
Most graphics IS vector based math run through matrices. Same with physics. Every vertex is essentially a vector from the mathematical perspective. But you're right, that doesn't quite capture the massively parallel simple core arch that we see today.
