Because it's a tool in search of a use case (or many use cases) and mathematics is the most natural use case for it. Mathematics is by definition the art of putting words on a page in a rigorously defined "correct manner" (i.e. in the form of a valid logical argument, a proof) and all LLMs do is put words on pages and evaluating if they're good words is by far easiest when there is a strict definition of right and wrong.