But you can make an equally succinct argument in the opposite direction, namely to treat time values as polar coordinates. Then they indeed form an abelian group:
a + 0:00 = a
15:00 + 13:00 = 02:00
12:00 + 12:00 = 00:00
The examples are for hours only, but you can extend it for minutes too. Showing that time values on a clock forms an abelian group under addition is a textbook example everyone uses when introducing group theory.
You're right, but time values with timezones are completely different types from time values without. They follow different rules and need to be treated differently.