Well, these hits on your ssh port target only the most low-hanging of the low-hanging fruit - i.e. common username/password combinations. Nobody's brute-forcing you anything, they're just trying some common user/pass pairs at random. Disable remote login by password in lieu of key authentication and you can sleep tightly while listening to the soothing hum of failed login attempts crashing upon your airtight SSH configuration.

In the long run, I've found that years pass and connections configs get lost and you forget which fancy port you used for your SSH connection on that server. Maybe YOU have an ironclad convention, but your co-worker had another one, and you can't remember what port he used. And he's left the company or died or joined a cult.

Kids, leave your SSH ports alone. A config is just a config. But keys are forever.