I think you need to be at a certain level of karma to 'earn' the right to downvote.

Flag is for inappropriate content / behaviour.

You get a downvote button, separate from "flag", once you reach a certain karma level (I forget the number, though)

Karma threshold for the downvote button used to be 500, IIRC. Only 449 upvotes to go!

I just need to post one banger Sam Altman tweet and I'll be sorted lol