"You can simply decline the conversation" and "it's not a big deal" are entirely subjective and based on the assumption that being extroverted is normal. Your rules of etiquette are that I need to go out of my way to give people the opportunity to talk to me so that I can tell them not to rather than just signaling to them that I don't want to (which saves us both time and lets me quietly enjoy some music on the meantime), which is just bizarre.