Nobody likes to feel lonely in a crowd. Everybody prefers the company of their herd.
However there is plenty to learn from those who are not like you, even if you struggle to understand them, or they you.
Variety is the spice of life. Avoiding the company of most others -- or the correlate of exclusively preferring the company of one group -- risks stagnation.
It's not elitist to like something that makes you happy, even if that thing is being around a group of wealthy, intelligent people. Elitism is about how you treat the other people. Being elitist would mean refusing to socialise with any group other than the one MIT crowd because they're too 'dumb' to be fun around. I don't think that's what jkestner is suggesting; that he found a group he's happy with doesn't mean he refuses contact with the other groups.
If you two were friends, this might come across as playful ribbing ('taking the piss' as they say in the UK). But since I don't think you are friends, it comes across as mean, on your part. And you demonstrate (usefully, I think) the mistake of confusing complex and/or erudite speech/vocab with arrogance.
But actually, I think the "attribution of arrogance" is a post hoc rationalization. In truth, some people feel utterly threatened by complex, erudite speech. It is hard to be confronted with a person who is having a thought that, without a lifetime of reading with intellectual curiosity (and a good memory), cannot be fully appreciated.
Speaking intelligently can be (often is) like serving a plate of delicate, complex food in front of someone who isn't interested in this "pansy bullshit" and just wants a hot dog. And it hurts the same way, because what you prepared for them really is amazing, and not only is the effort not appreciated, but the recipient interprets the offer as an insult. This is a very painful experience, and you learn to take great care to learn who is expecting a gourmet conversation and who is not.
Having been through that exact situation, I can't agree enough. With time one may learn to sense if the context is right instead of suffering a blow. Unless he's ready to invest a lot to reach common ground. People do change, but it's a bet.
There is complexity that obscures and complexity that enlightens. Some concepts can only be conveyed adequately by using intricately detailed explanations. The same is true for simplicity -- too much simplification and a discussion devolves into a battle against strawmen. Officials are guilty of this as well.
While I understand where you're coming from and agree to some extent, I think there is a difference between "speaking `intelligently`", "having an intelligent conversation" and "leaving a comment on the internet".
I would argue that there is no such thing as the first without the second, because you are not 'speaking `intelligently`' if there is no one who understands you and is able to make a conversant reply. In such a case you are only being arrogant.
The insecurity is strong with this one. You're projecting that the person's literary references are "name dropping things in largely irrelevant contexts". There is no basis for this assumption, other than your obvious bias to jump to the conclusion that this person is elitist and goes out of their way to seem elitist. Why? Are you sensitive that you didn't get into a school like MIT? Neither did I, but nothing he is saying comes across as even remotely elitist to me.
The desire to have a community and shared culture doesn't make one elitist. The GP could have as easily said they found bookish people they could relate to by becoming a librarian. Would that have sounded elitist or was the MIT name-dropping what upsets you?
"Elite" simply means "drawn out of the larger pool". Sure, there is a lot of social baggage around that word, as in a sense of superiority, but the in the general case there is no such implication.
What's wrong with hanging around with folks who get your jokes (and vice versa)? They don't have to form the totality of your identity.
Somehow it seems uncontroversial when someone finds a group of, say, fans of their favorite team (or rock band) and says, "now I am with my people!" Is there some peculiar difference in regards to books or maths jokes?
Is it elitist to want to be around other kids you can mentally relate to? I suffered in school until I transferred to a 7th grade public magnet school. For the first time in my life I felt like I belonged because I could be myself with the other kids, most of whom happened to be from poor immigrant families (it is not a class or culture issue).
Same thing with my wife. When we met, I discovered that I didn't have to censor the literary references... she read way more than I.
I love being around people I can be myself with.