I also grew up using pansy as in weakling, as in like the flower. Gay men can be pansies or not. Straight men can be pansies or not. Sometimes I am a pansy. Sometimes I am not.
There is a connection to offensive meanings even if it's not strictly a slur. It's the wrong word here anyway--really the opposite of what I would take as the intended meaning. The behavior being argued here certainly isn't weak; those arguing against it are saying that it's overly aggressive.
This is a homophobic term of abuse. Please reconsider attacking people by saying that they are gay.
http://www.urbandictionary.com/define.php?term=pansy