You are assuming those that didn't vote wanted to remain.

Not voting shows disengagement, indecision, or disdain. Anyone regretting not voting at the moment is likely reacting to the tone of the resultant hyperbole than any actual knowledge or opinion.