You're making an argument based on popular vote, but black people and white people wouldn't be able to intermarry either if our system was entirely based on the will of the masses.
That’s why there is Congress. That’s why there is a Supreme Court. That’s why there are independent state and local governments. That’s why there are term limits.
http://en.wikipedia.org/wiki/Tyranny_of_the_majority