This was what I really wish I had done. I tallied up all of the electives I didn't need/benefit from on top of the B.S.: it came up to around 12 courses, enough time to fit another major and more. A B.A. would have cut into that even further, but I was worried about the perceived harm it would have on me in the job market, even though now I realize that was such a false fear.
I was worried about the same thing, and I thought a lot about switching. Ended up keeping it because I enjoyed my other classes more, and got lucky in that nobody cared.