I agree that it's not a downside. It's just an interesting effect and experience! As far as Pimsleur, from people's comments here I'd certainly consider using their materials.

Regarding your experience in Japan, I think that's a demonstration of why learning foreign languages (and traveling)is so significant: It creates new cultural ties and has the power to jolt people out of their preconceptions and prejudices (both for learners and native speakers). It opens minds on all sides.