For this to be a downside, you have to believe that the person expressly wants to remove their email address from your view, but nevertheless wants to reconnect with you. That's possible, but I don't think a sane thing to build around.