I wouldn't be so sure about that. We know that FB works with third-party companies to track offline purchasing behavior of its users (data which is supposedly anonymized).

So, I wouldn't think it a far leap that Facebook might correlate non-member invitee e-mail addresses with, say, the activities/interests of the inviter. This could be happening right now for all we know, or it could happen in the future. Once they have the data, it's their choice.