Follow-up: To see which contacts you have uploaded (and to delete those) see: https://www.facebook.com/invite_history.php

Similarly, there's a page for messenger too (with the tip that you can turn off continuous uploading): https://www.facebook.com/mobile/messenger/contacts/