We've indexed billions of pages already at you.com to avoid this situation. The majority of our apps like Stackoverflow and Medium etc, we've indexed ourselves.