> Of course, Google announces itself as GoogleBot. It wouldn't surprise me if they did a second stealthy crawl to detect cloaking. (But I think they are honest when they say they don't, and just throw cheap human labor at it instead by having people browse suspect sites.)
They actually run secondary tests that aren’t GoogleBot. They’re quite easy to detect on very low traffic sites. If you only have a few hundred users, all of which you know personally, and suddenly over the range of a few hours a few users using Chrome visit the page, while it’s not linked or findable in any search engine, just shortly after users using the googlebot UA visited it, and with certain usage patterns – it’s quite obvious.
Detecting the Android Bouncer’s VM is equally easy, although that only happened by accident because, due to an automated action, my app crashed in that and submitted a crash report that was unusual, and I managed to extract parameters that’d allow detecting it (similar with other android virus scanners), but I only cared about that to be able to split those "devices" into a separate category in the crash tracker (all my apps are GPL licensed, and don’t do anything evil anyway)
They actually run secondary tests that aren’t GoogleBot. They’re quite easy to detect on very low traffic sites. If you only have a few hundred users, all of which you know personally, and suddenly over the range of a few hours a few users using Chrome visit the page, while it’s not linked or findable in any search engine, just shortly after users using the googlebot UA visited it, and with certain usage patterns – it’s quite obvious.
Detecting the Android Bouncer’s VM is equally easy, although that only happened by accident because, due to an automated action, my app crashed in that and submitted a crash report that was unusual, and I managed to extract parameters that’d allow detecting it (similar with other android virus scanners), but I only cared about that to be able to split those "devices" into a separate category in the crash tracker (all my apps are GPL licensed, and don’t do anything evil anyway)