Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The irony of using JavaScript to detect scraping or bots when the majority of them not used to trick ads don't ever execute any of it because they are a better curl.


Well, if you're determined to prevent scraping, it's rather easy to hide content from non-JS bots: simply pull in the content via Ajax or "encrypt" is and perform the decryption via JS.

So thinking about how to ward off bots that do go the extra mile makes sense. (From a scrape-protection POV at least)


And it's actually getting easier with every new shiny web API. Want to make sure only the latest Chrome can retrieve the content of your website? Why not run a Webassembly computation that will yield the correct URL to fetch. Or what about a Web Worker? There are endless possibilities, and the only sane way to scrape / index the web in 2017 is a full-fledged browser.


If you try too hard then you can accidently hide content from search engines too.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: