Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

That isn't the same at all. A publisher cannot use robots.txt, and much less paywalls, to indicate a part of text that can be shared in syndication.


A paywall can. The page displays the snippet the publication is allowing to be shared, while the paywall hides the rest. I believe this is what a few of the bigger US newspapers are doing right now.


Ok, but that would require regular readers to have credentials for the paywall. I understood the discussion to be about scraping publicly accessible sites.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: