Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Yes, I am aware. The point of my comment is that Wikipedia obviously does not block wget.


The point is that if it becomes a problem they'll just block that particular useragent.


The point is that you can use -U to specify arbitrary user-agent strings, and -E robots=off to ignore robots.txt.

User-agent blocking is completely braindead. It does nothing at all. The fact that somebody in 2012 can possibly think it works is astounding to me.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: