I'm running some tests currently https://deep.43z.one (how deep bots will crawl)... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		h43z 11 days ago \| parent \| context \| favorite \| on: I posted to HN, and then my website was flooded wi... I'm running some tests currently https://deep.43z.one (how deep bots will crawl) https://stalling.43z.one (how long they'll keep the connection open)
		help

robotmay 11 days ago [–]

Ooh that’s fun! I’ll have to add some stats like this to mine too. Anthropic keeping connections alive for 2 hours is wild. I don’t think they fall into my trap as I believe they respect robots.txt, which is at least an improvement on Meta.

Consider applying for YC's Winter 2027 batch! Applications are open till November 2.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact