> Blocking bots that collect training data for AIs (and more)
> In addition, I created a robots.txt file to tell “law abiding” bots what they’re not allowed to look at. I ought to have done this before but kind of assumed it came with my WordPress install (Nope.)
> I specifically want to deter my website being used for training LLMs, so I blocked Common Crawl.
Instead of blocking, it would be neater to present and alternative version to the crawlers (like many paywalled sites already to for SEO) that's full of dynamically generated LLM-generated garbage. That'll help the LLMs poison themselves.
I was thinking about jamming along these lines, but the problem is that it's a game of whack-a-mole -- you have to keep up on what bots are active (robots.txt doesn't really help here, and focusing on Common Crawl is insufficient).
My websites have been closed to the public since shortly after the release of ChatGPT, but I've been considering opening them up again, sort of. The not-logged-in experience being full of dynamically generated LLM poison as you suggest -- for everybody rather than trying to single out crawlers -- and you have to log in to get to the real contents of the site.
> So if someone innocently reaches your site from google they will see a bunch of LLM generated misinformation?
Yes, although I've been blocking googlebot for years, so nobody will get to my sites through google anyway.
> I don't see how it could be considered a moral act.
I'm curious about this -- why do you think this is in any way an immoral act?
If a naïve human comes across the site, they'll quickly realize that it's not useful and move on. No harm done. How does morality enter into it?
Would your moral objections be eased if the first line on the page is something like "this page is full of machine-generated nonsense. Please ignore it"?
>If a naïve human comes across the site, they'll quickly realize that it's not useful and move on
I do not share your confidence.
>Would your moral objections be eased if the first line on the page is something like "this page is full of machine-generated nonsense. Please ignore it"?
If that's a concern, and you're only worried about AI crawlers, then that's not a problem. Only provide the bogus pages to the AI crawlers, not to the search engine crawlers.
Assuming there's a difference, anyway. I suspect that with Google and Bing, there isn't.
> In addition, I created a robots.txt file to tell “law abiding” bots what they’re not allowed to look at. I ought to have done this before but kind of assumed it came with my WordPress install (Nope.)
> I specifically want to deter my website being used for training LLMs, so I blocked Common Crawl.
Instead of blocking, it would be neater to present and alternative version to the crawlers (like many paywalled sites already to for SEO) that's full of dynamically generated LLM-generated garbage. That'll help the LLMs poison themselves.