i think the best way to keep your site working for legit users is serving static cached pages to suspected bots. decide based off something like cloudflare bot score.

crawlers get redirected to content that might be stale but its still useful for them and costs you almost nothing to serve. put a warning banner on it so if a user (or smart agent) accidentally ends up on the bot version they can click on it and get anubis checked for the real site.

of course this only works for people like me who want their work to be used for training ai.