Sure, there are loads of easy solutions to this problem. Putting everything behind Cloudflare is the most common one, Anubis and its competitors are another. I could also block all data centers, countries with loads of CGNAT + botnets like Brazil, China, and probably India, and refuse any user agent not in the top 5 modern user agents with a little TLS fingerprinting to make sure.
I could probably also analyze all the paths across my servers and write some bespoke Nginx config files to filter out all requests with illegal queries, but that's harder, more fragile, and just leaves the servers open for the next time a scraper finds a new way to screw me over.
The bots I'm fighting have gone for the nuclear option, relying on botnets and spoofing to bypass basic filtering, so I don't see why I shouldn't do the same.
I could probably also analyze all the paths across my servers and write some bespoke Nginx config files to filter out all requests with illegal queries, but that's harder, more fragile, and just leaves the servers open for the next time a scraper finds a new way to screw me over.
The bots I'm fighting have gone for the nuclear option, relying on botnets and spoofing to bypass basic filtering, so I don't see why I shouldn't do the same.