This is harder to do than you think. An industrious developer who works on these things will quickly notice that their distributed crawler has started failing, look through logs and ultimately identify the problem. Then they'll switch to another set of IPs and continue, this time without requesting that link. The other issue is that you need to set an explicit rule that is aware of how each and every API endpoint should be accessed, and whether or not it should logically be directly accessible.
I say this as someone who has had to combat this specific technique - I'd suggest that if you believe it works, it's probably because you saw obvious scraping activity stop when you did it, but you were never aware of the more professional scraping that adapted to it or was never caught by it in the first place.
Whenever I've scraped a website, I am extremely careful not to request more pages than I need. A better method for blocking them is to flag requests for resources that do not proceed in a logical manner. For example, if you have an API endpoint that displays the information scrapers want, that endpoint should have a specific "route" through the user interface. If you find requests directly to that resource without first proceeding through the typical UI flow, that is more accurate for identifying a scraper.
This is still not foolproof, because the scraper can just script requests to the requires series of pages in order. But it's a good start for getting rid of most scrapers. The most effective method for getting rid of scrapers is IP agnostic behavior analysis, because it can catch e.g. scrapers trying to parallelize requests that increment across a proxy farm or requests that don't obey typical behavior constraints in the UI.
I say this as someone who has had to combat this specific technique - I'd suggest that if you believe it works, it's probably because you saw obvious scraping activity stop when you did it, but you were never aware of the more professional scraping that adapted to it or was never caught by it in the first place.
Whenever I've scraped a website, I am extremely careful not to request more pages than I need. A better method for blocking them is to flag requests for resources that do not proceed in a logical manner. For example, if you have an API endpoint that displays the information scrapers want, that endpoint should have a specific "route" through the user interface. If you find requests directly to that resource without first proceeding through the typical UI flow, that is more accurate for identifying a scraper.
This is still not foolproof, because the scraper can just script requests to the requires series of pages in order. But it's a good start for getting rid of most scrapers. The most effective method for getting rid of scrapers is IP agnostic behavior analysis, because it can catch e.g. scrapers trying to parallelize requests that increment across a proxy farm or requests that don't obey typical behavior constraints in the UI.