Yes, but it's slower and more expensive that way. If you want to run a few hundred parallel spider processes with full browser in each, it's neither easy nor cheap anymore. It takes a lot more resources than running it headless, plus adds a significant overhead to automate and control all that.
Ultimately, no protection is unbreakable, there's a work around for almost everything. If your site has thousands of pages (e.g. big online stores, that are common target for spidering) it's probably the best approach to make things as slow and complicated for spider author as possible. That's exactly the same logic like with captchas, they can be broken fairly easy nowadays, but they'll still slow down the spidering rate and pump up the cost.
> If you want to run a few hundred parallel spider processes with full browser in each, it's neither easy nor cheap anymore. It takes a lot more resources than running it headless, plus adds a significant overhead to automate and control all that.
This is not true. I've seen it done in AWS for less than $2,000 monthly, and I can do it from my home for less than $500 per month (minus the costs of my workstation and networking gear, which you'd amortize over the expected lifetime of the project). I have a home server with 128GB of RAM and an i7-6900K, with 125 static IPs and a bunch of Ubiquiti networking gear. You don't even need that much memory or compute power, but I also use my workstation for other research projects. I use my own static IPs to parallelize without having to sacrifice latency or cede control to a shady proxy farm.
It's a pretty straightforward setup - each static IP is given its own route across the switches from the gateway, and the switches have link aggregated connections for 40GbE bandwidth. You have pub sub and queuing, and each scraping target is its own headless Chrome process. The requests to the target from each process are round robin sent across the available interfaces. Then you've got parsing and a local database.
I'm not a particularly invasive scraper (I make my User Agent deliberately obvious with an explicit way to opt out), but it's really not true that it's prohibitively expensive. This is a pretty cheap setup; I don't even profit from the work, I use it for personal research. If someone was actively selling valuable data, this would absolutely be worth it.
Curious about your home setup, what ISP are you using at home that lets you have essentially a /25 block of public IPs, let alone 40GbE of bandwidth? Especially if this is costing you $500/month.
I have Verizon Fios set up as a business account to my home. However that bandwidth is for SAN to workstation data processing and analysis. There's only half a gigabit of external internet bandwidth, but I have hundreds of terabytes of data (and high tens of gigabytes are downloaded per day) in local storage.
With 128GB of RAM I can only load small amounts into memory for targeted analysis. Processing the rest of the data requires loading it directly from storage. To improve I/O performance I parallelize the transfer across link aggregated ethernet interfaces. Naturally that would cause disk reads to become a bottleneck; to take proper advantage of the network transfer speeds I hold all data in RAID 0 with 7200 RPM drives.
Certainly curious about getting 40GbE bandwidth, but the IP addresses doesn't seem like much of a hurdle. I was looking at business broadband a couple of weeks ago, and getting 13 static IPs was only £5 extra a month. Going all the way up to a leased line included unlimited "subject to internet regulations".
It's a matter of a personal perspective. If you're based in US and have a luxury of working with enterprise clients than sure. On the other hand, few years ago when I was doing data scrapping as a freelancer, price tag of $2K just for the infrastructure would be a huge show stopper for the most of my average clients back then. We were charging them way less than that.
Every spider process is a browser tab. People have hundreds tabs opened without any problem on ordinal computers. On a dedicated server with 128GB of ram you can run thousands of those and it will cost you 100$-200$/month.
True -- But the user is not interacting with all 100's of those tabs at the same time -- something an automated scraper would be.
Firefox + Chrome both lower the priority of background tabs, and may be doing other tricks so the background tabs can stick around and be switched to quickly.
Main purpose is to render the page, crawl the dom for the data, and then load the next page as fast as possible... so it's equivalent of having a thousand bookmarks and opening them all at the same time, and as soon each loads running the scraper script and reloading them with the new link. Try it for yourself, it's easy to script, and check the load...
To be honest (and speaking from experience), the only thing that would actually make life hard for a scraper is the last suggestion. I've never personally dealt with that, which is the only reason why I concede it.
Machine generated HTML is a pain, but I wouldn't call that "very difficult" - it's more like "annoying" in the same way that having to parse HTML instead of finding a neat JSON endpoint is.
And I'm not sure how randomly inserted HTML elements would help - if you're already parsing the HTML, you can extract the relevant data. Unless you're trying to parse the HTML with regex, in which case: https://stackoverflow.com/questions/1732348/regex-match-open...
The first two bullets combined can make it hard to parse the page reliably, but it's also making very hard the maintenance and using of any 'good' javascript and/or css too. Not worth the trouble usually.
In addition to dsacco's reply it should be mention that using images instead of text hurts accessibility and, depending on your industry and country, it may also run afoul of equal access laws.
Depending on a page and data it can make life a bit harder for the spider, but all that can be taken care of.
IMHO the best way to stop spiders is to control access to your pages. Require users to register and then login each time to see the data, and then have some pro-active monitoring & counter measures in place, monitoring the patterns per users, and per IPs, and across the whole system for an unexpected increase in activity or bot-like behavior (moving too fast or at unusually uniform speed, following links too sequentially, etc.).
If it's not an option, the next best approach is to make it hard for spider to fetch the whole dataset. Don't let user just browse all the pages, instead force them to use search, it makes it much harder to cover everything (and it will not affect normal users too much). You can, for example, return just the 100 products at once, and ask user to refine the search if there's more products than that. Then you can create some monitoring system to watch over unusual search queries that look like dictionary attacks. I've actually built a system like this for one client and it worked fairly well (combined with a few more tricks they were already using).
Of course, all of this can be tricked too, it's all a game of cat & mouse, trying constantly to outsmart the other side.
I don't believe Google's Recaptcha2 has been broken yet (at least not publicly). Though you can parcel it off to low wage workers who click on captchas all day long.
Ultimately, no protection is unbreakable, there's a work around for almost everything. If your site has thousands of pages (e.g. big online stores, that are common target for spidering) it's probably the best approach to make things as slow and complicated for spider author as possible. That's exactly the same logic like with captchas, they can be broken fairly easy nowadays, but they'll still slow down the spidering rate and pump up the cost.