The last one, I did 13 crawlers to keeping comparing prices of all drugstore products online. I'm selling it to drugstore ecommerces who wants to know when their competitor prices and when they're doing promotions, if competitors prices are higher... Well, I just automated the job of a person that was doing this job manually every single day, looking into competitor's site
If it is illegal, you have to ask to google maps to remove all houses from your database and just let in the houses who gave the permission to it.
The both sentence means the same bullshit, they are both wrong:
Google Maps: "The front-door of my house is faced to a public street. That does not mean you can take photos of it on Street View and use a very smart OCR, to read my house number. Not to mention that sometimes you give my house photo to others by Captcha asking the house number, c'mon!"
Linked: "My CV is half-public. That does not mean you can take crawler of it."
... This is only a cool discussion in North Korea or maybe in China. Not in the rest of the world.
I hate to be the bearer of bad new but...maybe. A reading of the Computer Fraud and Abuse Act could make robots.txt legally enforceable. And given the government's approach to CFAA cases a very aggressive interpretation, under the right circumstances (for example, when it provides evidence that the scraper knew that scrapint was not authorized), seems like a real possibility.
Among the many other things CFAA criminalizes, it makes it a crime to "intentionally access[] a computer without authorization or exceed[] authorized access . . and thereby obtain[] information from any protected computer;"
A "protected computer" is, among other things, a computer "which is used in or affecting interstate or foreign commerce or communication." That would probably cover just about any server on the Internet.
I wouldn't be too fast to jump to conclusion robots.txt legally enforceable. You would need to cite prior case law's. Without any case law's it make decision on a law error prone at best.
Saying something is illegal under the CFAA isn't really a stretch, it's one of the broadest statutes ever written. If you have an employee handbook at your job right now and are using the internet for your own entertainment you may be violating the CFAA.
The interesting question is are you violating the CFAA in a way that will cause the executive branch to exercise their discretion to prosecute, and moreso, is the CFAA even constitutional.
I'd talk to a lawyer. There are presumably some legal arguments that might apply. There definitely have been cases revolving around it, but I have no idea what the outcomes were.
From a moral point of view: robots.txt states an intent, and intentionally ignoring that is not a nice thing.
And often, "not a nice thing" translates into legal action. So, if this matters to you, you should find an expert :)
> From a moral point of view: robots.txt states an intent, and intentionally ignoring that is not a nice thing.
As a lawyer asked inane questions by friends, a common answer is: "I am no sure if its legal to do, but I'm absolutely sure you're a asshole if you do it."
It's probably an algorithm to get you some decent % of views early on to see what % of upvotes you get so it can move your post up or down accordingly.
The last one, I did 13 crawlers to keeping comparing prices of all drugstore products online. I'm selling it to drugstore ecommerces who wants to know when their competitor prices and when they're doing promotions, if competitors prices are higher... Well, I just automated the job of a person that was doing this job manually every single day, looking into competitor's site
If it is illegal, you have to ask to google maps to remove all houses from your database and just let in the houses who gave the permission to it.
The both sentence means the same bullshit, they are both wrong:
Google Maps: "The front-door of my house is faced to a public street. That does not mean you can take photos of it on Street View and use a very smart OCR, to read my house number. Not to mention that sometimes you give my house photo to others by Captcha asking the house number, c'mon!"
Linked: "My CV is half-public. That does not mean you can take crawler of it."
... This is only a cool discussion in North Korea or maybe in China. Not in the rest of the world.