While it's always a bit annoying to see a meme take up article slots on Hacker News regularly, I don't mind this one. I see sites like Mahalo as doing massive disservice to huge numbers of people --
* thousands of AdWords advertisers that have paid for their ads to be matched with content sites, not scraper pages with a huge ad to text ratio
* thousands of publishers whose work is being scraped, aggregated and outranked without as much as a backlink
* millions of web searchers that are hitting these pages instead of the real sources of the content they were searching for
And calling out companies that harm the fabric of the web for everyone else is worth doing.
The question is: why doesn't google do anything about this? There are countless such pages. Why not just blacklist domains? Or artificially give them lower pagerank?
I work for Google, and while I don't know anything about this particular situation, one invariant that I've been consistently impressed with is that the ads side of the company doesn't influence the search side of the company _at all_. I've seen well meaning ads support folks emailing people in search quality about an innocent problem with one of their customer's sites, and getting sternly told that they should never again send an email to those mailing lists.
Offtopic but I for one would like to see Google do this for personal information or at least provide the option to do so i.e. info from one G product does not leak to another without explicit opt-in.
Isn't there a search engine out there that takes into account the advertisements vs. content ratio? It would be trivial to get rid of Made for Adsense sites by scanning for the Adsense code used and seeing how often it's used on a page and the percentage of the page it takes up.
Google allows you to have three ads though. If you had a page with three ads and the best written 500 word article on it's topic, your definition of MFA sites would have it banned even though it is obviously a ton better than most sites around.
For smaller publishers, it's usually three blocks of ads plus two blocks of link units. For those not familiar with AdSense, link units are the horizontal or vertical lists which take the visitor to a search results page rather than straight to an advertiser's page.
I think a strict definition of MFA is always going to be a little grey, but there's clearly something wrong with pages that are nothing other than widgets, scraped content and the like.
An old boss told me, "When you're the man, the rules don't apply to you."
Not to say that Calacanis is the man, but to illustrate that rules are there to suit the rule makers and in this case, it suits the rule makers to ignore their own rules.
They went through a bunch of different iterations and stuff before they established as the site they are now. And look at the funding of Mahalo and Google. Same early people.
they do block domains, at least from their organic search engine results.
i accidentally got one of mine blocked when playing around with some random software that produced a site that was somehow against their TOS. i didn't have it live long, but long enough for the spiders to grab and flag. still waiting on a reconsideration request.
why not mahalo? who knows. if i had to venture a guess, i'd say that the larger the site and the more revenue google sees from that site due to adwords and the like, the higher up the chain the decision to block it has to come from.
I think Google prefers to programmatically detect and blacklist behaviours, without too many innocent casualties - even if working out how to do that takes longer. They aren't interested in playing whack-a-mole.
Can they then at least pick one scraping site from all the others (to prevent duplicate information)?
Although this is perfectly legal, there is quite a few big sites that scrapes Wikipedia and adds ad-sense. If you are searching for a narrow topic, page 1-2 is WP and WP copies.
As a sidenote - I see that Google Search has upvotes, comments and deletes in the search - yet it does not seem as if anyone uses it. Why? Why can't the user at least blacklist sites?
>If you are searching for a narrow topic, page 1-2 is WP and WP copies.
It would be good to have a "remove all pages with Wikipedia content" button on searches. Sometimes it seems half the results are simply Wikipedia copies.
They do often play whack-a-mole at the lower level too...but they also take human reviewers into account so it is not so expensive on engineering time.
Its really a shame how low Google's standards are for AdSense. I've tried advertising in the content network several times, but each time I end up wasting so much time blocking MFA sites that I give up.
Here's one thing to consider : Google's algorithms for detecting the actual value of clicks for advertisers has improved greatly over the past few years.
If Mahalo’s traffic was utter crap, it would be dropped.
"If Mahalo’s traffic was utter crap, it would be dropped."
That's not even close to true. If it were, there would be no AdSense for parked pages program. Their eye is still very much on their own bottom line, not the advertisers.
As an extension of that, sites which get most of their traffic from search often have the highest ad CPM values because the ad targeting is so relevant on the AdSense ads. And this is true even moreso now that Google is also using the referrer for ad targeting
Scraper sites do quite well on the ad revenues front because most of the traffic is search driven, and because the content ends up so bad that the ads look all the more appealing.
"Their eye is still very much on their own bottom line, not the advertisers."
I know for a fact that Google is balancing advertiser ROI against advertising spend and ensuring that they don't upset the apple cart.
However you are absolutely right that Google is maximizing their profit – having the ROI numbers in hand lets them drive down overall sales-from-adsense margins while still keeping the customer.
So my original point is that if Mahalo was flat out non-performing crap, it would be tossed.
An argument could be made though that they use the Mahalo stuff to grow ad inventory, while balancing out the ROI to their advantage since it likely converts on the lower end of the scale.
There's also the possibility that aggregator sites are actually useful for some portion of the Internet users (read: the "facebook login" set) and the advertisers that cater to them. Too bad "we" have to share Google with "them".
Not to be too cynical, but after several years in web marketing I've found that in this case "we" definitely do not outnumber "them." Not by any stretch of the imagination.
My conversion rates from the content network have never been as high as search. I try new campaigns targeting both networks every once in a while but always end up either opting out or setting a much lower bid on the content side.
I do not think anyone is asking for all of Mahalo to be blocked. The major issue is the insane amount of auto-generated pages that are not given a noindex call and are also added to the site map xml file.
These actions appear to be conscious choices to make sure that google will index these pages, which according to several sources, have nothing but "scraped" content and are therefore spam.
Google is providing a free service in a field in which there is intense competition. I don't see what basis anyone would have for filing a lawsuit for not being indexed in Google.
They could argue that Google has a monopoly on online search, and that excluding people from their search results is an illegal restraint of trade. I don't think that such a case would be a cut-and-dry victory for Google if the spammer (as in this case) has the money to fight them.
Ridiculous. I don't know how the law works but I do know that if Google was to somehow lose a case like that, their entire search engine would be worthless incredibly quickly with grey hat/black hat people going crazy.
Although I understand your points the AdWords and search issues is googles problem. People pay for a product with flaws in it and google is selling, not Mahalo. Mahalo has the freedom to do whatever they want with their pages and they can't be held responsible for what google in it's turn does with them.
Now, stealing other peoples work, that is obviously Mahalos doing and should be a case for the court.
I'm not really sure what the point with this article is. Mahalo spams google - so what? It's up to them to do this and up to google to prevent it. Business is business, even on the internet.
It's really too bad Google doesn't allow you to simply blacklist domains from your search results permanently. The thing that frustrates me the most about these SPAM sites is the fact they're constantly popping up and my down voting of the result seems to do absolutely nothing unless I'm using the exact same search query. Just let me blacklist Mahalo and other sites like it permanently. Better yet make it possible to subscribe to a blocklist so the community can pool it's resources and fight back.
I've requested this, too. I even switched to Yahoo! for a few hours because they let me blacklist expertsexchange.com (since changed to redirect to experts-exchange.com). I don't see the option on Yahoo!'s page anymore.
With Google custom search engine you can create your own personalized search service...built off the back of Google, but while delisting any sites you do not want results from.
And it only takes about 30 seconds to set one up. Then you embed the search box wherever you like.
Mahalo is not particularly interesting, not particularly evil, he doesn't really do anything, and yet - we keep talking about him and putting his shit on the front page of hacker news.
It's a mediocre aggregator/linkfarm, with some mechanical turk style incentives for humans to contribute, and a nice chunk of $ in the bank. Just ignore/mock him for another year or two until the funding runs dry.
You are absolutely right, it is mediocre crap. The issue is that through the spam techniques described in the article, he can now gain undeserved top 10 rankings for many phrases using pages that have no business being there.
If you are not part of the web development community then these discussions will most likely bore the hell out of you. However, if you are, and if you are aware of how many innocent sites Google bans or penalizes on a daily basis, or AdSense accounts that get canceled with no appeal for offenses much less than his, then this stuff actually matters.
One of the things I really miss in google is a persistent blocking preference a.k.a. site blacklist. Mahalo would go straight in there, along with expert-sexchange, sedo parking and a few others.
That is the angle Jason used when he created his steaming pile. It doesn't mean that anyone in the industry agrees with Jason's strategy.
And if you want to place the blame where it belongs remember that Google is the company funding all this content scraping with their ads programs.
I just tried searching for a recent post from an official Google blog (about AdSense using referral data for more relevant ad targeting) and found a scraper site with their ads outranking them for their own content. Pretty sad.
Not everything that uses the phrase "SEO" should be painted with the same brush. A lot of SEO best practices are making the HTML markup more semantic and human-readable. White-hat SEO dovetails pretty well with a human-readable web and accessibility standards, and is one of the better business cases for kicking the Flash habit. Just because Calacanis makes his name by abusing loopholes in the algorithms doesn't mean that's the only sort of thing that the name "SEO" applies to.
No, different datacenters will show different results... sometimes very different. It also matters if you are visiting Google.com or one of the country variants.
According to what Jason said in another comment, however, all of their pages are listed in their xml sitemap, and all of those are listed in a master xml sitemap index located here (warning! huge files if you follow the links in the first one!):
this is getting really old and we're not interested in doing anything black hat or even gray hat. as such we're doing the following:
1. we're removing (or building out) any page in our system created by our users with under 200 words of original content. This will take a couple of weeks but it's tarted.
2. we're not letting users create stub pages (short pages) until we can noindex them and put them in a different directory (i.e. /stubs/) so google can easily tell the difference between them.
these pages are < 1% of our revenue and low single digits of our traffic. we don't benefit from them materially, and I think we're being targeted by Aaron Wall and other SEOs for my "seo is bullshit" comment from 2005 or so.
I guess that is fine... I gotta live with the ramifications of what I say. however, for the record I don't believe that SEO is BS any more... when i said that it was when we were building joystiq and autoblog and we spent zero time on SEO.
All that being said, we're being targeted by a small group of folks who want to take us down. we're only going to get stronger from this because our hundreds of contributors are rallying around building out the short pages.
Topix, Kosmix, NYTimes and Zimbio are all making quality topic pages and are not getting attacked over it. not sure why there is some double standard.
regardless.... this is not a material thing for us. we're flushing all these pages and moving them to a different directory going forward so that search engines know where they are located (i.e. /stubs/ ).
thanks for the ass kicking.... having a horrible day today over this.
Jason, did you even read my article? This isn't about the traffic those autogenerated pages get, it's about the fact that through the minuscule amounts of PageRank that they are each capable of grabbing, you are now able to rank your mediocre pages with absolutely zero influence from the rest of the web.
We're not talking about stub pages, it's all the fully automated bullshit that you are generating. They not only need to be deindexed, they need to be nofollowed or removed altogether.
How is it you are out there playing the wounded puppy when apparently you haven't even read the articles or followed the reference links? You can't just skim this one and then craft a rebuttal and think you've addressed the issue. There's a lot of data in those paragraphs you apparently just skimmed over (if that even).
You have over 500,000 pages listed in your xml sitemap, and Google appears to have over 330,000 of them indexed. Click on this link, please, and actually go look at 10-12 of the pages we are talking about here:
Tell me how long it takes you, just by clicking through, to find even 3 pages that have any human interaction in them whatsoever.
Maybe, just maybe, you really don't have a clue what is happening. I personally don't believe that's the case, but if so then whoever it is you have working for you that set this up knows how to spam like a pro.
we're removing (or building out) any page in our system created by our users with under 200 words of original content. This will take a couple of weeks but it's tarted.
or
i'm also getting a list of every page under 300 words and having the page managers build them out in 30 days or deleting them.
There is a big difference between stating an observation and an attack. You can (and have) dismissed the Google guidelines as irrelevant. But you shouldn't slag off others as spammers if you are going to do far worse.
You are the one who attacked people. We are simply tracking how well you performed in your efforts.
Hold yourself to your own standards or close your mouth. Period.
Aaron: let's be real for a moment. You're a bit of a troll and you're certainly obsessed with me to an unhealthy level. I'm flattered, but it is getting old.
You pointed out some minor stuff we could do better, we're doing better. Now go back to your basement and hit F9 on Chatroulette.
Can't you just see Matt Cutts, sitting in his office, connecting to sites like HN through multiple proxy levels creating fake Matt accounts so he can leave truthful comments without his bosses at Google finding out... and when they question him he can just say, oh, of course that wasn't me!
Sure if Mahalo was really screwing with Google, Sequoia backing wouldn't help, but to say Google + Sequoia aren't incredibly buddy-buddy would be difficult to say.
It's obviously of interest and importance to a number of people involved in this field or troubled by poor quality material showing up in Google. If you're not one of those people, it's pretty easy to identify these links and not upvote them or visit the articles/comments, etc.
There are countless articles on HN that I have no interest in (e.g., I don't even know what Clojure is), but I just don't click through to them.
A lot of the comments in here seem like more of a personal attack than anything else. You might as well change the title of this post to "Jason Calacanis ruined the Internet" or something to that effect..
Can we stop the drama already? I think we're going to need a hose to control this mob.
* thousands of AdWords advertisers that have paid for their ads to be matched with content sites, not scraper pages with a huge ad to text ratio
* thousands of publishers whose work is being scraped, aggregated and outranked without as much as a backlink
* millions of web searchers that are hitting these pages instead of the real sources of the content they were searching for
And calling out companies that harm the fabric of the web for everyone else is worth doing.