Instead of a 90's style "search aggregator," how about a "SEO-less search" website? Basically, it would be a search engine that would aggressively quash SEO spam. It would be monetized through micropayments up-front through services like Paypal, Amazon Payments, and Google Checkout. There would also be search-results ads.
Here's the problem - how do you 1. define SEO spam, 2. detect SEO spam. It's not as simple as saying "deindex anything that has optimized title tags or lots of links or shows adsense" because plenty of useful websites optimise for search engines, show ads and have lots of links. What you'd end up with is the dregs of the web, because although you'd reduce the number of spam sites in the results, you'd likely also reduce the number of useful sites in the results because you're specifically targeting and removing the sites that optimise for search engines, which is 99% of the most successful sites on the web today.
SEO is not an inherently bad thing, it's valueless, spammy content that's bad. The real challenge is in determining whether a page is useful, original and relevant to the search query, and that's where Google et al are falling down at the moment.
It's not as simple as saying "deindex anything that has optimized title tags or lots of links or shows adsense"
No, it's not. But undoubtedly, SEO spammers are leaving behind the same kind of clues that email spammers do. Using techniques like Bayesian classifiers should be able to classify such pages. Just detecting all the pages that are slightly edited/SEO-optimized stackoverflow answers would be a good start for me.
you'd likely also reduce the number of useful sites in the results because you're specifically targeting and removing the sites that optimise for search engines, which is 99% of the most successful sites on the web today.
I'd be putting useful sites in the "useful" corpus.
SEO is not an inherently bad thing, it's valueless, spammy content that's bad.
Agreed, just as email marketing is not an inherently bad thing, just spammy emails.
There are billions and billions of pages in the index, let's say they produce a page classifier that can correctly identify spam an unrealistic 99% of the time. They'd still misidentify tens of millions of pages. Regarding your second point, how do you determine usefulness in a way that can't be gamed? My point is, this is a very, very, very hard problem to solve at their scale, a lot of posters on here are trivialising it.
99% classifier? So I get to have a search that's like Google/Bing for all the non-spammy searches, and it only gets 1/100th the spam for the spammy searches? I think I could sell a proposition like that!
Right, but you'll also disincentivize legitimate websites that happen to make their money through such ads or affiliate links. Where do you draw the line?
The solution is simple here: have an alternative search engine that makes their money a different way. Even though it would just be a niche, I bet there's a pretty healthy income to be had for a highly SEO-spam resistant paid search engine. (Supplemented with Google-style search ads.)
A small minority would use such a search engine, so the impact on legitimate sites would be minimal.
If your new search engine gets enough traffic to be worth it, spammers will just target your search engine too, and you'll have exactly the same problem Google has now, but presumably with far fewer engineering resources to throw at the problem.
If your new search engine gets enough traffic to be worth it, spammers will just target your search engine too
I think you're dead wrong here. I only need to catch a sliver of Google/Bing traffic for such a search engine to be worth it. Let's say I get a 1% share of traffic -- which I would characterize as wildly successful -- why would it be worth the while of SEO spammers? How many resources can you throw at something worth 1% of your income?
Because spam is cheap and easy, far easier to produce than to fight, you can spam in your sleep with automated tools. Let's take your 1% example, that's millions of searches a day, well worth a spammer's time and effort. Some people will spam your search engine just because you say they can't, and that kills your unique selling point straight away.
Because spam is cheap and easy, far easier to produce than to fight, you can spam in your sleep with automated tools.
What you are saying is that that someone could algorithmically determine the profitable targets of spam. This opens the door to an economic hack of spamming. Simply run the same algorithms yourself, then if the spam-profit potential of a particular search is above a certain threshold, simply present a "hammy" search based on well-known authoritative sites.
This scheme would work, precisely because spam is cheap and easy.
Some people will spam your search engine just because you say they can't, and that kills your unique selling point straight away.
This only works against the scheme I just described if you can spam my method for determining "well-known authoritative sites." If you curate that, then you can't. What I'm advocating is actually switching between curated and algorithmic search depending on spam-economics. This means that spamming such a site might end up being equivalent to "internally spamming Wikipedia and StackOverflow." I don't think that's going to be viable to SEO spammers for going after an extra 1% income.
Ah yes, "curated search". That's been tried, and it's literally impossible to keep up to date, web pages change all the time, what was an authoritative site yesterday is a phishing site or online pharmacy today. And you're wrong about the economics of spamming wikipedia or stackoverflow, spammers already target sites like that, there are even spammers who target HN, and your model would just make it even more worth it for them.
Ah yes, "curated search". That's been tried, and it's literally impossible to keep up to date
Weak. I'm not advocating "curated search" for everything. Your point is like trumpeting dynamic RAM over NAND gates and not mentioning caching.
And you're wrong about the economics of spamming wikipedia or stackoverflow, spammers already target sites like that, there are even spammers who target HN, and your model would just make it even more worth it for them.
Wow. that's kinda fuzzy thinking there. For it to be worth it for spammers to specifically target an anti-SEO engine, it would have to be successful in the first place, which undercuts the point you were trying to make.
Also, even though spammers target wikipedia, HN, and stackoverflow, why don't I perceive spam as a problem?
ok, I don't believe we're getting anywhere. My challenge to you is, if it's so easy, build it. Prove me wrong and make your millions while you're at it :)
If Google serves AdSense, it already knows how long you as a reader spends on site before clicking through an ad. Presumably, if statistically speaking, visitors have high probability of hitting the back button or clicking through immediately, then it is a red flag that the content wasn't useful.
That was why I wrote in some other thread that years ago I worked on a toy search engine, it was exactly for that purpose.
I'm still pretty sure that it's viable but the results would have to be at least as good as google for 99 out of 100 searches or so and even with the SEO spam killed that's a pretty high hurdle.
How about something that defaults to Google or Bing for searches that aren't spammy? How about coupling this with a browser add-on that detects spam in the search results (or uses a list of "known to be spammy" search terms) then does an auto 2nd query to the SEO-resistant search engine in those cases? This makes the hurdle quite easy to reach. Essentially, it gets you the best of both worlds.
I'd be tempted to monetize this through search ads and micropayments based on Amazon AWS hosting costs X2.
Through the search results it's involved in. Actually, you wouldn't detect this for single keywords, but for keyword tuples, which is exactly what SEO spammers who are scraping stackoverflow are optimizing on.
this implies that you can detect the spammy sites in the first place, take a ridiculously spammy keyword like "buy viagra", how would you detect useful sites in this mess:
http://www.google.com/search?sourceid=chrome&ie=UTF-8...
this implies that you can detect the spammy sites in the first place
In essence, you've just admitted the feasibility of rating the "spammyness" of keyword tuples, provided you can do the classifier.
Second, of course this is possible! SEO-spammers must have some way of determining what to spam. They must have some metric for determining how they can make money and for targeting their efforts for the most profitable scraped content, determining what keyword-tuples to optimize for. An anti-SEO spam engine just has to take the same steps. Just as email spammers put on their email-filterer hats to evade filters, anti-SEO spam engines just have to think like SEO spammers to figure out what SEO spammers are likely to target.
Let's suppose it's possible to tell that a keyword is spammy, what benefit does that provide? The person typing the keyword doesn't care if it's spammy, they just want useful results, quickly.
You can't just reverse the spammer's process, because it's a process followed by many legitimate, successful websites: Research keywords that drive traffic > produce content > build links > get traffic.