Don't write your own crawler. Use nutch. It is designed to scale and do mapreduc... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		mig on Jan 8, 2008 \| parent \| context \| favorite \| on: Ask YC: Any ideas about intelligent crawlers :) Don't write your own crawler. Use nutch. It is designed to scale and do mapreduce kind of parallel processing. I would strongly recommend you to take a look before writing your own. http://lucene.apache.org/nutch/

imsteve on Jan 9, 2008 [–]

mapreduce? Just how many requests will you be making to third-party sites at once? Sounds like a good way to get blocked fast.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact