http://incubator.apache.org/nutch/tutorial.html
Seems like a perfect open source crawler. Just wondering if it can be used with Links SQL.
Which crawlers do most of the webmasters use? I try to build a niche search engine with at least 300k sites
and look for a powerful crawler.
Seems like a perfect open source crawler. Just wondering if it can be used with Links SQL.
Which crawlers do most of the webmasters use? I try to build a niche search engine with at least 300k sites
and look for a powerful crawler.