Gossamer Forum
Home : Products : Gossamer Links : Development, Plugins and Globals :

Metacrawler Plugin - Will Pay

Quote Reply
Metacrawler Plugin - Will Pay
I really want to have a metacrawler plugin for the copy of Links SQL 2.0 that I am running on KaaZoo.com. Basically I just want to be able to grab results from alternate search engines if a person finds only a few or no listings in my directory, similar to how Yahoo grabs alternate results from Google. It would be nice to be able to easily change the search engine being queried and it would also be nice to have the option of querying multiple engines, similar to metacrawler.com. As a matter of fact, I really like the way that metacrawler.com works.

Is anyone already working on this? If not, can someone recommend someone who could develop this? Alex, would you guys be able work on this? How much would it cost me? I don't expect the work to be exclusive since I am sure you would like to resell the plugin or use it as a future added feature. ;)

Thx



Rob Bartlett
AAA Internet Publishing, Inc.
http://www.AAAInternet.com
Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
Alex is already developing a spider plugin but that may not be what you want.

I can create a plugin for you that uses a select list of search engines and the user can select which one they want to search......

Im not sure that is exactly what you want either..

Paul Wilson.
new - http://www.wiredon.net
Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
Basically the functionality that I want would be a hybrid between metacrawler.com and yahoo.com.

The search logic would work like this:
1. Search results are grabbed from the local directory first.
2. Alternate results are displayed below the local results if few or no results are found in the local directory.
3. The alternate results are found by searching alternate search engines, chosen/configured by myself, not the user.
4. Should have the option of giving weights to each search engine to help determine the relevancy of the results provided.
5. If a site is found at more than one of the selected search engines, relevancy should increase, but no duplicate listings/URLs in the results.
6. Each search result should have a small link to the search engine(s) providing the results. When this link is clicked on, the keyword(s) should be hotlinked to automatically perform the search, similar to the format used at Metacrawler.com.

I think that is about it.


Rob Bartlett
AAA Internet Publishing, Inc.
http://www.AAAInternet.com
Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
ewww......that is pretty complexed.

Alex is going to have to help you with that I think.

Paul Wilson.
new - http://www.wiredon.net
Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
The spider that is being developed might suffice for now, but I am not quite sure what capabilities it will have. I asked before but nobody responded. Guess I will wait a bit, a full-fledgeed metacrawler as I described would still be very nice. =)

Rob Bartlett
AAA Internet Publishing, Inc.
http://www.AAAInternet.com
Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
Definitely would be nice.........(but not cheap)

Paul Wilson.
new - http://www.wiredon.net
Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
Hi,

No the spider is quite different from what you asked. We should have it available by Monday or Tuesday, just working on help pages right now.

Basically the spider will let you index links you have in your directory, and make those results searchable via the regular search.cgi.

It does not interact with third party search engines at all.

However, with the spider done, this is something we could work on building. Please drop an email to jack@gossamer-threads.com if you are interested in getting one built.

Cheers,

Alex

--
Gossamer Threads Inc.
Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
Ok, I will e-mail him.

Alex, I don't really understand the value of the spider. Why would you want to index links in your own directory? Why spider them if they are already in the database? =P Or does the spider actually look at the content of the linked pages and index keywords/content from each of them?

That makes sense, is that what the spider does? If not could you please rephrase the description so I can get a better idea of its capabilities?

Thx



Rob Bartlett
AAA Internet Publishing, Inc.
http://www.AAAInternet.com
Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
Hi,

The spider is a full featured spider. It will go out and spider any URL's you give it (with hooks to easily spider your Links SQL entries). It visits each page, indexes it and queues all the links on those pages to be spidered later.

It hooks into Links SQL by allowing your visitors a better search. Instead of just searching on the URL/Title/Description of a site, you can search on content found on the site. For instance MySQLMan may not be mentioned in a listing about Gossamer Threads, but if the site had been spidered, you'd be able to find it.

That's a simplified view, it is a full featured spider and:

* obeys robots.txt file
* can enable/disable spidering remote sites
* can set filters/rules on whether content should be indexed, or spidered
* can import links into Links SQL directory
* can enhance search, or offer second search (i.e. search directory/search the web).
* can spider any urls, not just your directory sites.

Let me know if that makes sense..

Cheers,

Alex

--
Gossamer Threads Inc.
Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
Thanks for the detailed explanation Alex, it helps alot. =)

It looks like the spider will be very useful!



Rob Bartlett
AAA Internet Publishing, Inc.
http://www.AAAInternet.com
Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
Speaking of the robots.txt file, can someone recommend a good place to go to learn how to use this file on a website?

Thx

Rob Bartlett
AAA Internet Publishing, Inc.
http://www.AAAInternet.com
Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
http://doc.altavista.com/...st_haw_avoiding.html


Robert Blackstone
Webmaster of Scato Search
http://www.scato.com
Quote Reply
Re: Metacrawler Plugin - Will Pay In reply to
what spam controls does it have?

http://www.ASciFi.com/ - The Science Fiction Portal