Does anyone know of a spider script that will run on a UNIX server?

URL Spider Pro <http://www.innerprise.net/products.htm> is exactly what I am looking for except it doesnt run on UNIX machines (bummer)

Any feedback would be helpful. Thanks

You can run the application on your personal computer running Windows...then FTP the links to your UNIX server.

Regards,

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
* Be sure to visit the Resource Center for FAQ's, Modifications and Extra Goodies!!
* Search Forums!
* Say NO to Duplicate Threads. :)
----------------------

Elliot,

Thanks for the info. I didnt know that, I just read that it wouldn't run on UNIX. That's great, then maybe that program is exactly what I am looking for after all :-)

Have you or anyone else had experience with this program? Would you recommend it?

FDSE http://www.xav.com/scripts/search/ is pretty good. You don't mention Links and Spider is better if you want to add what the robot finds to Links. It shouldn't be to hard to do the same thing with FDSE data but I haven't got around to making a script for that yet.

Dave, thanks for the info on xav, I'll check it out as well.

Dave,

Decent suggestion...That script is already used in the regular Spider script in LINKS Modifications.

Also, it does not target categories or keywords very well...It will basically pull tons of useless URLS off the Net.

Regards,

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
* Be sure to visit the Resource Center for FAQ's, Modifications and Extra Goodies!!
* Search Forums!
* Say NO to Duplicate Threads. :)
----------------------

If I remember right that spider mod makes limited use of FDSE's capabilities, being used as a real time search instead of an indexer.

FDSE doesn't target keywords (isn't that how URL Spider works?). It does index keywords, which can be given preference when searching, but that usefulness is only as good as the keywords the page author included in the meta tag (if at all). You can edit entries but last I knew these are lost when a site is reindexed (my biggest beef with the script). You have control over what gets indexed, you can do it on a page by page basis, indexing only the pages you want, or let it go and use my delete mod to get rid of the stuff you don't want.

You could set it up where realms = categories and the db conversion should be relatively easy (I know talk is cheap but I will get around to it).

FDSE can also be a full text search engine for your site. Its not perfect and I have some beefs but I haven't found a better (remote/local) indexing robot for the price.

 
Not with the tests I've conducted with XAV script...it is quite limited. I actually re-wrote XAV INDEXED SEARCH to fix a lot of bugs and also to make it work more easily on all platforms.



Regards,

------------------
Eliot Lee
Anthro TECH,L.L.C
www.anthrotech.com
* Be sure to visit the Resource Center for FAQ's, Modifications and Extra Goodies!!
* Search Forums!
* Say NO to Duplicate Threads. :)
----------------------

OK, I bought the URL Spider Pro program. I have a targeted DB of over 5000 pages.

Since I have unix, I cant use the cgi script that from http://www.innerprise.net that is meant for this program.

I have the DB converted to a Links DB, so I can search the DB, and it works OK.

I would really like a results page that would give relevency to each results (i.e. 100% 89% etc...) Do you guys know of any search scripts that can do this? Can links search.cfi be edited to do this?

Thanks

Eliot,


FDSE is *not* XAV. Last I knew you blew FDSE of because you didn't care about remote indexing and couldn't transfer your XAV mods to FDSE.

I've used FDSE since its inception (and XAV for years prior). When I say that you have control over what gets indexed I speak from personal experience. As per the original post this relates to remote indexing as I have not used FDSE for local indexing.