it spiders the site.. adds the link and then the link it spiders..

then the admin spiders off the spidered links and does it pretty much til the whole site has been spidered..

jerry

Is it possible to get these scripts? Where can i get one? They look pretty cool, widgets.
Thanks

------------------
Ted Wilcox
http://www.cyber-find.com

He said we shouldn't expect earlier than Friday in another thread. So sometime near Friday. It looks like it can run now, besides not having a search for it, and other stuff.

oh.... .i forgot.. and i stopped working on this script.. i didn't like it.. i could start over and do it in mysql.. but i don't want to... (flat file is slow after getting to about 500k of data.. [for spider.cgi]

i think the best idea is to make 2 files.. 1 for the spidered already.. and one for the not yet.. so it will go faster.. and then it will delete the ones on the not spidered and put them in with their results on the spidered..

get it?

i'll send you the scripts on request.. jsu7785@email.com

jerry

Widgetz,
Hows it going with this?

ok.. i made it much much much faster.

if you went to

http://www.widgetz.com/url.db

you will notice they are all spidered..

http://www.widgetz.com/spider.db

none are spidered..

MUCH faster..

jerry

spider.cgi is now running at 25 links each time..

i wanna see where this version can get before handing it out..

so add urls.. i added gossamer-threads and there are a bunch of links..

jerry

oh... nice

i set it at 100 links per spider and it still finishes..

unlike the old one.. sometimes dying at 1

jerry

Very nice, Jerry.

Regards,

------------------
Eliot Lee
Founder and Editor
Anthro TECH, L.L.C
http://www.anthrotech.com/
info@anthrotech.com
==========================
Coconino Community College
http://www.coco.cc.az.us/
Web Technology
Coordinator
elee@coco.cc.az.us

yeah, it seems really fast to be opening two databases that have alot of links.

there is a bug.. (not remaking spider.db).. i'm not feeling good in the head.. had a fever since friday..

but i'll get to it tommorrow after school and send out the copies for the request...

jerry

Hey widgetz... How is it now?

Widgetz,
Since you stopped i guess because of school and that spider.db error, can you give me the code that gets the description and crap from a page. I don't really need the rest now. I could use the rest later though.

Lavon

Is there a way to define Keywords to go search on. This is my dilema; I have a topic specific directory and only want links pertaining to a certain subject.

------------------
http://www.rvcorner.com Complete RV and Camping directory