Gossamer Forum
Quote Reply
Spider Update
Hi!

We are working on a major upgrade of the spider code that should make it a lot more user friendly. We are in the final testing stages of this and it will be available shortly after the Links SQL update.

Sorry about the headaches people have been having.

Cheers,

Alex

--
Gossamer Threads Inc.
Quote Reply
Re: Spider Update In reply to
Does the spider only help in bringing in searching of the links already in your database, or also help in spidering and finding relevant links to add to the database?

Quote Reply
Re: Spider Update In reply to
Hi,

You can seed the spider with links in your database, or cut and paste links into a textarea. You then tell the spider to check those, and it will fetch them and queue up the pages they link to, and index the content making it available to your visitors. You can also specify rules determining whether a page is relevant and should be either a: indexed, or b: followed.

Hope this helps,

Alex

--
Gossamer Threads Inc.
Quote Reply
Re: Spider Update In reply to
still need clarification as to the limits of spider.

can I specify a URL for say the main page of another web site and have spider add to the spider db? If so, how do you specify levels of pages to scan?

thanks,
Ed

Quote Reply
Re: Spider Q? In reply to
I'd like some clarification on the Spider plugin:

Does it function like
WebBandit
http://www.softwaresolutions.nu/jwsg/webbandit/news.html

and/or

Bullseye
http://www.intelliseek.com/prod/bullseye/pro.htm

???

Thanks,

Leo
Quote Reply
Re: Spider Q? In reply to
Please don't post duplicate threads. One is enough. There was no need to post the thread in the Discussion Forum also.

Installations:http://www.wiredon.net/gt/
Favicon:http://www.wiredon.net/favicon/

Quote Reply
Re: Spider Q? In reply to
I apologize.

Do you happen to know the answer(s) to my question?

Thanks,

Leo
Quote Reply
Re: Spider Q? In reply to
I'd be curious about an answer to your question too.

Thanks!

Joe

Quote Reply
Re: Spider Q? In reply to
Anyone with some clarification?

I have been unable to decipher if the Spider plugin will search the Internet for links based on search words and allow them to be added to LinksSQL???

Beuller? Beuller? .......Beuller?

Leo
Quote Reply
Re: Spider Q? In reply to
The spider plugin will collect information from specific sites that you specify url to. It collects both meta description and data for each page on that site.

You can also give the spider some keywords to work with. Ex: only collect pages with the words "health", "doctor" and "medical" in them.

The spider then comes up with a list of links in a separate spider-database. You can choose to validate these links before they are added to the spider-database, or they can go directly in. If you want to, you can also integrate these links with your ordinary links-database.

When searching, if there are no hits from the ordinary links search, the spider database will be searched. You can also search only the spider-database if you want to.

Quote Reply
Re: Spider Q? In reply to
I'm still working on figuring out what the spider will and won't do, can and can't do, and such.

I have the current version running, and I've seeded it with the links from my database, since the site it's running on only had about 400 specific links, and from there, I've let it run.

I've found the "status" page to be pretty cryptic, and even the "database" page to be somewhat confusing to use. I haven't had enough time to play with it to get any hints in order though.

I haven't tried to import any links yet, that is later today. Looking at the links database collected, it has automatically parsed the pages pretty well, for descriptions, keywords and such.

I think the most beneficial thing would be to have one of the spider developers monitor the forum for a few days, jumping on the questions immediately. Might give some jump-start to the understanding of this item.



PUGDOGŪ Enterprises, Inc.
FAQ:http://LinkSQL.com/FAQ
Forum:http://LinkSQL.com/forum
Quote Reply
Re: Spider Q? In reply to
Thank you kindly for the info dump.
It was very helpful.

Leo