Gossamer Forum
Quote Reply
Spider 2.1 -rules
I have been playing around with the spider 2.1 for some time now. The installation worked fine, the status works, I can stop an pause the spider from web. Which is all fine, BUT:

The reason why I bought the spider was the promise that you can use spider rules to get only the specific pages that YOU want in your database.

I can not get this to work. I made a rule to get only the pages with a very rare word in the data. I knew that this word was only on two or three of the pages on my site. The I spidererd my site. - the rule didn't work at all, all pages on my site came up for validation!

So please... what are we doing wrong? Can anyone give some instructions on how to use the spider rules?

Quote Reply
Re: Spider 2.1 -rules In reply to
I think it's a case where the developers know how to do it, and it's easy, but it's not so easy to get the idea of how to do it as a user.

They really need to release some rule sets, and examples, so you can have some guide lines on how to set up certain kinds of rules.

I've played with the spider, but I can't seem to make it crawl in the direction I want, (only that direction) and I'd like to to only index/insert parts of the web page, using the programming/function interface, but haven't made it work yet.

I've been to distracted with the files plugin to spend a lot of time on it, but it is something that needs to be discussed, and examples, and generic situations posted, and explained.

PUGDOGŪ Enterprises, Inc.
FAQ:http://LinkSQL.com/FAQ
Plugins:http://LinkSQL.com/plugin
Quote Reply
Re: Spider 2.1 -rules In reply to
What I tried to is this: I set the rules:
If data contains "word", change score 10
I set score required for spidering to 10
Only three of my pages has this word in the data.

But spider.pl came up with all the pages, none were left out. In my opinion, the spider rules does not work at all, something must be wrong with the script, It is not just a question of how.

Quote Reply
Re: Spider 2.1 -rules In reply to
The rules work, I just dont' think they work the way you expect. That's why some sort of "jump start" guide to all this would be great.

PUGDOGŪ Enterprises, Inc.
FAQ:http://LinkSQL.com/FAQ
Plugins:http://LinkSQL.com/plugin
Quote Reply
Re: Spider 2.1 -rules In reply to
i have 2 rules defined:

Title Contains 'gay' -- Change Score by 3
Keywords Contains 'gay' -- Change Score by 1

Score required for spidering: 10
Score required for indexing: 20

why where all urls indexed, many of its without the item 'gay' ?

Quote Reply
Re: Spider 2.1 -rules In reply to
Where is the staff?
Why nobody helps my?

I payed $200 for a spider which doesn't works.
I was sending questions to this forum, e-mails and fax to support but nothing happens!

I don't like to waste my money.

I will get my money back.

-------------------
Heiko Mentzel
http://findgay.net/
-------------------