Hi Alex,
thank you for your answers.
Anyway, some points are still not quite clear to me.
1. resources
------------
Isn't 5% server load very much for a virtual server with say about 20 or so other virtual servers on the same machine?
You say, just the indexing causes some server load for a short time. What is short?
At the moment I got about 2.000 links in the Links database and indexing the whole thing lasts about 10 minutes. Assuming the spider will fetch say 20.000 pages - will indexing last 100 minutes then?
How much resources will be needed during that time?
I ask because my provider doesn't allow banner systems based on mysql for example for the reason that these are writing to the database for every page called (banner served) and that mysql wouldn't be optimized for writing but is for queries (only reading, i.e. retrieving select results). I wonder if they will allow use of the spider...
2. Restrict searching
---------------------
You say it's possible to let users search only pages from my server by passing spider=0 as argument. The spider help file says that then only the links database will be searched, not the spider database.
My question was about pages which are generated outside from LinksSQL and therefor are not part of the Links database. So, to make them searchable they have to be spidered. This means to me, they can't be searched when passing spider=0.
I would like to offer my users the possibility of a site wide search (i.e. search only pages with URLs starting with "http://www.archaeologie-online.de") in addition to the option to search the complete spider database.
So I can't use the filtering mechanism to exclude other pages than mine from spidering, because I want to have other pages in the database, too. I just want to restrict the search, not the spidering.
Would that be possible by using a variable in the search form like "url=http://www.archaeologie-online.de*" or something like that?
Generally speaking: Is it possible to search certain fields of the spider database as it is with LinksSQL's internal search?
Andreas
--------------------------------------------
http://www.archaeologie-online.de
thank you for your answers.
Anyway, some points are still not quite clear to me.
1. resources
------------
Isn't 5% server load very much for a virtual server with say about 20 or so other virtual servers on the same machine?
You say, just the indexing causes some server load for a short time. What is short?
At the moment I got about 2.000 links in the Links database and indexing the whole thing lasts about 10 minutes. Assuming the spider will fetch say 20.000 pages - will indexing last 100 minutes then?
How much resources will be needed during that time?
I ask because my provider doesn't allow banner systems based on mysql for example for the reason that these are writing to the database for every page called (banner served) and that mysql wouldn't be optimized for writing but is for queries (only reading, i.e. retrieving select results). I wonder if they will allow use of the spider...
2. Restrict searching
---------------------
You say it's possible to let users search only pages from my server by passing spider=0 as argument. The spider help file says that then only the links database will be searched, not the spider database.
My question was about pages which are generated outside from LinksSQL and therefor are not part of the Links database. So, to make them searchable they have to be spidered. This means to me, they can't be searched when passing spider=0.
I would like to offer my users the possibility of a site wide search (i.e. search only pages with URLs starting with "http://www.archaeologie-online.de") in addition to the option to search the complete spider database.
So I can't use the filtering mechanism to exclude other pages than mine from spidering, because I want to have other pages in the database, too. I just want to restrict the search, not the spidering.
Would that be possible by using a variable in the search form like "url=http://www.archaeologie-online.de*" or something like that?
Generally speaking: Is it possible to search certain fields of the spider database as it is with LinksSQL's internal search?
Andreas
--------------------------------------------
http://www.archaeologie-online.de
