Gossamer Forum
Home : Products : Gossamer Links : Pre Sales :

spider plug-in

Quote Reply
spider plug-in
Well I have this questiong about Spider Plug-in:

In the spider's domumentation it is said:

----------------------------------------------------------------------------------
Running the Spider


The spider script, "spider.pl", cannot be run from the web. The script must run from a telnet shell or as part of a cron job. After you have placed a links into the Spider's queue, run the script to start the system.
----------------------------------------------------------------------------------

Well, I don't have telnet shell access, but I would like to refer to following section of Fileman's documentation and ask if it would be possible to use it's "Command" feature to run spider.pl:

----------------------------------------------------------------------------------
Fileman -> System Commands


FileMan 2 lets you run system commands, providing a virtual shell account. Great for systems that don't provide any shell access (disabled in demo)
----------------------------------------------------------------------------------

??????????????

Thank you in ahead for any answers :-)

Robo

Last edited by:

Robo: Jun 3, 2002, 3:26 AM
Quote Reply
Re: [Robo] spider plug-in In reply to
Is there anybody to answer my PreSales questions? Perhaps you don't want me to buy the product? Unsure
Quote Reply
Re: [Robo] spider plug-in In reply to
I haven't found anyway to run it, but from the shell. The spider will daemonize itself and this doesn't work correctly when you try to kick it off from anything but the shell.

Sean
Quote Reply
Re: [Robo] spider plug-in In reply to
Sean is right, you can start it off from the fileman command prompt and on occasion I've seen it work properly. Issues occur with permissions and paths. Permissions because the shell user and the web user are usually different, path problems because the shell or the webserver might be "chrooted" - meaning that they end up with different directory structures.

(great icon by the way, i love that show, it's been ages since I've seen it though Smile )
Quote Reply
Re: [Aki] spider plug-in In reply to
I have read some old post on this forum and I read something that shell access is neccessary only to INSTALL the spider and then it can be controlled just over any browser. Is that correct? Because if yes, I can ask my hosting support staff to install it - no problem Wink.

If this is not correct, well, I have quite a big confidence in Gossamer Threads staff. Would it be possible, that someone from the Gossamer Threads try to install and run the Spider on my virtual-host server over Fileman virtual command and tell me if it will work? If everything would be fine, I would buy the product.
Quote Reply
Re: [Robo] spider plug-in In reply to
Another issue:

If you don't have an account that provides you shell access, you might find that running the spider will get your account shut down or blocked. I have not used the spider in awhile, but it runs as a daemon, and does a lot of stuff around the clock, or as long as you let it run. While it's not a total pig, and was designed to more or less coexist with other processes, it still uses up a lot of system resources, and on a shared system that can cause problems. (Resources include it's own CPU/Memory, database accesses and increased CPU on MySQL, outgoing/incoming bandwidth, etc.).

I think the initial suggestion was to run it in the off hours, like the verify links system.

It spawns processes that are not owned by the web server, so if you have an account that does not provide telnet/logon access, you might find your system has it's own root-daemon that kills off such processes (non-server processes without a logged on user).

Check with your hosting company and see if you can run this sort of "automatic" background process.


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [pugdog] spider plug-in In reply to
hello pugdog! Wink

thank you for taking time to respond me. you've cleared up the issue for me pretty well. then i will try spider plug-in when we will switch to dedicated server (perhaps next year o year after).

for the moment i will keep on working with another great script - Search Engine from Fluid Dynamics (www.xav.com). It is freeware, quite powerfull, very customizable and can be run from the browser. Yet it needs quite a lot time to index big sites. Anyway - it is very reliable and i would recommend it to anybody without the possibility to run the spider plug-in from the shell.

Thank you guys!

Robo
Quote Reply
Re: [Robo] spider plug-in In reply to
Dedicated servers are about the only way to go with higher end needs.

The prices are falling, and rather than being $8000 to set up, many dedicated servers are plug-in boxes that can be had for a fairly reasonable monthly (not upfront) fee.

Once you have your own server, you are pretty unlimited as to what you can do, and things you thought were not possible (due to security on a shared machine) suddenly become very, very easy :)


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.