Gossamer Forum
Home : Products : Gossamer Links : Version 1.x :

Build all and re-index problems

Quote Reply
Build all and re-index problems
:-) new play, problems.

my database are 65000 links. When i re-build index. He break down hat a view minutes, now count on the screen. How long i must wait or is that a bug?

The same are with the re-built all. he create a few pages and than nothing. he stay and stay ... i wait and wait:-)

you know what can that be?


Quote Reply
Re: Build all and re-index problems In reply to
Your script is timing out due to a large number of links. As stated in many Build Problem Threads in this Forum, you need to use telnet or create a crontab to build your directory and re-index your tables.

Regards,

Eliot Lee

Quote Reply
Re: Build all and re-index problems In reply to
Hello!

It is a timeout problem and if you could not build it you cannot. Simply accept that or change to an expensive Internet service provider and get a dedicated server. Otherwise it will not work.

I have mentioned several times that telnet does not work on shared servers in such cases, most likely. Perhaphs on virtual servers. Since it basically only looks at the CPU usage and if it is more than the set time limit that the ISP have setup it simply kills the process of nph-build.cgi and that has nothing to do with telnet. Atleast it did not function in my case.

To beleive for yourself empty all the links you have from the database, import everytime 1000 links each time and build then everytime. And then you will find the cutoff point of your server. May be in the third of the fourth import the nph-build.cgi will not build anymore. Below the build page it shows the time taken. This is the total time taken in the processor.


Quote Reply
Re: Build all and re-index problems In reply to
Uh...I beg to differ...I have final got the build script to work (on a semi-dedicated, BUT shared account). Although my problem now is stopping the build process from contiuously building the directory. I have 460 categories and 2328 links. So, it is possible to build via telnet on a shared server.

Regards,

Eliot Lee

Quote Reply
Re: Build all and re-index problems In reply to
Hello Eliot!

I agree to you in principle. You are correct. One can ofcourse build via telnet. But here is what I observed what happens:

Once you start the process of build via nph-build.cgi (Via browser or Telnet) it is an OWN and INDEPENDENT STAND ALONE process. It works on its own. It is not necessary to be connected with the server. This process continues untill the process is finished or it is killed. If the admin was not able to build via Browser then the first assumtion is the CPU usage restriction could be the problem. If that is the case, then understanding the process theory a build via Telnet does ofcourse not help as it is not a program specific i.e. Telnet.exe or Browser.exe specific problem BUT CPU restriction specific problem. I fail to beleive how Telnet removes CPU usage restrictions as against browser.

There came an email from my provider yelling that six nph-build.cgi processes have been started. Unknowingly. The admin of ISP did kill them and send me his interesting observation. This was during the import study every 5000 more. The stats are as follows (Some of them):

Hallo Herr Rajani,

einige CGI-Skripte auf Ihrem Account laufen irre und legen den ganzen Rechner lahm:

aktuelle Auslastung:

PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND

3089 www 16 0 76148 72M 540 R 0 96.2 57.7 18:48 nph-build.cgi

16821 www 0 0 45372 14M 48 D 0 0.0 11.6 5:10 nph-build.cgi
17266 www 0 0 73548 11M 68 D 0 0.0 9.3 0:05 nph-build.cgi
17286 www 0 0 73548 9.8M 64 D 0 0.0 7.8 0:05 nph-build.cgi
17383 www 0 0 73400 8576 68 D 0 0.0 6.6 0:05 nph-build.cgi
16601 www 0 0 45596 8292 60 D 0 0.0 6.4 17:23 nph-build.cgi
16818 www 0 0 73400 1596 940 S 0 2.1 1.2 4:27 editor.cgi
16669 www 0 0 92128 1608 940 S 0 1.9 1.2 6:05 editor.cgi
17754 www 12 0 76620 74M 1344 R 0 98.3 59.7 1:53 nph-build.cgi >>> we Killed the process after few minutes.

Links table had somewhere like 100 thousand links, if I correctly remember. The process 17754 and 3089 above shows that it required 74 MB of RAM and was using 96.2% power of a Pentium 700 for 18 minutes. I had a batch of 5000 links that were imported several times (Ofcourse UNIQUE de-activated everywhere :) ) for the reason of research and to prove the quality of scripts. Hypothetically if you have Pentium III 550 then it would take 21 minutes. PLEASE note that this was on a Virtual server and it also depends #on the loads it has to serve for other accounts. At that time there were only three account active not having a considerable loads on the server.Hence it may be fairly close to the reality, but I am not sure. If you have few hundred thousand links with few hundred categories then you need one hour to build. On the contrary, Links enters in a jubilient function of Fanatism if it has thousands of categories.

Please, Please Please also note that this statistics may vary from system to system and server to server and situation to situation.

So one can ofcourse build via Telnet. But not beyond the results of a Browser build. Thats what I meant. Alex mentioned earlier that I should log-in with my password and I may have a chance to build. That was also not the case. I did have the chance to build more with the help of Telnet on His Majesty`s advice but only a FEW links more! It did not seem to be a great idea when "A few links more" was not the remaining links to be build but few thousands more!!! ;)

The %CPU and %MEN to your H`onor still remains to be addressed in v2.0!!!
Quote Reply
Re: Build all and re-index problems In reply to
Hello!

Sorry for posting the message without the translation:

Hallo Herr Rajani,

einige CGI-Skripte auf Ihrem Account laufen irre und legen den ganzen Rechner lahm:

aktuelle Auslastung:

The above lines are in German and a fully authentic figurative/metaphoric translation of that line from my Internet service provider means as follows:

"Why d`ont you buy fishes and wash them on the heads of the programmer!!!"

Thanks

Rajani

Quote Reply
Re: Build all and re-index problems In reply to
I think you'll find that this long-running problem (for a few LSQL owners at least) is to be addressed in the next upgrade.

Currently LSQL builds ALL the categories in a SINGLE process, so the more categories and links you have, the longer this process takes (it also uses some of this same process time to calculate the category stats in the first part of the build process).

Links 2.0 (non-sql) has long since had a function called 'Build Staggered'. This builds the entire directory in shorts bursts by using small 'batches' of category pages and running the build process for a limited time on each batch. In effect this breaks down the build process into a large number of smaller 'single' processes and therefore the process never reaches the 'time out' that many ISP's have on their virtual accounts.

I have a large amount of webspace on my account, but a limited process time to run a 'single' process, so I'll welcome this feature with open arms when the new upgrade becomes available.

It may be worth your while asking your ISP to INCREASE the process time limit on your account. I have an agreement with my host that I will only run the 'build' script at times when the server is least busy and in turn they have agreed to allow me to have a longer 'time-out' on my account (hopefully I'll manage with this until the upgrade becomes available :-)

All the best
Shaun

Quote Reply
Re: Build all and re-index problems In reply to
Hi,

I have posted a build staggered modification in the resources. I hope it will help all who have problems with the building-time / cpu usage.

regards alexander

you can download it in:

http://www.silberling.com/lsql/staggerd.txt
Quote Reply
Re: Build all and re-index problems In reply to
Aloha qango,

Eh how can i INCREASE the process time limit on redhat linux ?

Regards Startpoint

Quote Reply
Re: Build all and re-index problems In reply to
Looks like a great mod... and as I've always said.... it's _not_ easy <G>

Lot's of little things to do. Good housekeeping, etc. :)


http://www.postcards.com
FAQ: http://www.postcards.com/FAQ/LinkSQL/

Quote Reply
Re: Build all and re-index problems In reply to
I've modified it slightly to us a Meta-refresh with a 15 second pause at each stage, this means I don't have to 'baby-sit' the build process and can do something else for a while until its finished :)

I'm pretty sure my ISP is going to like this mod too as I won't be hogging the server quite so much <hehehe!>

Pugdog, a worthy addition to your FAQ listings?

All the best
Shaun

Quote Reply
Re: Build all and re-index problems In reply to
Hi Shaun,

good idea to refresh the build process. Post your mod or mail it to me. I would like to insert it into the downloadable file.

regards, alexander