Gossamer Forum
Home : Products : Gossamer Links : Discussions :

build large site

Quote Reply
build large site
I have 8000 categories and 600000 links. I tried Build All and Build Changed and Build Staggered, none of them could finish building the whole site. The Build Staggered command only build part of my site before it says it's done. How can I solve this problem?

Can I build from shell command? will it faster than web based command? what is the shell command look like?

Thanks for any help.

Yanz

http://www.biomedgate.com
****Portal to life****
Quote Reply
Re: build large site In reply to
If you have a telnet access that will be your best way.
it sounds like your system is timing out.
The telnet takes a longer time but it should do the trick.

Regards
Abd

http://www.idleb.com
Quote Reply
Re: build large site In reply to
In Reply To:
The telnet takes a longer time but it should do the trick.
Actually telnet is a lot faster than building through the browser and also uses less CPU/Memory.

Regards,

Eliot Lee
Quote Reply
Re: build large site In reply to
Thanks you all for reply. Another question is how to write the shell command.

http://www.biomedgate.com
****Portal to life****
Quote Reply
Re: build large site In reply to
perl /path/to/nph-build.cgi --all

Paul Wilson.
http://www.wiredon.net
Quote Reply
Re: build large site In reply to
What you can simply put in your shell file is the following:

Code:

# Remove all New Files
rm /absolute/path/to/links/pages/New/*.shtml

# Re-build Links SQL Directory
/absolute/path/to/cgi-bin/links/admin/nph-build.cgi --all > /absolute/path/to/logs/nph-build.log

# Remove Log File
rm /absolute/path/to/logs/*


What this will do is the following:

1) Remove all the NEW files
2) Re-build your directory
3) Create a log file of your build process
4) Remove log file

You will need to replace /absolute/path/to with your absolute path. In addition, you will need to create a log folder/directory where the log file will be written to. The permission of this folder should be 777 (drwxrwxrwx). If you don't want to remove the log file, then delete the REMOVE log file codes...the reason I use this is to avoid getting emailed the log file and also removing extraneous files in my server.

BTW: There is a detailed tutorial about creating Crontab, which includes example codes for creating shell files, in the Links => FAQ in the Resources section.

Regards,

Eliot Lee
Quote Reply
Re: build large site In reply to
Hi, Eliot and Paul

Thanks for reply.

Now my problem is to build the lager site for the first time. I followed Paul's instruction and ran the shell command. It's much fast within shell but still got killed when finishing about 10000 links. This is the last resort, then what can I do to build the whole site w/o a dedicate server.

I tried the build stagered command. It only built part of my site and then it claimed all done. The stagered option is supposed to be an alternative for building large site. Why does it not work either?

Yanz



http://www.biomedgate.com
****Portal to life****
Quote Reply
Re: build large site In reply to
You need to ask your hosting company to increase the CPU timeout rate...most are defaulted to 60 seconds. You will need at least 15 minutes to build via telnet.

-OR-

If your hosting company will not do this...you will need to do build your directory locally on your machine by doing the following:

1) Download a web server (like Apache Win32)
2) Installing Perl
3) Installing MySQL
4) Installing DBI:: module
5) Installing Links SQL on your local machine
6) SQLDUMP your MySQL database where your Link tables are stored.
7) Execute the SQLDUMP file in your local system.
8) Then build your directory locally.
9) FTP your directory pages to your server.

Regards,

Eliot Lee
Quote Reply
Re: build large site In reply to
Hi,

The staggered should work. Can you send me an email with access to your admin area so I can see what is going on?

However, be warned, if this is happening it typically indicates that your server is not powerful enough (or has too many other users on it) to run the size of a directory you want.

Cheers,

Alex

--
Gossamer Threads Inc.