Gossamer Forum
Home : Products : Gossamer Links : Discussions :

Creating backup file - Build All

Quote Reply
Creating backup file - Build All
Using SSH connection and running "nph-build.cgi --all"

Q: how long should "Creating backup file" take?

Appears to start ok...I get this......


Started at Fri Dec 26 20:11:32 2003.

Building HTML Pages ...

Creating backup file ...


I wait and wait, leave my PC for an hour or so, and wait.....

How long should it take? Database has 2 million links....

Yesterday I did this with 1 million links in the database, "Creating backup file" appeared to take about 30 minutes.....build took about 18 hours.....

Today with 2 million links it just appears to "hang" - I gave in after 90 minutes

Q: should I be patient and wait longer?



advice & help welcome

Thanks
Colin Thompson
Quote Reply
Re: [colintho] Creating backup file - Build All In reply to
The more links, the longer it will take.

If 1 million links took 1/2 hour, 2 million is not going to take 1 hour. It could take 2 hours, or 6. It would depend on how indexes, tables, disk space, memory usage, and more are.

Before doing a back up, doing a repair tables, and doing a check of the database using the MySQL utilities wouuld be a good idea. The more links you have, the more important it is to keep the database "tuned".

There are some benchmarks at mysql.com, but "how long" is really a function of your machine. If 1 million links took 1/2 hour, then giving it 2-3 hours for 2 million wouldn't be unreasonable. That is a *LOT* of data to access and move around.

If you have access to your MySQL data directory, you could copy the tables and/or back up the tables directly, without doing an export. One nice thing about MySQL is you can copy the tables while it's running, and get a "snapshot" of your data. It's better than nothing, and requires far less system resources. If you can copy to another directory or better yet, to a different disk, you'll have a "snapshot" backup.

Just some thoughts.


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [pugdog] Creating backup file - Build All In reply to
Thanks

after some advice from "Andy" I set up a couple of cron jobs

build is now complete

Q: with 2 million links (DMOZ dump) the site is now very, very slow....what server specs should I be looking for as a minimum, to maintain or improve performance

thanks again.
Colin Thompson
Quote Reply
Re: [colintho] Creating backup file - Build All In reply to
>> Q: with 2 million links (DMOZ dump) the site is now very, very slow....what server specs should I be looking for as a minimum, to maintain or improve performance


The biggest you can afford.



I would suggest a Unix machine, with at least 2GB RAM, and if you are trying to serve a significant amount of hits, you might want to set it to run under mod_perl. You can check out http://GossamerHost.com for *realistic* pricing of packages, and rock-solid support and performance (a satisfied customer :) Anyone promising more for less, is *probably* not going to deliver 24/7/365. Remember, it's not what you are *allowed* to use under your plan, it's what's actually AVAILABLE for you to use :)

The CPU speed is less important (now adays) than the amount of RAM and the quality of your harddrives. If you are still having performance issues, in DISK I/O, you can use a performance RAID system, where data is stripped across different disks, so the read speed is higher than with doing seeks on a single drive. Of course, this is not cheap.

Bigger is *NOT* always better. Targeted sites seem to do better over all, and performance is higher. ie: people searching for hobby or fun things, are not likely to search business or money sites at the same time. A person looking up RC cars is not going to be really interested in gardening, etc. Getting rid of any/all adult links improves performance. I removed about 80,000 links from Girlfriends.com a couple of years back, and left about 35,000 links of "content" which improved performance significantly. You'll find most adult links are worthless, and they bog down a system. You'll do better finding a referral site, and getting a penny or less per referral for each adult surfer. (That is of course, unless you are running an adult site <G>)


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.
Quote Reply
Re: [pugdog] Creating backup file - Build All In reply to
Thanks for the advice, much appreciated.

As well as the "slow search" (which could be being made worse by the Links table - indexing set to NON-INDEXED, now in the process of change)

The “Add url” process, If a registered user logs in and clicks on “add url” - now returns error 500 internal server error

Q: is this due to the size of the database & server specs?

site is listed in google search results at No. 4 from 3 million results for directory add url

http://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=directory+add+url&btnG=Google+Search

So I really need to get the site working for “add url” visitors

Any advice?
Colin Thompson
Quote Reply
Re: [colintho] Creating backup file - Build All In reply to
What happens if you call add.cgi?ID=123 ? Do you still get a 500 IS Error?

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] Creating backup file - Build All In reply to
the "add url" form now starts after the db_gen_category_list is set to NO

Category: Regional/South America/Argentina/Provinces/Santa Cruz/Travel and Tourism

But - on clicking submit "add resource" the page / form just hangs, I can see in the base of the browser window "Opening page http://www.*****.com/cgi-bin/limks/add.cgi
Colin Thompson
Quote Reply
Re: [colintho] Creating backup file - Build All In reply to
Hi everyone, I have the same problem with frozen
Creating backup file...

I have about 10000 links.

Thanks
Quote Reply
Re: [rage] Creating backup file - Build All In reply to
Hi,

Have you tried running it via SSH?

cd /path/to/admin
perl nph-build.cgi --all

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] Creating backup file - Build All In reply to
Hi Andy,
it is working fine.
But why happened that? or I need everytime to build it from SSH?

Thanks a lot for your help.
Quote Reply
Re: [rage] Creating backup file - Build All In reply to
Hi,

Have you tried Build > Build Staggered ?

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Quote Reply
Re: [Andy] Creating backup file - Build All In reply to
Yes before SSH, I tryed and didn't work.
Now its working.

Thanks
Quote Reply
Re: [rage] Creating backup file - Build All In reply to
NP.

Just a suggestion - make sure in Setup > Misc , you don't have "debugging" enabled - as that can sometimes cause your site to go slow, and also builds.

Cheers

Andy (mod)
andy@ultranerds.co.uk
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package | GLinks ULTRA Package PRO
Links SQL Plugins | Website Design and SEO | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!