Using SSH connection and running "nph-build.cgi --all"

Q: how long should "Creating backup file" take?

Appears to start ok...I get this......

Started at Fri Dec 26 20:11:32 2003.

Building HTML Pages ...

Creating backup file ...

I wait and wait, leave my PC for an hour or so, and wait.....

How long should it take? Database has 2 million links....

Yesterday I did this with 1 million links in the database, "Creating backup file" appeared to take about 30 minutes.....build took about 18 hours.....

Today with 2 million links it just appears to "hang" - I gave in after 90 minutes

Q: should I be patient and wait longer?



advice & help welcome

Thanks
Colin Thompson

The more links, the longer it will take.

If 1 million links took 1/2 hour, 2 million is not going to take 1 hour. It could take 2 hours, or 6. It would depend on how indexes, tables, disk space, memory usage, and more are.

Before doing a back up, doing a repair tables, and doing a check of the database using the MySQL utilities wouuld be a good idea. The more links you have, the more important it is to keep the database "tuned".

There are some benchmarks at mysql.com, but "how long" is really a function of your machine. If 1 million links took 1/2 hour, then giving it 2-3 hours for 2 million wouldn't be unreasonable. That is a *LOT* of data to access and move around.

If you have access to your MySQL data directory, you could copy the tables and/or back up the tables directly, without doing an export. One nice thing about MySQL is you can copy the tables while it's running, and get a "snapshot" of your data. It's better than nothing, and requires far less system resources. If you can copy to another directory or better yet, to a different disk, you'll have a "snapshot" backup.

Just some thoughts.


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.

Thanks

after some advice from "Andy" I set up a couple of cron jobs

build is now complete

Q: with 2 million links (DMOZ dump) the site is now very, very slow....what server specs should I be looking for as a minimum, to maintain or improve performance

thanks again.
Colin Thompson

>> Q: with 2 million links (DMOZ dump) the site is now very, very slow....what server specs should I be looking for as a minimum, to maintain or improve performance


The biggest you can afford.



I would suggest a Unix machine, with at least 2GB RAM, and if you are trying to serve a significant amount of hits, you might want to set it to run under mod_perl. You can check out http://GossamerHost.com for *realistic* pricing of packages, and rock-solid support and performance (a satisfied customer :) Anyone promising more for less, is *probably* not going to deliver 24/7/365. Remember, it's not what you are *allowed* to use under your plan, it's what's actually AVAILABLE for you to use :)

The CPU speed is less important (now adays) than the amount of RAM and the quality of your harddrives. If you are still having performance issues, in DISK I/O, you can use a performance RAID system, where data is stripped across different disks, so the read speed is higher than with doing seeks on a single drive. Of course, this is not cheap.

Bigger is *NOT* always better. Targeted sites seem to do better over all, and performance is higher. ie: people searching for hobby or fun things, are not likely to search business or money sites at the same time. A person looking up RC cars is not going to be really interested in gardening, etc. Getting rid of any/all adult links improves performance. I removed about 80,000 links from Girlfriends.com a couple of years back, and left about 35,000 links of "content" which improved performance significantly. You'll find most adult links are worthless, and they bog down a system. You'll do better finding a referral site, and getting a penny or less per referral for each adult surfer. (That is of course, unless you are running an adult site <G>)


PUGDOG� Enterprises, Inc.

The best way to contact me is to NOT use Email.
Please leave a PM here.

Thanks for the advice, much appreciated.

As well as the "slow search" (which could be being made worse by the Links table - indexing set to NON-INDEXED, now in the process of change)

The “Add url” process, If a registered user logs in and clicks on “add url” - now returns error 500 internal server error

Q: is this due to the size of the database & server specs?

site is listed in google search results at No. 4 from 3 million results for directory add url

http://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=directory+add+url&btnG=Google+Search

So I really need to get the site working for “add url” visitors

Any advice?
Colin Thompson

What happens if you call add.cgi?ID=123 ? Do you still get a 500 IS Error?

Cheers

Andy (mod)
andy@ultranerds.co.uk

---

IMPORTANT: I've now moved to ultranerds.co.uk, and the .com will no longer work!
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package (plugins total "value" $3,325 & rising, for just $350)| GLinks ULTRA Package PRO (plugins total "value" $5,625 & rising, for just $500)
Support Forum | Links SQL Plugins | DMOZ Dumps | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Compare our different Plugin packages *new* Free CSS Templates

the "add url" form now starts after the db_gen_category_list is set to NO

Category: Regional/South America/Argentina/Provinces/Santa Cruz/Travel and Tourism

But - on clicking submit "add resource" the page / form just hangs, I can see in the base of the browser window "Opening page http://www.*****.com/cgi-bin/limks/add.cgi
Colin Thompson

Hi everyone, I have the same problem with frozen
Creating backup file...

I have about 10000 links.

Thanks

Hi,

Have you tried running it via SSH?

cd /path/to/admin
perl nph-build.cgi --all

Cheers

Andy (mod)
andy@ultranerds.co.uk

---

IMPORTANT: I've now moved to ultranerds.co.uk, and the .com will no longer work!
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package (plugins total "value" $3,325 & rising, for just $350)| GLinks ULTRA Package PRO (plugins total "value" $5,625 & rising, for just $500)
Support Forum | Links SQL Plugins | DMOZ Dumps | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Compare our different Plugin packages *new* Free CSS Templates

Hi Andy,
it is working fine.
But why happened that? or I need everytime to build it from SSH?

Thanks a lot for your help.

Hi,

Have you tried Build > Build Staggered ?

Cheers

Andy (mod)
andy@ultranerds.co.uk

---

IMPORTANT: I've now moved to ultranerds.co.uk, and the .com will no longer work!
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package (plugins total "value" $3,325 & rising, for just $350)| GLinks ULTRA Package PRO (plugins total "value" $5,625 & rising, for just $500)
Support Forum | Links SQL Plugins | DMOZ Dumps | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Compare our different Plugin packages *new* Free CSS Templates

Yes before SSH, I tryed and didn't work.
Now its working.

Thanks

NP.

Just a suggestion - make sure in Setup > Misc , you don't have "debugging" enabled - as that can sometimes cause your site to go slow, and also builds.

Cheers

Andy (mod)
andy@ultranerds.co.uk

---

IMPORTANT: I've now moved to ultranerds.co.uk, and the .com will no longer work!
Want to give me something back for my help? Please see my Amazon Wish List
GLinks ULTRA Package (plugins total "value" $3,325 & rising, for just $350)| GLinks ULTRA Package PRO (plugins total "value" $5,625 & rising, for just $500)
Support Forum | Links SQL Plugins | DMOZ Dumps | UltraNerds | ULTRAGLobals Plugin | Pre-Made Template Sets | FREE GLinks Plugins!
Compare our different Plugin packages *new* Free CSS Templates