Gossamer Forum
Home : Products : Gossamer Forum : Discussion :

WWWThreads Import Performance

Quote Reply
WWWThreads Import Performance
Heya,

I've been running the import script to get my WWWThreads data over to the GForum Tables. I have 9000+ users and 170,000+ posts. Users went ok but the posts have been at it for the better part of 18 hrs and I'm still not at the 50% mark!

I'm on a dedicated server and the rest of the site and mySQL applications seem to be doing fine. Any thoughts on how I can pump up the performance of this a bit. It's pretty crazy to have to wait 2 days to migrate data.... Any advice appreciated.

Safe swoops
Sangiro
Quote Reply
Re: [sangiro] WWWThreads Import Performance In reply to
What speed cpu do you have? It shouldn't be taking that long. It's been quite a while since I looked at the import code, but IIRC, our forums took less than an hour to import (we have a little more users and posts than you). Our server currently isn't all that powerful (a AMD Athlon 900MHz, 1GB memory), so unless you have a much slower system, I don't see why it's going so slow. If I have time later today, I'll try running the import on my own system and seeing if there has been code changes which have affected the import scripts. You are using v1.1.6 right?

Adrian
Quote Reply
Re: [brewt] WWWThreads Import Performance In reply to
Adrian,

Thanks for your reply. Ive got 512MB RAM and a 650 Mhz processor. A bit smaller than yours but I still would not expect it to run that much slower. It's Sunday night, I've been at this for the better part of 2 days as I said and I'm at 54% of 170,000 posts!

It's also definately getting slower as it moves along. It started off taking about 3 min to move 1%... now it takes more than 30 min.

I'm using v1.1.6.

Safe swoops
Sangiro
Quote Reply
Re: [sangiro] WWWThreads Import Performance In reply to
Check how much memory it's using. If it's eating into swap, it'll crawl, and that means there's something with the code as well Smile

Adrian
Quote Reply
Re: [brewt] WWWThreads Import Performance In reply to
Yup - it's killing off my CPU. One mySQL process (obviously this one) is using anything from 75% - 95% of the CPU. Eating into swap most of the time as well.

I'll appreciate any help you can give. Still trying to get the forums switched from WWWThreads to GForum before tomorrow morning. Smile

Safe swoops
Sangiro

Last edited by:

sangiro: Jun 23, 2002, 4:35 PM
Quote Reply
Re: [sangiro] WWWThreads Import Performance In reply to
Can you see how much memory it's using? If you're using a *nix based system, can you do:
top
then press M (shift m), to sort by memory.
And paste me the line

Adrian
Quote Reply
Re: [brewt] WWWThreads Import Performance In reply to
I'm running an import of our old WWWThreads database on my computer now, it's been running for about 2 hours, and it's hit 25%, using about 34MB of memory, about 8% CPU, and load is at about 1.0. This running on an AMD Athlon 950, 640MB RAM. I recall it being faster than that, I'm not sure what's going on. I'll turn on debugging and see what's going on.

Adrian
Quote Reply
Re: [brewt] WWWThreads Import Performance In reply to
Sorry I disappeared last night but it got too late here... I think the one that's dying off is CPU, not MEM. Here are the first lines of a top:

Code:
7:12am up 47 days, 12 min, 2 users, load average: 1.29, 1.14, 1.12
78 processes: 75 sleeping, 3 running, 0 zombie, 0 stopped
CPU states: 64.5% user, 35.4% system, 0.0% nice, 0.0% idle
Mem: 523876K av, 512380K used, 11496K free, 0K shrd, 79872K buff
Swap: 136512K av, 1444K used, 135068K free 305336K cached

PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND
7118 mysql 15 0 12696 12M 1528 R 0 96.6 2.4 349:17 mysqld
7115 dropzone 0 0 24164 23M 1376 S 0 2.1 4.6 7:25 perl
20446 dropzone 2 0 900 900 668 R 0 0.9 0.1 0:00 top
3 root 0 0 0 0 0 SW 0 0.1 0.0 290:21 kupdate
It's a UNIX system yes.

Safe swoops
Sangiro
Quote Reply
Re: [sangiro] WWWThreads Import Performance In reply to
Adrian,

After running it for another night I'm on my way to 75% but not there yet. Would really appreciate if you could take a look at this code. I'm being pummeled by my visitors - they all want to know why the forums have been down for 3 days!

Safe swoops
Sangiro
Quote Reply
Re: [sangiro] WWWThreads Import Performance In reply to
Hi,

For future reference, we've cleaned up the section of code that was causing the problem here which should significantly increase the WWWThreads import speed of large forums. As a side effect, this also speeds up posting by a second or two. The 1.1.6 version available as of now has the updated code.

Jason Rhinelander
Gossamer Threads
jason@gossamer-threads.com
Quote Reply
Re: [Jagerman] WWWThreads Import Performance In reply to
Thanks Jason! Also a big thank you to Adrain who put some time in yesterday to help me get this resolved. Smile

Safe swoops
Sangiro