Gossamer Forum
Home : Products : Gossamer Forum : Discussion :

are there limits to number of categories/forums

Quote Reply
are there limits to number of categories/forums
I tried to set up a bunch of categories (~7k) and 5 forums each (~35k) and I seemed to have taken the sails out of the mod-perl engine.

Has anyone had experience with this large an install ? Perhaps I just didn't set things up correctly loading directly into the gforum db although it worked just fine for the first few hundred.

If it can't be done I'll need to revise my plan.

Bob
Quote Reply
Re: [regert] are there limits to number of categories/forums In reply to
I once set up, just for the fun of it, a forum with a few thousand categories, and with a few ten-thounds of forums.

It worked OK, but I noticed that pages take very long to load by the browser (obviously, because the html that is produced is huge). My solution was to set all the categories to 'collapsed' in the script that made the forums. That way, loading was much faster.

Ivan
-----
Iyengar Yoga Resources / GT Plugins

Last edited by:

yogi: Jan 12, 2003, 3:14 AM
Quote Reply
Re: [yogi] are there limits to number of categories/forums In reply to
* wonders how well the template parser loop code will handle a 40,000+ element loop Shocked *

Actually, it should be able to handle it, but like yogi said, making all of the categories collapsed (ideally by setting "Show forums in main forum list" to "No" for all/most categories) ought to help. Are any of these 7000 categories nested, or is it a big smorgasbord of root categories?

Jason Rhinelander
Gossamer Threads
jason@gossamer-threads.com
Quote Reply
Re: [yogi] are there limits to number of categories/forums In reply to
Hi Ivan,
You are correct, the page display is prohibitively slow. I tried switching all categories to 'collapse' but still tooo slooooow.

I'm not certain I understand the reason for the delay though, since if I do a direct mysql query the response is instantaneous.

How/where does GF do it's lookup ? Is there a way I can put a mysql query string in the uri for GF to execute and display ?

Thanks
Bob

==== example mysql query
mysql> select forum_name,gforum_Category.cat_id_fk from gforum_Forum,gforum_Category where gforum_Forum.cat_id_fk=cat_id and cat_id=3668;
+------------+-----------+
| forum_name | cat_id_fk |
+------------+-----------+
| ADMIN | 0 |
| NURSING | 0 |
| AUTH | 0 |
| SCHEDULING | 0 |
| PT | 0 |
| AR | 0 |
+------------+-----------+
6 rows in set (0.00 sec)
Quote Reply
Re: [Jagerman] are there limits to number of categories/forums In reply to
Hi Jason,
The categories are at the root level since they are all of equal 'importance'. Would it make a difference if they were another level down ?

I'm not asking a rapid response in displaying the entire main forum listing. I'm calling the page like this:
http://my.org/gf-cgi/gforum.cgi?category=3600.

It brings up just forum 3600, but it take way too long.

The motivation for this is to use GF as an intra-office information location so that different groups of people (represented as Forums) can make comments/ask questions regarding different accounts (represented as Categories).

Comments are appreciated.

Bob
Quote Reply
Re: [regert] are there limits to number of categories/forums In reply to
Hi Bob,

Try doing a couple things for me. First, in gforum.cgi, add:

Code:
use Time::HiRes();

Put this somewhere near the top; just before the "GForum::init(..." line will work well.

After that, go into the Setup -> Miscellaneous section of the admin panel and enabling debugging. After having done these two things, your error log should start displaying all the queries that are run, and after each one something like:

Query execution took: 0.015813s

That should help determine where the slowdown is coming from.

Jason Rhinelander
Gossamer Threads
jason@gossamer-threads.com
Quote Reply
Re: [Jagerman] are there limits to number of categories/forums In reply to
Thanks for the help

Attached are last 100 lines from error log after going to:
http://XXX.org/gf-cgi/gforum.cgi?category=3668

and I am presented with the correct single forum:

======i added this line and removed blank lines
Gossamer Forum
Welcome, drb
2 registered users
Forum Name
Threads
Posts
Last Post
RYER, DARLENE - 3668
NURSING
Nursing notes
0
0
AUTH
Authorization
0
0
PT
Physical Therapy
0
0
AR
Accounts receivable
0
0
= Forum has new posts - Mark all read
= Forum has no new posts
===================== end of clip

From my brief look at the error log, it seems that all the categories and forums are retrieved when I only want one.

== clip from error log
GT::SQL::Driver::MYSQL::sth (15341): Executing query:
SELECT *
FROM gforum_Category, gforum_Forum
WHERE gforum_Forum.cat_id_fk = gforum_Category.cat_id
ORDER BY forum_sort_rank ASC
from GForum::Forum::tpl_list at GForum::Forum::tpl_list line 543
=====

Any way to reformulate this query ?

Bob

BTW: I added
Time::HiRes();
to gforum.cgi but I didn't notice any times being reported.
Quote Reply
Re: [Jagerman] are there limits to number of categories/forums In reply to
#Jason,
I found the following around line 468 of Forum.pm:

==========
#
# You can call with 'user' as the argument. If you do so, everything
# will happen as above, but all permissions will be checked (so forums
# that the user does not have access to will not be shown).
# Additionally, the 'show forums/subcats' option will be looked for.
# Any closed categories will not be recursed into. A list of categories
# can be provided after the 'user' argument - only those categories
# will be shown.
Note that any subcategories specified will also have
# their parent category in the list, however only the subcategory will
# show - not the parent category's other elements. Confused yet?
#
====
Sounds like if I use a uri with usr I can also specify a category.

Could you give me an exapmle of this ?

Thanks.
Quote Reply
Re: [regert] are there limits to number of categories/forums In reply to
I looked at the module code more closely and it seems that the entire Category and Forum tables are read in order to build a hierarchical structure of Categories and Forums.

If it would make resolution of my particular question easier, I would be willing to forego anything more complex than:
Code:
Category1
forum a
forum b
forum c etc
Category2
same forums as above
Category3
same forums
etc

Can someone tell me if a module like Forum.pm has access to the uri parameters ? If I have a uri such as
xxx/gforum.cgi?my_special_flag=GET_CATEGORY_1
can I get the value of my_special_flag from within Forum.pm ?

Thanks
Post deleted by regert In reply to
Quote Reply
Re: [regert] are there limits to number of categories/forums In reply to
I tried re-organizing the forums as above and figured out how to obtain cgi parameters (pretty easy via $IN->param() ). I meddled with Forum.pm and got it to perform a limited selection from Catgory and Forum tables and could get to a specific Category-Forum fairly easily.

However when I tried to post (or do anything else) I ran into the same slow performance (literally minutes) since it seems that every trip to Forum.pm results in a read of the join of the Forum and Category tables.

Anybody from GT have any suggestions. ?

Bob
Quote Reply
Re: [regert] are there limits to number of categories/forums In reply to
My suggestion would be to run the forum on a mod_perl enabled server. This should make a huge difference.

Ivan
-----
Iyengar Yoga Resources / GT Plugins
Quote Reply
Re: [yogi] are there limits to number of categories/forums In reply to
I am as running under mod-perl.

It did make a big difference.

But still slow when using thousands of Categories or Forums
Quote Reply
Re: [regert] are there limits to number of categories/forums In reply to
Hello - how did you solve the problem of speed and thousands of forums and categories?
Quote Reply
Re: [tora] are there limits to number of categories/forums In reply to
Hi,
I never was able to get this resolved. I spoke with a few people at GT and even tried to get someone to undertake it as a paid consulting job, without success.

I eventually gave up

Bob
Quote Reply
Re: [yogi] are there limits to number of categories/forums In reply to
All my categories are set to collapsed, and I think I have made the templates as efficient as possible, like commenting out the jump-list.

Is there somewhere in the perl script that I can speed things up? You mention setting all the categories to collapsed in the script - can I by-pass some of the checking?

Thanks,

Tora