Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Trac: Users

Blog is slow

 

 

Trac users RSS feed   Index | Next | Previous | View Threaded


Timm.Essigke at uni-bayreuth

Sep 23, 2009, 3:49 PM

Post #1 of 5 (430 views)
Permalink
Blog is slow

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dear Trac users,

we are happily using trac for our Intranet since a while. Generally
performance is fine, but the blog module is very slow and we have the
feeling it is getting slower over time.

I narrowed down the problem by some extra debug lines:

2009-09-23 16:24:11,276 Trac[model] DEBUG: TE: get_blog_posts() SQL 116:
'SELECT bp1.name, bp1.version, bp1.publish_time, bp1.author, bp1.title,
bp1.body, bp1.categories FROM fullblog_posts bp1 ,(SELECT name,
max(version) AS ver FROM fullblog_posts GROUP BY name) bp2 WHERE
bp1.version = bp2.ver AND bp1.name = bp2.name ORDER BY bp1.publish_time
DESC' (None)
2009-09-23 16:24:11,455 Trac[model] DEBUG: TE: get_blog_posts() SQL:
query finished
2009-09-23 16:24:11,467 Trac[model] DEBUG: TE: get_blog_posts(): returning
2009-09-23 16:24:30,565 Trac[session] DEBUG: Retrieving session for ID
u'XXX [at] DOMAIN'

The method get_blog_post is in
/usr/lib/python2.5/site-packages/TracFullBlogPlugin-0.1-py2.5.egg/tracfullblog/model.py.
First, I was suspecting the SELECT statement to take about 20 sec, but
by adding some extra debug lines it turned out to be fast. Instead trac
blocks after leaving the method, but I don't know how to debug python
code inside apache (e.g. get a call graph or run pdb). Can anybody tell
me what is done before the "Retrieving session" debug message is given?

I saw that about 200 MB of NFS traffic are transferred within the 20 sec
between the NFS server hosting the trac projects and the trac server.
Why so much data needs to pass the network? My trac.db is only 3.4 MB -
what else is required?
Usually, NFS is ten times faster in my setup (at least for large files).
Googleing was not helpful...

Here is my system information:
Trac: 0.11.5
Python: 2.5.4 (r254:67916, Feb 18 2009, 03:17:34) [GCC 4.3.3]
setuptools: 0.6c9
SQLite: 3.6.17
pysqlite: 2.3.2
Genshi: 0.6dev-r999
mod_python: 3.3.1
jQuery: 1.2.6

Thanks in advance!

Timm
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKuqXd9cEzJ2deIqIRAuP1AJ4xLK/X6rT6HeVuHD/MAIqrwa0LRACfWR9A
b4wLNtodquZZZ2egYho+O64=
=lswP
-----END PGP SIGNATURE-----

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Users" group.
To post to this group, send email to trac-users [at] googlegroups
To unsubscribe from this group, send email to trac-users+unsubscribe [at] googlegroups
For more options, visit this group at http://groups.google.com/group/trac-users?hl=en
-~----------~----~----~----~------~----~------~--~---


oddsimons at gmail

Sep 24, 2009, 3:38 PM

Post #2 of 5 (417 views)
Permalink
Re: Blog is slow [In reply to]

On Sep 24, 12:49 am, Timm Essigke <Timm.Essi...@uni-bayreuth.de>
wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dear Trac users,
>
> we are happily using trac for our Intranet since a while. Generally
> performance is fine, but the blog module is very slow and we have the
> feeling it is getting slower over time.
>
> I narrowed down the problem by some extra debug lines:
>
> 2009-09-23 16:24:11,276 Trac[model] DEBUG: TE: get_blog_posts() SQL 116:
> 'SELECT bp1.name, bp1.version, bp1.publish_time, bp1.author, bp1.title,
> bp1.body, bp1.categories FROM fullblog_posts bp1 ,(SELECT name,
> max(version) AS ver FROM fullblog_posts GROUP BY name) bp2 WHERE
> bp1.version = bp2.ver AND bp1.name = bp2.name ORDER BY bp1.publish_time
> DESC' (None)
> 2009-09-23 16:24:11,455 Trac[model] DEBUG: TE: get_blog_posts() SQL:
> query finished
> 2009-09-23 16:24:11,467 Trac[model] DEBUG: TE: get_blog_posts(): returning
> 2009-09-23 16:24:30,565 Trac[session] DEBUG: Retrieving session for ID
> u'...@DOMAIN.NET'
>
> The method get_blog_post is in
> /usr/lib/python2.5/site-packages/TracFullBlogPlugin-0.1-py2.5.egg/tracfullblog/model.py.
> First, I was suspecting the SELECT statement to take about 20 sec, but
> by adding some extra debug lines it turned out to be fast. Instead trac
> blocks after leaving the method, but I don't know how to debug python
> code inside apache (e.g. get a call graph or run pdb). Can anybody tell
> me what is done before the "Retrieving session" debug message is given?
>
> I saw that about 200 MB of NFS traffic are transferred within the 20 sec
> between the NFS server hosting the trac projects and the trac server.
> Why so much data needs to pass the network? My trac.db is only 3.4 MB -
> what else is required?
> Usually, NFS is ten times faster in my setup (at least for large files).
> Googleing was not helpful...
>
> Here is my system information:
> Trac:   0.11.5
> Python:         2.5.4 (r254:67916, Feb 18 2009, 03:17:34) [GCC 4.3.3]
> setuptools:     0.6c9
> SQLite:         3.6.17
> pysqlite:       2.3.2
> Genshi:         0.6dev-r999
> mod_python:     3.3.1
> jQuery: 1.2.6
>
> Thanks in advance!
>
> Timm

Hard to tell really without knowing the exact request you make, what
data you have, db backend, what other plugins and security policies
may be in place, site.html and whatever else may affect page
generation. The anatomy of a request is generally quite simple: find
the module (blog), have module process the request and collect the
data it needs (part of what you debug), and render the template using
the data provided. Intermixed in this is various calls to check
permissions +++.

I've got some scripts i use to profile requests, so perhaps you can
ping me on #trac IRC channel (I'm 'osimons') and we can try to debug/
profile your setup and see if anything interesting shows up?



:::simon

https://www.coderesort.com
http://www.ohloh.net/accounts/osimons

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Users" group.
To post to this group, send email to trac-users [at] googlegroups
To unsubscribe from this group, send email to trac-users+unsubscribe [at] googlegroups
For more options, visit this group at http://groups.google.com/group/trac-users?hl=en
-~----------~----~----~----~------~----~------~--~---


jjp at pobox

Mar 14, 2010, 1:12 PM

Post #3 of 5 (306 views)
Permalink
Re: Blog is slow [In reply to]

On Sep 24 2009, 6:38 pm, osimons <oddsim...@gmail.com> wrote:
> On Sep 24, 12:49 am, Timm Essigke <Timm.Essi...@uni-bayreuth.de>
> wrote:
> > Dear Trac users,
>
> > we are happily using trac for our Intranet since a while. Generally
> > performance is fine, but theblogmodule is veryslowand we have the
> > feeling it is getting slower over time.
>
> > I narrowed down the problem by some extra debug lines:
> >
> > ... snipped...
> >
> > Here is my system information:
> > Trac:   0.11.5
> > Python:         2.5.4 (r254:67916, Feb 18 2009, 03:17:34) [GCC 4.3.3]
> > setuptools:     0.6c9
> > SQLite:         3.6.17
> > pysqlite:       2.3.2
> > Genshi:         0.6dev-r999
> > mod_python:     3.3.1
> > jQuery: 1.2.6
>
> > Thanks in advance!
>
> > Timm
>
> Hard to tell really without knowing the exact request you make, what
> data you have, db backend, what other plugins and security policies
> may be in place, site.html and whatever else may affect page
> generation. The anatomy of a request is generally quite simple: find
> the module (blog), have module process the request and collect the
> data it needs (part of what you debug), and render the template using
> the data provided. Intermixed in this is various calls to check
> permissions +++.
>
> I've got some scripts i use to profile requests, so perhaps you can
> ping me on #trac IRC channel (I'm 'osimons') and we can try to debug/
> profile your setup and see if anything interesting shows up?

[.Resurrecting this old thread, since it seems to fit our symptoms
exactly.]

We experienced a similar slowdown in the blog plugin, getting
progressively worse as more posts were added. Our site has around
1,100 posts now, and any of the blog pages was taking upwards of 10s
to load.

With a little debugging, I discovered that the blog plugin was
checking the permissions of every single post ever for every page load
to display the statistics in the right hand column. While we have
permissions associated with some aspects of our trac instance, namely
wiki pages and subversion directories, we have no permissions at all
set on the blog, so this checking is unnecessary in our case. Looking
through the source, it appeared that if authorization was used at all,
then it was forced to be checked for every entry and there was no way
to configure around it. I hacked around this for our installation
with the following patch (to a possibly ancient r5676 of the blog
plugin). With the patch, loads of blog pages are back to sub second.

Is there a recommended way to use the blog plugin with large numbers
of posts and permissions (or a more appropriate place to ask the
question?).

Regards,
Josh Pieper

--- ../fullblogplugin.orig-20100314/0.11/tracfullblog/web_ui.py 2010-03-14 15:21:10.000000000 -0400
+++ 0.11/tracfullblog/web_ui.py 2010-03-14 15:42:24.000000000 -0400
@@ -350,10 +350,9 @@
data['context'] = Context.from_request(req, absurls=True)
return 'fullblog.rss', data, 'application/rss+xml'

data['blog_months'], data['blog_authors'], data['blog_categories'], \
data['blog_total'] = \
- blog_core.get_months_authors_categories(
- user=req.authname, perm=req.perm)
+ blog_core.get_months_authors_categories()
if 'BLOG_CREATE' in req.perm('blog'):
add_ctxtnav(req, 'New Post', href=req.href.blog('create'),
title="Create new Blog Post")

--
You received this message because you are subscribed to the Google Groups "Trac Users" group.
To post to this group, send email to trac-users [at] googlegroups
To unsubscribe from this group, send email to trac-users+unsubscribe [at] googlegroups
For more options, visit this group at http://groups.google.com/group/trac-users?hl=en.


olemis at gmail

Mar 15, 2010, 5:32 AM

Post #4 of 5 (305 views)
Permalink
Re: Blog is slow [In reply to]

On Wed, Sep 23, 2009 at 5:49 PM, Timm Essigke
<Timm.Essigke [at] uni-bayreuth> wrote:
>
[...]
>
> The method get_blog_post is in
> /usr/lib/python2.5/site-packages/TracFullBlogPlugin-0.1-py2.5.egg/tracfullblog/model.py.
> First, I was suspecting the SELECT statement to take about 20 sec, but
> by adding some extra debug lines it turned out to be fast. Instead trac
> blocks after leaving the method, but I don't know how to debug python
> code inside apache (e.g. get a call graph or run pdb).

What I usually do (while dev-ing) is to prepare a PyDev (Eclipse)
project and run tracd in debug mode (e.g. that's how I found the exact
location where old 1.4.x SVN bindings raised an ugly segmentation
fault while running Trac on a particular Windows PC ;o). If the
problem is not specific to Apache then this might help (and I suppose
that something similar happens with pdb et al ...)

;o)

--
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
Soporte para Hessian (RPC) en Trac -
http://feedproxy.google.com/~r/simelo-es/~3/1vNiBD5LjP8/soporte-para-hessian-rpc-en-trac.html

--
You received this message because you are subscribed to the Google Groups "Trac Users" group.
To post to this group, send email to trac-users [at] googlegroups
To unsubscribe from this group, send email to trac-users+unsubscribe [at] googlegroups
For more options, visit this group at http://groups.google.com/group/trac-users?hl=en.


oddsimons at gmail

Mar 17, 2010, 3:49 AM

Post #5 of 5 (296 views)
Permalink
Re: Blog is slow [In reply to]

On Mar 14, 9:12 pm, Josh Pieper <j...@pobox.com> wrote:
> We experienced a similar slowdown in the blog plugin, getting
> progressively worse as more posts were added.  Our site has around
> 1,100 posts now, and any of the blog pages was taking upwards of 10s
> to load.
>
> With a little debugging, I discovered that the blog plugin was
> checking the permissions of every single post ever for every page load
> to display the statistics in the right hand column.  While we have
> permissions associated with some aspects of our trac instance, namely
> wiki pages and subversion directories, we have no permissions at all
> set on the blog, so this checking is unnecessary in our case.  Looking
> through the source, it appeared that if authorization was used at all,
> then it was forced to be checked for every entry and there was no way
> to configure around it.  I hacked around this for our installation
> with the following patch (to a possibly ancient r5676 of the blog
> plugin).  With the patch, loads of blog pages are back to sub second.
>
> Is there a recommended way to use the blog plugin with large numbers
> of posts and permissions (or a more appropriate place to ask the
> question?).

2 things spring to mind:

1) Update to latest version of FullBlogPlugin from trac-hacks.org svn
repos. I've added some new database indexes since r5676, so I would
hope that you should see some improvement even without your patch.
Would be nice to know if that worked actually - could you update and
report back on general performance?

2) Your patch I cannot use, obviously. The plugin needs to respect
whatever permissions exists for any post, even permissions handled
through permission policy plugins that are not obviously visible. What
you could do, however, is implement your patch as a permission plugin
and insert it first into your [trac] permission_policies = ... (list)
- that way your plugin gets evaluated first and if it it just contains
this one line then the permission-checking round-trip should be really
low overhead and would mean you can avoid patching the plugin source::
if action == 'BLOG_VIEW': return True

Thanks for your input on performance, and I'd be happy to help adjust
what can be reasonably done to handle high volume and activity.
Particularly some sort of caching of the sidebar content would be
useful.



:::simon

https://www.coderesort.com
http://www.ohloh.net/accounts/osimons

--
You received this message because you are subscribed to the Google Groups "Trac Users" group.
To post to this group, send email to trac-users [at] googlegroups
To unsubscribe from this group, send email to trac-users+unsubscribe [at] googlegroups
For more options, visit this group at http://groups.google.com/group/trac-users?hl=en.

Trac users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.