Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Wikipedia: Wikitech

loading of extensions in 1.12 slow?

 

 

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded


travis at wikihow

May 20, 2008, 9:32 AM

Post #1 of 10 (679 views)
Permalink
loading of extensions in 1.12 slow?

Hi,

We're in the process of upgrading from 1.9 to 1.12 (I know I know I
know...), and it seems that the including of extensions is taking a long
time.

Profile from 1.9:

26.449 WebStart.php-conf
24.659 Setup.php
1.362 Setup.php-includes
2.117 Setup.php-misc1
1.020 Setup.php-memcached
0.570 Setup.php-SetupSession
0.117 Setup.php-globals
0.052 Setup.php-User
0.017 Setup.php-misc2
18.543 Setup.php-extensions

Seems fast, but 1.12:

594.761 WebStart.php-conf
0.012 WebStart.php-ob_start
207.409 Setup.php
50.838 Setup.php-includes
8.691 Setup.php-misc1
6.780 __autoload
9.632 Setup.php-memcached
0.685 __autoload
0.564 Setup.php-SetupSession
0.197 Setup.php-globals
0.046 Setup.php-User
0.023 Setup.php-misc2
128.051 Setup.php-extensions

This is pretty slow. It seems like for WebStart, having multiple inclusions
like this:

require_once('extensions/wikihow/SpecialThankAuthors.php');

slows things down. Has something changed, or did we miss flipping a switch
in the upgrade? What about Setup.php, does that have the same issue?

Thanks,
Travis
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Simetrical+wikilist at gmail

May 20, 2008, 10:01 AM

Post #2 of 10 (640 views)
Permalink
Re: loading of extensions in 1.12 slow? [In reply to]

On Tue, May 20, 2008 at 12:32 PM, Travis (wikiHow) <travis [at] wikihow> wrote:
> Hi,
>
> We're in the process of upgrading from 1.9 to 1.12 (I know I know I
> know...), and it seems that the including of extensions is taking a long
> time.

You may want to try asking Domas about this. I know there have been
performance problems in loading extensions caused by extremely
thorough localization in recent versions (e.g., Cite's
internationalization file is 117 KB in 64 languages), but I'm pretty
sure those hits have at least been deferred until the messages are
needed.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


roan.kattouw at home

May 20, 2008, 10:50 AM

Post #3 of 10 (638 views)
Permalink
Re: loading of extensions in 1.12 slow? [In reply to]

Simetrical schreef:
> On Tue, May 20, 2008 at 12:32 PM, Travis (wikiHow) <travis [at] wikihow> wrote:
>
>> Hi,
>>
>> We're in the process of upgrading from 1.9 to 1.12 (I know I know I
>> know...), and it seems that the including of extensions is taking a long
>> time.
>>
>
> You may want to try asking Domas about this. I know there have been
> performance problems in loading extensions caused by extremely
> thorough localization in recent versions (e.g., Cite's
> internationalization file is 117 KB in 64 languages), but I'm pretty
> sure those hits have at least been deferred until the messages are
> needed.
>
>
You might wanna read serialized/README , it can improve performance in
this regard. Also, upgrade your extensions if you haven't done so
already. A few months ago, pretty much all extensions in our SVN
repository were changed to only load their messages when they actually
need them, rather than loading them for every request.

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


travis at wikihow

Jun 6, 2008, 12:41 PM

Post #4 of 10 (602 views)
Permalink
Re: loading of extensions in 1.12 slow? [In reply to]

I updated all of our extensions, and redesigned our custom extensions so
that they lazy load messages and classes, it seemed to help, but didn't
address Setup.php, and is still fairly slow:

152.432 WebStart.php-conf
0.011 WebStart.php-ob_start
236.918 Setup.php
53.863 Setup.php-includes
9.218 Setup.php-misc1
6.839 __autoload
9.986 Setup.php-memcached
0.713 __autoload
0.608 Setup.php-SetupSession
0.212 Setup.php-globals
0.053 Setup.php-User
0.029 Setup.php-misc2
153.725 Setup.php-extensions

Is there any reason why the Cite extension takes 60ms to load every time?

61.447 Setup.php-extensions-wfCite

Any other ideas? Is there any reason why this load time has jumped from 1.9
to 1.12?

Thanks,
Travis



On Tue, May 20, 2008 at 1:50 PM, Roan Kattouw <roan.kattouw [at] home> wrote:

> Simetrical schreef:
> > On Tue, May 20, 2008 at 12:32 PM, Travis (wikiHow) <travis [at] wikihow>
> wrote:
> >
> >> Hi,
> >>
> >> We're in the process of upgrading from 1.9 to 1.12 (I know I know I
> >> know...), and it seems that the including of extensions is taking a long
> >> time.
> >>
> >
> > You may want to try asking Domas about this. I know there have been
> > performance problems in loading extensions caused by extremely
> > thorough localization in recent versions (e.g., Cite's
> > internationalization file is 117 KB in 64 languages), but I'm pretty
> > sure those hits have at least been deferred until the messages are
> > needed.
> >
> >
> You might wanna read serialized/README , it can improve performance in
> this regard. Also, upgrade your extensions if you haven't done so
> already. A few months ago, pretty much all extensions in our SVN
> repository were changed to only load their messages when they actually
> need them, rather than loading them for every request.
>
> Roan Kattouw (Catrope)
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Simetrical+wikilist at gmail

Jun 6, 2008, 1:26 PM

Post #5 of 10 (597 views)
Permalink
Re: loading of extensions in 1.12 slow? [In reply to]

On Fri, Jun 6, 2008 at 3:41 PM, Travis (wikiHow) <travis [at] wikihow> wrote:
> I updated all of our extensions, and redesigned our custom extensions so
> that they lazy load messages and classes, it seemed to help, but didn't
> address Setup.php, and is still fairly slow:
>
> 152.432 WebStart.php-conf
> 0.011 WebStart.php-ob_start
> 236.918 Setup.php
> 53.863 Setup.php-includes
> 9.218 Setup.php-misc1
> 6.839 __autoload
> 9.986 Setup.php-memcached
> 0.713 __autoload
> 0.608 Setup.php-SetupSession
> 0.212 Setup.php-globals
> 0.053 Setup.php-User
> 0.029 Setup.php-misc2
> 153.725 Setup.php-extensions
>
> Is there any reason why the Cite extension takes 60ms to load every time?
>
> 61.447 Setup.php-extensions-wfCite
>
> Any other ideas? Is there any reason why this load time has jumped from 1.9
> to 1.12?

I don't know. Maybe you could try doing some more detailed profiling
to see what the issue is? If there is an easily-fixable problem here,
I'm sure we'd all like to fix it. If I'm reading this correctly,
Setup.php is only taking 0.11 ms on Wikimedia, three orders of
magnitude less:

http://noc.wikimedia.org/cgi-bin/ng/report.py?db=bigpage&sort=real&sample=&compare=&limit=5000

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


brion at wikimedia

Jun 6, 2008, 1:41 PM

Post #6 of 10 (606 views)
Permalink
Re: loading of extensions in 1.12 slow? [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Travis (wikiHow) wrote:
> I updated all of our extensions, and redesigned our custom extensions so
> that they lazy load messages and classes, it seemed to help, but didn't
> address Setup.php, and is still fairly slow:
>
> 152.432 WebStart.php-conf
> 0.011 WebStart.php-ob_start
> 236.918 Setup.php
> 53.863 Setup.php-includes
> 9.218 Setup.php-misc1
> 6.839 __autoload
> 9.986 Setup.php-memcached
> 0.713 __autoload
> 0.608 Setup.php-SetupSession
> 0.212 Setup.php-globals
> 0.053 Setup.php-User
> 0.029 Setup.php-misc2
> 153.725 Setup.php-extensions
>
> Is there any reason why the Cite extension takes 60ms to load every time?
>
> 61.447 Setup.php-extensions-wfCite

Cite is triggering an unstub of the parser for its initialization (and
thus a bunch of other code gets loaded, making the profiling lopsided).

I've just broken out the setHook() calls to ParserFirstCallInit hooks
for Cite and several other extensions we use (in r35980). This has
already been done in a number of other extensions.

- -- brion vibber (brion @ wikimedia.org)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkhJoOkACgkQwRnhpk1wk44k8ACdGH34YZoVbhS4TeTrBOV8vD89
zd4An17iOAGkDaT4FCSUf+GlfxpIGCuy
=FC5X
-----END PGP SIGNATURE-----

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


travis at wikihow

Jun 9, 2008, 8:37 AM

Post #7 of 10 (596 views)
Permalink
Re: loading of extensions in 1.12 slow? [In reply to]

Hi,

It seems part of our problem in Setup.php is the setting of function hooks.
Just to clarify, this is the most efficient way to set a function hook,
right?

$wgExtensionFunctions[] = 'wfWikiHowParserFunction_Setup';

function wfWikiHowParserFunction_Setup() {
global $wgParser;
$wgParser->setFunctionHook( 'ARTICLESSTARTED', 'articlesstarted');
$wgParser->setFunctionHook( 'PATROLCOUNT', 'patrolcount');
}

We have 2 different extensions that set up function hooks like this, it
seems whichever one is executed first, takes 100ms, and the next one takes
about 7ms, is there something that's initializing with wgParser on the first
call? Is there anyway to cut down that first 100ms call?

Thanks
Travis



On Fri, Jun 6, 2008 at 4:26 PM, Simetrical
<Simetrical+wikilist [at] gmail<Simetrical%2Bwikilist [at] gmail>>
wrote:

> On Fri, Jun 6, 2008 at 3:41 PM, Travis (wikiHow) <travis [at] wikihow>
> wrote:
> > I updated all of our extensions, and redesigned our custom extensions so
> > that they lazy load messages and classes, it seemed to help, but didn't
> > address Setup.php, and is still fairly slow:
> >
> > 152.432 WebStart.php-conf
> > 0.011 WebStart.php-ob_start
> > 236.918 Setup.php
> > 53.863 Setup.php-includes
> > 9.218 Setup.php-misc1
> > 6.839 __autoload
> > 9.986 Setup.php-memcached
> > 0.713 __autoload
> > 0.608 Setup.php-SetupSession
> > 0.212 Setup.php-globals
> > 0.053 Setup.php-User
> > 0.029 Setup.php-misc2
> > 153.725 Setup.php-extensions
> >
> > Is there any reason why the Cite extension takes 60ms to load every time?
> >
> > 61.447 Setup.php-extensions-wfCite
> >
> > Any other ideas? Is there any reason why this load time has jumped from
> 1.9
> > to 1.12?
>
> I don't know. Maybe you could try doing some more detailed profiling
> to see what the issue is? If there is an easily-fixable problem here,
> I'm sure we'd all like to fix it. If I'm reading this correctly,
> Setup.php is only taking 0.11 ms on Wikimedia, three orders of
> magnitude less:
>
>
> http://noc.wikimedia.org/cgi-bin/ng/report.py?db=bigpage&sort=real&sample=&compare=&limit=5000
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


brion at wikimedia

Jun 9, 2008, 8:52 AM

Post #8 of 10 (583 views)
Permalink
Re: loading of extensions in 1.12 slow? [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Travis (wikiHow) wrote:
> It seems part of our problem in Setup.php is the setting of function hooks.
> Just to clarify, this is the most efficient way to set a function hook,
> right?
>
> $wgExtensionFunctions[] = 'wfWikiHowParserFunction_Setup';
>
> function wfWikiHowParserFunction_Setup() {
> global $wgParser;
> $wgParser->setFunctionHook( 'ARTICLESSTARTED', 'articlesstarted');
> $wgParser->setFunctionHook( 'PATROLCOUNT', 'patrolcount');
> }

This will force $wgParser to be unstubbed at setup time. This isn't the
end of the world, but it will have a couple of effects:

* If you don't have a working PHP opcode cache, this means that the big
Parser class gets loaded whether you need it or not

* Actual parser object initialization happens whether you need it or not

* Parser initialization will be accounted in the extension setup phase
in your profiling, which may skew the visible results.

In practice, this probably won't cause too much trouble. The parser is
going to get unstubbed on just about any request that outputs HTML;
it'll get used for UI message transformations even if you're outputting
a page from the parser cache.

But for 304 "not changed" results, and potentially for some API hits and
stuff, it might shave off a few milliseconds to be able to skip it.


To do that, you want to call setFunctionHook() from a parser first call
init hook, as currently done by a lot of our extensions, as mentioned in
my last post in this thread.

- -- brion

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkhNUa4ACgkQwRnhpk1wk45pugCgs6ESM+jWNVsJFRQUxlK4og0B
GZgAn3bIJwUZbjnHaZCwekCuvNMoDXr0
=Vnq+
-----END PGP SIGNATURE-----

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


travis at wikihow

Jun 9, 2008, 10:58 AM

Post #9 of 10 (591 views)
Permalink
Re: loading of extensions in 1.12 slow? [In reply to]

Great, that makes total sense and seems to have worked well. I think it
looks like for article views, 1.12 is performing better than 1.9 now, thanks
for the help.

For opcode caching, we're using eaccelerator, should we be using something
else? What does WP use?

Also, I noticed that the PHP sapi name for WP is just apache, while ours is
apache2handler. Could there be a difference in terms of performance between
the 2? How do you go about creating just an apache module?

Thanks,
Travis


On Mon, Jun 9, 2008 at 11:52 AM, Brion Vibber <brion [at] wikimedia> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Travis (wikiHow) wrote:
> > It seems part of our problem in Setup.php is the setting of function
> hooks.
> > Just to clarify, this is the most efficient way to set a function hook,
> > right?
> >
> > $wgExtensionFunctions[] = 'wfWikiHowParserFunction_Setup';
> >
> > function wfWikiHowParserFunction_Setup() {
> > global $wgParser;
> > $wgParser->setFunctionHook( 'ARTICLESSTARTED', 'articlesstarted');
> > $wgParser->setFunctionHook( 'PATROLCOUNT', 'patrolcount');
> > }
>
> This will force $wgParser to be unstubbed at setup time. This isn't the
> end of the world, but it will have a couple of effects:
>
> * If you don't have a working PHP opcode cache, this means that the big
> Parser class gets loaded whether you need it or not
>
> * Actual parser object initialization happens whether you need it or not
>
> * Parser initialization will be accounted in the extension setup phase
> in your profiling, which may skew the visible results.
>
> In practice, this probably won't cause too much trouble. The parser is
> going to get unstubbed on just about any request that outputs HTML;
> it'll get used for UI message transformations even if you're outputting
> a page from the parser cache.
>
> But for 304 "not changed" results, and potentially for some API hits and
> stuff, it might shave off a few milliseconds to be able to skip it.
>
>
> To do that, you want to call setFunctionHook() from a parser first call
> init hook, as currently done by a lot of our extensions, as mentioned in
> my last post in this thread.
>
> - -- brion
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkhNUa4ACgkQwRnhpk1wk45pugCgs6ESM+jWNVsJFRQUxlK4og0B
> GZgAn3bIJwUZbjnHaZCwekCuvNMoDXr0
> =Vnq+
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


brion at wikimedia

Jun 9, 2008, 11:22 AM

Post #10 of 10 (582 views)
Permalink
Re: loading of extensions in 1.12 slow? [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Travis (wikiHow) wrote:
> Great, that makes total sense and seems to have worked well. I think it
> looks like for article views, 1.12 is performing better than 1.9 now, thanks
> for the help.
>
> For opcode caching, we're using eaccelerator, should we be using something
> else? What does WP use?

We use APC; last I looked eaccelerator was pretty flake on PHP 5,
especially 5.1 and later, but that might have changed. YMMV. :)

> Also, I noticed that the PHP sapi name for WP is just apache, while ours is
> apache2handler. Could there be a difference in terms of performance between
> the 2? How do you go about creating just an apache module?

As I recall...

apache = Apache 1.3 module
apache2 = Apache 2.0 module
apache2handler = Apache 2.0 module using a different Apache API

There's probably not a huge difference in performance, but again YMMV. :)

If in doubt, try them out.

- -- brion
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkhNdNYACgkQwRnhpk1wk47IdACfVOiwG5DUGprQBQZ3PE8/xV0g
9hkAoMeew8am9JD7ha5Yp3yYA9YExuYU
=mWxa
-----END PGP SIGNATURE-----

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.