Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Wikipedia: Wikitech

English Wikipedia slow to load

 

 

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded


z at mzmcbride

Mar 23, 2012, 10:00 PM

Post #1 of 10 (838 views)
Permalink
English Wikipedia slow to load

Hi.

There are over a dozen reports of general slowness with the English
Wikipedia:
https://en.wikipedia.org/w/index.php?oldid=483631204#is_it_me_or_is_wiki_ver
y_slow.3F

As I browsed and edited the site today, I experienced similar issues
(painful slowness on intermittent page loads, loading half a page and then
stalling, etc.).

Is anyone from ops or engineering looking into this? I've filed a bug about
this as well: <https://bugzilla.wikimedia.org/show_bug.cgi?id=35448>.

It may be that this is a known issue (due to schema changes or whatever
else), but I don't think anyone from ops or engineering has commented on the
village pump (or elsewhere?). Can someone please take a look at what's going
on?

MZMcBride



_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


afeldman at wikimedia

Mar 24, 2012, 9:16 AM

Post #2 of 10 (812 views)
Permalink
Re: English Wikipedia slow to load [In reply to]

It looks like the problem started 3/21, see attached image. I'm about to
head to the airport for ~20 hours of travel so can't investigate further,
but may be related to parser changes made that day which line up with the
second bigger spike.

- 05:23 logmsgbot: tstarling synchronized
php-1.19/includes/parser/CoreParserFunctions.php
- 05:23 logmsgbot: tstarling synchronized
php-1.19/includes/parser/Parser.php
- 05:23 logmsgbot: tstarling synchronized
php-1.19/includes/parser/StripState.php
- 05:22 logmsgbot: tstarling synchronized
php-1.19/tests/parser/parserTests.txt

There was also a the release of a majorly changed MobileFrontend earlier
that day / prior night that line up with the first spike. The MF rewrite
doesn't perform well - MobileFrontend::DOMParse avg time went from 15ms to
150ms (~500ms at 99th) and it wouldn't be impossible for it to also impact
non-mobile performance.

DB, ES, and memcache latency look steady, so I suspect it's all application
side.

-Asher

On Fri, Mar 23, 2012 at 10:00 PM, MZMcBride <z [at] mzmcbride> wrote:

> Hi.
>
> There are over a dozen reports of general slowness with the English
> Wikipedia:
>
> https://en.wikipedia.org/w/index.php?oldid=483631204#is_it_me_or_is_wiki_ver
> y_slow.3F
>
> As I browsed and edited the site today, I experienced similar issues
> (painful slowness on intermittent page loads, loading half a page and then
> stalling, etc.).
>
> Is anyone from ops or engineering looking into this? I've filed a bug about
> this as well: <https://bugzilla.wikimedia.org/show_bug.cgi?id=35448>.
>
> It may be that this is a known issue (due to schema changes or whatever
> else), but I don't think anyone from ops or engineering has commented on
> the
> village pump (or elsewhere?). Can someone please take a look at what's
> going
> on?
>
> MZMcBride
>
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
Attachments: enwiki-latency.png (71.9 KB)


maxsem.wiki at gmail

Mar 24, 2012, 12:15 PM

Post #3 of 10 (784 views)
Permalink
Re: English Wikipedia slow to load [In reply to]

On 24.03.2012, 20:16 Asher wrote:

> There was also a the release of a majorly changed MobileFrontend earlier
> that day / prior night that line up with the first spike. The MF rewrite
> doesn't perform well - MobileFrontend::DOMParse avg time went from 15ms to
> 150ms (~500ms at 99th) and it wouldn't be impossible for it to also impact
> non-mobile performance.

> DB, ES, and memcache latency look steady, so I suspect it's all application
> side.

Thanks for the heads up, Asher. I've committed thorough profiling in
<https://gerrit.wikimedia.org/r/3696>, would appreciate if someone
deployed it.


--
Best regards,
Max Semenik ([[User:MaxSem]])


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


preilly at wikimedia

Mar 24, 2012, 12:33 PM

Post #4 of 10 (786 views)
Permalink
Re: English Wikipedia slow to load [In reply to]

On Sat, Mar 24, 2012 at 12:15 PM, Max Semenik <maxsem.wiki [at] gmail> wrote:
> On 24.03.2012, 20:16 Asher wrote:
>
>> There was also a the release of a majorly changed MobileFrontend earlier
>> that day / prior night that line up with the first spike.  The MF rewrite
>> doesn't perform well - MobileFrontend::DOMParse avg time went from 15ms to
>> 150ms (~500ms at 99th) and it wouldn't be impossible for it to also impact
>> non-mobile performance.
>
>> DB, ES, and memcache latency look steady, so I suspect it's all application
>> side.
>
> Thanks for the heads up, Asher. I've committed thorough profiling in
> <https://gerrit.wikimedia.org/r/3696>, would appreciate if someone
> deployed it.

I've reviewed and merged your profiling code. We need to create a
patch against the deployed version at:
https://svn.wikimedia.org/viewvc/mediawiki/branches/wmf/1.19wmf1/extensions/MobileFrontend/MobileFrontend.body.php?revision=114476&view=co

Once, that patch is created I'll apply it and push it live to production.
This should give us better insight into why this slowdown in
performance is occurring.

>
>
> --
> Best regards,
>  Max Semenik ([[User:MaxSem]])
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


preilly at wikimedia

Mar 24, 2012, 12:51 PM

Post #5 of 10 (784 views)
Permalink
Re: English Wikipedia slow to load [In reply to]

On Sat, Mar 24, 2012 at 12:33 PM, Patrick Reilly <preilly [at] wikimedia> wrote:
> On Sat, Mar 24, 2012 at 12:15 PM, Max Semenik <maxsem.wiki [at] gmail> wrote:
>> On 24.03.2012, 20:16 Asher wrote:
>>
>>> There was also a the release of a majorly changed MobileFrontend earlier
>>> that day / prior night that line up with the first spike.  The MF rewrite
>>> doesn't perform well - MobileFrontend::DOMParse avg time went from 15ms to
>>> 150ms (~500ms at 99th) and it wouldn't be impossible for it to also impact
>>> non-mobile performance.
>>
>>> DB, ES, and memcache latency look steady, so I suspect it's all application
>>> side.
>>
>> Thanks for the heads up, Asher. I've committed thorough profiling in
>> <https://gerrit.wikimedia.org/r/3696>, would appreciate if someone
>> deployed it.
>
> I've reviewed and merged your profiling code. We need to create a
> patch against the deployed version at:
> https://svn.wikimedia.org/viewvc/mediawiki/branches/wmf/1.19wmf1/extensions/MobileFrontend/MobileFrontend.body.php?revision=114476&view=co
>
> Once, that patch is created I'll apply it and push it live to production.
> This should give us better insight into why this slowdown in
> performance is occurring.

Okay, this revision:
https://www.mediawiki.org/wiki/Special:Code/MediaWiki/114477 was
pushed live at 12:46pm PDT.

So, we should be able to see a clear graph in about 30 minutes.

>
>>
>>
>> --
>> Best regards,
>>  Max Semenik ([[User:MaxSem]])
>>
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> Wikitech-l [at] lists
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


maxsem.wiki at gmail

Mar 24, 2012, 1:36 PM

Post #6 of 10 (796 views)
Permalink
Re: English Wikipedia slow to load [In reply to]

On 24.03.2012, 23:51 Patrick wrote:

> On Sat, Mar 24, 2012 at 12:33 PM, Patrick Reilly <preilly [at] wikimedia> wrote:
>> On Sat, Mar 24, 2012 at 12:15 PM, Max Semenik <maxsem.wiki [at] gmail> wrote:
>>> On 24.03.2012, 20:16 Asher wrote:
>>>
>>>> There was also a the release of a majorly changed MobileFrontend earlier
>>>> that day / prior night that line up with the first spike.  The MF rewrite
>>>> doesn't perform well - MobileFrontend::DOMParse avg time went from 15ms to
>>>> 150ms (~500ms at 99th) and it wouldn't be impossible for it to also impact
>>>> non-mobile performance.
>>>
>>>> DB, ES, and memcache latency look steady, so I suspect it's all application
>>>> side.
>>>
>>> Thanks for the heads up, Asher. I've committed thorough profiling in
>>> <https://gerrit.wikimedia.org/r/3696>, would appreciate if someone
>>> deployed it.
>>
>> I've reviewed and merged your profiling code. We need to create a
>> patch against the deployed version at:
>> https://svn.wikimedia.org/viewvc/mediawiki/branches/wmf/1.19wmf1/extensions/MobileFrontend/MobileFrontend.body.php?revision=114476&view=co
>>
>> Once, that patch is created I'll apply it and push it live to production.
>> This should give us better insight into why this slowdown in
>> performance is occurring.

> Okay, this revision:
> https://www.mediawiki.org/wiki/Special:Code/MediaWiki/114477 was
> pushed live at 12:46pm PDT.

> So, we should be able to see a clear graph in about 30 minutes.

And these graphs don't make any sense:
https://graphite.wikimedia.org/dashboard/MobileFrontend-DOMParse
the whole function's execution time is much larger than sum of its
pieces. What's going on?

--
Best regards,
Max Semenik ([[User:MaxSem]])


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Platonides at gmail

Mar 24, 2012, 3:28 PM

Post #7 of 10 (801 views)
Permalink
Re: English Wikipedia slow to load [In reply to]

On 24/03/12 21:36, Max Semenik wrote:
> And these graphs don't make any sense:
> https://graphite.wikimedia.org/dashboard/MobileFrontend-DOMParse
> the whole function's execution time is much larger than sum of its
> pieces. What's going on?

Maybe it's a recursive function?
I would have suspected that the parser fixes affected some common
extension making it slower.
Ganglia output shows that CPU load decreased, though. From a steady load
over 10k it became much more flaky, dropping to 9k (due to full
operations being slower?).

Is the revision hashing still running? If there are processes fetching
many old revisions from esternal storage it might be the hurting the
caches (such as taking "fresh" parsed pages out of memcached).

Another idea would be related to db24 failure, if some hosts failed the
syncing of db.php and still pointed to it.


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


maxsem.wiki at gmail

Mar 24, 2012, 4:22 PM

Post #8 of 10 (815 views)
Permalink
Re: English Wikipedia slow to load [In reply to]

On 25.03.2012, 2:28 Platonides wrote:

> On 24/03/12 21:36, Max Semenik wrote:
>> And these graphs don't make any sense:
>> https://graphite.wikimedia.org/dashboard/MobileFrontend-DOMParse
>> the whole function's execution time is much larger than sum of its
>> pieces. What's going on?

> Maybe it's a recursive function?

No, it's not - unless there's some crazy PHP quirk causing output
handler to be called several times.

> I would have suspected that the parser fixes affected some common
> extension making it slower.
> Ganglia output shows that CPU load decreased, though. From a steady load
> over 10k it became much more flaky, dropping to 9k (due to full
> operations being slower?).

> Is the revision hashing still running? If there are processes fetching
> many old revisions from esternal storage it might be the hurting the
> caches (such as taking "fresh" parsed pages out of memcached).

Parser cache is in MySQL, you can't displace anything out of it by
putting stuff into memcached.

> Another idea would be related to db24 failure, if some hosts failed the
> syncing of db.php and still pointed to it.



--
Best regards,
Max Semenik ([[User:MaxSem]])


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


lcarr at wikimedia

Mar 24, 2012, 6:05 PM

Post #9 of 10 (786 views)
Permalink
Re: English Wikipedia slow to load [In reply to]

This appears to have been due to a network link issue. One of the
uplinks from our row A switch in eqiad was spewing out input framing
errors, causing some packets to be dropped.

This link has been taken down for troubleshooting and the issue
appears to be gone. Please let us know ASAP if you still are seeing
any slowness.

Thanks
Leslie

On Sat, Mar 24, 2012 at 4:22 PM, Max Semenik <maxsem.wiki [at] gmail> wrote:
> On 25.03.2012, 2:28 Platonides wrote:
>
>> On 24/03/12 21:36, Max Semenik wrote:
>>> And these graphs don't make any sense:
>>> https://graphite.wikimedia.org/dashboard/MobileFrontend-DOMParse
>>> the whole function's execution time is much larger than sum of its
>>> pieces. What's going on?
>
>> Maybe it's a recursive function?
>
> No, it's not - unless there's some crazy PHP quirk causing output
> handler to be called several times.
>
>> I would have suspected that the parser fixes affected some common
>> extension making it slower.
>> Ganglia output shows that CPU load decreased, though. From a steady load
>> over 10k it became much more flaky, dropping to 9k (due to full
>> operations being slower?).
>
>> Is the revision hashing still running? If there are processes fetching
>> many old revisions from esternal storage it might be the hurting the
>> caches (such as taking "fresh" parsed pages out of memcached).
>
> Parser cache is in MySQL, you can't displace anything out of it by
> putting stuff into memcached.
>
>> Another idea would be related to db24 failure, if some hosts failed the
>> syncing of db.php and still pointed to it.
>
>
>
> --
> Best regards,
>  Max Semenik ([[User:MaxSem]])
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



--
Leslie Carr
Wikimedia Foundation
AS 14907, 43821

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Platonides at gmail

Mar 25, 2012, 7:59 AM

Post #10 of 10 (795 views)
Permalink
Re: English Wikipedia slow to load [In reply to]

Max Semenik wrote:
> Parser cache is in MySQL, you can't displace anything out of it by
> putting stuff into memcached.

You're wrong here.
Although there's usually plenty of space, so this isn't somehting to be
worried about.

(I would link to a graphic of memcached servers, but I'm unable to find
a ganglia entry for them)


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.