Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Wikipedia: Wikitech

State of the 1.19 deployment

 

 

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded


robla at wikimedia

Feb 29, 2012, 10:06 PM

Post #1 of 6 (373 views)
Permalink
State of the 1.19 deployment

On Wed, Feb 29, 2012 at 11:34 AM, Rob Lanphier <robla [at] wikimedia> wrote:
> Just a reminder that we're deploying MediaWiki 1.19 to most Wikipedias
> in a few hours (including enwiki), starting at 23:00 UTC (3pm PST).

This work is mostly done. The biggest known problem we have is with
language variants on the Chinese language wikis. See:
https://bugzilla.wikimedia.org/34832

We're going to give the original developer some time to respond on
this issue before taking action. For now, several zh* wikis are on
1.18

We're also keeping an eye on site performance. There's been a slight
regression in our parser cache hit rate:
http://bit.ly/w6Gy9t

The new diff colors have been temporarily reverted. Trevor and Timo
plan to spend some time looking into the subject. See:
http://www.mediawiki.org/wiki/Special:Code/MediaWiki/112750

Saper and Aaron have spent some time cleaning up areas where CheckUser
briefly stopped working properly:
https://bugzilla.wikimedia.org/show_bug.cgi?id=34838

Mostly unrelated to 1.19 (mainly correlated), we've rolled back the
Swift thumbnail deployment to correct many broken images. We'll be
bringing Swift back online in the coming days after we've purged all
of the broken images we know of from the system.

It'd be good to have more eyes on the recent bug list:
https://bugzilla.wikimedia.org/buglist.cgi?chfield=%5BBug%20creation%5D&query_format=advanced&chfieldfrom=1d&list_id=95623

Thanks everyone for your help in making this deployment as smooth as possible!

Rob

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


rkaldari at wikimedia

Feb 29, 2012, 11:34 PM

Post #2 of 6 (365 views)
Permalink
Re: State of the 1.19 deployment [In reply to]

I'm seeing a lot of ref errors from the Infobox settlement template due
to the 1.19 deploy. If anyone wants to take a look, here are some examples:
http://en.wikipedia.org/wiki/Nashville,_Tennessee
http://en.wikipedia.org/wiki/San_Francisco

Ryan Kaldari

On 2/29/12 10:06 PM, Rob Lanphier wrote:
> On Wed, Feb 29, 2012 at 11:34 AM, Rob Lanphier<robla [at] wikimedia> wrote:
>> Just a reminder that we're deploying MediaWiki 1.19 to most Wikipedias
>> in a few hours (including enwiki), starting at 23:00 UTC (3pm PST).
> This work is mostly done. The biggest known problem we have is with
> language variants on the Chinese language wikis. See:
> https://bugzilla.wikimedia.org/34832
>
> We're going to give the original developer some time to respond on
> this issue before taking action. For now, several zh* wikis are on
> 1.18
>
> We're also keeping an eye on site performance. There's been a slight
> regression in our parser cache hit rate:
> http://bit.ly/w6Gy9t
>
> The new diff colors have been temporarily reverted. Trevor and Timo
> plan to spend some time looking into the subject. See:
> http://www.mediawiki.org/wiki/Special:Code/MediaWiki/112750
>
> Saper and Aaron have spent some time cleaning up areas where CheckUser
> briefly stopped working properly:
> https://bugzilla.wikimedia.org/show_bug.cgi?id=34838
>
> Mostly unrelated to 1.19 (mainly correlated), we've rolled back the
> Swift thumbnail deployment to correct many broken images. We'll be
> bringing Swift back online in the coming days after we've purged all
> of the broken images we know of from the system.
>
> It'd be good to have more eyes on the recent bug list:
> https://bugzilla.wikimedia.org/buglist.cgi?chfield=%5BBug%20creation%5D&query_format=advanced&chfieldfrom=1d&list_id=95623
>
> Thanks everyone for your help in making this deployment as smooth as possible!
>
> Rob
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


rkaldari at wikimedia

Feb 29, 2012, 11:48 PM

Post #3 of 6 (374 views)
Permalink
Re: State of the 1.19 deployment [In reply to]

Nevermind, this was just a poorly timed change to Template:Infobox
settlement/areadisp that broke the infoboxes. It's fixed now.

Ryan Kaldari

On 2/29/12 11:34 PM, Ryan Kaldari wrote:
> I'm seeing a lot of ref errors from the Infobox settlement template
> due to the 1.19 deploy. If anyone wants to take a look, here are some
> examples:
> http://en.wikipedia.org/wiki/Nashville,_Tennessee
> http://en.wikipedia.org/wiki/San_Francisco
>
> Ryan Kaldari
>
> On 2/29/12 10:06 PM, Rob Lanphier wrote:
>> On Wed, Feb 29, 2012 at 11:34 AM, Rob Lanphier<robla [at] wikimedia>
>> wrote:
>>> Just a reminder that we're deploying MediaWiki 1.19 to most Wikipedias
>>> in a few hours (including enwiki), starting at 23:00 UTC (3pm PST).
>> This work is mostly done. The biggest known problem we have is with
>> language variants on the Chinese language wikis. See:
>> https://bugzilla.wikimedia.org/34832
>>
>> We're going to give the original developer some time to respond on
>> this issue before taking action. For now, several zh* wikis are on
>> 1.18
>>
>> We're also keeping an eye on site performance. There's been a slight
>> regression in our parser cache hit rate:
>> http://bit.ly/w6Gy9t
>>
>> The new diff colors have been temporarily reverted. Trevor and Timo
>> plan to spend some time looking into the subject. See:
>> http://www.mediawiki.org/wiki/Special:Code/MediaWiki/112750
>>
>> Saper and Aaron have spent some time cleaning up areas where CheckUser
>> briefly stopped working properly:
>> https://bugzilla.wikimedia.org/show_bug.cgi?id=34838
>>
>> Mostly unrelated to 1.19 (mainly correlated), we've rolled back the
>> Swift thumbnail deployment to correct many broken images. We'll be
>> bringing Swift back online in the coming days after we've purged all
>> of the broken images we know of from the system.
>>
>> It'd be good to have more eyes on the recent bug list:
>> https://bugzilla.wikimedia.org/buglist.cgi?chfield=%5BBug%20creation%5D&query_format=advanced&chfieldfrom=1d&list_id=95623
>>
>>
>> Thanks everyone for your help in making this deployment as smooth as
>> possible!
>>
>> Rob
>>
>> _______________________________________________
>> Wikitech-l mailing list
>> Wikitech-l [at] lists
>> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


jeroendedauw at gmail

Mar 1, 2012, 12:50 AM

Post #4 of 6 (360 views)
Permalink
Re: State of the 1.19 deployment [In reply to]

Hey,

> There's been a slight regression in our parser cache hit rate:
> http://bit.ly/w6Gy9t

This one is probably more informative for people not aware of the usual hit
rate http://goo.gl/YY80C

Looks to me that the miss rate went up over 500% - is that really just a
"slight" regression? :)

Cheers

--
Jeroen De Dauw
http://www.bn2vs.com
Don't panic. Don't be evil.
--
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


hashar+wmf at free

Mar 1, 2012, 2:09 AM

Post #5 of 6 (363 views)
Permalink
Re: State of the 1.19 deployment [In reply to]

Le 01/03/12 09:50, Jeroen De Dauw a écrit :
> Hey,
>
>> There's been a slight regression in our parser cache hit rate:
>> http://bit.ly/w6Gy9t
>
> This one is probably more informative for people not aware of the usual hit
> rate http://goo.gl/YY80C
>
> Looks to me that the miss rate went up over 500% - is that really just a
> "slight" regression? :)

We really want to use absolute time range: http://bit.ly/A7kcys


Anyway, they are absent misses. Probably a key changed somewhere in our
parser that magically invalidated roughly 15% of the parser cache. It
seems to slowly recover afterward.


By zooming and making the Y scale start at 50%, the event seems to have
occurred on February 17th just before 6am UTC.

I have uploaded a screenshot on mediawiki.org :

https://www.mediawiki.org/wiki/File:Parser_cache_hit_20120217.png

From the wikitech admin logs we have:


07:42 binasher:upgraded mysql on db40 to 5.1.53-facebook-r3753, enabled
innodb_use_purge_thread
----------------------------------------------------------------------
05:39 tstarling synchronized wmf-config/InitialiseSettings.php
05:38 tstarling synchronized wmf-config/CommonSettings.php
----------------------------------------------------------------------
05:35 Tim: on db40: reduced to 10M, should be causing massive delays,
but the site's not down and the purge rate is lower if anything. Going
to disable the mysql parser cache entirely.
----------------------------------------------------------------------
05:25 Tim: on db40: purge lag is still increasing at 108 per second, so
reducing innodb_max_purge_lag to 50M
----------------------------------------------------------------------
05:21 Tim: on db40: giving the innodb manual the benefit of the doubt
and following its advice, setting innodb_max_purge_lag to 100M, which
should give a delay of 4.5ms
----------------------------------------------------------------------
05:13 Tim: killing purgeParserCache.php since it is probably doing more
harm than good
----------------------------------------------------------------------
02:43 maplebed: deployed updated thumb_handler.php to ms5 to include
Content-Length in generated images
----------------------------------------------------------------------
02:34 logmsgbot: LocalisationUpdate completed (1.19) at Fri Feb 17
02:34:32 UTC 2012
----------------------------------------------------------------------
https://wikitech.wikimedia.org/view/Server_admin_log


" Going to disable the mysql parser cache entirely."

Seems to have been reenabled on Feb 29th at 00:20:

00:20 Tim: reimported schema files on db40 and re-enabled mysql parser cache


--
Antoine "hashar" Musso




_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


afeldman at wikimedia

Mar 1, 2012, 9:23 AM

Post #6 of 6 (352 views)
Permalink
Re: State of the 1.19 deployment [In reply to]

The larger time range includes the removal of the mysql based parsercache
which is the cause of the primary decline and not related to 1.19. The time
range of the originally mentioned graph just shows a bit of context before
the enwiki deployment up until the time I posted it to irc last night.

A key change as hashar suggests is a good theory but I'm not sure if the
hit rate is actually recovering when looking at -24 or -8 hours, more time
will tell.

This may not be valid since the mysql pcache wasn't re-enabled (from an
empty state) long before 1.19 but the rate of selects against it seems to
be up day over day. Maybe a new key is consistently fetched before it
would ever be set.

2012-02-29T11:24:00+00:00,10288.770238
2012-02-29T12:06:00+00:00,10992.754365
2012-02-29T12:48:00+00:00,11140.606746

2012-03-01T11:54:00+00:00,13912.613492
2012-03-01T12:36:00+00:00,13790.359524
2012-03-01T13:18:00+00:00,14176.010317

(from
http://ganglia.wikimedia.org/latest/graph.php?c=MySQL%20pmtpa&h=db40.pmtpa.wmnet&v=15608&m=mysql_com_select&r=custom&z=default&jr=&js=&st=1330622240&cs=2%2F28%2F2012%2010%3A23&ce=3%2F1%2F2012%200%3A56&vl=stmts&ti=mysql_com_select&csv=1
)

On Thursday, March 1, 2012, Antoine Musso wrote:

> Le 01/03/12 09:50, Jeroen De Dauw a écrit :
>
>> Hey,
>>
>> There's been a slight regression in our parser cache hit rate:
>>> http://bit.ly/w6Gy9t
>>>
>>
>> This one is probably more informative for people not aware of the usual
>> hit
>> rate http://goo.gl/YY80C
>>
>> Looks to me that the miss rate went up over 500% - is that really just a
>> "slight" regression? :)
>>
>
> We really want to use absolute time range: http://bit.ly/A7kcys
>
>
> Anyway, they are absent misses. Probably a key changed somewhere in our
> parser that magically invalidated roughly 15% of the parser cache. It seems
> to slowly recover afterward.
>
>
> By zooming and making the Y scale start at 50%, the event seems to have
> occurred on February 17th just before 6am UTC.
>
> I have uploaded a screenshot on mediawiki.org :
>
> https://www.mediawiki.org/**wiki/File:Parser_cache_hit_**20120217.png<https://www.mediawiki.org/wiki/File:Parser_cache_hit_20120217.png>
>
> From the wikitech admin logs we have:
>
>
> 07:42 binasher:upgraded mysql on db40 to 5.1.53-facebook-r3753, enabled
> innodb_use_purge_thread
> ------------------------------**------------------------------**----------
> 05:39 tstarling synchronized wmf-config/InitialiseSettings.**php
> 05:38 tstarling synchronized wmf-config/CommonSettings.php
> ------------------------------**------------------------------**----------
> 05:35 Tim: on db40: reduced to 10M, should be causing massive delays, but
> the site's not down and the purge rate is lower if anything. Going to
> disable the mysql parser cache entirely.
> ------------------------------**------------------------------**----------
> 05:25 Tim: on db40: purge lag is still increasing at 108 per second, so
> reducing innodb_max_purge_lag to 50M
> ------------------------------**------------------------------**----------
> 05:21 Tim: on db40: giving the innodb manual the benefit of the doubt and
> following its advice, setting innodb_max_purge_lag to 100M, which should
> give a delay of 4.5ms
> ------------------------------**------------------------------**----------
> 05:13 Tim: killing purgeParserCache.php since it is probably doing more
> harm than good
> ------------------------------**------------------------------**----------
> 02:43 maplebed: deployed updated thumb_handler.php to ms5 to include
> Content-Length in generated images
> ------------------------------**------------------------------**----------
> 02:34 logmsgbot: LocalisationUpdate completed (1.19) at Fri Feb 17
> 02:34:32 UTC 2012
> ------------------------------**------------------------------**----------
> https://wikitech.wikimedia.**org/view/Server_admin_log<https://wikitech.wikimedia.org/view/Server_admin_log>
>
>
> " Going to disable the mysql parser cache entirely."
>
> Seems to have been reenabled on Feb 29th at 00:20:
>
> 00:20 Tim: reimported schema files on db40 and re-enabled mysql parser
> cache
>
>
> --
> Antoine "hashar" Musso
>
>
>
>
> ______________________________**_________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/**mailman/listinfo/wikitech-l<https://lists.wikimedia.org/mailman/listinfo/wikitech-l>
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.