Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Wikipedia: Wikitech

upcoming 1.17 deployment and the xml dumps

 

 

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded


ariel at wikimedia

Feb 5, 2011, 10:56 PM

Post #1 of 7 (569 views)
Permalink
upcoming 1.17 deployment and the xml dumps

A little bit before the scheduled deployment of the 1.17 branch on our
production servers, I will be halting production of XML dumps.
Deployment is set for Tuesday Feb 8 at 07:00 UTC, so a few hours before
that I'll start shutting down processes.

This is a precautionary measure; after the deployment and any hasty
fixes that may be needed, I will be doing some testing to ensure that
dumps are not impacted, before we restart them. Barring some bizarre
problem, we should be back up and running within a day or two.

Ariel



_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


janesh at calcey

Feb 7, 2011, 10:21 PM

Post #2 of 7 (540 views)
Permalink
Re: upcoming 1.17 deployment and the xml dumps [In reply to]

----- Original Message -----
From: "Ariel T. Glenn" <ariel [at] wikimedia>
Newsgroups: gmane.science.linguistics.wikipedia.technical
To: <xmldatadumps-l [at] lists>; <wikitech-l [at] lists>
Sent: Sunday, February 06, 2011 12:26 PM
Subject: upcoming 1.17 deployment and the xml dumps


>A little bit before the scheduled deployment of the 1.17 branch on our
> production servers, I will be halting production of XML dumps.
> Deployment is set for Tuesday Feb 8 at 07:00 UTC, so a few hours before
> that I'll start shutting down processes.
>
> This is a precautionary measure; after the deployment and any hasty
> fixes that may be needed, I will be doing some testing to ensure that
> dumps are not impacted, before we restart them. Barring some bizarre
> problem, we should be back up and running within a day or two.
>
> Ariel
Hi Ariel,
Let us know if we can give some support (remotely) on the testing effort.
Thanks
Janesh

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


ariel at wikimedia

Feb 7, 2011, 10:26 PM

Post #3 of 7 (541 views)
Permalink
Re: upcoming 1.17 deployment and the xml dumps [In reply to]

Thanks for the offer!

What I plan to do is wait til things settle down a bit and then run
against a couple wikis with no activity and see if I get known good
results. After that I'll start one process up and ask folks to check
the next few files produced. If those pan out I"ll crank everything back
up again.

Ariel

Στις 08-02-2011, ημέρα Τρι, και ώρα 11:51 +0530, ο/η Janesh Kodikara
έγραψε:
> ----- Original Message -----
> From: "Ariel T. Glenn" <ariel [at] wikimedia>
> Newsgroups: gmane.science.linguistics.wikipedia.technical
> To: <xmldatadumps-l [at] lists>; <wikitech-l [at] lists>
> Sent: Sunday, February 06, 2011 12:26 PM
> Subject: upcoming 1.17 deployment and the xml dumps
>
>
> >A little bit before the scheduled deployment of the 1.17 branch on our
> > production servers, I will be halting production of XML dumps.
> > Deployment is set for Tuesday Feb 8 at 07:00 UTC, so a few hours before
> > that I'll start shutting down processes.
> >
> > This is a precautionary measure; after the deployment and any hasty
> > fixes that may be needed, I will be doing some testing to ensure that
> > dumps are not impacted, before we restart them. Barring some bizarre
> > problem, we should be back up and running within a day or two.
> >
> > Ariel
> Hi Ariel,
> Let us know if we can give some support (remotely) on the testing effort.
> Thanks
> Janesh
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l



_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


jmorken at shaw

Feb 9, 2011, 1:44 PM

Post #4 of 7 (540 views)
Permalink
Re: [Xmldatadumps-l] upcoming 1.17 deployment and the xml dumps [In reply to]

Hi Ariel,

I don't really understand why the dumps need to be halted as I thought the mediawiki code and database dump code were basically two separate entities already*. I guess the 1.17 branch code changes the structure of the database causing potential errors in the database dump? I also don't understand the "precautionary" logic of halting the dumps, as a dump with errors is better than no dump in the case where there are a limited supply of recent dumps due to the RAID server failure as well. If its only a couple day halt as you mentioned that's probably irrelevant, but it sounds like it may be a longer period of limited testing from your last wikitech email, which makes me wonder if it is even worth halting the dumps in the first place.. Also wouldn't potential dump errors be detected better if they continue to be produced and check them for errors, rather than halt them?

cheers,
Jamie



*

http://svn.wikimedia.org/viewvc/mediawiki/branches/REL1_17/

http://svn.wikimedia.org/viewvc/mediawiki/branches/ariel/xmldumps-backup/



----- Original Message -----
From: "Ariel T. Glenn" <ariel [at] wikimedia>
Date: Saturday, February 5, 2011 10:56 pm
Subject: [Xmldatadumps-l] upcoming 1.17 deployment and the xml dumps
To: xmldatadumps-l [at] lists, wikitech-l [at] lists

> A little bit before the scheduled deployment of the 1.17 branch
> on our
> production servers, I will be halting production of XML dumps.
> Deployment is set for Tuesday Feb 8 at 07:00 UTC, so a few hours
> beforethat I'll start shutting down processes.
>
> This is a precautionary measure; after the deployment and any hasty
> fixes that may be needed, I will be doing some testing to ensure that
> dumps are not impacted, before we restart them. Barring some bizarre
> problem, we should be back up and running within a day or two.
>
> Ariel
>
>
>
> _______________________________________________
> Xmldatadumps-l mailing list
> Xmldatadumps-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


roan.kattouw at gmail

Feb 9, 2011, 2:08 PM

Post #5 of 7 (531 views)
Permalink
Re: [Xmldatadumps-l] upcoming 1.17 deployment and the xml dumps [In reply to]

2011/2/9 Jamie Morken <jmorken [at] shaw>:
>
> Hi Ariel,
>
> I don't really understand why the dumps need to be halted as I thought the mediawiki code and database dump code were basically two separate entities already*. I guess the 1.17 branch code changes the structure of the database causing potential errors in the database dump? I also don't understand the "precautionary" logic of halting the dumps, as a dump with errors is better than no dump in the case where there are a limited supply of recent dumps due to the RAID server failure as well. If its only a couple day halt as you mentioned that's probably irrelevant, but it sounds like it may be a longer period of limited testing from your last wikitech email, which makes me wonder if it is even worth halting the dumps in the first place.. Also wouldn't potential dump errors be detected better if they continue to be produced and check them for errors, rather than halt them?
>
The dump code and MW code are intertwined, they're not separate. I
think the xmldumps-backup branch you're linking to is just a branch
Ariel is playing around in or working on dump-specific code or
something.

We really don't want any maintenance scripts running when doing 1.17
stuff, and the dumps use maintenance scripts.

Roan Kattouw (Catrope)

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


ariel at wikimedia

Feb 9, 2011, 2:14 PM

Post #6 of 7 (536 views)
Permalink
Re: [Xmldatadumps-l] upcoming 1.17 deployment and the xml dumps [In reply to]

We halted them because we can have bad data creep on during times when
the codebase is badly broken. I don't want to have to walk through and
detrmine later which 30 or 50 wiki dumps those are and toss them, so I
have them on hold til things are sorted out or until we have a date for
deployment that is a number of days off.

A dump with errors isn't better than no dump in that it is possible for
bad data to be carried forward into subsequent dumps, even with the
revision length check in the code.

The only certain check involves doing an md5sum of the revision text,
something that can only be accomplished right now by retrieving the text
from the database, thus making prefetch from the previous dump file a
pointless exercise.

After a brief meeting just now about deployment, it appears we are going
to make another stab at testing tomorrow at this time. (Check
http://techblog.wikimedia.org/ in a couple of hours for the details.)

After that we should have several days of a break; if that pans out,
I'll happily crank dumps back up for that interval.

Ariel

Στις 09-02-2011, ημέρα Τετ, και ώρα 13:44 -0800, ο/η Jamie Morken
έγραψε:
>
> Hi Ariel,
>
> I don't really understand why the dumps need to be halted as I thought
> the mediawiki code and database dump code were basically two separate
> entities already*. I guess the 1.17 branch code changes the structure
> of the database causing potential errors in the database dump? I also
> don't understand the "precautionary" logic of halting the dumps, as a
> dump with errors is better than no dump in the case where there are a
> limited supply of recent dumps due to the RAID server failure as well.
> If its only a couple day halt as you mentioned that's probably
> irrelevant, but it sounds like it may be a longer period of limited
> testing from your last wikitech email, which makes me wonder if it is
> even worth halting the dumps in the first place.. Also wouldn't
> potential dump errors be detected better if they continue to be
> produced and check them for errors, rather than halt them?
>
> cheers,
> Jamie
>
>
> *
> http://svn.wikimedia.org/viewvc/mediawiki/branches/REL1_17/
> http://svn.wikimedia.org/viewvc/mediawiki/branches/ariel/xmldumps-backup/
>
>
> ----- Original Message -----
> From: "Ariel T. Glenn" <ariel [at] wikimedia>
> Date: Saturday, February 5, 2011 10:56 pm
> Subject: [Xmldatadumps-l] upcoming 1.17 deployment and the xml dumps
> To: xmldatadumps-l [at] lists, wikitech-l [at] lists
>
> > A little bit before the scheduled deployment of the 1.17 branch
> > on our
> > production servers, I will be halting production of XML dumps.
> > Deployment is set for Tuesday Feb 8 at 07:00 UTC, so a few hours
> > beforethat I'll start shutting down processes.
> >
> > This is a precautionary measure; after the deployment and any hasty
> > fixes that may be needed, I will be doing some testing to ensure
> that
> > dumps are not impacted, before we restart them. Barring some bizarre
> > problem, we should be back up and running within a day or two.
> >
> > Ariel
> >
> >
> >
> > _______________________________________________
> > Xmldatadumps-l mailing list
> > Xmldatadumps-l [at] lists
> > https://lists.wikimedia.org/mailman/listinfo/xmldatadumps-l
> >



_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


ariel at wikimedia

Feb 9, 2011, 2:16 PM

Post #7 of 7 (537 views)
Permalink
Re: [Xmldatadumps-l] upcoming 1.17 deployment and the xml dumps [In reply to]

The xmldatadumps branch I have mods to is only the python code; the
maintenance scripts run out of the deployment branch. When 1.17 is
deployed those are the versions that run.

Ariel

Στις 09-02-2011, ημέρα Τετ, και ώρα 23:08 +0100, ο/η Roan Kattouw
έγραψε:
> 2011/2/9 Jamie Morken <jmorken [at] shaw>:
> >
> > Hi Ariel,
> >
> > I don't really understand why the dumps need to be halted as I thought the mediawiki code and database dump code were basically two separate entities already*. I guess the 1.17 branch code changes the structure of the database causing potential errors in the database dump? I also don't understand the "precautionary" logic of halting the dumps, as a dump with errors is better than no dump in the case where there are a limited supply of recent dumps due to the RAID server failure as well. If its only a couple day halt as you mentioned that's probably irrelevant, but it sounds like it may be a longer period of limited testing from your last wikitech email, which makes me wonder if it is even worth halting the dumps in the first place.. Also wouldn't potential dump errors be detected better if they continue to be produced and check them for errors, rather than halt them?
> >
> The dump code and MW code are intertwined, they're not separate. I
> think the xmldumps-backup branch you're linking to is just a branch
> Ariel is playing around in or working on dump-specific code or
> something.
>
> We really don't want any maintenance scripts running when doing 1.17
> stuff, and the dumps use maintenance scripts.
>
> Roan Kattouw (Catrope)



_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.