Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Wikipedia: Wikitech

Gerrit server issues

 

 

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded


innocentkiller at gmail

Sep 6, 2012, 2:11 PM

Post #1 of 12 (1144 views)
Permalink
Gerrit server issues

Hi everyone,

Just letting everyone know that we're having some issues with the Gerrit
server at the moment. Sometime since last night, a little over a dozen
repositories (listed below) have gone a wonky and are missing their
branches & tags (objects are intact).

We're working on recovering everything right now, and I'll be sure to let
everyone know the status when we're done.

In the meantime, if you use any of the following repositories, please
refrain from trying to use them until I give the all-clear (I don't want to
exacerbate any problems):

In mediawiki/extensions/*:
Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear,
OnlineStatusBar, Phalanx, RandomImageByCategory,
SemanticImageInput, ShoutWikiAds, SphinxSearch,
TranslationNotifcations

In operations/*:
debs/mysqlatfacebook, debs/wikimedia-lvs-realserver
debs/wikimedia-search-qa, debs/wikistats, software

Also note: operations/mediawiki-config was busted, but we went ahead
and just rebuilt the repo from the live copy so as to not interfere with
site operations.

I thank you for your patience, and please find me on IRC if you have
any questions or find other repositories that are misbehaving.

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


innocentkiller at gmail

Sep 6, 2012, 6:43 PM

Post #2 of 12 (1124 views)
Permalink
Re: Gerrit server issues [In reply to]

On Thu, Sep 6, 2012 at 5:11 PM, Chad <innocentkiller [at] gmail> wrote:
> Hi everyone,
>
> Just letting everyone know that we're having some issues with the Gerrit
> server at the moment. Sometime since last night, a little over a dozen
> repositories (listed below) have gone a wonky and are missing their
> branches & tags (objects are intact).
>
> We're working on recovering everything right now, and I'll be sure to let
> everyone know the status when we're done.
>
> In the meantime, if you use any of the following repositories, please
> refrain from trying to use them until I give the all-clear (I don't want to
> exacerbate any problems):
>
> In mediawiki/extensions/*:
> Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear,
> OnlineStatusBar, Phalanx, RandomImageByCategory,
> SemanticImageInput, ShoutWikiAds, SphinxSearch,
> TranslationNotifcations
>
> In operations/*:
> debs/mysqlatfacebook, debs/wikimedia-lvs-realserver
> debs/wikimedia-search-qa, debs/wikistats, software
>
> Also note: operations/mediawiki-config was busted, but we went ahead
> and just rebuilt the repo from the live copy so as to not interfere with
> site operations.
>
> I thank you for your patience, and please find me on IRC if you have
> any questions or find other repositories that are misbehaving.
>

Just to let everyone know, I'm still working on restoring these projects,
but they're not ready yet.

operations/software.git is back up, though, master was intact after all.

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


innocentkiller at gmail

Sep 6, 2012, 6:49 PM

Post #3 of 12 (1124 views)
Permalink
Re: Gerrit server issues [In reply to]

On Thu, Sep 6, 2012 at 9:43 PM, Chad <innocentkiller [at] gmail> wrote:
> On Thu, Sep 6, 2012 at 5:11 PM, Chad <innocentkiller [at] gmail> wrote:
>> Hi everyone,
>>
>> Just letting everyone know that we're having some issues with the Gerrit
>> server at the moment. Sometime since last night, a little over a dozen
>> repositories (listed below) have gone a wonky and are missing their
>> branches & tags (objects are intact).
>>
>> We're working on recovering everything right now, and I'll be sure to let
>> everyone know the status when we're done.
>>
>> In the meantime, if you use any of the following repositories, please
>> refrain from trying to use them until I give the all-clear (I don't want to
>> exacerbate any problems):
>>
>> In mediawiki/extensions/*:
>> Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear,
>> OnlineStatusBar, Phalanx, RandomImageByCategory,
>> SemanticImageInput, ShoutWikiAds, SphinxSearch,
>> TranslationNotifcations
>>
>> In operations/*:
>> debs/mysqlatfacebook, debs/wikimedia-lvs-realserver
>> debs/wikimedia-search-qa, debs/wikistats, software
>>
>> Also note: operations/mediawiki-config was busted, but we went ahead
>> and just rebuilt the repo from the live copy so as to not interfere with
>> site operations.
>>
>> I thank you for your patience, and please find me on IRC if you have
>> any questions or find other repositories that are misbehaving.
>>
>
> Just to let everyone know, I'm still working on restoring these projects,
> but they're not ready yet.
>
> operations/software.git is back up, though, master was intact after all.
>

And mediawiki/extensions/TranslationNotifications is back up too, master
was intact.

The remaining ones in extensions/* and debs/* are under repair.

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


santhosh.thottingal at gmail

Sep 7, 2012, 8:30 AM

Post #4 of 12 (1124 views)
Permalink
Re: Gerrit server issues [In reply to]

2012/9/7 Chad <innocentkiller [at] gmail>:
> And mediawiki/extensions/TranslationNotifications is back up too, master
> was intact.
>
> The remaining ones in extensions/* and debs/* are under repair.
>

It seems TranslationNotifications extension has some problem. Git
review is always failing with this error:

error: unpack failed: error Missing unknown
4ee9dc02a655cb376a8f20e6d5c1ee95a81b1a37

And remote rejected with unpack error.

ps: Some information about this corruption available at
http://asheepapart.blogspot.com/2011/10/gerrit-code-review-unpack-error-missing.html


Thanks
Santhosh Thottingal

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


innocentkiller at gmail

Sep 7, 2012, 9:45 AM

Post #5 of 12 (1124 views)
Permalink
Re: Gerrit server issues [In reply to]

On Fri, Sep 7, 2012 at 11:30 AM, Santhosh Thottingal
<santhosh.thottingal [at] gmail> wrote:
> 2012/9/7 Chad <innocentkiller [at] gmail>:
>> And mediawiki/extensions/TranslationNotifications is back up too, master
>> was intact.
>>
>> The remaining ones in extensions/* and debs/* are under repair.
>>
>
> It seems TranslationNotifications extension has some problem. Git
> review is always failing with this error:
>
> error: unpack failed: error Missing unknown
> 4ee9dc02a655cb376a8f20e6d5c1ee95a81b1a37
>
> And remote rejected with unpack error.
>
> ps: Some information about this corruption available at
> http://asheepapart.blogspot.com/2011/10/gerrit-code-review-unpack-error-missing.html
>

Ah indeed, that repo does need some work. This is the only thing
I'm working on today, and I figured out how to resolve it late last
night--things will be back to normal by the end of today.

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


innocentkiller at gmail

Sep 7, 2012, 2:12 PM

Post #6 of 12 (1119 views)
Permalink
Re: Gerrit server issues [In reply to]

On Thu, Sep 6, 2012 at 5:11 PM, Chad <innocentkiller [at] gmail> wrote:
> In mediawiki/extensions/*:
> Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear,
> OnlineStatusBar, Phalanx, RandomImageByCategory,
> SemanticImageInput, ShoutWikiAds, SphinxSearch,
> TranslationNotifcations
>

All of these have been fixed other than Nonlinear (more heavily
broken). TranslationNotifications' master is intact, but some of
the changes are still in a bad state and I need to finish cleaning
it up.

> In operations/*:
> debs/mysqlatfacebook, debs/wikimedia-lvs-realserver
> debs/wikimedia-search-qa, debs/wikistats, software
>

software, wikimedia-lvs-realserver, wikimedia-search-qa and
wikistats are all back and fine. mysqlatfacebook is very broken
like Nonlinear.

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


daniel at nadir-seen-fire

Sep 7, 2012, 3:48 PM

Post #7 of 12 (1128 views)
Permalink
Re: Gerrit server issues [In reply to]

On Fri, 07 Sep 2012 08:30:47 -0700, Santhosh Thottingal
<santhosh.thottingal [at] gmail> wrote:

> 2012/9/7 Chad <innocentkiller [at] gmail>:
>> And mediawiki/extensions/TranslationNotifications is back up too, master
>> was intact.
>>
>> The remaining ones in extensions/* and debs/* are under repair.
>>
>
> It seems TranslationNotifications extension has some problem. Git
> review is always failing with this error:
>
> error: unpack failed: error Missing unknown
> 4ee9dc02a655cb376a8f20e6d5c1ee95a81b1a37
>
> And remote rejected with unpack error.
>
> ps: Some information about this corruption available at
> http://asheepapart.blogspot.com/2011/10/gerrit-code-review-unpack-error-missing.html
>
>
> Thanks
> Santhosh Thottingal

I'd like to know what kind of blobs it is that Gerrit depends on so much
but git is happy to prune.

--
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


roan.kattouw at gmail

Sep 10, 2012, 9:52 PM

Post #8 of 12 (1111 views)
Permalink
Re: Gerrit server issues [In reply to]

On Fri, Sep 7, 2012 at 2:12 PM, Chad <innocentkiller [at] gmail> wrote:
> All of these have been fixed other than Nonlinear (more heavily
> broken). TranslationNotifications' master is intact, but some of
> the changes are still in a bad state and I need to finish cleaning
> it up.
>
TranslationNotifications has now been fixed.

> software, wikimedia-lvs-realserver, wikimedia-search-qa and
> wikistats are all back and fine. mysqlatfacebook is very broken
> like Nonlinear.
>
I fixed mysqlatfacebook, which had been completely garbage-collected
on manganese (the primary server). I restored the commits from formey
(the replication slave) where they'd been orphaned but not deleted.
Nonlinear is also fixed.

Roan

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


raimond.spekking at gmail

Sep 12, 2012, 1:53 PM

Post #9 of 12 (1101 views)
Permalink
Re: Gerrit server issues [In reply to]

Am 11.09.2012 06:52, schrieb Roan Kattouw:
> On Fri, Sep 7, 2012 at 2:12 PM, Chad <innocentkiller [at] gmail> wrote:
>> All of these have been fixed other than Nonlinear (more heavily
>> broken). TranslationNotifications' master is intact, but some of
>> the changes are still in a bad state and I need to finish cleaning
>> it up.
>>
> TranslationNotifications has now been fixed.
>
Could you please check the permissions? L10n-bot is not longer allowed
to merge its commits:

https://gerrit.wikimedia.org/r/#/c/23455/
https://gerrit.wikimedia.org/r/#/c/23565/

It fails with

Merging 23565...
error: Verified=1 not permitted
one or more approvals failed; review output above


Thanks. Raimond.
Attachments: signature.asc (0.24 KB)


innocentkiller at gmail

Sep 12, 2012, 2:16 PM

Post #10 of 12 (1102 views)
Permalink
Re: Gerrit server issues [In reply to]

On Wed, Sep 12, 2012 at 4:53 PM, Raimond Spekking
<raimond.spekking [at] gmail> wrote:
> Am 11.09.2012 06:52, schrieb Roan Kattouw:
>> On Fri, Sep 7, 2012 at 2:12 PM, Chad <innocentkiller [at] gmail> wrote:
>>> All of these have been fixed other than Nonlinear (more heavily
>>> broken). TranslationNotifications' master is intact, but some of
>>> the changes are still in a bad state and I need to finish cleaning
>>> it up.
>>>
>> TranslationNotifications has now been fixed.
>>
> Could you please check the permissions? L10n-bot is not longer allowed
> to merge its commits:
>
> https://gerrit.wikimedia.org/r/#/c/23455/
> https://gerrit.wikimedia.org/r/#/c/23565/
>
> It fails with
>
> Merging 23565...
> error: Verified=1 not permitted
> one or more approvals failed; review output above
>

The permissions were not inheriting from mediawiki/extensions,
they were instead inheriting from All-Projects. This is now fixed.

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Platonides at gmail

Sep 12, 2012, 3:51 PM

Post #11 of 12 (1100 views)
Permalink
Re: Gerrit server issues [In reply to]

http://blog.wikimedia.org/2012/09/07/recovery-of-broken-gerrit-repositories/
writes:
> While Git does make it very difficult to get rid of data forever, it is
> possible. For example, if we had let the cron run again, git gc would’ve
> merrily deleted all of the unreferenced objects since they were unused.

> So, now it just became a matter of sorting out which refs these commits
> could go to. Playing around with git show makes it pretty easy to figure
> out where the commits belong, and so this just turned into a process of
> editing the appropriate refs/* files to add the commit hash (and what
>seems so obvious now took several hours of hand-wringing, I assure you).
> So for the next several hours, I took the time to sort out these dangling
> commits, and the repos ended up working again.



Doesn't formey have core.logAllRefUpdates set to true?
Wouldn't that have prevented git gc from removing commits referenced in
the reflog? (at least until two weeks without the references passed, it
should probably have been run with --no-prune)


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


roan.kattouw at gmail

Sep 12, 2012, 4:07 PM

Post #12 of 12 (1095 views)
Permalink
Re: Gerrit server issues [In reply to]

On Wed, Sep 12, 2012 at 3:51 PM, Platonides <Platonides [at] gmail> wrote:
> Doesn't formey have core.logAllRefUpdates set to true?
> Wouldn't that have prevented git gc from removing commits referenced in
> the reflog? (at least until two weeks without the references passed, it
> should probably have been run with --no-prune)
>
I did indeed manage to recover some commits that were lost (apparently
gc'ed) on manganese but still present on formey, so that sounds
plausible.

Roan

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.