Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Wikipedia: Wikitech

Preliminary git module splitup notes (MediaWiki core & extensions)

 

 

First page Previous page 1 2 Next page Last page  View All Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded


brion at pobox

Oct 4, 2011, 6:33 PM

Post #1 of 32 (1211 views)
Permalink
Preliminary git module splitup notes (MediaWiki core & extensions)

I whipped up a few prelim notes on a possible git repository layout for
MediaWiki core and extensions, splitting out from SVN:

http://www.mediawiki.org/wiki/Git_conversion/Splitting_tests

I've run this past Siebrand to make sure that it should work for the
localization batch commits, and he seems to think it sounds sane.

You can check out extensions as separate repositories directly into
subfolders within core's 'extensions' dir for a ready-to-run system. But,
you *do* need to do either manually or scripted iteration over them to pull
updates or commit across repos. Git's submodules might be a useful way to
help automate checkouts, but they introduce their own complications for
maintenance.

There's a shell script on that page to make a sample checkout not including
any history -- just exporting latest code to make sure the layout makes
sense. Real conversion will probably want to include version history and
release branches/tags for both core and extensions.


Note that there's lots of other stuff in the MediaWiki subversion repo that
will want to be split out as well -- but for now our lives are simplified by
only worrying about MediaWiki and maintained extensions. ;)

Please do feel free to comment or ask questions -- things are malleable and
feedback will be *very* helpful in finalizing plans and building
documentation.

-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Platonides at gmail

Oct 5, 2011, 7:18 AM

Post #2 of 32 (1160 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

Brion Vibber wrote:
> You can check out extensions as separate repositories directly into
> subfolders within core's 'extensions' dir for a ready-to-run system. But,
> you *do* need to do either manually or scripted iteration over them to pull
> updates or commit across repos. Git's submodules might be a useful way to
> help automate checkouts, but they introduce their own complications for
> maintenance.
>
> There's a shell script on that page to make a sample checkout not including
> any history -- just exporting latest code to make sure the layout makes
> sense. Real conversion will probably want to include version history and
> release branches/tags for both core and extensions.


There are 615 extensions in trunk/extensions
With the new setup, would someone which has a checkout of everything
need to open 616 connections (network latency, ssh authentication, etc.)
whenever he wants to pull?
svn:externals already noticeably slow checkouts, and there's only a
bunch of them.


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


brion at pobox

Oct 5, 2011, 11:59 AM

Post #3 of 32 (1171 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On Wed, Oct 5, 2011 at 7:18 AM, Platonides <Platonides [at] gmail> wrote:

> There are 615 extensions in trunk/extensions
> With the new setup, would someone which has a checkout of everything
> need to open 616 connections (network latency, ssh authentication, etc.)
> whenever he wants to pull?
> svn:externals already noticeably slow checkouts, and there's only a
> bunch of them.
>

I'll do some tests; it may be possible to avoid having to do serial
connection setup/teardown by using SSH's ControlMaster[1] setting to
piggyback the 'new' connections on one that's already open.

[1] <
http://www.anchor.com.au/blog/2010/02/ssh-controlmaster-the-good-the-bad-the-ugly/
>

(Part of what makes svn:externals slow is that you're jumping over to
another site entirely -- that's an extra DNS lookup, possibly separate auth,
and who knows how well the other server performs.)

-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


lists at nadir-seen-fire

Oct 5, 2011, 12:41 PM

Post #4 of 32 (1152 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

11-10-05 11:59 AM, Brion Vibber wrote:
> On Wed, Oct 5, 2011 at 7:18 AM, Platonides <Platonides [at] gmail> wrote:
>
>> There are 615 extensions in trunk/extensions
>> With the new setup, would someone which has a checkout of everything
>> need to open 616 connections (network latency, ssh authentication, etc.)
>> whenever he wants to pull?
>> svn:externals already noticeably slow checkouts, and there's only a
>> bunch of them.
>>
> I'll do some tests; it may be possible to avoid having to do serial
> connection setup/teardown by using SSH's ControlMaster[1] setting to
> piggyback the 'new' connections on one that's already open.
>
> [1] <
> http://www.anchor.com.au/blog/2010/02/ssh-controlmaster-the-good-the-bad-the-ugly/
> (Part of what makes svn:externals slow is that you're jumping over to
> another site entirely -- that's an extra DNS lookup, possibly separate auth,
> and who knows how well the other server performs.)
>
> -- brion
;) And when all else fails, rsync a bunch of --bare repos and do your
pull on the local hd.

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Platonides at gmail

Oct 5, 2011, 1:06 PM

Post #5 of 32 (1155 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

Brion Vibber wrote:
> On Wed, Oct 5, 2011 at 7:18 AM, Platonides<Platonides [at] gmail> wrote:
>
>> There are 615 extensions in trunk/extensions
>> With the new setup, would someone which has a checkout of everything
>> need to open 616 connections (network latency, ssh authentication, etc.)
>> whenever he wants to pull?
>> svn:externals already noticeably slow checkouts, and there's only a
>> bunch of them.
>>
>
> I'll do some tests; it may be possible to avoid having to do serial
> connection setup/teardown by using SSH's ControlMaster[1] setting to
> piggyback the 'new' connections on one that's already open.

I know about ControlMaster (which we can only those of us with ssh+git
can benefit), but just launching a new process and waiting if there's
something new will slow-down. OTOH git skips the "recurse everything
locking all subfolders" step, so it may be equivalent.

Maybe there's some way for fetching updates from aggregate repositories
at once and I am just when everything is solved, though.


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Platonides at gmail

Oct 5, 2011, 1:20 PM

Post #6 of 32 (1153 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

Daniel Friesen wrote:
> ;) And when all else fails, rsync a bunch of --bare repos and do your
> pull on the local hd.
>
> ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

Sorry? Note that only a small subset of those with commit access do have
full ssh to run commands. And a good system shouldn't need those hacks
(which is why we are discussing in advance).


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


rlane32 at gmail

Oct 5, 2011, 1:27 PM

Post #7 of 32 (1161 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

> Sorry? Note that only a small subset of those with commit access do have
> full ssh to run commands. And a good system shouldn't need those hacks
> (which is why we are discussing in advance).
>

I don't see why we can't open up access to basically everyone. I'd
like to make it so that everyone who makes a Labs account also gets
access via SSH. I'd like Labs account access to be fairly open.

We can configure Gerrit to only allow read access to people with an
account. I'd really rather that everyone with an account has the
ability to push a merge request and make branches as well, though.

- Ryan

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


brion at pobox

Oct 5, 2011, 1:59 PM

Post #8 of 32 (1163 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On Oct 5, 2011 1:03 PM, "Platonides" <Platonides [at] gmail> wrote:
> I know about ControlMaster (which we can only those of us with ssh+git
> can benefit), but just launching a new process and waiting if there's
> something new will slow-down. OTOH git skips the "recurse everything
> locking all subfolders" step, so it may be equivalent.
>
> Maybe there's some way for fetching updates from aggregate repositories
> at once and I am just when everything is solved, though.

Submodules may actually work well for this, as long as something propagates
the ext updates to the composite repo. The checked-out commit id of each
submodule is stored in the tree, so if no changes were seen from that one
containing repo it shouldn't have to pull anything from the submodule's
repo.

(Not yet tested)

-- brion
>
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


brion at pobox

Oct 5, 2011, 2:30 PM

Post #9 of 32 (1159 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On Wed, Oct 5, 2011 at 1:59 PM, Brion Vibber <brion [at] pobox> wrote:

> On Oct 5, 2011 1:03 PM, "Platonides" <Platonides [at] gmail> wrote:
> > I know about ControlMaster (which we can only those of us with ssh+git
> > can benefit), but just launching a new process and waiting if there's
> > something new will slow-down. OTOH git skips the "recurse everything
> > locking all subfolders" step, so it may be equivalent.
> >
> > Maybe there's some way for fetching updates from aggregate repositories
> > at once and I am just when everything is solved, though.
>
> Submodules may actually work well for this, as long as something propagates
> the ext updates to the composite repo. The checked-out commit id of each
> submodule is stored in the tree, so if no changes were seen from that one
> containing repo it shouldn't have to pull anything from the submodule's
> repo.
>
> (Not yet tested)
>

Ok, did some quick tests fetching updates for 16 repos sitting on my
Gitorious account.

Ping round-trip from my office desktop to Gitorious's server is 173ms,
making the theoretical *absolute best* possible time involving a round-trip
for each at 2-3 seconds.


Running a simple loop of 'git fetch' over each repo (auth'ing with my ssh
key, passphrase already provided) takes 53 seconds (about 3 seconds per
repo). This does a separate ssh setup & poke into git for each repo.

Clearly unacceptable for 600+ extensions. :)


Turning on ControlMaster and starting a long-running git clone in the
background, then running the same 'git fetch' loop took the time down to
about 10 seconds (<1s per repo). ControlMaster lets those looped 'git
fetch's piggyback on the existing SSH connection, but still has to start up
git and run several round-trips.

Better, but still doesn't scale to hundreds of extensions: several minutes
for a null update is too frustrating!


Checking them out as submodules via 'git submodule add' and then issuing a
single 'git submodule update' command takes... 0.15 seconds. Nice!

Looks like it does indeed see that there's no changes, so nothing has to be
pulled from the upstream repos. Good!


The downside is that maintaining submodules means constantly pushing commits
to the containing repo so it knows there are updates. :(

Probably the most user-friendly way to handle this is with a wrapper script
that can do a single query to fetch the current branch head positions of a
bunch of repos, then does fetch/pull only on the ones that have changed.

This could still end up pulling from 600+ repos -- if there are actually
changes in them all! -- but should make typical cases a *lot* faster.

We should check in a little more detail how Android & other big projects
using multiple git repos are doing their helper tools to see if we can just
use something that already does this or if we have to build it ourselves. :)

-- brion
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


happy.melon.wiki at gmail

Oct 5, 2011, 3:07 PM

Post #10 of 32 (1154 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On 5 October 2011 22:30, Brion Vibber <brion [at] pobox> wrote:

> On Wed, Oct 5, 2011 at 1:59 PM, Brion Vibber <brion [at] pobox> wrote:
>
> This could still end up pulling from 600+ repos -- if there are actually
> changes in them all! -- but should make typical cases a *lot* faster.
>

Pushed localisation updates from translatewiki would produce precisely this
effect (minor but nonzero changes to hundreds of repos) on a daily or at
least weekly basis. :-(

--HM
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


s.mazeland at xs4all

Oct 5, 2011, 3:12 PM

Post #11 of 32 (1159 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

Op 5 okt. 2011 om 15:07 heeft Happy Melon <happy.melon.wiki [at] gmail> het volgende geschreven:

> On 5 October 2011 22:30, Brion Vibber <brion [at] pobox> wrote:
>
>> On Wed, Oct 5, 2011 at 1:59 PM, Brion Vibber <brion [at] pobox> wrote:
>>
>> This could still end up pulling from 600+ repos -- if there are actually
>> changes in them all! -- but should make typical cases a *lot* faster.
>>
>
> Pushed localisation updates from translatewiki would produce precisely this
> effect (minor but nonzero changes to hundreds of repos) on a daily or at
> least weekly basis. :-(

Actually, we plan to start pushing updates automagically every day, so that will be a reality.

Is that ssh session share thingy also available for Windows users?

Siebrand
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


conrad.irwin at gmail

Oct 5, 2011, 3:12 PM

Post #12 of 32 (1154 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On Wed, Oct 5, 2011 at 3:07 PM, Happy Melon <happy.melon.wiki [at] gmail>wrote:

> On 5 October 2011 22:30, Brion Vibber <brion [at] pobox> wrote:
>
> > On Wed, Oct 5, 2011 at 1:59 PM, Brion Vibber <brion [at] pobox> wrote:
> >
> > This could still end up pulling from 600+ repos -- if there are actually
> > changes in them all! -- but should make typical cases a *lot* faster.
> >
>

What's the use-case for checking out all of the extensions? Presumably that
would only be needed if you wanted to do some global clean-up, so this won't
affect the majority of people too extremely.

Conrad
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


overlordq at gmail

Oct 5, 2011, 3:41 PM

Post #13 of 32 (1159 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On Wed, Oct 5, 2011 at 5:12 PM, Siebrand Mazeland <s.mazeland [at] xs4all> wrote:
> Is that ssh session share thingy also available for Windows users?
>

If you use one of the windows clients that support it[1], I would assume so.

1 - http://en.wikipedia.org/wiki/Comparison_of_SSH_clients#Technical

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


valhallasw at arctus

Oct 6, 2011, 12:06 AM

Post #14 of 32 (1156 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On 6 October 2011 00:12, Siebrand Mazeland <s.mazeland [at] xs4all> wrote:
> Is that ssh session share thingy also available for Windows users?

As Windows users generally use putty (although I'm not 100% sure about
git-mingw, in this respect, but it works at least with tortoisesvn,
and probably also with tortoisegit): pageant is what you're looking
for. It's available from the putty website [1].

Best,
Merlijn

[1] http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


lists at nadir-seen-fire

Oct 6, 2011, 12:55 AM

Post #15 of 32 (1160 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On 11-10-05 03:07 PM, Happy Melon wrote:
> On 5 October 2011 22:30, Brion Vibber <brion [at] pobox> wrote:
>
>> On Wed, Oct 5, 2011 at 1:59 PM, Brion Vibber <brion [at] pobox> wrote:
>>
>> This could still end up pulling from 600+ repos -- if there are actually
>> changes in them all! -- but should make typical cases a *lot* faster.
>>
> Pushed localisation updates from translatewiki would produce precisely this
> effect (minor but nonzero changes to hundreds of repos) on a daily or at
> least weekly basis. :-(
>
> --HM
I wish we could stop pushing TWN updates as code commits.
I'm getting tired of trying to look through an extension's log for
changes and having to sort through it where 3/4 of the commits are TWN
updates.

Someone mentioned an idea at one point of having only English in core
(as the source for translations) and then pulling localization updates
in with some script, part of building the tarballs would be pulling in
the localizations and bundling them.

--
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


maxsem.wiki at gmail

Oct 6, 2011, 9:07 AM

Post #16 of 32 (1152 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On 06.10.2011, 2:12 Conrad Irwin wrote:

> What's the use-case for checking out all of the extensions? Presumably that
> would only be needed if you wanted to do some global clean-up, so this won't
> affect the majority of people too extremely.

Actually, it's pretty common, most if not all active core devs do
that because every change to core APIs must be accompanied with a grep
through extensions to check if it's OK or something will need to be
fixed.

--
Best regards,
Max Semenik ([[User:MaxSem]])


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


z at mzmcbride

Oct 6, 2011, 9:59 PM

Post #17 of 32 (1151 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

Max Semenik wrote:
> On 06.10.2011, 2:12 Conrad Irwin wrote:
>> What's the use-case for checking out all of the extensions? Presumably that
>> would only be needed if you wanted to do some global clean-up, so this won't
>> affect the majority of people too extremely.
>
> Actually, it's pretty common, most if not all active core devs do
> that because every change to core APIs must be accompanied with a grep
> through extensions to check if it's OK or something will need to be
> fixed.

s/grep/ack-grep/ ;-)

MZMcBride



_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


vasilvv at gmail

Oct 6, 2011, 10:43 PM

Post #18 of 32 (1149 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On Wed, Oct 5, 2011 at 5:33 AM, Brion Vibber <brion [at] pobox> wrote:
> You can check out extensions as separate repositories directly into
> subfolders within core's 'extensions' dir for a ready-to-run system. But,
> you *do* need to do either manually or scripted iteration over them to pull
> updates or commit across repos. Git's submodules might be a useful way to
> help automate checkouts, but they introduce their own complications for
> maintenance.

That does not sound like The Bright Git Future.

--vvv

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


lists at nadir-seen-fire

Oct 7, 2011, 4:48 AM

Post #19 of 32 (1149 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On 11-10-06 10:43 PM, Victor Vasiliev wrote:
> On Wed, Oct 5, 2011 at 5:33 AM, Brion Vibber <brion [at] pobox> wrote:
>> You can check out extensions as separate repositories directly into
>> subfolders within core's 'extensions' dir for a ready-to-run system. But,
>> you *do* need to do either manually or scripted iteration over them to pull
>> updates or commit across repos. Git's submodules might be a useful way to
>> help automate checkouts, but they introduce their own complications for
>> maintenance.
> That does not sound like The Bright Git Future.
>
> --vvv
;) No, "The Bright Git Future" is when I can commit from my server, pull
the changes to my local working copy, and push them to the central repo
from there. Since I develop on my servers, but don't trust them with my
private keys.

I currently do this with absolute hacks involving ssh up on both working
copies, piping svn diff through ssh into patch, commit, then another svn
up. The fact I have unfinished code lying around in my working copies
just makes things even more fun (I always make use of git's lovely index
which lets me pick piece by piece what parts of the diff to actually
commit). Not to mention that the svn diff trick has issues if I have a
new file.

This of course leads to lovely commits like:
This: https://www.mediawiki.org/wiki/Special:Code/MediaWiki/96668
And this: https://www.mediawiki.org/wiki/Special:Code/MediaWiki/96273
Humorous ones like this:
https://www.mediawiki.org/wiki/Special:Code/MediaWiki/97180
^_^ And this kind of lovely commit:
https://www.mediawiki.org/wiki/Special:Code/MediaWiki/85242

By the way, when we switch to get I'll finally be able to get rid of
half the reason I make some commits without bothering to test them ;).
Since it will no longer be a huge hassle to make the change in a place I
can actually test it.

~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


vasilvv at gmail

Oct 7, 2011, 5:27 AM

Post #20 of 32 (1145 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On Fri, Oct 7, 2011 at 3:48 PM, Daniel Friesen
<lists [at] nadir-seen-fire> wrote:
> I currently do this with absolute hacks involving ssh up on both working
> copies, piping svn diff through ssh into patch, commit, then another svn
> up. The fact I have unfinished code lying around in my working copies
> just makes things even more fun (I always make use of git's lovely index
> which lets me pick piece by piece what parts of the diff to actually
> commit). Not to mention that the svn diff trick has issues if I have a
> new file.

I also use two working copies (one read-only for development and one
for commiting). This is mostly due to lack of svn stash and local
branches (I have at least two large patches on my local copy I'm
forced to manually unmerge by editing svn diffs).

--vvv

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


hashar+wmf at free

Oct 9, 2011, 2:08 AM

Post #21 of 32 (1136 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On 07/10/11 14:27, Victor Vasiliev wrote:
> I also use two working copies (one read-only for development and one
> for commiting). This is mostly due to lack of svn stash and local
> branches (I have at least two large patches on my local copy I'm
> forced to manually unmerge by editing svn diffs).

You could git init your existing repository to have a git stash on top
of your existing svn working copie.
Whenever you want to svn update: stash your changes with git, svn
update, git commit, unstash and merge :)

--
Ashar Voultoiz


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


hashar+wmf at free

Oct 9, 2011, 2:17 AM

Post #22 of 32 (1126 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On 06/10/11 09:55, Daniel Friesen wrote:
> I wish we could stop pushing TWN updates as code commits.
> I'm getting tired of trying to look through an extension's log for
> changes and having to sort through it where 3/4 of the commits are TWN
> updates.
>
> Someone mentioned an idea at one point of having only English in core
> (as the source for translations) and then pulling localization updates
> in with some script, part of building the tarballs would be pulling in
> the localizations and bundling them.

That might be me, or at least I have the same concern about logs
specially when bisecting changes.

Much like we moved extension out of core, we might want to have
translations moved out too. We do not really need 0day translations for
day to day MW hacking.

Having all translations (core+ext) in one repository will makes life
easier for the translation team and developers maintaining their
extension outside of our repository will be able to take advantage of
the translation system.

To update live site, we will just have to fetch the change from that
repository and merge that to the wmf branches.
To release a new tarball, we could fetch the latest revision from the
translation repository.


--
Ashar Voultoiz




--
Ashar Voultoiz


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Platonides at gmail

Oct 9, 2011, 11:20 AM

Post #23 of 32 (1122 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

Ashar Voultoiz wrote:
> That might be me, or at least I have the same concern about logs
> specially when bisecting changes.
>
> Much like we moved extension out of core, we might want to have
> translations moved out too. We do not really need 0day translations for
> day to day MW hacking.

What about commits which add a message?
Most i18n commits are irrelevant for normal coding. But there are code
commits which carry along i18n changes.

And I don't like solutions like keeping English at a different repo, or
having to split a commit in two independent repositories. Maybe
subrepositories can help here?


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


lists at nadir-seen-fire

Oct 9, 2011, 11:52 AM

Post #24 of 32 (1134 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On 11-10-09 11:20 AM, Platonides wrote:
> Ashar Voultoiz wrote:
>> That might be me, or at least I have the same concern about logs
>> specially when bisecting changes.
>>
>> Much like we moved extension out of core, we might want to have
>> translations moved out too. We do not really need 0day translations for
>> day to day MW hacking.
> What about commits which add a message?
> Most i18n commits are irrelevant for normal coding. But there are code
> commits which carry along i18n changes.
Commits adding messages aren't generally done by TWN so those are of
course really code changes which would be committed.
The few times TWN might actually change an en message I don't mind
seeing it. It won't be like the mass of non-en stuff that's in the way
when browsing logs.

> And I don't like solutions like keeping English at a different repo, or
> having to split a commit in two independent repositories. Maybe
> subrepositories can help here?
A submodule can only handle entire dirs. I don't know about putting the
canonical English in another repo, then it's not simply twn occasionally
crossing repos, it's every single developer making any change that
happens to also add a new message or modify a message crossing two repos
to change the canonical form. Piles of non-automated commits will need
to double commit and include a submodule commit id change in the parent
repo's commit.

What I've been thinking isn't so much putting translations in another
repo, or even TWN doing any commits anywhere at all. Frankly the whole
idea of TWN reading and writing .php files has felt completely messed up
to me anyways. Sure our canonical message forms can be in .php, but
having the semi-automated system we use to translate to every other
language we support output php files feels like a relic of a time before
it existed and a band-aid hack just to make it possible for TWN to do
translations back then.
I'd like to make TWN the proper source for all the translations. Rather
than TWN spitting out php for non-en, we have a proper generated output
format for translations, and MediaWiki uses that instead of .php for our
translations. Instead of TWN having to make this a commit somewhere, I
think we should pull those translations right from TWN once we need them.

If we make an out-of-repo pull for i18n our preferred way of getting
i18n instead of having it pushed to core as commits we can also make the
method of getting absolute up-to-date translations even on non-trunk
versions a non-hack.

--
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


hashar+wmf at free

Oct 9, 2011, 12:50 PM

Post #25 of 32 (1128 views)
Permalink
Re: Preliminary git module splitup notes (MediaWiki core & extensions) [In reply to]

On 09/10/11 20:20, Platonides wrote:
> What about commits which add a message?

Since English is our reference language, we would keep it in core. If
you wanted to provide other translations (Portuguese for you, or French
for me), it would need to be done in the i18n repository.

> Most i18n commits are irrelevant for normal coding. But there are code
> commits which carry along i18n changes.

Those i18n commits would have to be done on the new repository.

> And I don't like solutions like keeping English at a different repo, or
> having to split a commit in two independent repositories. Maybe
> subrepositories can help here?

Per above, we should keep English in the core repository.

--
Ashar Voultoiz


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

First page Previous page 1 2 Next page Last page  View All Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.