Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Wikipedia: Wikitech

Nightly tarballs?

 

 

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded


mah at everybody

Aug 1, 2012, 12:29 PM

Post #1 of 18 (2998 views)
Permalink
Nightly tarballs?

Is there any interest in having nightly snapshots of MediaWiki available?

I realize people could just use git, but this poses a problem for users
who are familiar with extracting, say, a .xip file, and making MediaWiki
work, but are stymied by the esoteric nature of git.

This would be similar to Mozilla's nightlies:
(http://nightly.mozilla.org/) and may also be a stepping stone for
people to get into development, or at least patch submission.

This all came up because I had the chance to provide a snapshot to help
solve a problem in 1.19
(http://thread.gmane.org/gmane.org.wikimedia.mediawiki/40011, shortened:
http://hexm.de/kt).

I used the make-release script
(http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/make-release/,
shortened: http://hexm.de/ku) and put the snapshot up at
http://mah.everybody.org/snapshots/.

I'm willing to set this up to run on wmflabs.org or on my own server if
there is interest. This may also be a good way to measure the "need"
for a point release -- for example, if the nightly starts including
fixes for annoying bugs that affect a lot of people, then a point
release is probably needed. (I'm looking at you, Bug #24985.)

--
http://hexmode.com/

Human evil is not a problem. It is a mystery. It cannot be solved.
-- When Atheism Becomes Religion, Chris Hedges

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


p858snake at gmail

Aug 1, 2012, 12:48 PM

Post #2 of 18 (2934 views)
Permalink
Re: Nightly tarballs? [In reply to]

I believe there is already a process of this happening on the TS
(don't remember the address), or it did used to happen.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


mah at everybody

Aug 1, 2012, 12:52 PM

Post #3 of 18 (2936 views)
Permalink
Re: Nightly tarballs? [In reply to]

On 08/01/2012 03:48 PM, K. Peachey wrote:
> I believe there is already a process of this happening on the TS
> (don't remember the address), or it did used to happen.

I believe it. Now, to find it and publicize it. (And maybe get some
stats on the usage of it.)


--
http://hexmode.com/

Human evil is not a problem. It is a mystery. It cannot be solved.
-- When Atheism Becomes Religion, Chris Hedges

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


mah at everybody

Aug 1, 2012, 12:55 PM

Post #4 of 18 (2936 views)
Permalink
Re: Nightly tarballs? [In reply to]

On 08/01/2012 03:48 PM, K. Peachey wrote:
> I believe there is already a process of this happening on the TS
> (don't remember the address), or it did used to happen.

Found it: http://toolserver.org/~vvv/mw-nightly/


--
http://hexmode.com/

Human evil is not a problem. It is a mystery. It cannot be solved.
-- When Atheism Becomes a Religion, Chris Hedges

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


valhallasw at arctus

Aug 1, 2012, 3:23 PM

Post #5 of 18 (2929 views)
Permalink
Re: Nightly tarballs? [In reply to]

On 1 August 2012 21:55, Mark A. Hershberger <mah [at] everybody> wrote:
> Found it: http://toolserver.org/~vvv/mw-nightly/

Please note this is hosted in vvv's personal space. Due to the nature
of toolserver accounts, which are locked if the account is not
renewed, the address has a very low 'bus factor'[1] - so it it's
better not to promote this address publicly.

Best,
Merlijn

[1] http://en.wikipedia.org/wiki/Bus_factor

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


lists at nadir-seen-fire

Aug 1, 2012, 3:39 PM

Post #6 of 18 (2930 views)
Permalink
Re: Nightly tarballs? [In reply to]

On Wed, 01 Aug 2012 15:23:37 -0700, Merlijn van Deen
<valhallasw [at] arctus> wrote:

> On 1 August 2012 21:55, Mark A. Hershberger <mah [at] everybody> wrote:
>> Found it: http://toolserver.org/~vvv/mw-nightly/
>
> Please note this is hosted in vvv's personal space. Due to the nature
> of toolserver accounts, which are locked if the account is not
> renewed, the address has a very low 'bus factor'[1] - so it it's
> better not to promote this address publicly.
>
> Best,
> Merlijn
>
> [1] http://en.wikipedia.org/wiki/Bus_factor

Also, gitweb has a snapshot feature so you don't exactly even need a
nightly:
https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=snapshot;h=HEAD;sf=tgz

--
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


Platonides at gmail

Aug 1, 2012, 3:50 PM

Post #7 of 18 (2934 views)
Permalink
Re: Nightly tarballs? [In reply to]

On 02/08/12 00:39, Daniel Friesen wrote:
> Also, gitweb has a snapshot feature so you don't exactly even need a
> nightly:
> https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=snapshot;h=HEAD;sf=tgz

I don't think we need a separate nightly page. If someone wants a
"nightly", he can use this git download. Differences with release
mediawiki are minimal.
We could make a page with the steps to download it and providing the url
(which I agree is not pretty), but that's all.


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


mah at everybody

Aug 1, 2012, 3:51 PM

Post #8 of 18 (2931 views)
Permalink
Re: Nightly tarballs? [In reply to]

On 08/01/2012 06:39 PM, Daniel Friesen wrote:
> Also, gitweb has a snapshot feature so you don't exactly even need a
> nightly:
> https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=snapshot;h=HEAD;sf=tgz

Excellent. Glad the solution here was just fixing my ignorance.

Mark.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


vasilvv at gmail

Aug 1, 2012, 3:56 PM

Post #9 of 18 (2929 views)
Permalink
Re: Nightly tarballs? [In reply to]

On Thu, Aug 2, 2012 at 2:39 AM, Daniel Friesen
<lists [at] nadir-seen-fire> wrote:
> On Wed, 01 Aug 2012 15:23:37 -0700, Merlijn van Deen <valhallasw [at] arctus>
> wrote:
>
>> On 1 August 2012 21:55, Mark A. Hershberger <mah [at] everybody> wrote:
>>>
>>> Found it: http://toolserver.org/~vvv/mw-nightly/
>>
>>
>> Please note this is hosted in vvv's personal space. Due to the nature
>> of toolserver accounts, which are locked if the account is not
>> renewed, the address has a very low 'bus factor'[1] - so it it's
>> better not to promote this address publicly.
>>
>> Best,
>> Merlijn
>>
>> [1] http://en.wikipedia.org/wiki/Bus_factor
>
>
> Also, gitweb has a snapshot feature so you don't exactly even need a
> nightly:
> https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=snapshot;h=HEAD;sf=tgz
>

Also, the tool linked works as a redirect to the exact URL you specified :)

— Victor.

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


mah at everybody

Aug 1, 2012, 4:17 PM

Post #10 of 18 (2929 views)
Permalink
Re: Nightly tarballs? [In reply to]

On 08/01/2012 06:50 PM, Platonides wrote:
> I don't think we need a separate nightly page. If someone wants a
> "nightly", he can use this git download. Differences with release
> mediawiki are minimal.

The release *does* include some standard extensions that are not in the
gitweb link.

But, yes, having the gitweb link makes this less of an issue.

> We could make a page with the steps to download it and providing the url
> (which I agree is not pretty), but that's all.

Right. I'll do that soon if no one else does. If someone else does it,
then I would ask that they post a followup here.

Mark.



_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


innocentkiller at gmail

Aug 1, 2012, 4:26 PM

Post #11 of 18 (2923 views)
Permalink
Re: Nightly tarballs? [In reply to]

On Wed, Aug 1, 2012 at 7:17 PM, Mark A. Hershberger <mah [at] everybody> wrote:
> On 08/01/2012 06:50 PM, Platonides wrote:
>> I don't think we need a separate nightly page. If someone wants a
>> "nightly", he can use this git download. Differences with release
>> mediawiki are minimal.
>
> The release *does* include some standard extensions that are not in the
> gitweb link.
>
> But, yes, having the gitweb link makes this less of an issue.
>

Let's please not link to the gitweb tars. They're not cached, so
each one would be generated on demand.

-Chad

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


krinklemail at gmail

Aug 1, 2012, 5:06 PM

Post #12 of 18 (2926 views)
Permalink
Re: Nightly tarballs? [In reply to]

We already have hourly snapshots of the stable master though:

https://toolserver.org/~krinkle/mwSnapshots/#!/mediawiki-core/master

(and it includes release branches, feature branches and wmf branches).

That could be expanded to keep old version (right now it only keeps the
latest one).

-- KJrinkle

On Wed, Aug 1, 2012 at 12:29 PM, Mark A. Hershberger <mah [at] everybody>wrote:

> Is there any interest in having nightly snapshots of MediaWiki available?
>
> I realize people could just use git, but this poses a problem for users
> who are familiar with extracting, say, a .xip file, and making MediaWiki
> work, but are stymied by the esoteric nature of git.
>
> This would be similar to Mozilla's nightlies:
> (http://nightly.mozilla.org/) and may also be a stepping stone for
> people to get into development, or at least patch submission.
>
> This all came up because I had the chance to provide a snapshot to help
> solve a problem in 1.19
> (http://thread.gmane.org/gmane.org.wikimedia.mediawiki/40011, shortened:
> http://hexm.de/kt).
>
> I used the make-release script
> (http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/make-release/,
> shortened: http://hexm.de/ku) and put the snapshot up at
> http://mah.everybody.org/snapshots/.
>
> I'm willing to set this up to run on wmflabs.org or on my own server if
> there is interest. This may also be a good way to measure the "need"
> for a point release -- for example, if the nightly starts including
> fixes for annoying bugs that affect a lot of people, then a point
> release is probably needed. (I'm looking at you, Bug #24985.)
>
> --
> http://hexmode.com/
>
> Human evil is not a problem. It is a mystery. It cannot be solved.
> -- When Atheism Becomes Religion, Chris Hedges
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l [at] lists
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
>
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


lists at nadir-seen-fire

Aug 1, 2012, 6:08 PM

Post #13 of 18 (2933 views)
Permalink
Re: Nightly tarballs? [In reply to]

Expanding it might be a good idea. Though you should probably rethink the
tool and do things differently.

When you wrote that we discussed Git's built in `git archive` which can
create a tarball of an arbitrary repo.
But besides that what we didn't think of was Gitweb. Which already has
snapshot capabilities.
The hourly snapshot tool doesn't really serve much purpose:
- Gitweb HEAD snapshots are always completely up to date while the
snapshot lags by an hour
- Gitweb can give snapshots of every branch in the repo so the hourly
snapshot doesn't add anything there
- Gitweb can give snapshots of every repo instead of just core. So we
already essentially have snapshots of extensions built into Gitweb while
the hourly tool only does core.
- Gitweb can make a snapshot of any commit, not just the latest commit

So the only thing that a tool really has to offer is a clean UI and some
information like the checksum.
Those are probably what we should focus on.

- Drop the notion of hourly snapshots; Either use Gitweb snapshot links
directly or package snapshots dynamically when a new one is requested.
- Support getting snapshots from old commits in the repo.
- Consider expanding the tool to support other repos.

Because this will basically be a UI we shouldn't be half-hearted about the
UI. The tool is pointless if we don't go all the way.
Don't leave 'Branches' out in the open like that. This is a user tool
rather than a Git tool.
So instead of listing things like "master" and "REL1_19" use a mapping to
give them proper labels like "Development" or "Alpha" and "MediaWiki
1.19.x".
Also cleanly group things together. ie: Take all the REL#_## branches and
put them in a "Release snapshots" group. wmf/ in "WMF branch snapshots".
And things like Wikidata and iwtransclusion in a "Experimental project
snapshots" group.
Also drop the select field and instead make it a list of links that users
can actually see.


I wonder if we should turn this into a special page on MW.org.

--
~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

On Wed, 01 Aug 2012 17:06:08 -0700, Krinkle <krinklemail [at] gmail> wrote:

> We already have hourly snapshots of the stable master though:
>
> https://toolserver.org/~krinkle/mwSnapshots/#!/mediawiki-core/master
>
> (and it includes release branches, feature branches and wmf branches).
>
> That could be expanded to keep old version (right now it only keeps the
> latest one).
>
> -- KJrinkle
>
> On Wed, Aug 1, 2012 at 12:29 PM, Mark A. Hershberger
> <mah [at] everybody>wrote:
>
>> Is there any interest in having nightly snapshots of MediaWiki
>> available?
>>
>> I realize people could just use git, but this poses a problem for users
>> who are familiar with extracting, say, a .xip file, and making MediaWiki
>> work, but are stymied by the esoteric nature of git.
>>
>> This would be similar to Mozilla's nightlies:
>> (http://nightly.mozilla.org/) and may also be a stepping stone for
>> people to get into development, or at least patch submission.
>>
>> This all came up because I had the chance to provide a snapshot to help
>> solve a problem in 1.19
>> (http://thread.gmane.org/gmane.org.wikimedia.mediawiki/40011, shortened:
>> http://hexm.de/kt).
>>
>> I used the make-release script
>> (http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/make-release/,
>> shortened: http://hexm.de/ku) and put the snapshot up at
>> http://mah.everybody.org/snapshots/.
>>
>> I'm willing to set this up to run on wmflabs.org or on my own server if
>> there is interest. This may also be a good way to measure the "need"
>> for a point release -- for example, if the nightly starts including
>> fixes for annoying bugs that affect a lot of people, then a point
>> release is probably needed. (I'm looking at you, Bug #24985.)
>>
>> --
>> http://hexmode.com/
>>
>> Human evil is not a problem. It is a mystery. It cannot be solved.
>> -- When Atheism Becomes Religion, Chris Hedges

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


robla at wikimedia

Aug 1, 2012, 6:10 PM

Post #14 of 18 (2924 views)
Permalink
Re: Nightly tarballs? [In reply to]

On Wed, Aug 1, 2012 at 12:29 PM, Mark A. Hershberger <mah [at] everybody> wrote:
> I'm willing to set this up to run on wmflabs.org or on my own server if
> there is interest. This may also be a good way to measure the "need"
> for a point release -- for example, if the nightly starts including
> fixes for annoying bugs that affect a lot of people, then a point
> release is probably needed. (I'm looking at you, Bug #24985.)

For testing reasons, I think this would be a great thing to have. I
would love to have a process where Jenkins kicks off a nightly
tarball, running make-release and then uploading it to Swift somehow.
I asked Ben offlist if it would be possible to set up a non-wiki
public front end, and he acknowledged it would be and that we'll
probably need it for other stuff (though nothing is planned, and no
one has explicitly asked before now).

The reason I bring Swift into this is because I'm pretty sure we don't
have any place else that would have the capacity we need to feel
comfortable hosting something like this. Even then, we'd still
probably want to do pruning (keep a couple weeks of tarballs, then
nuke the old stuff).

Don't read "I think this would be great" to mean that this is a high
priority thing, since I can't promise this won't get stalled behind
other priorities, but it's an interesting idea. The next step, I
think, is to figure out exactly how hosting on Swift would work.

Rob

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


krinklemail at gmail

Aug 1, 2012, 6:42 PM

Post #15 of 18 (2923 views)
Permalink
Re: Nightly tarballs? [In reply to]

On Wed, Aug 1, 2012 at 4:26 PM, Chad <innocentkiller [at] gmail> wrote:

> On Wed, Aug 1, 2012 at 7:17 PM, Mark A. Hershberger <mah [at] everybody>
> wrote:
> > On 08/01/2012 06:50 PM, Platonides wrote:
> >> I don't think we need a separate nightly page. If someone wants a
> >> "nightly", he can use this git download. Differences with release
> >> mediawiki are minimal.
> >
> > The release *does* include some standard extensions that are not in the
> > gitweb link.
> >
> > But, yes, having the gitweb link makes this less of an issue.
> >
>
> Let's please not link to the gitweb tars. They're not cached, so
> each one would be generated on demand.
>
> -Chad
>
>
That's why I created mwSnapshots are a replacement for vvv's mw-nightly
(which, before it was taken down, afaik still only worked with SVN).

https://toolserver.org/~krinkle/mwSnapshots/#!/mediawiki-core/master

And it does cache :)

-- Krinkle
_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


mah at everybody

Aug 2, 2012, 7:57 AM

Post #16 of 18 (2911 views)
Permalink
Re: Nightly tarballs? [In reply to]

Rob Laphier writes:

> For testing reasons, I think this would be a great thing to have.
...
> Don't read "I think this would be great" to mean that this is a high
> priority thing, since I can't promise this won't get stalled behind
> other priorities

Fair enough.

Given that Chad has pointed out the un-cached nature of the gitweb
links, though, this sounds like the thing to do.

Since it sounds like the WMF is at least willing to provide the
infrastructure (Git, Jenkins and Swift), can those of us outside the WMF
who want to make this happen work with Chad, Antoine, and Ben to figure
out how we can handle the nitty-gritty work that needs to be done?

It sounds like some knowledge transfer (which I hope would be minimally
invasive) and the already-existing infrastructure would be a all that is
required from the WMF to make this happen. The rest (changes for
Jenkins) could be done by volunteers.

--
http://hexmode.com/

Human evil is not a problem. It is a mystery. It cannot be solved.
-- When Atheism Becomes a Religion, Chris Hedges

_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


hashar+wmf at free

Aug 2, 2012, 9:06 AM

Post #17 of 18 (2909 views)
Permalink
Re: Nightly tarballs? [In reply to]

Le 01/08/12 21:29, Mark A. Hershberger a écrit :
> Is there any interest in having nightly snapshots of MediaWiki available?
<snip>

I have created a new Jenkins job to generate nightly snapshots of
mediawiki/core.git :

https://integration.mediawiki.org/ci/job/Nightly%20-%20MediaWiki%20Core/

Files are available at:
https://integration.mediawiki.org/nightly/mediawiki/core/

mediawiki-latest.zip being a symbolic link to the latest build.

The job is triggered everyday at 3am UTC which has been a low activity
hour for the last 4 months or so.

There is no process to cleanup the old snapshot, something we will have
to think about later.

The build script needed a git-archive macro:
https://gerrit.wikimedia.org/r/#/c/17416/
And the Jenkins + ant target is at:
https://gerrit.wikimedia.org/r/#/c/17417/

cheers,


--
Antoine "hashar" Musso


_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


z at mzmcbride

Aug 27, 2012, 5:23 PM

Post #18 of 18 (2742 views)
Permalink
Re: Nightly tarballs? [In reply to]

Daniel Friesen wrote:
> I wonder if we should turn this into a special page on MW.org.

I'm not sure about a Special page, but I created a very short page on
MediaWiki.org: <https://www.mediawiki.org/wiki/Nightlies>.

MZMcBride



_______________________________________________
Wikitech-l mailing list
Wikitech-l [at] lists
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Wikipedia wikitech RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.