Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Gentoo: Dev

RFC: an eclass for github snapshots?

 

 

Gentoo dev RSS feed   Index | Next | Previous | View Threaded


mgorny at gentoo

May 29, 2011, 11:27 PM

Post #1 of 17 (581 views)
Permalink
RFC: an eclass for github snapshots?

Hello,

Right now, a quick 'grep -l github.*tarball' shows that there are about
147 ebuilds in portage using github snapshots. This evaluates to 83
different packages.

The problem with github is that it suffixes the tarballs with
a complete git commit id. This means that the `S' variable
in the ebuild needs to refer to a long hash changing randomly. Right
now, the problem is handled in a number of ways:

1) (from app-admin/rudy)

S="${WORKDIR}/solutious-${PN}-*"

I'm surprised if that actually works. I'd say that seems not
PMS-compliant but in fact PMS seems to almost not mention S.

2) (app-emacs/calfw and suggested solution for Sunrise)

src_unpack() {
unpack ${A}
mv kiwanami-emacs-calfw-* ${P} || die
}

3) (app-misc/bgrep)

GITHUB_USER="tmbinc"
GITHUB_HASH="49b098be9548d174023ad05c10f6af9d02b8e18e"
MY_P="${GITHUB_USER}-${PN}-${GITHUB_HASH:0:7}"
S="${WORKDIR}/${MY_P}"

4) (app-misc/tmux-mem-cpu-load)

src_prepare() {
cd "${WORKDIR}"/thewtex-${PN}-*
S=$(pwd)
}


What I'd like to do is creating a small github.eclass, encapsulating
a common, nice way of handling the S issue. I guess the best solution
would be to git with something like 2) above, with the eclass providing
github_src_unpack() for EAPIs 2+.

Maybe the eclass could be extended to support other kinds of snapshots.
I'm not sure if there aren't other services providing snapshots similar
to github.

--
Best regards,
Michał Górny
Attachments: signature.asc (0.31 KB)


flameeyes at gmail

May 30, 2011, 2:35 AM

Post #2 of 17 (557 views)
Permalink
Re: RFC: an eclass for github snapshots? [In reply to]

Il giorno lun, 30/05/2011 alle 08.27 +0200, Michał Górny ha scritto:
>
> S="${WORKDIR}/solutious-${PN}-*"
>
> I'm surprised if that actually works. I'd say that seems not
> PMS-compliant but in fact PMS seems to almost not mention S.

Because you didn't follow the whole eclass tree ;)

ruby-ng handles the star as a special case, given how many of those we
had to use over time, so it is that, expanding SRC_URI, not the package
manager.

--
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/


ulm at gentoo

May 30, 2011, 4:21 AM

Post #3 of 17 (559 views)
Permalink
Re: RFC: an eclass for github snapshots? [In reply to]

>>>>> On Mon, 30 May 2011, Diego Elio Pettenò wrote:
> Il giorno lun, 30/05/2011 alle 08.27 +0200, Michał Górny ha scritto:
>> S="${WORKDIR}/solutious-${PN}-*"
>>
>> I'm surprised if that actually works. I'd say that seems not
>> PMS-compliant but in fact PMS seems to almost not mention S.

> Because you didn't follow the whole eclass tree ;)

> ruby-ng handles the star as a special case, given how many of those
> we had to use over time, [...]

But it is not compliant with PMS:
"If S is assigned in the global scope of an ebuild, then the
restrictions of section 12.2 for global variables apply." (section 12.1)
"Global variables must only contain invariant values." (section 12.2)

Ulrich


flameeyes at gmail

May 30, 2011, 4:44 AM

Post #4 of 17 (554 views)
Permalink
Re: RFC: an eclass for github snapshots? [In reply to]

Il giorno lun, 30/05/2011 alle 13.21 +0200, Ulrich Mueller ha scritto:
>
> But it is not compliant with PMS:
> "If S is assigned in the global scope of an ebuild, then the
> restrictions of section 12.2 for global variables apply." (section
> 12.1)
> "Global variables must only contain invariant values." (section 12.2)

It is fixed for EAPI=4, but we didn't want to break the previous API
that worked quite fine for over an year.

--
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/


dberkholz at gentoo

Jun 7, 2011, 7:17 AM

Post #5 of 17 (536 views)
Permalink
Re: Re: RFC: an eclass for github snapshots? [In reply to]

On 13:21 Mon 30 May , Ulrich Mueller wrote:
> >>>>> On Mon, 30 May 2011, Diego Elio Pettenò wrote:
> > Il giorno lun, 30/05/2011 alle 08.27 +0200, Michał Górny ha scritto:
> >> S="${WORKDIR}/solutious-${PN}-*"
> >>
> >> I'm surprised if that actually works. I'd say that seems not
> >> PMS-compliant but in fact PMS seems to almost not mention S.
>
> > Because you didn't follow the whole eclass tree ;)
>
> > ruby-ng handles the star as a special case, given how many of those
> > we had to use over time, [...]
>
> But it is not compliant with PMS:
> "If S is assigned in the global scope of an ebuild, then the
> restrictions of section 12.2 for global variables apply." (section 12.1)
> "Global variables must only contain invariant values." (section 12.2)

It seems compliant to me, as S is assigned an invariant value that
happens to contain the character '*', which is overwritten with a new
value as a local variable in ebuild functions. Sample code in listing
12.1 in my copy of the PMS seems to suggest this is perfectly fine
behavior as long as the global invariant is restored after each
function.

--
Thanks,
Donnie

Donnie Berkholz
Sr. Developer, Gentoo Linux
Blog: http://dberkholz.com


ulm at gentoo

Jun 7, 2011, 11:16 AM

Post #6 of 17 (539 views)
Permalink
Re: Re: RFC: an eclass for github snapshots? [In reply to]

>>>>> On Tue, 7 Jun 2011, Donnie Berkholz wrote:

>> But it is not compliant with PMS:
>> "If S is assigned in the global scope of an ebuild, then the
>> restrictions of section 12.2 for global variables apply." (section 12.1)
>> "Global variables must only contain invariant values." (section 12.2)

> It seems compliant to me, as S is assigned an invariant value that
> happens to contain the character '*', which is overwritten with a new
> value as a local variable in ebuild functions. Sample code in listing
> 12.1 in my copy of the PMS seems to suggest this is perfectly fine
> behavior as long as the global invariant is restored after each
> function.

Even if it fulfills the restrictions for global variables, it is still
an abuse of the spec, because PMS defines S as follows:
"The full path to the temporary build directory, used by src_compile,
src_install etc."

And for EAPI 4 it will fail, because S is required to exist as initial
working directory in most src_* phase functions.

Ulrich


graaff at gentoo

Jun 7, 2011, 11:01 PM

Post #7 of 17 (529 views)
Permalink
Re: Re: RFC: an eclass for github snapshots? [In reply to]

On Tue, 2011-06-07 at 20:16 +0200, Ulrich Mueller wrote:

> Even if it fulfills the restrictions for global variables, it is still
> an abuse of the spec, because PMS defines S as follows:
> "The full path to the temporary build directory, used by src_compile,
> src_install etc."

I don't see how setting S violates this specification. For each ruby
implementation that we build for the definition of S holds. It just has
a different value for each implementation.

> And for EAPI 4 it will fail, because S is required to exist as initial
> working directory in most src_* phase functions.

Correct, so in EAPI 4 we now set the RUBY_S variable to handle the
initial setup, and then we set S as part of the environment setup when
we are iterating through each ruby implementation within each of the PMS
phases.

Kind regards,

Hans
Attachments: signature.asc (0.22 KB)


ulm at gentoo

Jun 8, 2011, 12:17 AM

Post #8 of 17 (530 views)
Permalink
Re: Re: RFC: an eclass for github snapshots? [In reply to]

>>>>> On Wed, 08 Jun 2011, Hans de Graaff wrote:

>> Even if it fulfills the restrictions for global variables, it is
>> still an abuse of the spec, because PMS defines S as follows:
>> "The full path to the temporary build directory, used by
>> src_compile, src_install etc."

> I don't see how setting S violates this specification. For each ruby
> implementation that we build for the definition of S holds. It just
> has a different value for each implementation.

The value of S that is assigned in global scope (i.e. the one
containing the wildcard) violates it.

Ulrich


graaff at gentoo

Jun 8, 2011, 8:43 AM

Post #9 of 17 (525 views)
Permalink
Re: Re: RFC: an eclass for github snapshots? [In reply to]

On Wed, 2011-06-08 at 09:17 +0200, Ulrich Mueller wrote:

> The value of S that is assigned in global scope (i.e. the one
> containing the wildcard) violates it.

Ah, right, I initially read Donnie's quotation from PMS as an
endorsement for our approach, but that is only true for our EAPI=4
solution where we only change S within a function.

That leaves the question what to do with the approach for EAPI=2,3. I'd
rather not risk breaking ebuilds by removing this support just for a
violation of PMS if there is no real problem exposed by it.

Kind regards,

Hans
Attachments: signature.asc (0.22 KB)


ciaran.mccreesh at googlemail

Jun 8, 2011, 8:50 AM

Post #10 of 17 (522 views)
Permalink
Re: Re: RFC: an eclass for github snapshots? [In reply to]

On Wed, 08 Jun 2011 17:43:38 +0200
Hans de Graaff <graaff [at] gentoo> wrote:
> That leaves the question what to do with the approach for EAPI=2,3.
> I'd rather not risk breaking ebuilds by removing this support just
> for a violation of PMS if there is no real problem exposed by it.

The 'invariant' restriction on S in PMS is, strictly speaking, stronger
than it has to be. However, working out exactly what set of weaker
rules would be ok proved to be too difficult -- historically, Portage
has had various different behaviours for global scope variables that
are assigned variable values. Thus, PMS is the way it is there because
we know for sure that if you follow those rules you're safe; if you
don't, you'll definitely cause problems for EAPI 4, and you may or may
not get away with it for earlier EAPIs.

It's a bit like assuming that it's ok to dereference a null pointer
and get a zero, since that's what one particular system does...

--
Ciaran McCreesh
Attachments: signature.asc (0.19 KB)


graaff at gentoo

Jun 8, 2011, 12:20 PM

Post #11 of 17 (527 views)
Permalink
Re: Re: RFC: an eclass for github snapshots? [In reply to]

On Wed, 2011-06-08 at 16:50 +0100, Ciaran McCreesh wrote:
> On Wed, 08 Jun 2011 17:43:38 +0200
> Hans de Graaff <graaff [at] gentoo> wrote:
> > That leaves the question what to do with the approach for EAPI=2,3.
> > I'd rather not risk breaking ebuilds by removing this support just
> > for a violation of PMS if there is no real problem exposed by it.
>
> The 'invariant' restriction on S in PMS is, strictly speaking, stronger
> than it has to be. However, working out exactly what set of weaker
> rules would be ok proved to be too difficult -- historically, Portage
> has had various different behaviours for global scope variables that
> are assigned variable values. Thus, PMS is the way it is there because
> we know for sure that if you follow those rules you're safe; if you
> don't, you'll definitely cause problems for EAPI 4, and you may or may
> not get away with it for earlier EAPIs.
>
> It's a bit like assuming that it's ok to dereference a null pointer
> and get a zero, since that's what one particular system does...

Thanks for the background on this particular part of the specification.

I think I'll add an eqawarn to the eclass for EAPI=2,3 and migrate
ebuilds over naturally. I'll bump the remaining ones in 6 months or so.
That also gives overlays some time to move to EAPI=4.

Kind regards,

Hans
Attachments: signature.asc (0.22 KB)


leho at kraav

Mar 11, 2012, 10:25 AM

Post #12 of 17 (390 views)
Permalink
Re: RFC: an eclass for github snapshots? [In reply to]

On Monday, May 30, 2011 9:30:02 AM UTC+3, Michał Górny wrote:
>
> Right now, a quick 'grep -l github.*tarball' shows that there are about
> 147 ebuilds in portage using github snapshots. This evaluates to 83
> different packages.
>
> The problem with github is that it suffixes the tarballs with
> a complete git commit id. This means that the `S' variable
> in the ebuild needs to refer to a long hash changing randomly. Right
> now, the problem is handled in a number of ways:
>
> 1) (from app-admin/rudy)
> 2) (app-emacs/calfw and suggested solution for Sunrise)
> 3) (app-misc/bgrep)
> 4) (app-misc/tmux-mem-cpu-load)
>
> What I'd like to do is creating a small github.eclass, encapsulating
> a common, nice way of handling the S issue. I guess the best solution
> would be to git with something like 2) above, with the eclass providing
> github_src_unpack() for EAPIs 2+.

What is the current situation with this one? Every once in a while I run into a github ebuild I need to create and I am not really sure what to do with it.

Right now 2) seems like the safest approach. But did anything get into EAPI?


zmedico at gentoo

Mar 11, 2012, 11:15 AM

Post #13 of 17 (391 views)
Permalink
Re: RFC: an eclass for github snapshots? [In reply to]

On 03/11/2012 10:25 AM, Leho Kraav wrote:
> On Monday, May 30, 2011 9:30:02 AM UTC+3, Michał Górny wrote:
>>
>> Right now, a quick 'grep -l github.*tarball' shows that there are about
>> 147 ebuilds in portage using github snapshots. This evaluates to 83
>> different packages.
>>
>> The problem with github is that it suffixes the tarballs with
>> a complete git commit id. This means that the `S' variable
>> in the ebuild needs to refer to a long hash changing randomly. Right
>> now, the problem is handled in a number of ways:
>>
>> 1) (from app-admin/rudy)
>> 2) (app-emacs/calfw and suggested solution for Sunrise)
>> 3) (app-misc/bgrep)
>> 4) (app-misc/tmux-mem-cpu-load)
>>
>> What I'd like to do is creating a small github.eclass, encapsulating
>> a common, nice way of handling the S issue. I guess the best solution
>> would be to git with something like 2) above, with the eclass providing
>> github_src_unpack() for EAPIs 2+.
>
> What is the current situation with this one? Every once in a while I run into a github ebuild I need to create and I am not really sure what to do with it.
>
> Right now 2) seems like the safest approach. But did anything get into EAPI?

No, there's no EAPI extension for that yet, and no bug filed for it here:

https://bugs.gentoo.org/show_bug.cgi?id=174380
--
Thanks,
Zac


mgorny at gentoo

Mar 11, 2012, 11:27 AM

Post #14 of 17 (384 views)
Permalink
Re: RFC: an eclass for github snapshots? [In reply to]

On Sun, 11 Mar 2012 10:25:38 -0700 (PDT)
Leho Kraav <leho [at] kraav> wrote:

> On Monday, May 30, 2011 9:30:02 AM UTC+3, Michał Górny wrote:
> >
> > Right now, a quick 'grep -l github.*tarball' shows that there are
> > about 147 ebuilds in portage using github snapshots. This evaluates
> > to 83 different packages.
> >
> > The problem with github is that it suffixes the tarballs with
> > a complete git commit id. This means that the `S' variable
> > in the ebuild needs to refer to a long hash changing randomly. Right
> > now, the problem is handled in a number of ways:
> >
> > 1) (from app-admin/rudy)
> > 2) (app-emacs/calfw and suggested solution for Sunrise)
> > 3) (app-misc/bgrep)
> > 4) (app-misc/tmux-mem-cpu-load)
> >
> > What I'd like to do is creating a small github.eclass, encapsulating
> > a common, nice way of handling the S issue. I guess the best
> > solution would be to git with something like 2) above, with the
> > eclass providing github_src_unpack() for EAPIs 2+.
>
> What is the current situation with this one? Every once in a while I
> run into a github ebuild I need to create and I am not really sure
> what to do with it.
>
> Right now 2) seems like the safest approach. But did anything get
> into EAPI?

You mean eclass? I submitted one for review but didn't get much of
positive feedback on it. I'll commit it anyway soon, just let me double
check and do some testing.

--
Best regards,
Michał Górny
Attachments: signature.asc (0.31 KB)


yngwin at gmail

Mar 11, 2012, 6:36 PM

Post #15 of 17 (365 views)
Permalink
Re: RFC: an eclass for github snapshots? [In reply to]

On 12 March 2012 02:27, Michał Górny <mgorny [at] gentoo> wrote:
> On Sun, 11 Mar 2012 10:25:38 -0700 (PDT)
> Leho Kraav <leho [at] kraav> wrote:
>
>> On Monday, May 30, 2011 9:30:02 AM UTC+3, Michał Górny wrote:
>> >
>> > Right now, a quick 'grep -l github.*tarball' shows that there are
>> > about 147 ebuilds in portage using github snapshots. This evaluates
>> > to 83 different packages.
>> >
>> > The problem with github is that it suffixes the tarballs with
>> > a complete git commit id. This means that the `S' variable
>> > in the ebuild needs to refer to a long hash changing randomly. Right
>> > now, the problem is handled in a number of ways:
>> >
>> > 1) (from app-admin/rudy)
>> > 2) (app-emacs/calfw and suggested solution for Sunrise)
>> > 3) (app-misc/bgrep)
>> > 4) (app-misc/tmux-mem-cpu-load)
>> >
>> > What I'd like to do is creating a small github.eclass, encapsulating
>> > a common, nice way of handling the S issue. I guess the best
>> > solution would be to git with something like 2) above, with the
>> > eclass providing github_src_unpack() for EAPIs 2+.
>>
>> What is the current situation with this one? Every once in a while I
>> run into a github ebuild I need to create and I am not really sure
>> what to do with it.
>>
>> Right now 2) seems like the safest approach. But did anything get
>> into EAPI?
>
> You mean eclass? I submitted one for review but didn't get much of
> positive feedback on it. I'll commit it anyway soon, just let me double
> check and do some testing.

+1 from me. I think it would be useful to have a standard way of handling this.

Cheers,
Ben | yngwin


mgorny at gentoo

Mar 12, 2012, 2:47 AM

Post #16 of 17 (363 views)
Permalink
Re: RFC: an eclass for github snapshots? [In reply to]

On Mon, 12 Mar 2012 09:36:00 +0800
Ben <yngwin [at] gmail> wrote:

> On 12 March 2012 02:27, Michał Górny <mgorny [at] gentoo> wrote:
> > On Sun, 11 Mar 2012 10:25:38 -0700 (PDT)
> > Leho Kraav <leho [at] kraav> wrote:
> >
> >> On Monday, May 30, 2011 9:30:02 AM UTC+3, Michał Górny wrote:
> >> >
> >> > Right now, a quick 'grep -l github.*tarball' shows that there are
> >> > about 147 ebuilds in portage using github snapshots. This
> >> > evaluates to 83 different packages.
> >> >
> >> > The problem with github is that it suffixes the tarballs with
> >> > a complete git commit id. This means that the `S' variable
> >> > in the ebuild needs to refer to a long hash changing randomly.
> >> > Right now, the problem is handled in a number of ways:
> >> >
> >> > 1) (from app-admin/rudy)
> >> > 2) (app-emacs/calfw and suggested solution for Sunrise)
> >> > 3) (app-misc/bgrep)
> >> > 4) (app-misc/tmux-mem-cpu-load)
> >> >
> >> > What I'd like to do is creating a small github.eclass,
> >> > encapsulating a common, nice way of handling the S issue. I
> >> > guess the best solution would be to git with something like 2)
> >> > above, with the eclass providing github_src_unpack() for EAPIs
> >> > 2+.
> >>
> >> What is the current situation with this one? Every once in a while
> >> I run into a github ebuild I need to create and I am not really
> >> sure what to do with it.
> >>
> >> Right now 2) seems like the safest approach. But did anything get
> >> into EAPI?
> >
> > You mean eclass? I submitted one for review but didn't get much of
> > positive feedback on it. I'll commit it anyway soon, just let me
> > double check and do some testing.
>
> +1 from me. I think it would be useful to have a standard way of
> handling this.

Attaching my current conceptual eclass. I've tested it with github,
gitweb and bitbucket. It won't work with gitorious but their snapshot
download mechanism is broken anyway (they like to submit 'try again
later' in plaintext).

--
Best regards,
Michał Górny
Attachments: vcs-snapshot.eclass (1.39 KB)
  signature.asc (0.31 KB)


mgorny at gentoo

Mar 19, 2012, 1:41 AM

Post #17 of 17 (351 views)
Permalink
Re: RFC: an eclass for github snapshots? [In reply to]

On Mon, 12 Mar 2012 10:47:41 +0100
Michał Górny <mgorny [at] gentoo> wrote:

> On Mon, 12 Mar 2012 09:36:00 +0800
> Ben <yngwin [at] gmail> wrote:
>
> > On 12 March 2012 02:27, Michał Górny <mgorny [at] gentoo> wrote:
> > > On Sun, 11 Mar 2012 10:25:38 -0700 (PDT)
> > > Leho Kraav <leho [at] kraav> wrote:
> > >
> > >> On Monday, May 30, 2011 9:30:02 AM UTC+3, Michał Górny wrote:
> > >> >
> > >> > Right now, a quick 'grep -l github.*tarball' shows that there
> > >> > are about 147 ebuilds in portage using github snapshots. This
> > >> > evaluates to 83 different packages.
> > >> >
> > >> > The problem with github is that it suffixes the tarballs with
> > >> > a complete git commit id. This means that the `S' variable
> > >> > in the ebuild needs to refer to a long hash changing randomly.
> > >> > Right now, the problem is handled in a number of ways:
> > >> >
> > >> > 1) (from app-admin/rudy)
> > >> > 2) (app-emacs/calfw and suggested solution for Sunrise)
> > >> > 3) (app-misc/bgrep)
> > >> > 4) (app-misc/tmux-mem-cpu-load)
> > >> >
> > >> > What I'd like to do is creating a small github.eclass,
> > >> > encapsulating a common, nice way of handling the S issue. I
> > >> > guess the best solution would be to git with something like 2)
> > >> > above, with the eclass providing github_src_unpack() for EAPIs
> > >> > 2+.
> > >>
> > >> What is the current situation with this one? Every once in a
> > >> while I run into a github ebuild I need to create and I am not
> > >> really sure what to do with it.
> > >>
> > >> Right now 2) seems like the safest approach. But did anything get
> > >> into EAPI?
> > >
> > > You mean eclass? I submitted one for review but didn't get much of
> > > positive feedback on it. I'll commit it anyway soon, just let me
> > > double check and do some testing.
> >
> > +1 from me. I think it would be useful to have a standard way of
> > handling this.
>
> Attaching my current conceptual eclass. I've tested it with github,
> gitweb and bitbucket. It won't work with gitorious but their snapshot
> download mechanism is broken anyway (they like to submit 'try again
> later' in plaintext).

Committed.

--
Best regards,
Michał Górny
Attachments: signature.asc (0.31 KB)

Gentoo dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.