Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Gentoo: Dev

Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds

  

 
Gentoo dev RSS feed   Index | Next | Previous | View Threaded


grobian at gentoo

Oct 18, 2009, 2:11 AM

Post #1 of 22 (800 views)
Permalink
Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds
Recently, Prefix changes have been committed to the gentoo-x86 tree, it
was rather ambitious on my part, where I moved stuff that we are not
maintainer of ourself. It should have been communicated better for
these ebuilds. This is a formal apology for springing that onto you.
This will attempt to solve the confusion and answer questions that have
been raised.

Gentoo Prefix has mainly the following characteristics:
- allow for unprivileged installations,
- in an offset-prefix called "the prefix", often referred to as EPREFIX,
in autoconf words: ./configure --prefix=${EPREFIX}/usr

In the Prefix Portage branch, that we have in use [1], we extended and
adapted regular Portage to support above two characteristics. I will
ignore the details of unprivileged installations in this email.

The offset-prefix changes in Portage introduce three new variables that
are available in the ebuild environment:
- EPREFIX: the configured offset-prefix
- ED: shorthand for ${D%/}${EPREFIX}/
- EROOT: ${ROOT%/}${EPREFIX}/

The offset-prefix changes require extensive changes to most eclasses,
and minor changes to many ebuilds. This is mainly because "awareness"
of the offset-prefix has to be added to places where ebuilds manually
deal with file-system locations. In particular:
- configure calls that specify some argument to find a component need to
do so in the offset prefix, e.g.
--with-libxml2="${EPREFIX}"/usr/$(get_libdir)
- almost every place where ${D} is used, the offset-prefix ${EPREFIX}
has to be added. Because this is lengthy, Prefix Portage provides a
variable ${ED} that is the shortcut for ${D} plus the offset-prefix.
- the same holds for occurrences of ${ROOT}, where ${EROOT} is available
as shortcut

Why do we need both ${ED} and ${D}? Technically we don't, because we
can use ${EPREFIX} all the time. However, one cannot say that ${D}
includes ${EPREFIX} for Prefix Portage, because that means the following
src_install() function no longer works correctly:

src_install() {
emake DESTDIR="${D}" install || die
mv "${D}"/usr/bin/{,exuberant-}ctags || die
}

While the "mv" does a great job if ${D} would include ${EPREFIX} here,
the make install will cause double offset to be written, since configure
recorded the ${EPREFIX} before in src_compile using the argument
--prefix="${EPREFIX}"/usr. In a previous version of Prefix Portage, the
variable EDEST was available as workaround for this, so the example
would read like this:

src_install() {
emake DESTDIR="${EDEST}" install || die
mv "${D}"/usr/bin/{,exuberant-}ctags || die
}

Apart from that this approach got very tricky and confusing in eclasses
and ebuilds that do special mungling in their src_install, it makes it
harder to reconstruct the variable in normal Portage and hence to make
existing ebuilds forward compatible.

The lengthy forward compatible version of the example src_install
function would look like this:

src_install() {
emake DESTDIR="${D}" install || die
mv "${D%/}${EPREFIX}"/usr/bin/{,exuberant-}ctags || die
}

As mentioned before, this is pretty lengthy, and quickly becomes too
much work and unreadable when ${D} is used more.
To avoid the confusion that implicit ${EPREFIX} in ${D} in the ${EDEST}
approach brought, ED and EROOT were chosen because they are easy to
understand and easy to reconstruct outside Prefix Portage. To help
with this, the Prefix profiles use.force the "prefix" USE-flag.
Non-Prefix profiles have this flag masked and unset in the base profile.
This USE-flag can be used to do conditional code for Prefix consumers.
In case of our ${ED} and ${EROOT} convenience variables, we can use it
to define ${ED} and ${EROOT} in case a normal Portage is used. For our
example function:

src_install() {
use prefix || local ED=${D}
emake DESTDIR="${D}" install || die
mv "${ED}"/usr/bin/{,exuberant-}ctags || die
}

Here, Prefix Portage (using a Prefix profile) will not set ${ED}, but
still do as intended because ${ED} is set correctly by Prefix Portage.
Normal Portage will set the convenience variable such that it does not
cause a sandbox violation due to the missing image path.

We will consult the maintainers of packages we would like to make
compatible with Gentoo Portage before adding the changes. In the
future, you will likely see more bug reports and more requests on IRC
from us.

At this point, we have reached a critical mass where we cannot maintain
the Prefix overlay much longer with its size and usage. We either
continue to grow on, requiring less maintenance on tree synchronisation,
or we stop the project -- an option we really don't like.
Bringing back most, and ideally all, ebuilds to gentoo-x86 significantly
improves the workload caused by synchronisation, leaving more time for
the real issues. Examples are fixing and porting packages and getting
the Prefix Portage branch merged with regular Portage some day. At that
point, the variables EPREFIX, ED and EROOT can become available in a
next EAPI as well.


[1] http://sources.gentoo.org/viewcvs.py/portage/main/branches/prefix/

--
Fabian Groffen
Gentoo on a different level


scarabeus at gentoo

Oct 18, 2009, 4:57 AM

Post #2 of 22 (772 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
Hi,
You know i am totaly supporting prefix but i have one point.
Why on earth portage simply does not detect the prefix enviroment is being run
and then INTERNALY switch D->ED and other variables. It would be much easier
that way to migrate all stuff in portage instead of doing this || shebang.
Mostly when it is done by eclasses its quite cool, but when you get into
changing lots of ebuilds its quite hard for maintaining.

Even the multilib overlay guys rather modify the portage than changing a load
of ebuilds.

Just my 2 cents.

Tomas


grobian at gentoo

Oct 18, 2009, 5:31 AM

Post #3 of 22 (765 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
On 18-10-2009 13:57:10 +0200, Tomáš Chvátal wrote:
> Hi,
> You know i am totaly supporting prefix but i have one point.
> Why on earth portage simply does not detect the prefix enviroment is being run
> and then INTERNALY switch D->ED and other variables. It would be much easier
> that way to migrate all stuff in portage instead of doing this || shebang.
> Mostly when it is done by eclasses its quite cool, but when you get into
> changing lots of ebuilds its quite hard for maintaining.
>
> Even the multilib overlay guys rather modify the portage than changing a load
> of ebuilds.

Of course we would like to do that, but that was rejected for EAPI=3, so
it will at least take until EAPI=4 is implemented, which is not the
forseeable future, given that EAPI=3 isn't a fact yet either.


--
Fabian Groffen
Gentoo on a different level


grobian at gentoo

Oct 19, 2009, 12:44 PM

Post #4 of 22 (763 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
On 18-10-2009 14:31:15 +0200, Fabian Groffen wrote:
> On 18-10-2009 13:57:10 +0200, Tomáš Chvátal wrote:
> > Hi,
> > You know i am totaly supporting prefix but i have one point.
> > Why on earth portage simply does not detect the prefix enviroment is being run
> > and then INTERNALY switch D->ED and other variables. It would be much easier
> > that way to migrate all stuff in portage instead of doing this || shebang.
> > Mostly when it is done by eclasses its quite cool, but when you get into
> > changing lots of ebuilds its quite hard for maintaining.
> >
> > Even the multilib overlay guys rather modify the portage than changing a load
> > of ebuilds.
>
> Of course we would like to do that, but that was rejected for EAPI=3, so
> it will at least take until EAPI=4 is implemented, which is not the
> forseeable future, given that EAPI=3 isn't a fact yet either.

I was just informed that it is also a possibility to do an EAPI bump
just for these variables, which would mean we can avoid replicating
setting ED and EROOT in ebuilds.

The suggestion was to just introduce EAPI=3 with these variables, and
making everything which is scheduled for current EAPI=3 just EAPI=4. I
was told we could quite quickly have a Portage in the tree that would
set ED and EROOT for EAPI=3 that way.

Are there any objections to this? If not, I'd like to put this on the
agenda for the next council meeting.


--
Fabian Groffen
Gentoo on a different level


betelgeuse at gentoo

Oct 24, 2009, 12:37 PM

Post #5 of 22 (773 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
Fabian Groffen wrote:
> On 18-10-2009 14:31:15 +0200, Fabian Groffen wrote:
>> On 18-10-2009 13:57:10 +0200, Tomáš Chvátal wrote:
>>> Hi,
>>> You know i am totaly supporting prefix but i have one point.
>>> Why on earth portage simply does not detect the prefix enviroment is being run
>>> and then INTERNALY switch D->ED and other variables. It would be much easier
>>> that way to migrate all stuff in portage instead of doing this || shebang.
>>> Mostly when it is done by eclasses its quite cool, but when you get into
>>> changing lots of ebuilds its quite hard for maintaining.
>>>
>>> Even the multilib overlay guys rather modify the portage than changing a load
>>> of ebuilds.
>> Of course we would like to do that, but that was rejected for EAPI=3, so
>> it will at least take until EAPI=4 is implemented, which is not the
>> forseeable future, given that EAPI=3 isn't a fact yet either.
>
> I was just informed that it is also a possibility to do an EAPI bump
> just for these variables, which would mean we can avoid replicating
> setting ED and EROOT in ebuilds.
>

It's possible.

> The suggestion was to just introduce EAPI=3 with these variables, and
> making everything which is scheduled for current EAPI=3 just EAPI=4. I
> was told we could quite quickly have a Portage in the tree that would
> set ED and EROOT for EAPI=3 that way.
>

Maybe 2+prefix is a more describing name? This would avoid changing what
EAPI 3 means.

> Are there any objections to this? If not, I'd like to put this on the
> agenda for the next council meeting.
>

As the council decided to add new stuff in the last meeting if zac is
starting to implement new EAPIs this could go into EAPI 3 too.

Regards,
Petteri
Attachments: signature.asc (0.26 KB)


grobian at gentoo

Oct 24, 2009, 1:00 PM

Post #6 of 22 (755 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
On 24-10-2009 22:37:30 +0300, Petteri Räty wrote:
> > The suggestion was to just introduce EAPI=3 with these variables, and
> > making everything which is scheduled for current EAPI=3 just EAPI=4. I
> > was told we could quite quickly have a Portage in the tree that would
> > set ED and EROOT for EAPI=3 that way.
>
> Maybe 2+prefix is a more describing name? This would avoid changing what
> EAPI 3 means.

Naming is up to others, from my point of view.

> > Are there any objections to this? If not, I'd like to put this on the
> > agenda for the next council meeting.
>
> As the council decided to add new stuff in the last meeting if zac is
> starting to implement new EAPIs this could go into EAPI 3 too.

Yes, this was implicit, the next EAPI should contain the same support
too.


--
Fabian Groffen
Gentoo on a different level


ulm at gentoo

Nov 13, 2009, 3:43 AM

Post #7 of 22 (717 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
In its November meeting [1], the council has unanimously expressed
support for this proposal [2].

However, there is need for additional discussion. From the council
meeting log I could extract the following open questions:

1. What are the implications for non-prefix devs and users?

2. Should the Prefix team be allowed to do the necessary changes to
ebuilds themselves, or should it be done by the respective
maintainers?

3. Are there any backwards compatibility or upgrade path issues for
eclasses that must still accept EAPI 0 (where the new ED, EROOT,
and EPREFIX variables are not defined)?

4. EAPI numbering: Would this simply be added as an additional
feature to EAPI 3? Or should we have an intermediate EAPI slot,
e.g. 2.1 or 3 (and current EAPI 3 renamed to 4 in the latter
case)?

5. Who is going to write the exact specification (PMS patch) for
this EAPI feature?

6. (Any question that I've missed?)

Let's start the discussion now, in order to work out these details
before the next council meeting (December 7th).

Ulrich

[1] <http://www.gentoo.org/proj/en/council/meeting-logs/20091109.txt>
(topic was discussed from 21:32 to 22:11 in the log's timezone)
[2] <http://archives.gentoo.org/gentoo-dev/msg_2a62689c71f95e4de5699a330b8b5524.xml>


calchan at gentoo

Nov 19, 2009, 4:26 PM

Post #8 of 22 (700 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
On Sun, Oct 18, 2009 at 2:11 AM, Fabian Groffen <grobian [at] gentoo> wrote:
> This is a formal apology for springing that onto you.

Thanks a lot. Not everybody can do such a thing as a public apology. I
will nonetheless ask the council to vote on the following during next
meeting:
Ask Fabian to change his signature from:
"Gentoo on a different level"
To:
"Failure in a different level"
;o)

2009/10/18 Tomáš Chvátal <scarabeus [at] gentoo>:
> Why on earth portage simply does not detect the prefix enviroment is being run
> and then INTERNALY switch D->ED and other variables.

If that means we can get away without touching ebuilds, apart from
changing their EAPI variable, then that's absolutely what we have to
do. I'd like things to be done the right way though (see below).

On Fri, Nov 13, 2009 at 4:43 AM, Ulrich Mueller <ulm [at] gentoo> wrote:
> However, there is need for additional discussion. From the council
> meeting log I could extract the following open questions:

It would be preferable for the discussion to happen on this list
before the meeting or we'll end up postponing again due to having more
questions coming up at that time.

> 2. Should the Prefix team be allowed to do the necessary changes to
> ebuilds themselves, or should it be done by the respective
> maintainers?

I think here it's obvious that anybody who is an ebuild dev and sees
anything to fix (prefix or else) is encouraged to go ahead and do it,
as we've always done. The recommendation is and will always be to talk
to the current maintainers out of politeness and to be extra careful
(i.e. usually letting the maintainers do it) in case of
system/tricky/exotic package. We don't give full cvs access to the
whole tree to all ebuild devs for nothing.

> 4. EAPI numbering: Would this simply be added as an additional
> feature to EAPI 3? Or should we have an intermediate EAPI slot,
> e.g. 2.1 or 3 (and current EAPI 3 renamed to 4 in the latter
> case)?

Here I'd add to the choices: why not release an intermediate EAPI with
the prefix stuff and whatever that has already been done for EAPI3?
The exact name of a potential intermediate EAPI is a non-problem.
However I would prefer if it were a number like 2.1 or 2.5 or even 3,
because although we currently treat the EAPI variable as a simple
string we may change our mind later and find it handy someday to use
operators on them such as >=2.1.

> 5. Who is going to write the exact specification (PMS patch) for
> this EAPI feature?

I thought I asked Fabian to work on that at the end of the meeting. In
case I didn't then consider this as me officially asking him if he can
take care of it. Fabian is this OK with you?

Also I think it would be nice if somebody took care of a portage
patch, since I hear it is rather simple. Fabian again? Or Zac? Any
other volunteers?

I would prefer to have all the pieces in places before the next
meeting so that we can vote on the real thing and have prefix
implemented the right way before the end of the year.

> 6. (Any question that I've missed?)

Here are a few that I gathered from others (my comments are between
parentheses):

> How are dynamically linked set*id programs going to work?

> How are scripts using #!shebangs going to work?
> You write an ebuild, and you DEPEND upon >=foo-3, because the build
> process includes some foo code. The foo code is executed via
> scripts using #!/usr/bin/foo. Normally, this is fine.
> But on prefix, /usr/bin/foo might be a crappy, OS X mangled foo-2
> that's no good. So even though you've got the foo-3 dep met, it'll be
> met in /opt/Gentoo/blah, so your package will fail.

> How are ebuilds to be marked as supporting prefix or not?
(Here I'm guessing that changing the EAPI variable will do)

> Why is there only a single permitted installation path?
(I'm under the impression this is a limitation of the windows
installer but not of prefix itself. So patching the installer would
fix that)

> What exactly is expected from a prefix-compliant package manager to
> support full prefix installs, as opposed to just supporting installs
> to / with prefix-aware ebuilds?
(The PMS patch should answer that)

Denis.


darkside at gentoo

Nov 19, 2009, 5:42 PM

Post #9 of 22 (698 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
Some questions answered. snipped the rest.

Denis Dupeyron wrote:
> 2009/10/18 Tomáš Chvátal <scarabeus [at] gentoo>:
>> Why on earth portage simply does not detect the prefix enviroment is being run
>> and then INTERNALY switch D->ED and other variables.
>
> If that means we can get away without touching ebuilds, apart from
> changing their EAPI variable, then that's absolutely what we have to
> do. I'd like things to be done the right way though (see below).

When you change econf to do --prefix=${EPREFIX}/usr then you cannot
simply s/D/ED/ for everything. I hope this makes sense when you think
about it. ;)

src_install() {
emake DESTDIR="${D}" install || die
mv "${ED}"/usr/bin/{,exuberant-}ctags || die
}

But then again, some ebuilds need no changing once you fix econf to do
the work, which is nice.

>
> On Fri, Nov 13, 2009 at 4:43 AM, Ulrich Mueller <ulm [at] gentoo> wrote:
>> However, there is need for additional discussion. From the council
>> meeting log I could extract the following open questions:
>
> It would be preferable for the discussion to happen on this list
> before the meeting or we'll end up postponing again due to having more
> questions coming up at that time.

We are willing to talk, but it always seems like the Council is "not
prepared" no matter what we do. Hope everyone involved can change that.

>
>> 2. Should the Prefix team be allowed to do the necessary changes to
>> ebuilds themselves, or should it be done by the respective
>> maintainers?
>
> I think here it's obvious that anybody who is an ebuild dev and sees
> anything to fix (prefix or else) is encouraged to go ahead and do it,
> as we've always done. The recommendation is and will always be to talk
> to the current maintainers out of politeness and to be extra careful
> (i.e. usually letting the maintainers do it) in case of
> system/tricky/exotic package. We don't give full cvs access to the
> whole tree to all ebuild devs for nothing.

It is quite obvious that we are not trying to make trouble. Talk is
cheap, so we prefer that. But, we see no need to ask permission to add
~prefix keywords, same as other arch teams.

Currently, 'repoman -d full' will fail in some packages. We are fixing this.

> Also I think it would be nice if somebody took care of a portage
> patch, since I hear it is rather simple. Fabian again? Or Zac? Any
> other volunteers?
>
> I would prefer to have all the pieces in places before the next
> meeting so that we can vote on the real thing and have prefix
> implemented the right way before the end of the year.

portage devs and prefix devs have agreed that it is rather 'easy' to
merge the prefix-portage branch. Just waiting.. ;) We have access to
check into the portage repo, so this should not hold anything up
regarding any decisions.

>
>> 6. (Any question that I've missed?)

>> How are scripts using #!shebangs going to work?
>> You write an ebuild, and you DEPEND upon >=foo-3, because the build
>> process includes some foo code. The foo code is executed via
>> scripts using #!/usr/bin/foo. Normally, this is fine.
>> But on prefix, /usr/bin/foo might be a crappy, OS X mangled foo-2
>> that's no good. So even though you've got the foo-3 dep met, it'll be
>> met in /opt/Gentoo/blah, so your package will fail.

The prefix-portage branch has a nice feature that fixes shebangs
automatically to be ${EPREFIX}/foo instead of /foo. It has even caught
some Gentoo Linux bugs.

>
>> How are ebuilds to be marked as supporting prefix or not?
> (Here I'm guessing that changing the EAPI variable will do)

Gentoo Prefix has keywords. So if EAPI 3 has ED/EROOT support but the
ebuild doesn't use them then the ebuild does not need an EAPI bump. In
this case, please rephrase your question to be "How are ebuilds to be
marked as working on a prefix arch or not?" and then it is clear that it
is the same as Gentoo Linux.

>
>> Why is there only a single permitted installation path?
> (I'm under the impression this is a limitation of the windows
> installer but not of prefix itself. So patching the installer would
> fix that)

My installation path on my 6-8 prefix arches is in my NFS home. If you
are referring to the Windows special installation package, well..that is
just a "stage4" installer with binary packages. The windows installer is
no where near the heart of Gentoo Prefix, instead it is a product of
Gentoo Prefix and a convenience factor offered by another Prefix dev. It
showcases the possibilities quite well, IMO.

You can set EPREFIX to anything. One of our users even set it to "/" -
which we don't endorse but it is possible. :)

>
>> What exactly is expected from a prefix-compliant package manager to
>> support full prefix installs, as opposed to just supporting installs
>> to / with prefix-aware ebuilds?
> (The PMS patch should answer that)
>
> Denis.
>


grobian at gentoo

Nov 20, 2009, 12:45 AM

Post #10 of 22 (692 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
On 13-11-2009 12:43:25 +0100, Ulrich Mueller wrote:
> In its November meeting [1], the council has unanimously expressed
> support for this proposal [2].
>
> However, there is need for additional discussion. From the council
> meeting log I could extract the following open questions:
>
> 1. What are the implications for non-prefix devs and users?

for devs: take note of EPREFIX, ED and EROOT, be aware what they mean
and eventually, use them in the right way. This is not hard, since a
simple script[3] based on some heuristics can do the work in 98% of the
cases right.
for users: not an single bit, unless they look inside ebuilds

> 2. Should the Prefix team be allowed to do the necessary changes to
> ebuilds themselves, or should it be done by the respective
> maintainers?

The Prefix team is the equivalent of an arch team, only handling a big
bunch of arches [4]. Hence, keywording should be out of the question,
and just allowed. Adding simple patches (to normal non-critical
ebuilds) used to be allowed AFAIR, so I see that as ok too. In other
words, like darkside mentioned, I see our activities like those of an
arch team.

> 3. Are there any backwards compatibility or upgrade path issues for
> eclasses that must still accept EAPI 0 (where the new ED, EROOT,
> and EPREFIX variables are not defined)?

The backward and forward compatability is handled with conditional
code like:
use prefix || local ED=${D}
or
[[ -z ${ED} ]] && local ED=${D}

Since Zac checked in yesterday blacklisting of EPREFIX, EROOT and ED in
the main Portage branch, these variables hopefully soon become protected
in the sense that an externally set ED does not result in weird
behaviour of a Prefix aware eclass or ebuild when using the latter
approach, which is true forward compatible.
For your information, this was exactly what I asked for from the council.

> 4. EAPI numbering: Would this simply be added as an additional
> feature to EAPI 3? Or should we have an intermediate EAPI slot,
> e.g. 2.1 or 3 (and current EAPI 3 renamed to 4 in the latter
> case)?

I don't care about this, but a fast EAPI is necessary to 1) rely on ED not
coming from outside, and 2) since we're defining ED, EROOT and EPREFIX
anyway, initialise ED and EROOT such that we don't need any conditional
code in ebuilds at all (and can just use ED where necessary).

> 5. Who is going to write the exact specification (PMS patch) for
> this EAPI feature?

We agreed that I/Prefix team will give this a shot.

> 6. (Any question that I've missed?)

I guess most additional questions go beyond, and actually address the
real Prefix work, and its implications to Gentoo and many ebuilds, since
we have to touch around 2300 ebuilds, most with trivial changes, some
with heavy changes (think of the 32 patches that one needs to compile
Python...)

> Let's start the discussion now, in order to work out these details
> before the next council meeting (December 7th).
>
> Ulrich
>
> [1] <http://www.gentoo.org/proj/en/council/meeting-logs/20091109.txt>
> (topic was discussed from 21:32 to 22:11 in the log's timezone)
> [2] <http://archives.gentoo.org/gentoo-dev/msg_2a62689c71f95e4de5699a330b8b5524.xml>
>

[3] http://overlays.gentoo.org/proj/alt/browser/trunk/prefix-overlay/scripts/eapify
[4] http://stats.prefix.freens.org/keywords-packages.png

--
Fabian Groffen
Gentoo on a different level


grobian at gentoo

Nov 20, 2009, 1:03 AM

Post #11 of 22 (694 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
On 19-11-2009 19:42:11 -0600, Jeremy Olexa wrote:
> Some questions answered. snipped the rest.

readded questions where necessary

> Denis Dupeyron wrote:
> > 2009/10/18 Tomáš Chvátal <scarabeus [at] gentoo>:
> >> Why on earth portage simply does not detect the prefix enviroment is being run
> >> and then INTERNALY switch D->ED and other variables.
> >
> > If that means we can get away without touching ebuilds, apart from
> > changing their EAPI variable, then that's absolutely what we have to
> > do. I'd like things to be done the right way though (see below).
>
> When you change econf to do --prefix=${EPREFIX}/usr then you cannot
> simply s/D/ED/ for everything. I hope this makes sense when you think
> about it. ;)
>
> src_install() {
> emake DESTDIR="${D}" install || die
> mv "${ED}"/usr/bin/{,exuberant-}ctags || die
> }
>
> But then again, some ebuilds need no changing once you fix econf to do
> the work, which is nice.
>
> >
> > On Fri, Nov 13, 2009 at 4:43 AM, Ulrich Mueller <ulm [at] gentoo> wrote:
> >> However, there is need for additional discussion. From the council
> >> meeting log I could extract the following open questions:
> >
> > It would be preferable for the discussion to happen on this list
> > before the meeting or we'll end up postponing again due to having more
> > questions coming up at that time.
>
> We are willing to talk, but it always seems like the Council is "not
> prepared" no matter what we do. Hope everyone involved can change that.
>
> >
> >> 2. Should the Prefix team be allowed to do the necessary changes to
> >> ebuilds themselves, or should it be done by the respective
> >> maintainers?
> >
> > I think here it's obvious that anybody who is an ebuild dev and sees
> > anything to fix (prefix or else) is encouraged to go ahead and do it,
> > as we've always done. The recommendation is and will always be to talk
> > to the current maintainers out of politeness and to be extra careful
> > (i.e. usually letting the maintainers do it) in case of
> > system/tricky/exotic package. We don't give full cvs access to the
> > whole tree to all ebuild devs for nothing.
>
> It is quite obvious that we are not trying to make trouble. Talk is
> cheap, so we prefer that. But, we see no need to ask permission to add
> ~prefix keywords, same as other arch teams.
>
> Currently, 'repoman -d full' will fail in some packages. We are fixing this.

> > 4. EAPI numbering: Would this simply be added as an additional
> > feature to EAPI 3? Or should we have an intermediate EAPI slot,
> > e.g. 2.1 or 3 (and current EAPI 3 renamed to 4 in the latter
> > case)?
>
> Here I'd add to the choices: why not release an intermediate EAPI with
> the prefix stuff and whatever that has already been done for EAPI3?
> The exact name of a potential intermediate EAPI is a non-problem.
> However I would prefer if it were a number like 2.1 or 2.5 or even 3,
> because although we currently treat the EAPI variable as a simple
> string we may change our mind later and find it handy someday to use
> operators on them such as >=3D2.1.

While I think that this is a totally different discussion, unrelated to
Prefix (well, on a side-note, we once had things like EAPI="prefix 1",
which denoted both the features from "prefix" and "1"), it is good to
isolate things such that it is obvious what an EAPI stands for.

> > 5. Who is going to write the exact specification (PMS patch) for
> > this EAPI feature?
>
> I thought I asked Fabian to work on that at the end of the meeting. In
> case I didn't then consider this as me officially asking him if he can
> take care of it. Fabian is this OK with you?

Yes, I agreed coming up with some patch. I admit I haven't yet even
looked into it.

> Also I think it would be nice if somebody took care of a portage

Technically, Portage trunk already contains the most important bits that
allow us to continue since yesterday. The rest is waiting for a formal
approval of the council, and then it will most probably be me and Zac
fighting to get the prefix branch merged into trunk.

> > patch, since I hear it is rather simple. Fabian again? Or Zac? Any
> > other volunteers?
> >
> > I would prefer to have all the pieces in places before the next
> > meeting so that we can vote on the real thing and have prefix
> > implemented the right way before the end of the year.
>
> portage devs and prefix devs have agreed that it is rather 'easy' to
> merge the prefix-portage branch. Just waiting.. ;) We have access to
> check into the portage repo, so this should not hold anything up
> regarding any decisions.

just to repeat: the implementation already exists in the "prefix" branch
of the portage repository.

> >> 6. (Any question that I've missed?)
>
> >> How are scripts using #!shebangs going to work?
> >> You write an ebuild, and you DEPEND upon >=foo-3, because the build
> >> process includes some foo code. The foo code is executed via
> >> scripts using #!/usr/bin/foo. Normally, this is fine.
> >> But on prefix, /usr/bin/foo might be a crappy, OS X mangled foo-2
> >> that's no good. So even though you've got the foo-3 dep met, it'll be
> >> met in /opt/Gentoo/blah, so your package will fail.
>
> The prefix-portage branch has a nice feature that fixes shebangs
> automatically to be ${EPREFIX}/foo instead of /foo. It has even caught
> some Gentoo Linux bugs.

Next to that, it is part of the Prefix team's job to make sure that
whatever is installed, does not reference the host system when this is
not absolutely necessary. All of the questions you raise here, are
part of the job to get an ebuild "ported" from gx86 to Prefix. In fact,
the "crappy, on OS X mangled foo-2" was the main drive to get the
project going.

> >> How are ebuilds to be marked as supporting prefix or not?
> > (Here I'm guessing that changing the EAPI variable will do)
>
> Gentoo Prefix has keywords. So if EAPI 3 has ED/EROOT support but the
> ebuild doesn't use them then the ebuild does not need an EAPI bump. In
> this case, please rephrase your question to be "How are ebuilds to be
> marked as working on a prefix arch or not?" and then it is clear that it
> is the same as Gentoo Linux.
>
> >
> >> Why is there only a single permitted installation path?
> > (I'm under the impression this is a limitation of the windows
> > installer but not of prefix itself. So patching the installer would
> > fix that)
>
> My installation path on my 6-8 prefix arches is in my NFS home. If you
> are referring to the Windows special installation package, well..that is
> just a "stage4" installer with binary packages. The windows installer is
> no where near the heart of Gentoo Prefix, instead it is a product of
> Gentoo Prefix and a convenience factor offered by another Prefix dev. It
> showcases the possibilities quite well, IMO.
>
> You can set EPREFIX to anything. One of our users even set it to "/" -
> which we don't endorse but it is possible. :)
>
> >
> >> What exactly is expected from a prefix-compliant package manager to
> >> support full prefix installs, as opposed to just supporting installs
> >> to / with prefix-aware ebuilds?
> > (The PMS patch should answer that)

Agreed


--
Fabian Groffen
Gentoo on a different level


calchan at gentoo

Nov 25, 2009, 3:43 PM

Post #12 of 22 (671 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
Things seem to be progressing nicely on this front. We have answers to
the questions people had and they look satisfactory to me.

One thing that I think would be valuable is a document that explains
the average dev how to make his/her ebuilds prefix compliant with
links to more details when necessary. I understand that there's the
trivial situations and the less trivial ones. In the latter case it
would be nice to explain why the case isn't trivial and how to fix it.
Using python as an example could be one way to do it. I'm thinking of
something practical that could possibly be patched into devmanual. If
such a document already exists then please just point us to it.

On Fri, Nov 20, 2009 at 2:03 AM, Fabian Groffen <grobian [at] gentoo> wrote:
>> I thought I asked Fabian to work on that at the end of the meeting. In
>> case I didn't then consider this as me officially asking him if he can
>> take care of it. Fabian is this OK with you?
>
> Yes, I agreed coming up with some patch.  I admit I haven't yet even
> looked into it.

Great, thanks. If you can have it ready some time before the meeting
so that all devs can get a chance to review it before the council
members vote on it that will be even better. If you need help don't
hesitate to contact me. I'll try and look for the right people to help
you depending on what you need.

> Technically, Portage trunk already contains the most important bits that
> allow us to continue since yesterday.  The rest is waiting for a formal
> approval of the council, and then it will most probably be me and Zac
> fighting to get the prefix branch merged into trunk.

Good.

> Next to that, it is part of the Prefix team's job to make sure that
> whatever is installed, does not reference the host system when this is
> not absolutely necessary.

Could you give some examples of when it is absolutely necessary?

Denis.


calchan at gentoo

Nov 25, 2009, 4:01 PM

Post #13 of 22 (667 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
It looks like this question is still unanswered:

On Thu, Nov 19, 2009 at 5:26 PM, Denis Dupeyron <calchan [at] gentoo> wrote:
>> How are dynamically linked set*id programs going to work?

Denis.


grobian at gentoo

Nov 26, 2009, 12:53 AM

Post #14 of 22 (664 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
On 25-11-2009 16:43:32 -0700, Denis Dupeyron wrote:
> Things seem to be progressing nicely on this front. We have answers to
> the questions people had and they look satisfactory to me.
>
> One thing that I think would be valuable is a document that explains
> the average dev how to make his/her ebuilds prefix compliant with
> links to more details when necessary. I understand that there's the
> trivial situations and the less trivial ones. In the latter case it
> would be nice to explain why the case isn't trivial and how to fix it.
> Using python as an example could be one way to do it. I'm thinking of
> something practical that could possibly be patched into devmanual. If
> such a document already exists then please just point us to it.

I'm working on a PMS patch at this very moment. I hope to finish a
first version today.

The Python ebuild is a very bad example, IMO. Not that the style there
is horrendous or something, but more that a couple of things are mixed
up in there, which are not directly a Prefix issue, but a derivative of
that -- support for arches which cannot be supported without Prefix.

We can explain some cases though. The *blas-* family comes to mind as
we recently had to alter them for Prefix.

> > Next to that, it is part of the Prefix team's job to make sure that
> > whatever is installed, does not reference the host system when this is
> > not absolutely necessary.
>
> Could you give some examples of when it is absolutely necessary?

Simple example is perl. If you install a script, for instance ekeyword,
then it is important that this script doesn't say '#!/usr/bin/perl' in
its shebang. "/usr/bin/perl" may simply not exist, but more importantly
it is not the perl that Portage has installed and also installed all
required dependencies for. Hence, ekeyword should be installed such
that it references the perl from the offset installation, e.g.
"/home/joe/gentoo/usr/bin/perl".

"/bin/sh" is another nice one. Technically this should work fine, as
almost every system has this shell available, however in practice it
won't work that well, because many people assume that "/bin/sh" is a
POSIX shell, which is not true on every UNIX that exists, as far as I
know. People may also rely on things from the Gentoo environment in
such scripts, which of course are not present in the host provided
"/bin/sh", hence also here we need to call "/home/joe/gentoo/bin/sh"
(for the chosen offset-prefix "/home/joe/gentoo").

Outside scripts, also libraries have this same principle. An
application linking against libexpat should do so against the Prefix
installed version of expat. If the host system provides
"/usr/lib/libexpat.so" and the upstream configure script is hardwired to
look in "/usr/local/lib /usr/lib64 /usr/lib32 /usr/lib /usr/sfw/lib /sw"
it will never find libexpat from "/home/joe/gentoo/usr/lib". In such
case either an --with-expat="${EPREFIX}"/usr option needs to be passed
to configure (if it exists), or if nothing possible configure be hacked
not to use this hardwired list of paths, but instead fall back on the
default, which for a Prefix toolchain includes the offset library search
paths before the host system's. In any case, it must compile and link
against the Prefix installed libexpat, since the one from the host
system usually is very archaic and not under Portage's control.


--
Fabian Groffen
Gentoo on a different level


grobian at gentoo

Nov 26, 2009, 1:02 AM

Post #15 of 22 (664 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
On 25-11-2009 17:01:19 -0700, Denis Dupeyron wrote:
> It looks like this question is still unanswered:
>
> On Thu, Nov 19, 2009 at 5:26 PM, Denis Dupeyron <calchan [at] gentoo> wrote:
> >> How are dynamically linked set*id programs going to work?

Depends on how the host OS/libc handles this :)

If you're root, you can install a program that any user can call and
run as if that user were root. If you're a normal user, you just
install a program that any user can call and run as if that user were
you.

If you refer to GNU glibc ignoring or greatly limiting variables like
LD_LIBRARY_PATH, then there is no problem in that for Prefix, since
these variables are considered harmful anyway. We don't use them.
Prefix uses RUNPATH (RPATH) in ELF objects to have the loader find the
correct libraries at runtime without any tricks. This is for instance
handy when someone calls a binary from a Prefix without having properly
initialised the Prefix environment. While this is not what we advise
people to do, it can work fine for applications like subversion.


--
Fabian Groffen
Gentoo on a different level


1i5t5.duncan at cox

Nov 26, 2009, 2:01 AM

Post #16 of 22 (665 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
Fabian Groffen posted on Thu, 26 Nov 2009 09:53:04 +0100 as excerpted:

>> > Next to that, it is part of the Prefix team's job to make sure that
>> > whatever is installed, does not reference the host system when this
>> > is not absolutely necessary.
>>
>> Could you give some examples of when it is absolutely necessary?
>
> Simple example is perl. If you install a script, for instance ekeyword,
> then it is important that this script doesn't say '#!/usr/bin/perl' in
> its shebang. "/usr/bin/perl" may simply not exist, but more importantly
> it is not the perl that Portage has installed and also installed all
> required dependencies for. Hence, ekeyword should be installed such
> that it references the perl from the offset installation, e.g.
> "/home/joe/gentoo/usr/bin/perl".
>
> "/bin/sh" is another nice one.

At least here, that it would ordinarily be best to reference the prefix
system was taken for granted, so when it's "absolutely necessary" to
reference the host system is the interesting case, and how I parsed the
request. You provided examples of just the opposite, the case I (and
evidently Denis, if I parsed the request correctly) assumed to be normal,
where referencing the prefix is strongly desirable or "absolutely
necessary".

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman


grobian at gentoo

Nov 26, 2009, 2:10 AM

Post #17 of 22 (665 views)
Permalink
Re: Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
On 26-11-2009 10:01:24 +0000, Duncan wrote:
> > required dependencies for. Hence, ekeyword should be installed such
> > that it references the perl from the offset installation, e.g.
> > "/home/joe/gentoo/usr/bin/perl".
> >
> > "/bin/sh" is another nice one.
>
> At least here, that it would ordinarily be best to reference the prefix
> system was taken for granted, so when it's "absolutely necessary" to
> reference the host system is the interesting case, and how I parsed the
> request. You provided examples of just the opposite, the case I (and
> evidently Denis, if I parsed the request correctly) assumed to be normal,
> where referencing the prefix is strongly desirable or "absolutely
> necessary".

I see, thanks for bringing that up. To clarify (I hope this is):

Gentoo Prefix tries to be as much self-sufficient as possible, and hence
applications *must* not reference the host system, unless absolutely
necessary, such as for e.g. /lib/libc.so.

If Gentoo Prefix would not do this, Portage's dependencies would become
useless, and many uncontrollable and unpredictable errors may arise both
at compile time as well as at run time.


--
Fabian Groffen
Gentoo on a different level


1i5t5.duncan at cox

Nov 26, 2009, 2:37 AM

Post #18 of 22 (659 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
Fabian Groffen posted on Thu, 26 Nov 2009 11:10:09 +0100 as excerpted:

> Gentoo Prefix tries to be as much self-sufficient as possible, and hence
> applications *must* not reference the host system, unless absolutely
> necessary, such as for e.g. /lib/libc.so.

Thanks. Host libc /does/ make sense as "absolutely necessary.

Are there any less obvious ones, say of the type that might reach out and
grab an unsuspecting dev trying to make his ebuilds prefix compliant? It
seems to me that enumerating all (or all non-corner) cases where
referencing the host is desired/mandatory, with a blanket rule saying
reference prefix unless it's a known exception, should be by /far/ the
easiest alternative, here.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman


grobian at gentoo

Nov 26, 2009, 2:51 AM

Post #19 of 22 (656 views)
Permalink
Re: Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
On 26-11-2009 10:37:10 +0000, Duncan wrote:
> Fabian Groffen posted on Thu, 26 Nov 2009 11:10:09 +0100 as excerpted:
>
> > Gentoo Prefix tries to be as much self-sufficient as possible, and hence
> > applications *must* not reference the host system, unless absolutely
> > necessary, such as for e.g. /lib/libc.so.
>
> Thanks. Host libc /does/ make sense as "absolutely necessary.
>
> Are there any less obvious ones, say of the type that might reach out and

Some that you may find are:
/lib/libm.so
/lib/libsocket.so
/lib/libpthread.so
/lib/libnsl.so

On a side note, we have a question about this in our
prefix-ebuild-quiz[1] (question 5 from the second section).

> grab an unsuspecting dev trying to make his ebuilds prefix compliant? It
> seems to me that enumerating all (or all non-corner) cases where
> referencing the host is desired/mandatory, with a blanket rule saying
> reference prefix unless it's a known exception, should be by /far/ the
> easiest alternative, here.

I think there's unfortunately no simple way to tell what should be in
and what unfortunately has to be out. It depends a lot on the host
system. I feel -- but I can't back this up with hard evidence -- that
it are usually the libs that are not in *DEPEND that can only be
available in the host system. Basically because they usually are part
of the libc, which we assume to be installed.


[1] http://dev.gentoo.org/~grobian/prefix-quiz

--
Fabian Groffen
Gentoo on a different level


1i5t5.duncan at cox

Nov 26, 2009, 4:36 AM

Post #20 of 22 (656 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
Fabian Groffen posted on Thu, 26 Nov 2009 11:51:06 +0100 as excerpted:

>> Are there any less obvious ones
>
> Some that you may find are:
> /lib/libm.so
> /lib/libsocket.so
> /lib/libpthread.so
> /lib/libnsl.so
>
> On a side note, we have a question about this in our
> prefix-ebuild-quiz[1] (question 5 from the second section).

> I think there's unfortunately no simple way to tell what should be in
> and what unfortunately has to be out. It depends a lot on the host
> system. I feel -- but I can't back this up with hard evidence -- that
> it are usually the libs that are not in *DEPEND that can only be
> available in the host system. Basically because they usually are part
> of the libc, which we assume to be installed.
>
>
> [1] http://dev.gentoo.org/~grobian/prefix-quiz

Thanks again. That quiz was particularly enlightening (to this gentoo,
not prefix, user) on the type of stuff you think about, as a part of what
prefix /does/.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman


grobian at gentoo

Nov 26, 2009, 5:43 AM

Post #21 of 22 (656 views)
Permalink
Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
On 25-11-2009 16:43:32 -0700, Denis Dupeyron wrote:
> > Yes, I agreed coming up with some patch.  I admit I haven't yet even
> > looked into it.
>
> Great, thanks. If you can have it ready some time before the meeting
> so that all devs can get a chance to review it before the council
> members vote on it that will be even better. If you need help don't
> hesitate to contact me. I'll try and look for the right people to help
> you depending on what you need.

Here is the patch. I tried to cover all places.


--
Fabian Groffen
Gentoo on a different level
Attachments: prefix-r0.patch (14.4 KB)


grobian at gentoo

Nov 26, 2009, 7:26 AM

Post #22 of 22 (659 views)
Permalink
Re: Re: Gentoo Prefix: on EPREFIX, ED and EROOT inside ebuilds [In reply to]
On 26-11-2009 12:36:47 +0000, Duncan wrote:
> > I think there's unfortunately no simple way to tell what should be in
> > and what unfortunately has to be out. It depends a lot on the host
> > system. I feel -- but I can't back this up with hard evidence -- that
> > it are usually the libs that are not in *DEPEND that can only be
> > available in the host system. Basically because they usually are part
> > of the libc, which we assume to be installed.
> >
> >
> > [1] http://dev.gentoo.org/~grobian/prefix-quiz
>
> Thanks again. That quiz was particularly enlightening (to this gentoo,
> not prefix, user) on the type of stuff you think about, as a part of what
> prefix /does/.

The following bug is a classical example of something which just happens
to work by accident, but is broken. It also illustrates what kind of
changes people can expect to have to be made to their ebuilds to get
them to work for Prefix:

http://bugs.gentoo.org/show_bug.cgi?id=293901

Because the ebuild gives a direct pointer to a place in the filesystem,

--with-apr=/usr/bin/apr-1-config

the ebuild needs a change such that it points to the corresponding place
in the offset-prefix installation, e.g. under "${EPREFIX}".

--with-apr="${EPREFIX}"/usr/bin/apr-1-config

When this variable is empty, there's obviously no change to the
behaviour of the ebuild.


--
Fabian Groffen
Gentoo on a different level

Gentoo dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.