Mailing List Archive: Gentoo: Dev

RFC: new feature to disable merging into stray locations

Index | Next | Previous | View Threaded

mgorny at gentoo

Apr 29, 2012, 11:57 PM

Post #1 of 18 (222 views)
Permalink

RFC: new feature to disable merging into stray locations

Hello,

Since lately Gentoo devs force you to replace collision-protect with
protect-owned [1] and sometimes packages just spit out files randomly
on the filesystem due to random errors, I thought it may be a good idea
to provide a new feature limiting the locations where packages can
install.

In order to do that, we should first compose a complete include/exclude
list where packages can install. I'd suggest the following:

+ /bin
+ /boot (but maybe just subdirectories so packages can't overwrite
kernels?)
[.potentially + /dev? but that's useful only when tmpfs isn't mounted]
+ /etc
+ /lib, /lib32, /lib64
+ /opt
+ /sbin
[potentially + /service for ugly daemontools]
+ /usr
+ /var
- /usr/local
- /usr/portage

What are your thoughts on this?

[1]:https://bugs.gentoo.org/show_bug.cgi?id=410691#c4

--
Best regards,
Michał Górny

Attachments:

signature.asc (0.31 KB)

ulm at gentoo

Apr 30, 2012, 1:12 AM

Post #2 of 18 (221 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

>>>>> On Mon, 30 Apr 2012, Micha� G�rny wrote:
> Since lately Gentoo devs force you to replace collision-protect with
> protect-owned [1] and sometimes packages just spit out files randomly
> on the filesystem due to random errors, I thought it may be a good idea
> to provide a new feature limiting the locations where packages can
> install.

If the eclass doesn't work with FEATURES="collision-protect" then it
needs to be fixed.

Ulrich

> [1]:https://bugs.gentoo.org/show_bug.cgi?id=410691#c4

nelchael at gentoo

Apr 30, 2012, 1:23 AM

Post #3 of 18 (224 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

On 30/04/12 10:12, Ulrich Mueller wrote:
>>>>>> On Mon, 30 Apr 2012, Michał Górny wrote:
>> Since lately Gentoo devs force you to replace collision-protect with
>> protect-owned [1] and sometimes packages just spit out files randomly
>> on the filesystem due to random errors, I thought it may be a good idea
>> to provide a new feature limiting the locations where packages can
>> install.
>
> If the eclass doesn't work with FEATURES="collision-protect" then it
> needs to be fixed.

Long story short: older eclass compiled Python byte code in live file system,
new one does it in src_install so .pyo/.pyc gets properly recorded, so the
package *has to overwrite* files that are not owned by anyone (no package owns
them). I've talked to Zac and:

>>>
There's not much else you can do. However, FEATURES="protect-owned" is
enabled by default, and it will work fine if the .pyo and .pyc files are
not owned by anything. I don't know why people use
FEATURES="collision-protect", but at least you can tell them that
they'll still have a reasonable level of protection from
FEATURES="protect-owned" (it protect against file collisions between
packages).
<<<

If you want to help feel free to suggest *how* to fix it, patches are especially
welcomed :)

--
Krzysztof Pawlik <nelchael at gentoo.org> key id: 0xF6A80E46
desktop-misc, java, vim, kernel, python, apache...

Attachments:

signature.asc (0.54 KB)

nelchael at gentoo

Apr 30, 2012, 1:24 AM

Post #4 of 18 (223 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

On 30/04/12 08:57, Michał Górny wrote:
> Hello,
>
> Since lately Gentoo devs force you to replace collision-protect with
> protect-owned [1] and sometimes packages just spit out files randomly
> on the filesystem due to random errors, I thought it may be a good idea
> to provide a new feature limiting the locations where packages can
> install.

You're not forced to do anything.

> In order to do that, we should first compose a complete include/exclude
> list where packages can install. I'd suggest the following:
>
> + /bin
> + /boot (but maybe just subdirectories so packages can't overwrite
> kernels?)
> [.potentially + /dev? but that's useful only when tmpfs isn't mounted]
> + /etc
> + /lib, /lib32, /lib64
> + /opt
> + /sbin
> [potentially + /service for ugly daemontools]
> + /usr
> + /var
> - /usr/local
> - /usr/portage
>
> What are your thoughts on this?
>
> [1]:https://bugs.gentoo.org/show_bug.cgi?id=410691#c4

I think it's feature creep - you can just set INSTALL_MASK="/usr/local
/usr/portage", no need for new features.

PS. I (and few other folks) don't have /usr/portage.

--
Krzysztof Pawlik <nelchael at gentoo.org> key id: 0xF6A80E46
desktop-misc, java, vim, kernel, python, apache...

Attachments:

signature.asc (0.54 KB)

ulm at gentoo

Apr 30, 2012, 1:39 AM

Post #5 of 18 (224 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

>>>>> On Mon, 30 Apr 2012, Krzysztof Pawlik wrote:

> On 30/04/12 10:12, Ulrich Mueller wrote:
>> If the eclass doesn't work with FEATURES="collision-protect" then it
>> needs to be fixed.

> Long story short: older eclass compiled Python byte code in live
> file system, new one does it in src_install so .pyo/.pyc gets
> properly recorded,

That's a good change.

> so the package *has to overwrite* files that are not owned by anyone
> (no package owns them).

The usual approach to this problem was to have some cleanup code in
pkg_setup or pkg_preinst that would remove the old files.

Ulrich

mgorny at gentoo

Apr 30, 2012, 2:28 AM

Post #6 of 18 (214 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

On Mon, 30 Apr 2012 10:23:03 +0200
Krzysztof Pawlik <nelchael [at] gentoo> wrote:

> On 30/04/12 10:12, Ulrich Mueller wrote:
> >>>>>> On Mon, 30 Apr 2012, Michał Górny wrote:
> >> Since lately Gentoo devs force you to replace collision-protect
> >> with protect-owned [1] and sometimes packages just spit out files
> >> randomly on the filesystem due to random errors, I thought it may
> >> be a good idea to provide a new feature limiting the locations
> >> where packages can install.
> >
> > If the eclass doesn't work with FEATURES="collision-protect" then it
> > needs to be fixed.
>
> Long story short: older eclass compiled Python byte code in live file
> system, new one does it in src_install so .pyo/.pyc gets properly
> recorded, so the package *has to overwrite* files that are not owned
> by anyone (no package owns them). I've talked to Zac and:
>
> >>>
> There's not much else you can do. However, FEATURES="protect-owned" is
> enabled by default, and it will work fine if the .pyo and .pyc files
> are not owned by anything. I don't know why people use
> FEATURES="collision-protect", but at least you can tell them that
> they'll still have a reasonable level of protection from
> FEATURES="protect-owned" (it protect against file collisions between
> packages).
> <<<
>
> If you want to help feel free to suggest *how* to fix it, patches are
> especially welcomed :)

Just make it warn or something like that. Or even fail but only when
such files exist. There is no point in forcing me to disable
collision-protect when the ebuilds were migrated already.

--
Best regards,
Michał Górny

Attachments:

signature.asc (0.31 KB)

mgorny at gentoo

Apr 30, 2012, 2:30 AM

Post #7 of 18 (214 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

On Mon, 30 Apr 2012 10:24:58 +0200
Krzysztof Pawlik <nelchael [at] gentoo> wrote:

> On 30/04/12 08:57, Michał Górny wrote:
> > Hello,
> >
> > Since lately Gentoo devs force you to replace collision-protect with
> > protect-owned [1] and sometimes packages just spit out files
> > randomly on the filesystem due to random errors, I thought it may
> > be a good idea to provide a new feature limiting the locations
> > where packages can install.
>
> You're not forced to do anything.

I am not? How come random ebuilds die in pkg_pretend() for me then?

>
> > In order to do that, we should first compose a complete
> > include/exclude list where packages can install. I'd suggest the
> > following:
> >
> > + /bin
> > + /boot (but maybe just subdirectories so packages can't overwrite
> > kernels?)
> > [.potentially + /dev? but that's useful only when tmpfs isn't
> > mounted]
> > + /etc
> > + /lib, /lib32, /lib64
> > + /opt
> > + /sbin
> > [potentially + /service for ugly daemontools]
> > + /usr
> > + /var
> > - /usr/local
> > - /usr/portage
> >
> > What are your thoughts on this?
> >
> > [1]:https://bugs.gentoo.org/show_bug.cgi?id=410691#c4
>
> I think it's feature creep - you can just set INSTALL_MASK="/usr/local
> /usr/portage", no need for new features.
>
> PS. I (and few other folks) don't have /usr/portage.

INSTALL_MASK won't make committing such an ebuild a fatal error. It
will just hide problems.

--
Best regards,
Michał Górny

Attachments:

signature.asc (0.31 KB)

nelchael at gentoo

Apr 30, 2012, 2:37 AM

Post #8 of 18 (212 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

On 30/04/12 11:28, Michał Górny wrote:
> On Mon, 30 Apr 2012 10:23:03 +0200
> Krzysztof Pawlik <nelchael [at] gentoo> wrote:
>
>> On 30/04/12 10:12, Ulrich Mueller wrote:
>>>>>>>> On Mon, 30 Apr 2012, Michał Górny wrote:
>>>> Since lately Gentoo devs force you to replace collision-protect
>>>> with protect-owned [1] and sometimes packages just spit out files
>>>> randomly on the filesystem due to random errors, I thought it may
>>>> be a good idea to provide a new feature limiting the locations
>>>> where packages can install.
>>>
>>> If the eclass doesn't work with FEATURES="collision-protect" then it
>>> needs to be fixed.
>>
>> Long story short: older eclass compiled Python byte code in live file
>> system, new one does it in src_install so .pyo/.pyc gets properly
>> recorded, so the package *has to overwrite* files that are not owned
>> by anyone (no package owns them). I've talked to Zac and:
>>
>>>>>
>> There's not much else you can do. However, FEATURES="protect-owned" is
>> enabled by default, and it will work fine if the .pyo and .pyc files
>> are not owned by anything. I don't know why people use
>> FEATURES="collision-protect", but at least you can tell them that
>> they'll still have a reasonable level of protection from
>> FEATURES="protect-owned" (it protect against file collisions between
>> packages).
>> <<<
>>
>> If you want to help feel free to suggest *how* to fix it, patches are
>> especially welcomed :)
>
> Just make it warn or something like that. Or even fail but only when
> such files exist. There is no point in forcing me to disable
> collision-protect when the ebuilds were migrated already.

Does this work for you:

$ cvs diff python-distutils-ng.eclass
Index: python-distutils-ng.eclass
===================================================================
RCS file: /var/cvsroot/gentoo-x86/eclass/python-distutils-ng.eclass,v
retrieving revision 1.12
diff -u -r1.12 python-distutils-ng.eclass
--- python-distutils-ng.eclass 30 Apr 2012 08:25:31 -0000 1.12
+++ python-distutils-ng.eclass 30 Apr 2012 09:36:48 -0000
@@ -304,11 +304,10 @@
# Phase function: pkg_pretend
python-distutils-ng_pkg_pretend() {
if has "collision-protect" ${FEATURES}; then
- eerror "Due to previous eclass compiling Python files outside of
src_install"
- eerror "(and not recording resulting .pyc and .pyo files as
owned by any package)"
- eerror "merging this package with \"collision-protect\" in
FEATURES will result"
- eerror "in an error, please switch to using \"protect-owned\"
instead."
- die "\"collision-protect\" in FEATURES detected"
+ ewarn "Due to previous eclass compiling Python files outside of
src_install"
+ ewarn "(and not recording resulting .pyc and .pyo files as owned
by any package)"
+ ewarn "merging this package with \"collision-protect\" in
FEATURES may result"
+ ewarn "in an error, please switch to using \"protect-owned\"
instead."
fi
}

(it's s/eerror/ewarn/ & die removal).

I'm still wondering how many people will read this warning when they hit
collision error.

--
Krzysztof Pawlik <nelchael at gentoo.org> key id: 0xF6A80E46
desktop-misc, java, vim, kernel, python, apache...

Attachments:

signature.asc (0.54 KB)

nelchael at gentoo

Apr 30, 2012, 2:40 AM

Post #9 of 18 (211 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

On 30/04/12 11:30, Michał Górny wrote:
> On Mon, 30 Apr 2012 10:24:58 +0200
> Krzysztof Pawlik <nelchael [at] gentoo> wrote:
>
>> On 30/04/12 08:57, Michał Górny wrote:
>>> Hello,
>>>
>>> Since lately Gentoo devs force you to replace collision-protect with
>>> protect-owned [1] and sometimes packages just spit out files
>>> randomly on the filesystem due to random errors, I thought it may
>>> be a good idea to provide a new feature limiting the locations
>>> where packages can install.
>>
>> You're not forced to do anything.
>
> I am not? How come random ebuilds die in pkg_pretend() for me then?
>
>>
>>> In order to do that, we should first compose a complete
>>> include/exclude list where packages can install. I'd suggest the
>>> following:
>>>
>>> + /bin
>>> + /boot (but maybe just subdirectories so packages can't overwrite
>>> kernels?)
>>> [.potentially + /dev? but that's useful only when tmpfs isn't
>>> mounted]
>>> + /etc
>>> + /lib, /lib32, /lib64
>>> + /opt
>>> + /sbin
>>> [potentially + /service for ugly daemontools]
>>> + /usr
>>> + /var
>>> - /usr/local
>>> - /usr/portage
>>>
>>> What are your thoughts on this?
>>>
>>> [1]:https://bugs.gentoo.org/show_bug.cgi?id=410691#c4
>>
>> I think it's feature creep - you can just set INSTALL_MASK="/usr/local
>> /usr/portage", no need for new features.
>>
>> PS. I (and few other folks) don't have /usr/portage.
>
> INSTALL_MASK won't make committing such an ebuild a fatal error. It
> will just hide problems.

How would your proposed feature prevent ebuilds that install something to
/usr/local from being committed?

Other valid case are private repos - I have a few ebuilds that install stuff to
/<some_random_dir> (proprietary stuff).

--
Krzysztof Pawlik <nelchael at gentoo.org> key id: 0xF6A80E46
desktop-misc, java, vim, kernel, python, apache...

Attachments:

signature.asc (0.54 KB)

nelchael at gentoo

Apr 30, 2012, 2:42 AM

Post #10 of 18 (215 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

On 30/04/12 10:39, Ulrich Mueller wrote:
>>>>>> On Mon, 30 Apr 2012, Krzysztof Pawlik wrote:
>
>> On 30/04/12 10:12, Ulrich Mueller wrote:
>>> If the eclass doesn't work with FEATURES="collision-protect" then it
>>> needs to be fixed.
>
>> Long story short: older eclass compiled Python byte code in live
>> file system, new one does it in src_install so .pyo/.pyc gets
>> properly recorded,
>
> That's a good change.

:)

>> so the package *has to overwrite* files that are not owned by anyone
>> (no package owns them).
>
> The usual approach to this problem was to have some cleanup code in
> pkg_setup or pkg_preinst that would remove the old files.

Interesting idea - I see two problems:
- how I would get the list of files for already installed version (the file
list can (and will) differ between versions)
- PM has a manifest file that lists all installed file, it can check against
it, not against live file system, so just removing the files may not be enough
(just my speculations - I'm not sure of this)

--
Krzysztof Pawlik <nelchael at gentoo.org> key id: 0xF6A80E46
desktop-misc, java, vim, kernel, python, apache...

Attachments:

signature.asc (0.54 KB)

mgorny at gentoo

Apr 30, 2012, 2:43 AM

Post #11 of 18 (214 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

On Mon, 30 Apr 2012 11:37:53 +0200
Krzysztof Pawlik <nelchael [at] gentoo> wrote:

> On 30/04/12 11:28, Michał Górny wrote:
> > On Mon, 30 Apr 2012 10:23:03 +0200
> > Krzysztof Pawlik <nelchael [at] gentoo> wrote:
> >
> >> On 30/04/12 10:12, Ulrich Mueller wrote:
> >>>>>>>> On Mon, 30 Apr 2012, Michał Górny wrote:
> >>>> Since lately Gentoo devs force you to replace collision-protect
> >>>> with protect-owned [1] and sometimes packages just spit out files
> >>>> randomly on the filesystem due to random errors, I thought it may
> >>>> be a good idea to provide a new feature limiting the locations
> >>>> where packages can install.
> >>>
> >>> If the eclass doesn't work with FEATURES="collision-protect" then
> >>> it needs to be fixed.
> >>
> >> Long story short: older eclass compiled Python byte code in live
> >> file system, new one does it in src_install so .pyo/.pyc gets
> >> properly recorded, so the package *has to overwrite* files that
> >> are not owned by anyone (no package owns them). I've talked to Zac
> >> and:
> >>
> >>>>>
> >> There's not much else you can do. However,
> >> FEATURES="protect-owned" is enabled by default, and it will work
> >> fine if the .pyo and .pyc files are not owned by anything. I don't
> >> know why people use FEATURES="collision-protect", but at least you
> >> can tell them that they'll still have a reasonable level of
> >> protection from FEATURES="protect-owned" (it protect against file
> >> collisions between packages).
> >> <<<
> >>
> >> If you want to help feel free to suggest *how* to fix it, patches
> >> are especially welcomed :)
> >
> > Just make it warn or something like that. Or even fail but only when
> > such files exist. There is no point in forcing me to disable
> > collision-protect when the ebuilds were migrated already.
>
> Does this work for you:
>
> $ cvs diff python-distutils-ng.eclass
> Index: python-distutils-ng.eclass
> ===================================================================
> RCS file: /var/cvsroot/gentoo-x86/eclass/python-distutils-ng.eclass,v
> retrieving revision 1.12
> diff -u -r1.12 python-distutils-ng.eclass
> --- python-distutils-ng.eclass 30 Apr 2012 08:25:31 -0000 1.12
> +++ python-distutils-ng.eclass 30 Apr 2012 09:36:48 -0000
> @@ -304,11 +304,10 @@
> # Phase function: pkg_pretend
> python-distutils-ng_pkg_pretend() {
> if has "collision-protect" ${FEATURES}; then
> - eerror "Due to previous eclass compiling Python files
> outside of src_install"
> - eerror "(and not recording resulting .pyc and .pyo
> files as owned by any package)"
> - eerror "merging this package with
> \"collision-protect\" in FEATURES will result"
> - eerror "in an error, please switch to using
> \"protect-owned\" instead."
> - die "\"collision-protect\" in FEATURES detected"
> + ewarn "Due to previous eclass compiling Python files
> outside of src_install"
> + ewarn "(and not recording resulting .pyc and .pyo
> files as owned by any package)"
> + ewarn "merging this package with
> \"collision-protect\" in FEATURES may result"
> + ewarn "in an error, please switch to using
> \"protect-owned\" instead."
> fi
> }
>
> (it's s/eerror/ewarn/ & die removal).

Yes, it will let me use the eclass at all.

> I'm still wondering how many people will read this warning when they
> hit collision error.

You can make it fatal if you detect collisions yourself but that's
repeating a lot of work.

--
Best regards,
Michał Górny

Attachments:

signature.asc (0.31 KB)

mgorny at gentoo

Apr 30, 2012, 2:47 AM

Post #12 of 18 (214 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

On Mon, 30 Apr 2012 11:40:26 +0200
Krzysztof Pawlik <nelchael [at] gentoo> wrote:

> On 30/04/12 11:30, Michał Górny wrote:
> > On Mon, 30 Apr 2012 10:24:58 +0200
> > Krzysztof Pawlik <nelchael [at] gentoo> wrote:
> >
> >> On 30/04/12 08:57, Michał Górny wrote:
> >>> Hello,
> >>>
> >>> Since lately Gentoo devs force you to replace collision-protect
> >>> with protect-owned [1] and sometimes packages just spit out files
> >>> randomly on the filesystem due to random errors, I thought it may
> >>> be a good idea to provide a new feature limiting the locations
> >>> where packages can install.
> >>
> >> You're not forced to do anything.
> >
> > I am not? How come random ebuilds die in pkg_pretend() for me then?
> >
> >>
> >>> In order to do that, we should first compose a complete
> >>> include/exclude list where packages can install. I'd suggest the
> >>> following:
> >>>
> >>> + /bin
> >>> + /boot (but maybe just subdirectories so packages can't overwrite
> >>> kernels?)
> >>> [.potentially + /dev? but that's useful only when tmpfs isn't
> >>> mounted]
> >>> + /etc
> >>> + /lib, /lib32, /lib64
> >>> + /opt
> >>> + /sbin
> >>> [potentially + /service for ugly daemontools]
> >>> + /usr
> >>> + /var
> >>> - /usr/local
> >>> - /usr/portage
> >>>
> >>> What are your thoughts on this?
> >>>
> >>> [1]:https://bugs.gentoo.org/show_bug.cgi?id=410691#c4
> >>
> >> I think it's feature creep - you can just set
> >> INSTALL_MASK="/usr/local /usr/portage", no need for new features.
> >>
> >> PS. I (and few other folks) don't have /usr/portage.
> >
> > INSTALL_MASK won't make committing such an ebuild a fatal error. It
> > will just hide problems.
>
> How would your proposed feature prevent ebuilds that install
> something to /usr/local from being committed?

If we enable this by default, devs will get errors for that.

> Other valid case are private repos - I have a few ebuilds that
> install stuff to /<some_random_dir> (proprietary stuff).

Feel free to disable it. That's what FEATURES are for.

--
Best regards,
Michał Górny

Attachments:

signature.asc (0.31 KB)

mgorny at gentoo

Apr 30, 2012, 2:50 AM

Post #13 of 18 (212 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

On Mon, 30 Apr 2012 11:42:56 +0200
Krzysztof Pawlik <nelchael [at] gentoo> wrote:

> >> so the package *has to overwrite* files that are not owned by
> >> anyone (no package owns them).
> >
> > The usual approach to this problem was to have some cleanup code in
> > pkg_setup or pkg_preinst that would remove the old files.
>
> Interesting idea - I see two problems:
> - how I would get the list of files for already installed version
> (the file list can (and will) differ between versions)

As PMS doesn't give us any better tools, you can just look at ${ROOT}
in pkg_preinst().

> - PM has a manifest file that lists all installed file, it can check
> against it, not against live file system, so just removing the files
> may not be enough (just my speculations - I'm not sure of this)

It is enough in case of Portage. Especially that those files aren't
listed in those manifests.

--
Best regards,
Michał Górny

Attachments:

signature.asc (0.31 KB)

nelchael at gentoo

Apr 30, 2012, 3:00 AM

Post #14 of 18 (212 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

On 30/04/12 11:47, Michał Górny wrote:
> On Mon, 30 Apr 2012 11:40:26 +0200
> Krzysztof Pawlik <nelchael [at] gentoo> wrote:
>
>> On 30/04/12 11:30, Michał Górny wrote:
>>> On Mon, 30 Apr 2012 10:24:58 +0200
>>> Krzysztof Pawlik <nelchael [at] gentoo> wrote:
>>>
>>>> On 30/04/12 08:57, Michał Górny wrote:
>>>>> Hello,
>>>>>
>>>>> Since lately Gentoo devs force you to replace collision-protect
>>>>> with protect-owned [1] and sometimes packages just spit out files
>>>>> randomly on the filesystem due to random errors, I thought it may
>>>>> be a good idea to provide a new feature limiting the locations
>>>>> where packages can install.
>>>>
>>>> You're not forced to do anything.
>>>
>>> I am not? How come random ebuilds die in pkg_pretend() for me then?
>>>
>>>>
>>>>> In order to do that, we should first compose a complete
>>>>> include/exclude list where packages can install. I'd suggest the
>>>>> following:
>>>>>
>>>>> + /bin
>>>>> + /boot (but maybe just subdirectories so packages can't overwrite
>>>>> kernels?)
>>>>> [.potentially + /dev? but that's useful only when tmpfs isn't
>>>>> mounted]
>>>>> + /etc
>>>>> + /lib, /lib32, /lib64
>>>>> + /opt
>>>>> + /sbin
>>>>> [potentially + /service for ugly daemontools]
>>>>> + /usr
>>>>> + /var
>>>>> - /usr/local
>>>>> - /usr/portage
>>>>>
>>>>> What are your thoughts on this?
>>>>>
>>>>> [1]:https://bugs.gentoo.org/show_bug.cgi?id=410691#c4
>>>>
>>>> I think it's feature creep - you can just set
>>>> INSTALL_MASK="/usr/local /usr/portage", no need for new features.
>>>>
>>>> PS. I (and few other folks) don't have /usr/portage.
>>>
>>> INSTALL_MASK won't make committing such an ebuild a fatal error. It
>>> will just hide problems.
>>
>> How would your proposed feature prevent ebuilds that install
>> something to /usr/local from being committed?
>
> If we enable this by default, devs will get errors for that.

Could it be added instead to repoman? I'm testing all ebuilds before committing
like this:

$ ebuild xhtml2pdf-0.0.3.ebuild manifest clean install

repoman could look at build.log or ${D} and warn if it can't do those checks.

>> Other valid case are private repos - I have a few ebuilds that
>> install stuff to /<some_random_dir> (proprietary stuff).
>
> Feel free to disable it. That's what FEATURES are for.

Yes, I could.

--
Krzysztof Pawlik <nelchael at gentoo.org> key id: 0xF6A80E46
desktop-misc, java, vim, kernel, python, apache...

Attachments:

signature.asc (0.54 KB)

tommy at gentoo

Apr 30, 2012, 6:21 AM

Post #15 of 18 (205 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

Krzysztof Pawlik schrieb:
> On 30/04/12 10:39, Ulrich Mueller wrote:
>>>>>>> On Mon, 30 Apr 2012, Krzysztof Pawlik wrote:
>>
>>> On 30/04/12 10:12, Ulrich Mueller wrote:
>>>> If the eclass doesn't work with FEATURES="collision-protect" then it
>>>> needs to be fixed.
>>
>>> Long story short: older eclass compiled Python byte code in live
>>> file system, new one does it in src_install so .pyo/.pyc gets
>>> properly recorded,
>>
>> That's a good change.
>
> :)
>
>>> so the package *has to overwrite* files that are not owned by anyone
>>> (no package owns them).
>>
>> The usual approach to this problem was to have some cleanup code in
>> pkg_setup or pkg_preinst that would remove the old files.
>
> Interesting idea - I see two problems:
> - how I would get the list of files for already installed version (the file
> list can (and will) differ between versions)

Simple workaround in pkg_preinst (just example code, nothing to copy/paste):

if has_version < first_good_version ; then
for i in $(find ${D} -name *.{pyc,pyo}) ; do
[[ -e ${i/${D}/${ROOT}/} ]] && rm ${i/${D}/${ROOT}/}
done
fi

Alternatively, you could take all .py files of the installed versions
and blindly remove the pyo/pyc files for them in pkg_preinst (this
should also prevent leaving dead files around).

--

Thomas Sachau
Gentoo Linux Developer

Attachments:

signature.asc (0.37 KB)

ulm at gentoo

Apr 30, 2012, 6:36 AM

Post #16 of 18 (209 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

>>>>> On Mon, 30 Apr 2012, Thomas Sachau wrote:

> Krzysztof Pawlik schrieb:
>> On 30/04/12 10:39, Ulrich Mueller wrote:
>>>>> If the eclass doesn't work with FEATURES="collision-protect"
>>>>> then it needs to be fixed.
>>>
>>>> Long story short: older eclass compiled Python byte code in live
>>>> file system, new one does it in src_install so .pyo/.pyc gets
>>>> properly recorded, so the package *has to overwrite* files that
>>>> are not owned by anyone (no package owns them).
>>>
>>> The usual approach to this problem was to have some cleanup code
>>> in pkg_setup or pkg_preinst that would remove the old files.
>>
>> Interesting idea - I see two problems:
>> - how I would get the list of files for already installed version
>> (the file list can (and will) differ between versions)

> Simple workaround in pkg_preinst (just example code, nothing to
> copy/paste):

> if has_version < first_good_version ; then
> for i in $(find ${D} -name *.{pyc,pyo}) ; do
> [[ -e ${i/${D}/${ROOT}/} ]] && rm ${i/${D}/${ROOT}/}
> done
> fi

This wouldn't work, because collision protection runs _before_
pkg_preinst.

However, one could imagine a (somewhat dirty) workaround:
- At the end of src_install, rename all *.pyc and *.pyo in ${D} to
names that don't collide with the ones in ${ROOT}, so that collision
protection won't be triggered.
- At the beginning of pkg_preinst, rename them back to their original
names.
- Run above code.

> Alternatively, you could take all .py files of the installed
> versions and blindly remove the pyo/pyc files for them in
> pkg_preinst (this should also prevent leaving dead files around).

s/preinst/setup/ otherwise the same argument as above would apply.

Ulrich

ferringb at gmail

May 1, 2012, 5:36 AM

Post #17 of 18 (200 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

On Mon, Apr 30, 2012 at 03:36:17PM +0200, Ulrich Mueller wrote:
> >>>>> On Mon, 30 Apr 2012, Thomas Sachau wrote:
>
> > Krzysztof Pawlik schrieb:
> >> On 30/04/12 10:39, Ulrich Mueller wrote:
> >>>>> If the eclass doesn't work with FEATURES="collision-protect"
> >>>>> then it needs to be fixed.
> >>>
> >>>> Long story short: older eclass compiled Python byte code in live
> >>>> file system, new one does it in src_install so .pyo/.pyc gets
> >>>> properly recorded, so the package *has to overwrite* files that
> >>>> are not owned by anyone (no package owns them).
> >>>
> >>> The usual approach to this problem was to have some cleanup code
> >>> in pkg_setup or pkg_preinst that would remove the old files.
> >>
> >> Interesting idea - I see two problems:
> >> - how I would get the list of files for already installed version
> >> (the file list can (and will) differ between versions)
>
> > Simple workaround in pkg_preinst (just example code, nothing to
> > copy/paste):
>
> > if has_version < first_good_version ; then
> > for i in $(find ${D} -name *.{pyc,pyo}) ; do
> > [[ -e ${i/${D}/${ROOT}/} ]] && rm ${i/${D}/${ROOT}/}
> > done
> > fi
>
> This wouldn't work, because collision protection runs _before_
> pkg_preinst.
>
> However, one could imagine a (somewhat dirty) workaround:
> - At the end of src_install, rename all *.pyc and *.pyo in ${D} to
> names that don't collide with the ones in ${ROOT}, so that collision
> protection won't be triggered.
> - At the beginning of pkg_preinst, rename them back to their original
> names.
> - Run above code.
>
> > Alternatively, you could take all .py files of the installed
> > versions and blindly remove the pyo/pyc files for them in
> > pkg_preinst (this should also prevent leaving dead files around).
>
> s/preinst/setup/ otherwise the same argument as above would apply.

The checks above can misbehave.

Pretty sure I've already suggested this route for this specific issue,
but QA vars in the ebuild that portage can use for hints is a sane,
non-cluster-fucky way to deal with this. Something along the lines of
QA_COLLISION_PROTECT_IGNORE='*.pyc'

Still has the potential for things going boom, but warnings can be
setup for when collision-protect is on, and the portage version
doesn't look new enough (those being details; point is the general
approach).
~harring

nelchael at gentoo

May 3, 2012, 2:07 AM

Post #18 of 18 (198 views)
Permalink

Re: RFC: new feature to disable merging into stray locations [In reply to]

On 01/05/12 14:36, Brian Harring wrote:
> On Mon, Apr 30, 2012 at 03:36:17PM +0200, Ulrich Mueller wrote:
>>>>>>> On Mon, 30 Apr 2012, Thomas Sachau wrote:
>>
>>> Krzysztof Pawlik schrieb:
>>>> On 30/04/12 10:39, Ulrich Mueller wrote:
>>>>>>> If the eclass doesn't work with FEATURES="collision-protect"
>>>>>>> then it needs to be fixed.
>>>>>
>>>>>> Long story short: older eclass compiled Python byte code in live
>>>>>> file system, new one does it in src_install so .pyo/.pyc gets
>>>>>> properly recorded, so the package *has to overwrite* files that
>>>>>> are not owned by anyone (no package owns them).
>>>>>
>>>>> The usual approach to this problem was to have some cleanup code
>>>>> in pkg_setup or pkg_preinst that would remove the old files.
>>>>
>>>> Interesting idea - I see two problems:
>>>> - how I would get the list of files for already installed version
>>>> (the file list can (and will) differ between versions)
>>
>>> Simple workaround in pkg_preinst (just example code, nothing to
>>> copy/paste):
>>
>>> if has_version < first_good_version ; then
>>> for i in $(find ${D} -name *.{pyc,pyo}) ; do
>>> [[ -e ${i/${D}/${ROOT}/} ]] && rm ${i/${D}/${ROOT}/}
>>> done
>>> fi
>>
>> This wouldn't work, because collision protection runs _before_
>> pkg_preinst.
>>
>> However, one could imagine a (somewhat dirty) workaround:
>> - At the end of src_install, rename all *.pyc and *.pyo in ${D} to
>> names that don't collide with the ones in ${ROOT}, so that collision
>> protection won't be triggered.
>> - At the beginning of pkg_preinst, rename them back to their original
>> names.
>> - Run above code.
>>
>>> Alternatively, you could take all .py files of the installed
>>> versions and blindly remove the pyo/pyc files for them in
>>> pkg_preinst (this should also prevent leaving dead files around).
>>
>> s/preinst/setup/ otherwise the same argument as above would apply.
>
> The checks above can misbehave.
>
> Pretty sure I've already suggested this route for this specific issue,
> but QA vars in the ebuild that portage can use for hints is a sane,
> non-cluster-fucky way to deal with this. Something along the lines of
> QA_COLLISION_PROTECT_IGNORE='*.pyc'
>
> Still has the potential for things going boom, but warnings can be
> setup for when collision-protect is on, and the portage version
> doesn't look new enough (those being details; point is the general
> approach).

I've sent an e-mail to gentoo-portage-dev [at] g with a patch that adds a new
variable that causes collision-protect/protect-owned to ignore specific file
patterns for unowned files. Behaviour is similar to already existing
COLLISION_IGNORE.

--
Krzysztof Pawlik <nelchael at gentoo.org> key id: 0xF6A80E46
desktop-misc, java, vim, kernel, python, apache...

Attachments:

signature.asc (0.54 KB)

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads