Mailing List Archive: Gentoo: Dev

mtime preservation

1 2

View All

Index | Next | Previous | View Threaded

calchan at gentoo

Nov 23, 2009, 10:49 AM

Post #1 of 37 (807 views)
Permalink

mtime preservation

I'm going to try and sum up the situation of the thread started in
[1]. Feel free to correct me or add to the summary in replies.

There seems to be two main ideas. I have removed the authors' name in
quotes below in order to make sure we talk about the ideas and not who
proposed them.

1- All packages are treated equally. Some files have their mtime
preserved, some don't. We need to agree on what files have their mtime
preserved and at what phase the mtime is frozen. What we seemed to
have agreed on is that the wording in PMS needs to be unambiguous
enough that it doesn't defeat the purpose. Here's a first proposition:

"The package manager must preserve modification times of regular files.
This includes files being compressed before merging. Exceptions to
this are:

(Now we need to enumerate the exceptions:)

* files newly created by the package manager,
(This will cover splitdebug, for example. (And please don't tell me
that the wording is flawed because the PM could save a file's contents
in some buffer, then delete the file and create it newly. This would
be as unreasonable as the rot-13 example.) )

* binary object files being stripped of symbols.

(Anything else missing from above list?)"

2- Here's the second idea, shamelessly pasted (note that it says EAPI4
below instead of EAPI3 but this is irrelevant to the idea):

"Thus, I would let EAPI 4 ebuilds call dopreservemtimes (with an API
similar to docompress) in both src_install and pkg_preinst. Doing so
would instruct the package manager that it must preserve mtimes
(including subsecond, if supported on the filesystem) for a particular
set of paths, even if doing so means no stripping etc. All other mtimes
may be rewritten as the package manager sees fit, and from EAPI 4
onwards must be rewritten at merge time for anything predating the
start of the build."

In both cases nanosecond resolution may be required and is a problem
due to python. The following workaround can be used until the
nanosecond issue is fixed in python:

"Alternatively, we could simply make portage spawn the mv binary
whenever rename fails (it fails when the source and destination are
on different devices). Although it's relatively slow, it should
solve the problem."

My intention is to ask the council to vote on which method is
preferable in two weeks. I will also ask the council on whether we
still want mtime preservation for EAPI3 or if we now think it's better
to push it to EAPI4. Please discuss.

Denis.

[1] http://archives.gentoo.org/gentoo-council/msg_becc15a2882cc7e4f1079b2f9f4dcaad.xml

zmedico at gentoo

Nov 23, 2009, 12:39 PM

Post #2 of 37 (786 views)
Permalink

Re: mtime preservation [In reply to]

Denis Dupeyron wrote:
> 1- All packages are treated equally. Some files have their mtime
> preserved, some don't. We need to agree on what files have their mtime
> preserved and at what phase the mtime is frozen.

I'd vote for method 1.

> My intention is to ask the council to vote on which method is
> preferable in two weeks. I will also ask the council on whether we
> still want mtime preservation for EAPI3 or if we now think it's better
> to push it to EAPI4. Please discuss.

You can probably do method 1 retroactively for all EAPIs, since the
few existing packages which require mtime preservation are
presumably broken already anyway (any damage is already done), and
packages which don't require mtime preservation are not hurt by it.
--
Thanks,
Zac

ferringb at gmail

Nov 23, 2009, 3:19 PM

Post #3 of 37 (787 views)
Permalink

Re: mtime preservation [In reply to]

On Mon, Nov 23, 2009 at 11:49:25AM -0700, Denis Dupeyron wrote:
> 2- Here's the second idea, shamelessly pasted (note that it says EAPI4
> below instead of EAPI3 but this is irrelevant to the idea):
>
> "Thus, I would let EAPI 4 ebuilds call dopreservemtimes (with an API
> similar to docompress) in both src_install and pkg_preinst. Doing so
> would instruct the package manager that it must preserve mtimes
> (including subsecond, if supported on the filesystem) for a particular
> set of paths, even if doing so means no stripping etc. All other mtimes
> may be rewritten as the package manager sees fit, and from EAPI 4
> onwards must be rewritten at merge time for anything predating the
> start of the build."

Someone mind explaining to me why we're making mtime preservation so
nasty? Having to enumerate every pathway that requires mtime
preservation is pretty arduous for the ebuild dev, meaning it's
unlikely they'll get it right, leading to bugs.

The thing I'm not understanding here is that pkgcore since day one has
preserved mtime- I've yet to see any complaints about that nor any
issues caused by it. Portage shifted over a year or two back, same
thing, haven't heard complaints.

The thing that's bugging me about this whole discussion is that we're
spending more time focusing on stupid things the manager could do to
(basically intentionally) screw up the mtime preservation.

I know it won't fly, but realistically stating "the package manager
must preserve mtime- if there are instances where it does not (due to
some feature, perhaps splitdebug or some form of compression) it is
the package managers responsibility to ensure this does not break the
ebuilds resultant merge/runtime invocation".

Via such wording an exemption is created and it's made clear it's the
managers responsibility to keep things playing nice... if the manager
can't do that, then the feature/functionality that is changing the
mtime (resulting in the pkg on disk failing) must be fixed or
disabled.

The nice thing about this wording is that it basically matches what
the case is now, and what has worked for quite a few years.

> In both cases nanosecond resolution may be required and is a problem
> due to python. The following workaround can be used until the
> nanosecond issue is fixed in python:

It'd be nice if someone enumerated merge scenarios where nanosecond
resolution is actually required. Seems like a white elephant to me,
especially in combination w/ the fact that the target fs may not
support nanosecond.

Basically, if it's required the manager support nanosecond resolution
for merging, ebuilds must be able to rely on that- end result, any
merging to FS's that do not support nanosecond (this includes the
intermediate ${D}) are no longer supported. Unless I'm missing
something, iso-9660 doesn't look to support nanosecond resolution.
Beyond that, does cramfs/squashfs? If not, taking this to the
logical conclusion the livecds aren't technically supported as a merge
environment.

Essentially If we're going to require the manager support NS
resolution w/out requiring all FS's involved support NS level
resolution, in reality ebuilds can't rely on NS level resolution.

Thus drop the requirement since we gain nothing from it- if it's not a
gurantee ebuilds can rely on then they should be writing to the lowest
common denominator- second level resolution. That may sound heinous,
but it's also being realistic.

> "Alternatively, we could simply make portage spawn the mv binary
> whenever rename fails (it fails when the source and destination are
> on different devices). Although it's relatively slow, it should
> solve the problem."

Yeah... no. Slowing down the main manager for a thereotical edge case
doesn't seem particularly useful to me ;)

~harring

1i5t5.duncan at cox

Nov 24, 2009, 6:26 AM

Post #4 of 37 (778 views)
Permalink

Re: mtime preservation [In reply to]

Brian Harring posted on Mon, 23 Nov 2009 15:19:00 -0800 as excerpted:

>> "Alternatively, we could simply make portage spawn the mv binary
>> whenever rename fails (it fails when the source and destination are on
>> different devices). Although it's relatively slow, it should solve the
>> problem."
>
> Yeah... no. Slowing down the main manager for a thereotical edge case
> doesn't seem particularly useful to me ;)

I say go for second resolution now, basically using the #1 proposal, and
apply it to all eapis for the reasons zac outlined, but with a fixed
effective date say 90 or 180 days out from the council resolution to give
PMs time to come into compliance. To try for nanosecond resolution now
is simply letting the complex perfect be the enemy of the simple good,
when what we have at present is the no-good.

Then make a donsmtimes like the #2 proposal, that individual ebuilds can
call for specific paths if it's really necessary. It's more work for the
ebuilder, but that's in relative measure to the slowdown it'll cause on
two out of three PMs including our core PM, so a bit of discouragement
from using it could be considered a good thing, tho it would be there, if
required.

If at some point in the future ns resolution becomes a much bigger issue
and many ebuilds are using the donsmtimes call, there's nothing stopping
us from looking at further tightening up the requirements for the general
case, probably as part of an eapi, then.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

ciaran.mccreesh at googlemail

Nov 24, 2009, 2:21 PM

Post #5 of 37 (778 views)
Permalink

Re: mtime preservation [In reply to]

On Mon, 23 Nov 2009 15:19:00 -0800
Brian Harring <ferringb [at] gmail> wrote:
> Someone mind explaining to me why we're making mtime preservation so
> nasty? Having to enumerate every pathway that requires mtime
> preservation is pretty arduous for the ebuild dev, meaning it's
> unlikely they'll get it right, leading to bugs.

It's not in the least bit nasty. It's requiring people to be explicit
about special requirements.

> The thing I'm not understanding here is that pkgcore since day one
> has preserved mtime- I've yet to see any complaints about that nor
> any issues caused by it. Portage shifted over a year or two back,
> same thing, haven't heard complaints.

You can't have been listening very hard, then. The complaint is that it
results in files with dumb mtimes being merged to /.

> I know it won't fly, but realistically stating "the package manager
> must preserve mtime- if there are instances where it does not (due to
> some feature, perhaps splitdebug or some form of compression) it is
> the package managers responsibility to ensure this does not break the
> ebuilds resultant merge/runtime invocation".
>
> Via such wording an exemption is created and it's made clear it's the
> managers responsibility to keep things playing nice... if the manager
> can't do that, then the feature/functionality that is changing the
> mtime (resulting in the pkg on disk failing) must be fixed or
> disabled.
>
> The nice thing about this wording is that it basically matches what
> the case is now, and what has worked for quite a few years.

Wording such as that just isn't suitable for a specification. It
requires implementers to guess what future ebuilds are going to
rely upon (and ebuilds regularly do rely upon weird quirks), and makes
it impossible to produce a package manager that can be shown to be
compliant.

> > In both cases nanosecond resolution may be required and is a problem
> > due to python. The following workaround can be used until the
> > nanosecond issue is fixed in python:
>
> It'd be nice if someone enumerated merge scenarios where nanosecond
> resolution is actually required. Seems like a white elephant to me,
> especially in combination w/ the fact that the target fs may not
> support nanosecond.

POSIX considers several of the non-nanosecond resolution calls to be
deprecated. Going "la la la I can't hear you!" because Python happens
to have utterly screwed this up is just going to lead to problems when
programs start using sub-second validity checks -- 'make' already does,
and it's given rise to various build-directory-related issues.

> Basically, if it's required the manager support nanosecond resolution
> for merging, ebuilds must be able to rely on that- end result, any
> merging to FS's that do not support nanosecond (this includes the
> intermediate ${D}) are no longer supported. Unless I'm missing
> something, iso-9660 doesn't look to support nanosecond resolution.
> Beyond that, does cramfs/squashfs? If not, taking this to the
> logical conclusion the livecds aren't technically supported as a
> merge environment.

No, we're after a requirement that the package manager must not screw up
nanosecond-resolution timestamps, and must not partially preserve and
partially not preserve them.

> > "Alternatively, we could simply make portage spawn the mv binary
> > whenever rename fails (it fails when the source and destination are
> > on different devices). Although it's relatively slow, it should
> > solve the problem."
>
> Yeah... no. Slowing down the main manager for a thereotical edge
> case doesn't seem particularly useful to me ;)

...or you could just fix Python.

--
Ciaran McCreesh

Attachments:

signature.asc (0.19 KB)

calchan at gentoo

Nov 25, 2009, 1:13 PM

Post #6 of 37 (774 views)
Permalink

Re: mtime preservation [In reply to]

A quick note to tell you that I have tried to look for examples of
breakage due to how mtime preservation is currently implemented in
portage. Unfortunately I didn't find anything, maybe because I'm not
knowledgeable enough or because I can't afford to spend any more time
on this. If any of you can provide an example then please do so.

In case nobody could show any sign of such breakage I would have to
add to the list of two propositions to vote on:
3- Do nothing.

We can always point to code that shows the implementation is
suboptimal but we have to understand that some of us who are less
proactive at fixing issues may want to procrastinate until breakage
actually happens.

Denis.

ciaran.mccreesh at googlemail

Nov 25, 2009, 1:27 PM

Post #7 of 37 (772 views)
Permalink

Re: Re: mtime preservation [In reply to]

On Wed, 25 Nov 2009 14:13:58 -0700
Denis Dupeyron <calchan [at] gentoo> wrote:
> A quick note to tell you that I have tried to look for examples of
> breakage due to how mtime preservation is currently implemented in
> portage. Unfortunately I didn't find anything, maybe because I'm not
> knowledgeable enough or because I can't afford to spend any more time
> on this. If any of you can provide an example then please do so.

Portage uses two ways of merging a file: os.rename() and the manual
way.

os.rename() correctly preserves mtimes.

Python's os.utime call, which is what Portage uses to preserve mtimes
for files that it installs the hard way, and which is not a wrapper for
the Unix utime call, uses an IEEE 754 double to handle timestamps. But
a double only gives sixteen accurate decimal digits, and you've got ten
to the left of the decimal point, leaving only six reliably preserved,
with the remaining three digits being corrupted.

Thus, packages can end up being installed with some files having
accurately-preserved mtimes, and some having corrupted mtimes, which
will lead to mtime comparisons giving incorrect results.

--
Ciaran McCreesh

Attachments:

signature.asc (0.19 KB)

1i5t5.duncan at cox

Nov 25, 2009, 1:52 PM

Post #8 of 37 (774 views)
Permalink

Re: mtime preservation [In reply to]

Ciaran McCreesh posted on Wed, 25 Nov 2009 21:27:18 +0000 as excerpted:

> Portage uses two ways of merging a file: os.rename() and the manual way.
>
> os.rename() correctly preserves mtimes.
>
> Python's os.utime call, which is what Portage uses to preserve mtimes
> for files that it installs the hard way, and which is not a wrapper for
> the Unix utime call, uses an IEEE 754 double to handle timestamps. But a
> double only gives sixteen accurate decimal digits, and you've got ten to
> the left of the decimal point, leaving only six reliably preserved, with
> the remaining three digits being corrupted.
>
> Thus, packages can end up being installed with some files having
> accurately-preserved mtimes, and some having corrupted mtimes, which
> will lead to mtime comparisons giving incorrect results.

That's a great explanation (thanks, I now know the details to the degree
I'd be interested), but what was asked for was examples of breakage, aka
actual bugs. I seem to remember either you or someone else mentioning
that there had been a couple issues already, and I'll take the claim at
face value, but actual bug numbers would be nice. =:^)

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

ciaran.mccreesh at googlemail

Nov 25, 2009, 2:13 PM

Post #9 of 37 (772 views)
Permalink

Re: Re: mtime preservation [In reply to]

On Wed, 25 Nov 2009 21:52:00 +0000 (UTC)
Duncan <1i5t5.duncan [at] cox> wrote:
> That's a great explanation (thanks, I now know the details to the
> degree I'd be interested), but what was asked for was examples of
> breakage, aka actual bugs.

Why? You can easily look and see that it's broken. Examples will merely
be dismissed as one-off cases that can be worked around, or as relying
upon a string of coincidences that will "obviously" never really
happen, right up until they do, at which point they'll be dismissed
with a WORKSFORME. What you have is a proof that it's broken, which is
far better than an example.

--
Ciaran McCreesh

Attachments:

signature.asc (0.19 KB)

ulm at gentoo

Nov 25, 2009, 2:59 PM

Post #10 of 37 (772 views)
Permalink

Re: mtime preservation [In reply to]

>>>>> On Wed, 25 Nov 2009, Ciaran McCreesh wrote:

> On Wed, 25 Nov 2009 21:52:00 +0000 (UTC)
> Duncan <1i5t5.duncan [at] cox> wrote:
>> That's a great explanation (thanks, I now know the details to the
>> degree I'd be interested), but what was asked for was examples of
>> breakage, aka actual bugs.

> Why? You can easily look and see that it's broken.

Only for a suitable definition of "broken".

> Examples will merely be dismissed as one-off cases that can be
> worked around, or as relying upon a string of coincidences that will
> "obviously" never really happen, right up until they do, at which
> point they'll be dismissed with a WORKSFORME.

Real examples would be issues like bugs 83877 [1] or 263387 [2].
Nothing that could be easily dismissed or worked around. Both issues
are fixed with Portage since a long time.

I don't know of any example where non-preservation of nanosecond
timestamps would cause problems.

> What you have is a proof that it's broken, which is far better than
> an example.

So we have a proven theorem, but unfortunately the cases where it is
applicable form an empty set. ;-)

Ulrich

[1] <http://bugs.gentoo.org/83877#c36>
[2] <http://bugs.gentoo.org/263387>

ciaran.mccreesh at googlemail

Nov 25, 2009, 4:15 PM

Post #11 of 37 (773 views)
Permalink

Re: mtime preservation [In reply to]

On Wed, 25 Nov 2009 23:59:45 +0100
Ulrich Mueller <ulm [at] gentoo> wrote:
> Real examples would be issues like bugs 83877 [1] or 263387 [2].
> Nothing that could be easily dismissed or worked around. Both issues
> are fixed with Portage since a long time.

Yes, those are examples of packages relying upon something that is
undefined behaviour, and that behaves differently depending upon the
Portage version you use.

> I don't know of any example where non-preservation of nanosecond
> timestamps would cause problems.

Not non-preservation. Partial and inconsistent corruption.

srcfile gets installed with its mtime preserved, because it is installed
using rename.

objfile, whose mtime should be a bit later than srctime's, gets
installed with its mtime corrupted to be slightly less than it should
be, and less than srctime's, because it is installed the slow way.

A program checks that objfile's mtime is greater than or equal to
srcfile's mtime. That check will fail, and reinstalls will randomly fix
and break it depending upon which way the corruption goes.

--
Ciaran McCreesh

Attachments:

signature.asc (0.19 KB)

zmedico at gentoo

Nov 25, 2009, 4:49 PM

Post #12 of 37 (773 views)
Permalink

Re: mtime preservation [In reply to]

Ciaran McCreesh wrote:
> On Wed, 25 Nov 2009 23:59:45 +0100
> Ulrich Mueller <ulm [at] gentoo> wrote:
>> Real examples would be issues like bugs 83877 [1] or 263387 [2].
>> Nothing that could be easily dismissed or worked around. Both issues
>> are fixed with Portage since a long time.
>
> Yes, those are examples of packages relying upon something that is
> undefined behaviour, and that behaves differently depending upon the
> Portage version you use.
>
>> I don't know of any example where non-preservation of nanosecond
>> timestamps would cause problems.
>
> Not non-preservation. Partial and inconsistent corruption.

Wouldn't "loss of precision" be a more accurate description? Of the
known packages which require timestamp preservation, do any of them
use sub-second precision in their timestamp comparisons?
--
Thanks,
Zac

ferringb at gmail

Nov 25, 2009, 5:14 PM

Post #13 of 37 (774 views)
Permalink

Re: mtime preservation [In reply to]

On Wed, Nov 25, 2009 at 04:49:17PM -0800, Zac Medico wrote:
> Ciaran McCreesh wrote:
> > On Wed, 25 Nov 2009 23:59:45 +0100
> > Ulrich Mueller <ulm [at] gentoo> wrote:
> >> Real examples would be issues like bugs 83877 [1] or 263387 [2].
> >> Nothing that could be easily dismissed or worked around. Both issues
> >> are fixed with Portage since a long time.
> >
> > Yes, those are examples of packages relying upon something that is
> > undefined behaviour, and that behaves differently depending upon the
> > Portage version you use.
> >
> >> I don't know of any example where non-preservation of nanosecond
> >> timestamps would cause problems.
> >
> > Not non-preservation. Partial and inconsistent corruption.
>
> Wouldn't "loss of precision" be a more accurate description? Of the
> known packages which require timestamp preservation, do any of them
> use sub-second precision in their timestamp comparisons?

This discussion in generall is daft. No package can rely on
nanonsecond resolution for installation because the most common FS out
there (ext3) does *second* level resolution only. As such, I can
pretty much gurantee there is *zero* packages out there that require
nanosecond resolution for installation.

It's an academic discussion, and pointless. We don't mandate the
filesystems PMS implementations are run on- as such we cannot make a
gurantee to ebuilds that nanosecond resolution is available. It's
daft to encode in the spec NS resolution when it's essentially
impossible to gurantee it- I'm honestly not sure why we're having this
discussion beyond the "python sucks" angle.

~brian

ferringb at gmail

Nov 25, 2009, 5:28 PM

Post #14 of 37 (774 views)
Permalink

Re: mtime preservation [In reply to]

On Tue, Nov 24, 2009 at 10:21:06PM +0000, Ciaran McCreesh wrote:
> On Mon, 23 Nov 2009 15:19:00 -0800
> Brian Harring <ferringb [at] gmail> wrote:
> > Someone mind explaining to me why we're making mtime preservation so
> > nasty? Having to enumerate every pathway that requires mtime
> > preservation is pretty arduous for the ebuild dev, meaning it's
> > unlikely they'll get it right, leading to bugs.
>
> It's not in the least bit nasty. It's requiring people to be explicit
> about special requirements.

I honestly wish that explicitness you're pushing for mtime were
applied to all parts of mtime.

Why is this one special? Two out of three do this already, and it
works. Paludis doesn't preserve mtime- and it's approach to randomly
resetting mtimes is known to cause issues (bug 263387, python eclass
having to do pyc/pyo compilation in postinst, etc).

> > The thing I'm not understanding here is that pkgcore since day one
> > has preserved mtime- I've yet to see any complaints about that nor
> > any issues caused by it. Portage shifted over a year or two back,
> > same thing, haven't heard complaints.
>
> You can't have been listening very hard, then. The complaint is that it
> results in files with dumb mtimes being merged to /.

As far as I can tell, no one but you is complaining about the mtimes.

Your complaints primarily have been that tarball'd mtimes can pass
through to the fs- I've not seen any comments from you in bug 264130
that actually enumerate spots were this is anything but an aesthetic
complaint. Specifically your aesthetics- the pkgs in question have
functioned fine w/ odd mtimes passing through.

> > I know it won't fly, but realistically stating "the package manager
> > must preserve mtime- if there are instances where it does not (due to
> > some feature, perhaps splitdebug or some form of compression) it is
> > the package managers responsibility to ensure this does not break the
> > ebuilds resultant merge/runtime invocation".
> >
> > Via such wording an exemption is created and it's made clear it's the
> > managers responsibility to keep things playing nice... if the manager
> > can't do that, then the feature/functionality that is changing the
> > mtime (resulting in the pkg on disk failing) must be fixed or
> > disabled.
> >
> > The nice thing about this wording is that it basically matches what
> > the case is now, and what has worked for quite a few years.
>
> Wording such as that just isn't suitable for a specification. It
> requires implementers to guess what future ebuilds are going to
> rely upon (and ebuilds regularly do rely upon weird quirks), and makes
> it impossible to produce a package manager that can be shown to be
> compliant.

That wording is explicit that PMS cannot lock down all potentials.
This is no different then PMS's stance on VDB (and similar in
reasoning)- ebuilds are reliant on vdb internals yet PMS explicitly
ducked documenting it. Double standards suck.

Further, drop the specificity BS. You've intentionally left parts of
PMS vague- trailing '/' for ebuild vars is a good example (ssmtp
breaks if ${D} vs ${D}/, perl likely has the same breakage). These
lead to actual incompatibilities, and because PMS isn't explicit,
there is no 'right' implementation for it.

The daft thing about you picking at this wording is that the scenarios
you can come up with are all hypothetical. Yes, a manager could
randomize mtimes when it's doing splitdebug.

The manager wouldn't however because the only reason to do things like
that is to intentionally cause issues.

~brian

1i5t5.duncan at cox

Nov 25, 2009, 9:04 PM

Post #15 of 37 (774 views)
Permalink

Re: mtime preservation [In reply to]

Ciaran McCreesh posted on Wed, 25 Nov 2009 22:13:27 +0000 as excerpted:

> Examples will merely be
> dismissed as one-off cases that can be worked around, or as relying upon
> a string of coincidences that will "obviously" never really happen,
> right up until they do, at which point they'll be dismissed with a
> WORKSFORME. What you have is a proof that it's broken, which is far
> better than an example.

Actually, that "dismissed with WORKSFORME" strikes a chord, here. There
was a very strange parallel make bug that I filed that was closed with
that. I'd have really liked to see someone with some skill tackle it, as
that was the only one I've ever seen that had striped fail and working
zones, and I've have loved to see some logic as to why... (If -j10
failed, -j3 and -j15 might succeed, -l24 fail again, and -j33 succeed
again...) Unfortunately, flameeyes, the only one I know who really gets
into such things, was fresh out of the hospital at the time, and I think
it was beyond the maintainer's abilities, so WORKSFORME was about the
best that could be done. I've long since changed and changed again my
makeopts, and don't remember the pkg now, tho I could probably find it in
my old bug mail if I needed to.

So I gotta admit you have a point, with that one.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

1i5t5.duncan at cox

Nov 25, 2009, 9:07 PM

Post #16 of 37 (773 views)
Permalink

Re: mtime preservation [In reply to]

Ulrich Mueller posted on Wed, 25 Nov 2009 23:59:45 +0100 as excerpted:

> Real examples would be issues like bugs 83877 [1] or 263387 [2]. Nothing
> that could be easily dismissed or worked around. Both issues are fixed
> with Portage since a long time.

Thanks.

> I don't know of any example where non-preservation of nanosecond
> timestamps would cause problems.

That's really what I was hoping for, a bug where the nanosecond times
thing made a difference. But Ciaran does have a point about such things
being potentially closed as WORKSFORME or the like, as I've seen that in
other cases, see my just earlier post.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

zmedico at gentoo

Nov 25, 2009, 9:26 PM

Post #17 of 37 (773 views)
Permalink

Re: mtime preservation [In reply to]

Brian Harring wrote:
> This discussion in generall is daft. No package can rely on
> nanonsecond resolution for installation because the most common FS out
> there (ext3) does *second* level resolution only. As such, I can
> pretty much gurantee there is *zero* packages out there that require
> nanosecond resolution for installation.

Your "guarantee" is filesystem-specific. However, if we can
establish that all known packages with timestamp preservation
requirements do their timestamp comparisons with 1-second
granularity, then we'll have a much safer (filesystem-independent)
assumption.
--
Thanks,
Zac

1i5t5.duncan at cox

Nov 25, 2009, 9:30 PM

Post #18 of 37 (772 views)
Permalink

Re: mtime preservation [In reply to]

Brian Harring posted on Wed, 25 Nov 2009 17:14:27 -0800 as excerpted:

> On Wed, Nov 25, 2009 at 04:49:17PM -0800, Zac Medico wrote:
>> Ciaran McCreesh wrote:
>> > On Wed, 25 Nov 2009 23:59:45 +0100
>> > Ulrich Mueller <ulm [at] gentoo> wrote:
>> >> Real examples would be issues like bugs 83877 [1] or 263387 [2].
>> >> Nothing that could be easily dismissed or worked around. Both issues
>> >> are fixed with Portage since a long time.
>> >
>> > Yes, those are examples of packages relying upon something that is
>> > undefined behaviour, and that behaves differently depending upon the
>> > Portage version you use.
>> >
>> >> I don't know of any example where non-preservation of nanosecond
>> >> timestamps would cause problems.
>> >
>> > Not non-preservation. Partial and inconsistent corruption.
>>
>> Wouldn't "loss of precision" be a more accurate description? Of the
>> known packages which require timestamp preservation, do any of them use
>> sub-second precision in their timestamp comparisons?
>
> This discussion in generall is daft. No package can rely on nanonsecond
> resolution for installation because the most common FS out there (ext3)
> does *second* level resolution only. As such, I can pretty much
> gurantee there is *zero* packages out there that require nanosecond
> resolution for installation.

One of the reasons I was asking for a bug number, was because there's two
possible failure modes, and I was hoping reading them would clue me (or
at least those who are making the decisions here) in on which one
actually occurs.

Your posts suggest a mode where subsecond times are always truncated. In
such a case, there should be no issue.

But it's also possible that times are compared as-is. If that's the
case, then there should be no issue on second-resolution filesystems
(such as ext3, in your example), because the filesystem would effectively
truncate those, reducing to case-one, above. But on higher resolution
filesystems, there might very well be issues, due to the subtle double
resolution issue Ciaran pointed out, especially when compared against
renames where the mtimes were accurately preserved.

Now it could well be that it's possible to turn case two into case one by
adding a simple option to the the command line or the like. Or maybe
not. Again, that's why I wanted specific examples, so people could see
if that were tried or not, and I would have to write all this up and
possibly look like a moron if I'm getting it wrong, somehow. But since
the examples don't seem to be forthcoming, and in view of the IMO
legitimate point about such bugs very possibly being closed as
WORKSFORME, I guess I post this and hope someone can either explain that
I have it all wrong, or say definitively that such command-line time
truncation option workarounds are generally practical, or not, thus
potentially affecting the direction of the debate.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

1i5t5.duncan at cox

Nov 25, 2009, 9:33 PM

Post #19 of 37 (774 views)
Permalink

Re: mtime preservation [In reply to]

Zac Medico posted on Wed, 25 Nov 2009 21:26:59 -0800 as excerpted:

> Brian Harring wrote:
>> This discussion in generall is daft. No package can rely on
>> nanonsecond resolution for installation because the most common FS out
>> there (ext3) does *second* level resolution only. As such, I can
>> pretty much gurantee there is *zero* packages out there that require
>> nanosecond resolution for installation.
>
> Your "guarantee" is filesystem-specific. However, if we can establish
> that all known packages with timestamp preservation requirements do
> their timestamp comparisons with 1-second granularity, then we'll have a
> much safer (filesystem-independent) assumption.

Thanks. That's basically what I'm wondering also, well, if they do it
with one second granularity, or if it's possible to make them do it that
with with a simple command-line sed, adding an option, but here it's
stated in different (fewer) words.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

ferringb at gmail

Nov 25, 2009, 9:35 PM

Post #20 of 37 (773 views)
Permalink

Re: mtime preservation [In reply to]

On Wed, Nov 25, 2009 at 09:26:59PM -0800, Zac Medico wrote:
> Brian Harring wrote:
> > This discussion in generall is daft. No package can rely on
> > nanonsecond resolution for installation because the most common FS out
> > there (ext3) does *second* level resolution only. As such, I can
> > pretty much gurantee there is *zero* packages out there that require
> > nanosecond resolution for installation.
>
> Your "guarantee" is filesystem-specific. However, if we can
> establish that all known packages with timestamp preservation
> requirements do their timestamp comparisons with 1-second
> granularity, then we'll have a much safer (filesystem-independent)
> assumption.

I've no complaints with mandating that ebuilds can rely on second
level resolution- it's a valid gurantee as far as I'm concerned.
Further any fs that can't offer it involves the user doing something
seriously wonky, thus their problem if the ebuild horks.

If/when the major filesystems out there all do NS level resolution,
and are in common deployment, I'd have no arguement extending the
spec to mandating NS level resolution. I've serious problems w/
mandating NS resolution in PMS prior to that however.

~harring

ulm at gentoo

Nov 25, 2009, 11:51 PM

Post #21 of 37 (773 views)
Permalink

Re: mtime preservation [In reply to]

>>>>> On Thu, 26 Nov 2009, Ciaran McCreesh wrote:

>> Real examples would be issues like bugs 83877 [1] or 263387 [2].
>> Nothing that could be easily dismissed or worked around. Both issues
>> are fixed with Portage since a long time.

> Yes, those are examples of packages relying upon something that is
> undefined behaviour,

That was the reason why I filed bug 264130, in the first place. ;-)

> and that behaves differently depending upon the Portage version you
> use.

Fact is that all versions of Portage in the tree get it right for both
packages, but Paludis fails for them.

> objfile, whose mtime should be a bit later than srctime's, gets
> installed with its mtime corrupted to be slightly less than it
> should be, and less than srctime's, because it is installed the slow
> way.

The major usage cases where mtimes play a role (*.py vs *.pyc, *.lisp
vs *.fasl, and *.el vs *.elc) install source and object files in the
same directory. So why would the PM use different methods for
installing them?

> A program checks that objfile's mtime is greater than or equal to
> srcfile's mtime. That check will fail, and reinstalls will randomly
> fix and break it depending upon which way the corruption goes.

Again, you don't come up with a concrete example where such breakage
would occur.

I'm starting to get tired of this whole discussion. Maybe we should
just follow Calchan's suggestion and do nothing. Portage works, after
all.

Ulrich

ulm at gentoo

Nov 25, 2009, 11:59 PM

Post #22 of 37 (773 views)
Permalink

Re: mtime preservation [In reply to]

>>>>> On Wed, 25 Nov 2009, Zac Medico wrote:

>> Not non-preservation. Partial and inconsistent corruption.

> Wouldn't "loss of precision" be a more accurate description?

Yes. Or even "rounding".

> Of the known packages which require timestamp preservation, do any
> of them use sub-second precision in their timestamp comparisons?

I can speak for Emacs only, where the comparison code (in fileio.c) is
as follows:

if (stat (SDATA (absname1), &st) < 0)
return Qnil;

mtime1 = st.st_mtime;

if (stat (SDATA (absname2), &st) < 0)
return Qt;

return (mtime1 > st.st_mtime) ? Qt : Qnil;

It uses stat(2), therefore nanoseconds are ignored.

Ulrich

ulm at gentoo

Nov 26, 2009, 1:07 AM

Post #23 of 37 (772 views)
Permalink

Re: mtime preservation [In reply to]

>>>>> On Thu, 26 Nov 2009, Ulrich Mueller wrote:

>>>>> On Wed, 25 Nov 2009, Zac Medico wrote:
>> Of the known packages which require timestamp preservation, do any
>> of them use sub-second precision in their timestamp comparisons?

> I can speak for Emacs only, where the comparison code (in fileio.c) is
> as follows: [...]

> It uses stat(2), therefore nanoseconds are ignored.

I've also checked with dev-lisp/sbcl and dev-lisp/cmucl. They use
stat(2) or fstat(2), i.e. compare timestamps on the seconds level.

Ulrich

bronksos at gmail

Nov 26, 2009, 1:54 AM

Post #24 of 37 (772 views)
Permalink

Re: mtime preservation [In reply to]

On 08:59 2009-11-26 +0100, Ulrich Mueller wrote:
>
> I can speak for Emacs only, where the comparison code (in fileio.c) is
> as follows:
>
[snip]
>
> It uses stat(2), therefore nanoseconds are ignored.
>

So, when that is updated to require nanosecond resolution where
supported, we're off to having this discussion again?

--
Pozdrawiam,
Łukasz P. Michalik

ulm at gentoo

Nov 26, 2009, 3:47 AM

Post #25 of 37 (773 views)
Permalink

Re: mtime preservation [In reply to]

>>>>> On Thu, 26 Nov 2009, �ukasz Michalik wrote:

>> I can speak for Emacs only, where the comparison code (in fileio.c) is
>> as follows:
>>
> [snip]
>>
>> It uses stat(2), therefore nanoseconds are ignored.

> So, when that is updated to require nanosecond resolution where
> supported, we're off to having this discussion again?

No, then my second argument still holds, since *.el and *.elc are
installed into the same directory.

Ulrich

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads