Mailing List Archive: Perl: porters

For 5.12: Make Perl Y2038 safe

Index | Next | Previous | View Threaded

schwern at pobox

Jan 20, 2008, 2:32 AM

Post #1 of 17 (477 views)
Permalink

For 5.12: Make Perl Y2038 safe

In a little under 30 years Unix will run out of time, the 2038 problem is upon
us. Any software that has to calculate dates out past 30 years in the future
cannot use Perl on a 32bit Unix system.

Time this was fixed.

C might be stuck with a signed 32bit integer for time but Perl is held to no
such thing. We just use the system date libraries out of convenience. Time
to change that.

--
100. Claymore mines are not filled with yummy candy, and it is wrong
to tell new soldiers that they are.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
http://skippyslist.com/?page_id=3

matthew.persico at gmail

Jan 31, 2008, 7:42 PM

Post #2 of 17 (443 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

On Jan 20, 2008 5:32 AM, Michael G Schwern <schwern[at]pobox.com> wrote:
> In a little under 30 years Unix will run out of time, the 2038 problem is upon
> us. Any software that has to calculate dates out past 30 years in the future
> cannot use Perl on a 32bit Unix system.
>
> Time this was fixed.

Yes, especially since if you have to do 30 year mortgage calcs, you
will run out of time THIS year.

I also recall in the latest edition of, I think, SDTimes, that some
language has already fixed this, but damned if I can recall or
<opinion> search the SDTimes site; what a piece of junk, doubly so for
a software-related org </opinion>

>
> C might be stuck with a signed 32bit integer for time but Perl is held to no
> such thing. We just use the system date libraries out of convenience. Time
> to change that.
>
>
> --
> 100. Claymore mines are not filled with yummy candy, and it is wrong
> to tell new soldiers that they are.
> -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
> http://skippyslist.com/?page_id=3
>
>

--
Matthew O. Persico

mark at mark

Jan 31, 2008, 8:44 PM

Post #3 of 17 (440 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

Matthew Persico wrote:
> On Jan 20, 2008 5:32 AM, Michael G Schwern <schwern[at]pobox.com> wrote:
>
>> In a little under 30 years Unix will run out of time, the 2038 problem is upon
>> us. Any software that has to calculate dates out past 30 years in the future
>> cannot use Perl on a 32bit Unix system.
>>
>> Time this was fixed.
>>
>
> Yes, especially since if you have to do 30 year mortgage calcs, you
> will run out of time THIS year.
>
> I also recall in the latest edition of, I think, SDTimes, that some
> language has already fixed this, but damned if I can recall or
> <opinion> search the SDTimes site; what a piece of junk, doubly so for
> a software-related org </opinion>
>
>

I thought the answer to this was 64-bit:

$ grep TIME_T_TYPE /usr/include/bits/*.h
/usr/include/bits/types.h:__STD_TYPE __TIME_T_TYPE __time_t; /*
Seconds since the Epoch. */
/usr/include/bits/typesizes.h:#define __TIME_T_TYPE
__SLONGWORD_TYPE

$ grep SLONGWORD_TYPE /usr/include/bits/*.h
/usr/include/bits/types.h:#define __SLONGWORD_TYPE long int

I don't think Perl itself represents a problem in this regard, as Perl
will return whatever C returns, and Perl compiled for a 64-bit target
will return whatever C for a 64-bit target uses for time_t, which is
64-bit for the machine I am typing this on now. My machine is already
2038-safe from the perspective of system calls, C, and Perl. However:

>> C might be stuck with a signed 32bit integer for time but Perl is held to no
>> such thing. We just use the system date libraries out of convenience. Time
>> to change that

This is not a C-specific issue. C doesn't track time. It is the
operating system that defines gettimeofday() and what it returns. If it
returns 32-bit, Perl will not be able to magically solve this. 64-bit
operating systems usually return 64-bit gettimeofday() system call
results, making C, and Perl, compiled for these platforms, fine. Also,
this issue of 2038 is primarily a UNIX issue. I believe Windows, and IBM
mainframes, only provided time_t values as a 32-bit offset from the
"epoch" in 1970 as a compatibility function to allow applications
designed for UNIX to run on them. IBM supported sub-second granularity
and >32 bits worth of time decades ago. My memory is faint on this (1999
was the last time I looked at it), but the BIOS in most modern day
personal computers and the "get time" function executed on system
startup does not query or store time as UNIX defines either - I don't
think they are affected by 2038.

The real problem is going to be applications that hard code an
assumption about the size of time_t into their data structures. An
example of this is database systems, or binary file formats, that
allocate only 4 bytes to store a time_t. All of these will break, and
changes to C, or Perl, are not going to fix this problem.

Database such as PostgreSQL have already upgraded to avoid this problem.
PostgreSQL stores time as either a 64-bit float or a 64-bit integer, and
I believe they marked zero as the start of the year 2000 UTC. In the
default configuration, using floating point, this has the very odd
effect that they support an extremely wide range of values in terms of
ability to represent time, however, time is most accurate, closest to
the year 2000. Pretty clever in my opinion, except that I hate using
floating point for any fixed quantities so I prefer their 64-bit option.

So - it's only a problem for dumb programs - the same as Y2K. Y2K could
not be prevented through the programming language. I don't think Perl
ever had a serious Y2K problem, and yet Perl programs failed in the year
2000. Perl gave users the ability to calculate time past 2000 all along,
but people still stored two digits years, and were still ignoring the
fact that localtime()'s year value is 1900 + the year value to get the
real year.

Cheers,
mark

--
Mark Mielke <mark[at]mielke.cc>

bet at rahul

Jan 31, 2008, 10:36 PM

Post #4 of 17 (440 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

2008-02-01T04:44:19 Mark Mielke:
> I don't think Perl itself represents a problem in this regard,
> as Perl will return whatever C returns, and Perl compiled for a
> 64-bit target will return whatever C for a 64-bit target uses for
> time_t, which is 64-bit for the machine I am typing this on now.

Hear, hear!

I adored my DEC UDB Alpha-based gizmo I enjoyed in the mid-90's for
many reasons, but the one that really lodged in my memory was doing
"perl -le" invocations demonstrating that 64bit time_t is cool-ly
close to hitting both ends of theoretical cosmological time.

-Bennett

rgiersig at cpan

Feb 1, 2008, 3:01 AM

Post #5 of 17 (440 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

Mark Mielke wrote:
> Matthew Persico wrote:
>> On Jan 20, 2008 5:32 AM, Michael G Schwern <schwern[at]pobox.com> wrote:
>>> C might be stuck with a signed 32bit integer for time but Perl is
>>> held to no
>>> such thing. We just use the system date libraries out of
>>> convenience. Time
>>> to change that
>
> This is not a C-specific issue. C doesn't track time. It is the
> operating system that defines gettimeofday() and what it returns. If it
> returns 32-bit, Perl will not be able to magically solve this.

But Perl can auto-upgrade time_t to 64 bit and use that internally. It
could also detect if the C libs have a Y2038 problem and not use them
(or do away with them alltogether). I thinks that is what Michael was
meaning.

So you actually should only look at the system time if you need to know
what time it is NOW. All other things (calculations, conversions etc)
can be dealt with internally in a Y2038-safe way.

> The real problem is going to be applications that hard code an
> assumption about the size of time_t into their data structures.

Nobody assumes that we can fix this in Perl.

But now I have a wish: put datetime handling and calculation into the
core. Unfortunately there is no perfect candidate for that on CPAN. Oh
well, one may dream...

Roland

mark at mark

Feb 1, 2008, 5:32 AM

Post #6 of 17 (441 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

Roland Giersig wrote:
> Mark Mielke wrote:
>> Matthew Persico wrote:
>>> On Jan 20, 2008 5:32 AM, Michael G Schwern <schwern[at]pobox.com> wrote:
>>>> C might be stuck with a signed 32bit integer for time but Perl is
>>>> held to no
>>>> such thing. We just use the system date libraries out of
>>>> convenience. Time
>>>> to change that
>>
>> This is not a C-specific issue. C doesn't track time. It is the
>> operating system that defines gettimeofday() and what it returns. If
>> it returns 32-bit, Perl will not be able to magically solve this.
>
> But Perl can auto-upgrade time_t to 64 bit and use that internally. It
> could also detect if the C libs have a Y2038 problem and not use them
> (or do away with them alltogether). I thinks that is what Michael was
> meaning.
>
> So you actually should only look at the system time if you need to
> know what time it is NOW. All other things (calculations, conversions
> etc) can be dealt with internally in a Y2038-safe way.

It's already done for 64-bit machines.

Also, if 32-bit machines exist still at that time, I expect time_t to be
64-bits even on those machines - long long values with 64-bit arithmetic
operations emulated in 32-bit software. Not that this is already done
today for "large file" support on UNIX file systems. 32-bit
architectures are able to access 64-bit file offsets with software
emulated 64-bit values. Using a different internal value for time than
the UNIX-like operating systems is unnecessary. Even 32-bit Perl could
store time as a double instead of a 32-bit integer in a fix.

>
>> The real problem is going to be applications that hard code an
>> assumption about the size of time_t into their data structures.
>
> Nobody assumes that we can fix this in Perl.
>
> But now I have a wish: put datetime handling and calculation into the
> core. Unfortunately there is no perfect candidate for that on CPAN. Oh
> well, one may dream...

Yep - Perl date manipulation is week. time(), localtime() and gmtime()
come from C - not Perl. I once found Date::Manip from CPAN useful,
although it seems clumsy... :-(

Cheers,
mark

--
Mark Mielke <mark[at]mielke.cc>

sartak at gmail

Feb 1, 2008, 5:46 AM

Post #7 of 17 (441 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

On 2/1/08, Roland Giersig <rgiersig[at]cpan.org> wrote:
> Mark Mielke wrote:
> > Matthew Persico wrote:
> >> On Jan 20, 2008 5:32 AM, Michael G Schwern <schwern[at]pobox.com> wrote:
> >>> C might be stuck with a signed 32bit integer for time but Perl is
> >>> held to no
> >>> such thing. We just use the system date libraries out of
> >>> convenience. Time
> >>> to change that
> >
> > This is not a C-specific issue. C doesn't track time. It is the
> > operating system that defines gettimeofday() and what it returns. If it
> > returns 32-bit, Perl will not be able to magically solve this.
>
> But Perl can auto-upgrade time_t to 64 bit and use that internally. It
> could also detect if the C libs have a Y2038 problem and not use them
> (or do away with them alltogether). I thinks that is what Michael was
> meaning.

It would be good to at least warn if gettimeofday or time(NULL)
returns a negative number. There might be some strange OS out there
that regularly returns < 0 for now, but that can be dealt with by
whoever is porting.

>
> So you actually should only look at the system time if you need to know
> what time it is NOW. All other things (calculations, conversions etc)
> can be dealt with internally in a Y2038-safe way.
>
> > The real problem is going to be applications that hard code an
> > assumption about the size of time_t into their data structures.
>
> Nobody assumes that we can fix this in Perl.
>
> But now I have a wish: put datetime handling and calculation into the
> core. Unfortunately there is no perfect candidate for that on CPAN. Oh
> well, one may dream...

DateTime is damn fine. But I don't think it should be cored. (We've
had that discussion enough times.. :))

>
> Roland
>

Shawn

autarch at urth

Feb 1, 2008, 12:17 PM

Post #8 of 17 (438 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

On Fri, 1 Feb 2008, Sartak wrote:

>> So you actually should only look at the system time if you need to know
>> what time it is NOW. All other things (calculations, conversions etc)
>> can be dealt with internally in a Y2038-safe way.
>>
>>> The real problem is going to be applications that hard code an
>>> assumption about the size of time_t into their data structures.
>>
>> Nobody assumes that we can fix this in Perl.
>>
>> But now I have a wish: put datetime handling and calculation into the
>> core. Unfortunately there is no perfect candidate for that on CPAN. Oh
>> well, one may dream...
>
> DateTime is damn fine. But I don't think it should be cored. (We've
> had that discussion enough times.. :))

Seconded, no DateTime in the core. Frankly, from a module maintainer's
perspective, having your module added to the core is just more work, and
there's no upside. I don't _care_ if some people don't use DateTime,
that's their loss, so I don't need to see it encouraged via coring.

This is compounded by the fact that Perl still uses Perforce, making
working with the core code more annoying than any other project I've ever
dealt with.

Yes, I _know_ all the rsync/whatever workarounds, and I don't care. It's
still more annoying ;)

DateTime is a particularly bad candidate for being added to the core since
it is dependent on DateTime::TimeZone, which in turn has a release cycle
largely determined by the Olson time zone database, which releases much
more often than Perl itself.

-dave

/*===================================================
VegGuide.Org www.BookIRead.com
Your guide to all that's veg. My book blog
===================================================*/

jand at activestate

Feb 1, 2008, 12:28 PM

Post #9 of 17 (439 views)
Permalink

RE: For 5.12: Make Perl Y2038 safe [In reply to]

On Thu, 31 Jan 2008, Matthew Persico wrote:
> On Jan 20, 2008 5:32 AM, Michael G Schwern <schwern[at]pobox.com> wrote:
> I also recall in the latest edition of, I think, SDTimes, that some
> language has already fixed this, but damned if I can recall or
> <opinion> search the SDTimes site; what a piece of junk, doubly so for
> a software-related org </opinion>

I don't know which language SDTimes was referring to, but I know that
Tcl 8.5 has fixed the 2038 date issue for its own core functions and
modules even in 32-bit releases of the interpreter:

http://www.ddj.com/linux-open-source/205201480

Cheers,
-Jan

matt at sergeant

Feb 1, 2008, 2:27 PM

Post #10 of 17 (441 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

On 1-Feb-08, at 6:01 AM, Roland Giersig wrote:

> But now I have a wish: put datetime handling and calculation into
> the core. Unfortunately there is no perfect candidate for that on
> CPAN. Oh well, one may dream...

You're not looking hard enough. Perl 5.10 got a new module.

Matt.

craig.a.berry at gmail

Feb 1, 2008, 4:42 PM

Post #11 of 17 (440 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

On Feb 1, 2008 5:01 AM, Roland Giersig <rgiersig[at]cpan.org> wrote:
> Mark Mielke wrote:
> > Matthew Persico wrote:
> >> On Jan 20, 2008 5:32 AM, Michael G Schwern <schwern[at]pobox.com> wrote:
> >>> C might be stuck with a signed 32bit integer for time but Perl is
> >>> held to no
> >>> such thing. We just use the system date libraries out of
> >>> convenience. Time
> >>> to change that
> >
> > This is not a C-specific issue. C doesn't track time. It is the
> > operating system that defines gettimeofday() and what it returns. If it
> > returns 32-bit, Perl will not be able to magically solve this.

It's true the C standard doesn't cover gettimeofday(), but POSIX does
and says that it will fill in a timeval struct containing the seconds
since the epoch in a member of type time_t. Both standards cover
time_t, but just say it's a numeric value "of an appropriate length."
There is nothing that says it has to be the type of a native integer
(or even an integer at all), so whatever the local machine clock can
or can't do is irrelevant, or should be. In any case, it doesn't
really matter how people got started making time_t a 32-bit signed
integer; what matters is that changing that will break a lot of
things.

> But Perl can auto-upgrade time_t to 64 bit and use that internally. It
> could also detect if the C libs have a Y2038 problem and not use them
> (or do away with them alltogether). I thinks that is what Michael was
> meaning.

Yes, this might work for a subset of the 2038 problems, and is at
least worth thinking about. The place to start would be one or more
Configure probes along the lines of time_t_size, time_t_upper_limit or
similar.

> So you actually should only look at the system time if you need to know
> what time it is NOW. All other things (calculations, conversions etc)
> can be dealt with internally in a Y2038-safe way.
>
> > The real problem is going to be applications that hard code an
> > assumption about the size of time_t into their data structures.

It's not applications, it's C implementations, though certainly
applications that maintain time values in permanent storage have
additional problems as you point out. Changing the size of time_t is a
change to the C compiler, run-time, and headers and breaks binary
compatibility because it changes the size of a struct returned from
standard library functions. For example, as of Visual Studio 2005,
Windows defaults to a 64-bit time_t, though you can override that to
get compatibility with previous releases:

http://msdn2.microsoft.com/en-us/library/1f4c8f33(VS.80).aspx

It sure looks to me like if you build with the current tools and link
against an older library that hasn't been recompiled, you're going to
be passing a larger timeval struct to something that expects a smaller
one. Perhaps there are tricks with macros and internal function names
behind the scenes that would prevent that, at least by generating an
error at link time.

Note that the docs I pointed to are for Win32, not Win64; there is no
particular correlation between the size of time_t and the native word
size of the machine or between either of those and the size of an int
in C.

Whether in C or in Perl, the size of time_t can be whatever you want
it to be, but changing the size breaks binary compatibility, which is
why Schwern put "For 5.12" in the subject line.

mark at mark

Feb 1, 2008, 6:10 PM

Post #12 of 17 (438 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

Craig A. Berry wrote:
> Note that the docs I pointed to are for Win32, not Win64; there is no
> particular correlation between the size of time_t and the native word
> size of the machine or between either of those and the size of an int
> in C.
>

Correct. The part that "breaks" for Perl, in this regard, is that Perl
for 32-bit stores integers as 32-bit, and I strongly suspect that any
high bits in gettimeofday() would be lost.

If Perl for 32-bit is to support >32-bit epoch offset values, Perl
itself must support >32-bit integers, or Perl must store the epoch
offset in a different format such as a floating point value, binary
data, a decimal string, or a BigInt. In any of the latter cases, it
makes manipulation of the epoch offset difficult, and this is one of the
primary benefits of an epoch offset. Ease of manipulation.

Cheers,
mark

--
Mark Mielke <mark[at]mielke.cc>

matt at sergeant

Feb 1, 2008, 8:34 PM

Post #13 of 17 (441 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

On 20-Jan-08, at 5:32 AM, Michael G Schwern wrote:

> In a little under 30 years Unix will run out of time, the 2038
> problem is upon us. Any software that has to calculate dates out
> past 30 years in the future cannot use Perl on a 32bit Unix system.
>
> Time this was fixed.
>
> C might be stuck with a signed 32bit integer for time but Perl is
> held to no such thing. We just use the system date libraries out
> of convenience. Time to change that.

Since this appears to be being questioned as to whether or not it's a
good idea - let me just stick my oar in and say I 100% support this
notion that it could, and should be fixed in core perl and the perl
hacker should be isolated from the C library regarding issues with
time_t's 32 bitness.

Matt (author of perl 5.10's Time::Piece).

mark at mark

Feb 1, 2008, 8:46 PM

Post #14 of 17 (440 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

Matt Sergeant wrote:
> Since this appears to be being questioned as to whether or not it's a
> good idea - let me just stick my oar in and say I 100% support this
> notion that it could, and should be fixed in core perl and the perl
> hacker should be isolated from the C library regarding issues with
> time_t's 32 bitness.
> Matt (author of perl 5.10's Time::Piece).
As opposed to DateTime, which exists - how do you envision a "fix" for
32-bit platforms? You can use floating point values today, and whenever
32-bit platforms start to commonly return 32-bit times from
gettimeofday() (they have 30 years to change) Perl can return a floating
point value. So what is the problem? That localtime() doesn't work? A
re-write of localtime() means understanding time zones, and having a
portable definition of time. This is what DateTime gives you.

If DateTime was in the core, would you be happy? I believe the owner of
DateTime has provided his own recommendation that DateTime *not* be
included in the core.

If you have followed Java, I think you might have noticed the issue with
a requirement that Java be upgraded due to the U.S. time zone changes
that occurred recently. Why should all of Java be upgraded, or why
should patches be provided to Java, just because a time zone changes?
The why is because Java embeds the time zone database in its run time
distribution. Is this good or bad? It's convenient, but it's also
troublesome.

On my Fedora 7 machine, perl-DateTime is a package available with yum
update. It doesn't need to be part of the Perl core to be useful, and
not being part of the Perl core means that it can have an independent
release cycle. As per the owner of DateTime's comment - the time zone
database changes more frequently than Perl is released. My own addition
to this is that most people do not regularly update Perl. At my place of
work, they are still using Perl 5.6.0 and there are still spots of Perl
5.004. Why should these places have an out-of-date understanding of time
zones?

I think it is clear to me that nothing is broken. Any Perl programmers
that need to manipulate dates and times as anything except seconds with
a range of now until the next reboot, should be using a module such as
DateTime, and should store dates in a portable format. 32-bit integers
are not portable formats nor do they have sufficient range.

Cheers,
mark

--
Mark Mielke <mark[at]mielke.cc>

matt at sergeant

Feb 2, 2008, 6:38 AM

Post #15 of 17 (439 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

On 1-Feb-08, at 11:46 PM, Mark Mielke wrote:

> Matt Sergeant wrote:
>> Since this appears to be being questioned as to whether or not
>> it's a good idea - let me just stick my oar in and say I 100%
>> support this notion that it could, and should be fixed in core
>> perl and the perl hacker should be isolated from the C library
>> regarding issues with time_t's 32 bitness.
>> Matt (author of perl 5.10's Time::Piece).

> As opposed to DateTime, which exists - how do you envision a "fix"
> for 32-bit platforms?

There's absolutely no reason that Perl's time() functions have to be
based on time_t. Simple as that.

Of course the devil is in the details, but the guys on here are
smarter than me - I'm sure they could figure it out.

Matt.

mark at mark

Feb 2, 2008, 6:53 AM

Post #16 of 17 (439 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

Matt Sergeant wrote:
> There's absolutely no reason that Perl's time() functions have to be
> based on time_t. Simple as that.

To the contrary - the only way to know the time of day is from the
operating system, and UNIX-like operating systems provide this as a
time_t. By 2038, all operating systems at that time will likely defined
time_t as 64-bits, and all versions of Perl that assume time_t is 32-bit
will need to be recompiled to function beyond the 2038 boundary. There
is almost 30 years to fix this problem, and the problem MUST be
addressed at the operating system level. Perl CANNOT fix this on its own
unless it makes its own bogus assumptions such as a negative time value
indicates a time beyond 2038.

Your question is really the same as Michael's - your question is "why is
the time valued stored in a Perl data structure as an IV instead of an
NV, or if it is an IV because it fits on 32-bit platforms today, why
when I perform arithmetic operations on it does it stay an IV and wrap
to negative instead of being auto-promoted to an NV?" This question is a
basic question about Perl numbers and has nothing to do with time. One
could ask this question about any thing. Why can I only store 32-bits
worth of flag bits in a Perl integer before results start to become
unpredictable? What a bogus limit some "normal" user might say.

A second issue was raised by somebody else about how localtime() might
not work outside the 1970 - 2038 range even if time_t is > 32-bits. I
suggest to you that DateTime already provides a practical solution to
this that you can use today.

Cheers,
mark

--
Mark Mielke <mark[at]mielke.cc>

matt at sergeant

Feb 2, 2008, 10:16 AM

Post #17 of 17 (439 views)
Permalink

Re: For 5.12: Make Perl Y2038 safe [In reply to]

On 2-Feb-08, at 9:53 AM, Mark Mielke wrote:

> To the contrary - the only way to know the time of day is from the
> operating system, and UNIX-like operating systems provide this as a
> time_t.

That's only relevant for obtaining the current time. I think we're
fine for that - I'm not really worried about my perl code working in
2038. This is only relevant for arithmetic.

The rest of the perl time functions (localtime, gmtime,
POSIX::strftime) don't need to be defined in terms of time_t. We are
perfectly capable of re-implementing them.

Matt.

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com