Mailing List Archive: Python: Dev

Drop the new time.wallclock() function?

1 2 3

View All

Index | Next | Previous | View Threaded

victor.stinner at gmail

Mar 13, 2012, 4:57 PM

Post #1 of 67 (875 views)
Permalink

Drop the new time.wallclock() function?

Hi,

I added two functions to the time module in Python 3.3: wallclock()
and monotonic(). I'm unable to explain the difference between these
two functions, even if I wrote them :-) wallclock() is suppose to be
more accurate than time() but has an unspecified starting point.
monotonic() is similar except that it is monotonic: it cannot go
backward. monotonic() may not be available or fail whereas wallclock()
is available/work, but I think that the two functions are redundant.

I prefer to keep only monotonic() because it is not affected by system
clock update and should help to fix issues on NTP update in functions
implementing a timeout.

What do you think?

--

monotonic() has 3 implementations:
* Windows: QueryPerformanceCounter() with QueryPerformanceFrequency()
* Mac OS X: mach_absolute_time() with mach_timebase_info()
* UNIX: clock_gettime(CLOCK_MONOTONIC_RAW) or clock_gettime(CLOCK_MONOTONIC)

wallclock() has 3 implementations:
* Windows: QueryPerformanceCounter() with QueryPerformanceFrequency(),
with a fallback to GetSystemTimeAsFileTime() if
QueryPerformanceFrequency() failed
* UNIX: clock_gettime(CLOCK_MONOTONIC_RAW),
clock_gettime(CLOCK_MONOTONIC) or clock_gettime(CLOCK_REALTIME), with
a fallback to gettimeofday() if clock_gettime(*) failed
* Otherwise: gettimeofday()

(wallclock should also use mach_absolute_time() on Mac OS X)

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

fuzzyman at voidspace

Mar 13, 2012, 5:03 PM

Post #2 of 67 (863 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

On 13 Mar 2012, at 16:57, Victor Stinner wrote:

> Hi,
>
> I added two functions to the time module in Python 3.3: wallclock()
> and monotonic(). I'm unable to explain the difference between these
> two functions, even if I wrote them :-) wallclock() is suppose to be
> more accurate than time() but has an unspecified starting point.
> monotonic() is similar except that it is monotonic: it cannot go
> backward. monotonic() may not be available or fail whereas wallclock()
> is available/work, but I think that the two functions are redundant.
>
> I prefer to keep only monotonic() because it is not affected by system
> clock update and should help to fix issues on NTP update in functions
> implementing a timeout.
>
> What do you think?
>

I am in the middle of adding a feature to unittest that involves timing of individual tests. I want the highest resolution cross platform measure of wallclock time - and time.wallclock() looked ideal. If monotonic may not exist or can fail why would that be better?

Michael

> --
>
> monotonic() has 3 implementations:
> * Windows: QueryPerformanceCounter() with QueryPerformanceFrequency()
> * Mac OS X: mach_absolute_time() with mach_timebase_info()
> * UNIX: clock_gettime(CLOCK_MONOTONIC_RAW) or clock_gettime(CLOCK_MONOTONIC)
>
> wallclock() has 3 implementations:
> * Windows: QueryPerformanceCounter() with QueryPerformanceFrequency(),
> with a fallback to GetSystemTimeAsFileTime() if
> QueryPerformanceFrequency() failed
> * UNIX: clock_gettime(CLOCK_MONOTONIC_RAW),
> clock_gettime(CLOCK_MONOTONIC) or clock_gettime(CLOCK_REALTIME), with
> a fallback to gettimeofday() if clock_gettime(*) failed
> * Otherwise: gettimeofday()
>
> (wallclock should also use mach_absolute_time() on Mac OS X)
>
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>

--
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing
http://www.sqlite.org/different.html

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

nadeem.vawda at gmail

Mar 13, 2012, 5:18 PM

Post #3 of 67 (860 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

So wallclock() falls back to a not-necessarily-monotonic time source
if necessary,
while monotonic() raises an exception in that case? ISTM that these
don't need to
be separate functions - rather, we can have one function that takes a
flag (called
require_monotonic, or something like that) telling it which failure mode to use.
Does that make sense?

Cheers,
Nadeem
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Mar 13, 2012, 5:27 PM

Post #4 of 67 (858 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

On Tue, Mar 13, 2012 at 4:57 PM, Victor Stinner
<victor.stinner [at] gmail> wrote:
> I added two functions to the time module in Python 3.3: wallclock()
> and monotonic(). I'm unable to explain the difference between these
> two functions, even if I wrote them :-) wallclock() is suppose to be
> more accurate than time() but has an unspecified starting point.
> monotonic() is similar except that it is monotonic: it cannot go
> backward. monotonic() may not be available or fail whereas wallclock()
> is available/work, but I think that the two functions are redundant.
>
> I prefer to keep only monotonic() because it is not affected by system
> clock update and should help to fix issues on NTP update in functions
> implementing a timeout.
>
> What do you think?

I think wallclock() is an awkward name; in other contexts I've seen
"wall clock time" used to mean the time that a clock on the wall would
show, i.e. local time. This matches definition #1 of
http://www.catb.org/jargon/html/W/wall-time.html (while yours matches
#2 :-).

I agree that it's better to have only one of these. I also think if we
offer it we should always have it -- if none of the implementations
are available, I guess you could fall back on returning time.time(),
with some suitable offset so people don't think it is always the same.
Maybe it could be called realtime()?

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

victor.stinner at gmail

Mar 13, 2012, 5:29 PM

Post #5 of 67 (858 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

On 14/03/2012 01:18, Nadeem Vawda wrote:
> So wallclock() falls back to a not-necessarily-monotonic time source
> if necessary,
> while monotonic() raises an exception in that case? ISTM that these
> don't need to
> be separate functions - rather, we can have one function that takes a
> flag (called
> require_monotonic, or something like that) telling it which failure mode to use.
> Does that make sense?

I don't think that time.monotonic() can fail in practice and it is
available for all modern platforms (Windows, Mac OS X and OS implemented
clock_gettime()).

On Windows, time.monotonic() fails with an OSError if
QueryPerformanceFrequency() failed. QueryPerformanceFrequency() can fail
if "the installed hardware does not support a high-resolution
performance counter" according to Microsoft documentation. Windows uses
the CPU RDTSC instruction, or the ACPI power management timer or event
the old 8245 PIT. I think that you have at least one of this device on
your computer.

I suppose that you can use a manual fallback to time.time() if
time.monotonic() failed. If time.monotonic() fails, it fails directly at
the first call. Example of a fallback working with Python < 3.3:

try:
time.monotonic()
except (OSError, AttributeError):
get_time = time.time
else:
get_time = time.monotonic

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

kristjan at ccpgames

Mar 13, 2012, 5:45 PM

Post #6 of 67 (863 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

The reason I originally suggested "wallclock" was because that term is often used to distinguish time measurements (delta) that show real world time from those showing CPU or Kernel time. "number.crunch() took 2 seconds wallclock time but only 1 second CPU!". The original problem was that time.clock() was "wallclock" on some platforms but "cpu" on others, IIRC.
But monotonic is probably even better. I agree removing one or the other, probably wallclock.
K

-----Original Message-----
From: python-dev-bounces+kristjan=ccpgames.com [at] python [mailto:python-dev-bounces+kristjan=ccpgames.com [at] python] On Behalf Of Guido van Rossum
Sent: 13. mars 2012 17:27
To: Victor Stinner
Cc: Python Dev
Subject: Re: [Python-Dev] Drop the new time.wallclock() function?

I think wallclock() is an awkward name; in other contexts I've seen "wall clock time" used to mean the time that a clock on the wall would show, i.e. local time. This matches definition #1 of http://www.catb.org/jargon/html/W/wall-time.html (while yours matches
#2 :-).

I agree that it's better to have only one of these. I also think if we offer it we should always have it -- if none of the implementations are available, I guess you could fall back on returning time.time(), with some suitable offset so people don't think it is always the same.
Maybe it could be called realtime()?

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

victor.stinner at gmail

Mar 13, 2012, 6:03 PM

Post #7 of 67 (858 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

> I agree that it's better to have only one of these. I also think if we
> offer it we should always have it -- if none of the implementations
> are available, I guess you could fall back on returning time.time(),
> with some suitable offset so people don't think it is always the same.
> Maybe it could be called realtime()?

For a concrete use case, see for example:
http://bugs.python.org/issue14222

I just wrote two patches, for the queue and threading modules, using
time.monotonic() if available, with a fallback to time.time(). My
patches call time.monotonic() to ensure that it doesn't fail with
OSError.

I suppose that most libraries and programs will have to implement a
similar fallback.

We may merge both functions with a flag to be able to disable the
fallback. Example:

- time.realtime(): best-effort monotonic, with a fallback
- time.realtime(monotonic=True): monotonic, may raise OSError or
NotImplementedError

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

tismer at stackless

Mar 13, 2012, 6:06 PM

Post #8 of 67 (866 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

On 3/13/12 5:45 PM, Kristj�n Valur J�nsson wrote:
> The reason I originally suggested "wallclock" was because that term is often used to distinguish time measurements (delta) that show real world time from those showing CPU or Kernel time. "number.crunch() took 2 seconds wallclock time but only 1 second CPU!". The original problem was that time.clock() was "wallclock" on some platforms but "cpu" on others, IIRC.
> But monotonic is probably even better. I agree removing one or the other, probably wallclock.
> K
>
> -----Original Message-----
> From: python-dev-bounces+kristjan=ccpgames.com [at] python [mailto:python-dev-bounces+kristjan=ccpgames.com [at] python] On Behalf Of Guido van Rossum
> Sent: 13. mars 2012 17:27
> To: Victor Stinner
> Cc: Python Dev
> Subject: Re: [Python-Dev] Drop the new time.wallclock() function?
>
> I think wallclock() is an awkward name; in other contexts I've seen "wall clock time" used to mean the time that a clock on the wall would show, i.e. local time. This matches definition #1 of http://www.catb.org/jargon/html/W/wall-time.html (while yours matches
> #2 :-).
>
> I agree that it's better to have only one of these. I also think if we offer it we should always have it -- if none of the implementations are available, I guess you could fall back on returning time.time(), with some suitable offset so people don't think it is always the same.
> Maybe it could be called realtime()?
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/tismer%40stackless.com

Btw., have you considered virtual machines?
I happen to run windows in Parallels or Virtualbox quite often.
There the "wall clock" stuff notoriously does not work.

It would be good (but difficult?) if the supposed-to-be-accurate
clock could test itself, if it works at all, and replace itself
with a fallback.

In my case, this causes quite a few PyPy tests to fail ;-)

ciao -- Chris

--
Christian Tismer :^)<mailto:tismer [at] stackless>
tismerysoft GmbH : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/
14482 Potsdam : PGP key -> http://pgp.uni-mainz.de
work +49 173 24 18 776 mobile +49 173 24 18 776 fax n.a.
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

yselivanov.ml at gmail

Mar 13, 2012, 6:09 PM

Post #9 of 67 (861 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

If we need to decide to which function should be kept - I vote for monotonic.
It's extremely useful (even essential) to track timeouts in various schedulers
implementations, for example. Quick search also shows the demand for it, as
there are questions on stackoverflow.com and few packages on PyPI.

-
Yury

On 2012-03-13, at 9:03 PM, Victor Stinner wrote:

>> I agree that it's better to have only one of these. I also think if we
>> offer it we should always have it -- if none of the implementations
>> are available, I guess you could fall back on returning time.time(),
>> with some suitable offset so people don't think it is always the same.
>> Maybe it could be called realtime()?
>
> For a concrete use case, see for example:
> http://bugs.python.org/issue14222
>
> I just wrote two patches, for the queue and threading modules, using
> time.monotonic() if available, with a fallback to time.time(). My
> patches call time.monotonic() to ensure that it doesn't fail with
> OSError.
>
> I suppose that most libraries and programs will have to implement a
> similar fallback.
>
> We may merge both functions with a flag to be able to disable the
> fallback. Example:
>
> - time.realtime(): best-effort monotonic, with a fallback
> - time.realtime(monotonic=True): monotonic, may raise OSError or
> NotImplementedError
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

nadeem.vawda at gmail

Mar 13, 2012, 6:10 PM

Post #10 of 67 (858 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

On Wed, Mar 14, 2012 at 3:03 AM, Victor Stinner
<victor.stinner [at] gmail> wrote:
> I suppose that most libraries and programs will have to implement a
> similar fallback.
>
> We may merge both functions with a flag to be able to disable the
> fallback. Example:
>
> - time.realtime(): best-effort monotonic, with a fallback
> - time.realtime(monotonic=True): monotonic, may raise OSError or
> NotImplementedError

This was my suggestion - I think it's useful to have the fallback
available (since most users will want it), but at the same time we
should also cater to users who need a clock that is *guaranteed* to
be monotonic.

As an aside, I think "monotonic" is a better name than "realtime";
it conveys the functions purpose more clearly. Then we could call
the flag "strict".

Cheers,
Nadeem
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

kristjan at ccpgames

Mar 13, 2012, 6:11 PM

Post #11 of 67 (849 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

Interesting thought.
Althougn I don't see how that could fail on windows, if the QPC function is really just talking to a clock chip, surely that hasn't been virtualized.
Is there an actual example of windows hardware where this api fails (virtual or not?) Perhaps there is no real need to have a fallback mechanism, and it would even be best to write such a mechanism inside the function itself, and just return getsystemtimeasfiletime() instead.

K

-----Original Message-----
From: Christian Tismer [mailto:tismer [at] stackless]
Sent: 13. mars 2012 18:07
To: Kristj�n Valur J�nsson
Cc: Guido van Rossum; Victor Stinner; Python Dev
Subject: Re: [Python-Dev] Drop the new time.wallclock() function?

Btw., have you considered virtual machines?
I happen to run windows in Parallels or Virtualbox quite often.
There the "wall clock" stuff notoriously does not work.

It would be good (but difficult?) if the supposed-to-be-accurate clock could test itself, if it works at all, and replace itself with a fallback.

In my case, this causes quite a few PyPy tests to fail ;-)

ciao -- Chris

--
Christian Tismer :^)<mailto:tismer [at] stackless>
tismerysoft GmbH : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/
14482 Potsdam : PGP key -> http://pgp.uni-mainz.de
work +49 173 24 18 776 mobile +49 173 24 18 776 fax n.a.
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

andrew.svetlov at gmail

Mar 13, 2012, 6:12 PM

Post #12 of 67 (858 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

On Tue, Mar 13, 2012 at 5:29 PM, Victor Stinner
<victor.stinner [at] gmail> wrote:
> I suppose that you can use a manual fallback to time.time() if
> time.monotonic() failed. If time.monotonic() fails, it fails directly at the
> first call. Example of a fallback working with Python < 3.3:
>
> try:
> time.monotonic()
> except (OSError, AttributeError):
> get_time = time.time
> else:
> get_time = time.monotonic
>

I like 'fallback' solution while `get_time` is not the best name for
high precision timer from my perspective.
Can you call it `monotonic` or `realtime`?
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Mar 13, 2012, 6:34 PM

Post #13 of 67 (849 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

On Tue, Mar 13, 2012 at 6:03 PM, Victor Stinner
<victor.stinner [at] gmail> wrote:
>> I agree that it's better to have only one of these. I also think if we
>> offer it we should always have it -- if none of the implementations
>> are available, I guess you could fall back on returning time.time(),
>> with some suitable offset so people don't think it is always the same.
>> Maybe it could be called realtime()?
>
> For a concrete use case, see for example:
> http://bugs.python.org/issue14222
>
> I just wrote two patches, for the queue and threading modules, using
> time.monotonic() if available, with a fallback to time.time(). My
> patches call time.monotonic() to ensure that it doesn't fail with
> OSError.
>
> I suppose that most libraries and programs will have to implement a
> similar fallback.

It seems horrible to force everyone to copy the same silly block of
code. The time module itself should implement this once.

> We may merge both functions with a flag to be able to disable the
> fallback. Example:
>
> �- time.realtime(): best-effort monotonic, with a fallback
> �- time.realtime(monotonic=True): monotonic, may raise OSError or
> NotImplementedError

I have no opinions on this or other API details. But please make the
function always exist and return something vaguely resembling a
monotonic real-time clock. (BTW IMO the docs should state explicitly
that it returns a float.)

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

tismer at stackless

Mar 13, 2012, 10:24 PM

Post #14 of 67 (848 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

The performancecounter is a thing that typically gets intercepted by
the VM infrastructure and does no longer work as a reliable timing
source. In PyPy there are tests which check certain assumptions
how much the performancecounter must advance at least between
a few opcodes, and that does not work in a VM.

cheers - Chris

On 3/13/12 6:11 PM, Kristj�n Valur J�nsson wrote:
> Interesting thought.
> Althougn I don't see how that could fail on windows, if the QPC function is really just talking to a clock chip, surely that hasn't been virtualized.
> Is there an actual example of windows hardware where this api fails (virtual or not?) Perhaps there is no real need to have a fallback mechanism, and it would even be best to write such a mechanism inside the function itself, and just return getsystemtimeasfiletime() instead.
>
> K
>
> -----Original Message-----
> From: Christian Tismer [mailto:tismer [at] stackless]
> Sent: 13. mars 2012 18:07
> To: Kristj�n Valur J�nsson
> Cc: Guido van Rossum; Victor Stinner; Python Dev
> Subject: Re: [Python-Dev] Drop the new time.wallclock() function?
>
> Btw., have you considered virtual machines?
> I happen to run windows in Parallels or Virtualbox quite often.
> There the "wall clock" stuff notoriously does not work.
>
> It would be good (but difficult?) if the supposed-to-be-accurate clock could test itself, if it works at all, and replace itself with a fallback.
>
> In my case, this causes quite a few PyPy tests to fail ;-)
>
> ciao -- Chris
>

--
Christian Tismer :^)<mailto:tismer [at] stackless>
tismerysoft GmbH : Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/
14482 Potsdam : PGP key -> http://pgp.uni-mainz.de
work +49 173 24 18 776 mobile +49 173 24 18 776 fax n.a.
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

jyasskin at gmail

Mar 13, 2012, 10:42 PM

Post #15 of 67 (844 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

On Tue, Mar 13, 2012 at 5:03 PM, Michael Foord
<fuzzyman [at] voidspace> wrote:
>
> On 13 Mar 2012, at 16:57, Victor Stinner wrote:
>
>> Hi,
>>
>> I added two functions to the time module in Python 3.3: wallclock()
>> and monotonic(). I'm unable to explain the difference between these
>> two functions, even if I wrote them :-) wallclock() is suppose to be
>> more accurate than time() but has an unspecified starting point.
>> monotonic() is similar except that it is monotonic: it cannot go
>> backward. monotonic() may not be available or fail whereas wallclock()
>> is available/work, but I think that the two functions are redundant.
>>
>> I prefer to keep only monotonic() because it is not affected by system
>> clock update and should help to fix issues on NTP update in functions
>> implementing a timeout.
>>
>> What do you think?
>>
>
>
> I am in the middle of adding a feature to unittest that involves timing of individual tests. I want the highest resolution cross platform measure of wallclock time - and time.wallclock() looked ideal. If monotonic may not exist or can fail why would that be better?
>

Isn't the highest resolution cross platform measure of "wallclock"
time spelled "time.clock()"? Its docs say "this is the function to use
for benchmarking Python or timing algorithms", and it would be a shame
to add and teach a new function rather than improving clock()'s
definition.

Jeffrey
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

jyasskin at gmail

Mar 13, 2012, 11:16 PM

Post #16 of 67 (849 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

On Tue, Mar 13, 2012 at 6:10 PM, Nadeem Vawda <nadeem.vawda [at] gmail> wrote:
> On Wed, Mar 14, 2012 at 3:03 AM, Victor Stinner
> <victor.stinner [at] gmail> wrote:
>> I suppose that most libraries and programs will have to implement a
>> similar fallback.
>>
>> We may merge both functions with a flag to be able to disable the
>> fallback. Example:
>>
>> - time.realtime(): best-effort monotonic, with a fallback
>> - time.realtime(monotonic=True): monotonic, may raise OSError or
>> NotImplementedError
>
> This was my suggestion - I think it's useful to have the fallback
> available (since most users will want it), but at the same time we
> should also cater to users who need a clock that is *guaranteed* to
> be monotonic.
>
> As an aside, I think "monotonic" is a better name than "realtime";
> it conveys the functions purpose more clearly. Then we could call
> the flag "strict".

While you're bikeshedding:

Some of the drafts of the new C++ standard had a monotonic_clock,
which was guaranteed to only go forwards, but which could be affected
by system clock updates that went forwards. Because of problems in
defining timeouts using an adjustable clock, C++11 instead defines a
"steady_clock", which ticks as steadily as the machine/OS/library can
ensure, and is definitely not affected by any time adjustments:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2010/n3191.htm. I
realize that Unix's clock_gettime(CLOCK_MONOTONIC) already includes
the steadiness criterion, but the word itself doesn't actually include
the meaning.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

kristjan at ccpgames

Mar 13, 2012, 11:32 PM

Post #17 of 67 (847 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

To quote:
"On Unix, return the current processor time as a floating point number expressed in seconds. The precision, and in fact the very definition of the meaning of "processor time", depends on that of the C function of the same name,"

The problem is that it is defined to return "processor time." This is historical baggage that comes from just writing a python wrapper around the unix "clock" function. Of course, "processor time" is quite useless when one is trying to write timeout algorithms or other such things that need to time out in real time, not just cpu cycles.
K

-----Original Message-----
From: python-dev-bounces+kristjan=ccpgames.com [at] python [mailto:python-dev-bounces+kristjan=ccpgames.com [at] python] On Behalf Of Jeffrey Yasskin
Sent: 13. mars 2012 22:42
To: Michael Foord
Cc: Python Dev
Subject: Re: [Python-Dev] Drop the new time.wallclock() function?

Isn't the highest resolution cross platform measure of "wallclock"
time spelled "time.clock()"? Its docs say "this is the function to use for benchmarking Python or timing algorithms", and it would be a shame to add and teach a new function rather than improving clock()'s definition.

Jeffrey
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Mar 14, 2012, 2:16 AM

Post #18 of 67 (828 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

On Wed, 14 Mar 2012 02:03:42 +0100
Victor Stinner <victor.stinner [at] gmail> wrote:
>
> We may merge both functions with a flag to be able to disable the
> fallback. Example:
>
> - time.realtime(): best-effort monotonic, with a fallback
> - time.realtime(monotonic=True): monotonic, may raise OSError or
> NotImplementedError

That's a rather awful name. time.time() is *the* real time.

time.monotonic(fallback=False) would be a better API.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

stefan at bytereef

Mar 14, 2012, 2:30 AM

Post #19 of 67 (826 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

Antoine Pitrou <solipsis [at] pitrou> wrote:
> time.monotonic(fallback=False) would be a better API.

+1

Stefan Krah

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

victor.stinner at gmail

Mar 14, 2012, 5:27 AM

Post #20 of 67 (827 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

2012/3/14 Antoine Pitrou <solipsis [at] pitrou>:
> On Wed, 14 Mar 2012 02:03:42 +0100
> Victor Stinner <victor.stinner [at] gmail> wrote:
>>
>> We may merge both functions with a flag to be able to disable the
>> fallback. Example:
>>
>> - time.realtime(): best-effort monotonic, with a fallback
>> - time.realtime(monotonic=True): monotonic, may raise OSError or
>> NotImplementedError
>
> That's a rather awful name. time.time() is *the* real time.
>
> time.monotonic(fallback=False) would be a better API.

I would prefer to enable the fallback by default with a warning in the
doc, just because it is more convinient and it is what user want even
if they don't know that they need a fallback :-)

Enabling the fallback by default allow to write such simple code:

try:
from time import monotonic as get_time
except ImportError:
# Python < 3.3
from time import time as get_time

Use time.monotonic(strict=True) if you need a truly monotonic clock.

monotonic() may not be the best name in this case. Jeffrey Yasskin
proposed time.steady_clock(), so time.steady_clock(monotonic=False)?

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Mar 14, 2012, 5:27 AM

Post #21 of 67 (827 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

On Wed, 14 Mar 2012 13:27:19 +0100
Victor Stinner <victor.stinner [at] gmail> wrote:
>
> monotonic() may not be the best name in this case. Jeffrey Yasskin
> proposed time.steady_clock(), so time.steady_clock(monotonic=False)?

I don't know what "steady" is supposed to mean here, so perhaps the
best solution is to improve the doc?

Also, "monotonic=False" implies that it won't be monotonic, which is
false.

Regards

Antoine.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

anacrolix at gmail

Mar 14, 2012, 9:11 AM

Post #22 of 67 (817 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

I have some observations regarding this:

Victor's existing time.monotonic and time.wallclock make use of
QueryPerformanceCounter, and CLOCK_MONOTONIC_RAW as possible. Both of
these are hardware-based counters, their monotonicity is just a
convenient property of the timer sources. Furthermore, time values can
actually vary depending on the processor the call is served on.
time.hardware()? time.monotonic_raw()?

There are bug reports on Linux that CLOCK_MONOTONIC isn't always
monotonic. This is why CLOCK_MONOTONIC_RAW was created. There's also
the issue of time leaps (forward), which also isn't a problem with the
latter form. time.monotonic(raw_only=False)?

The real value of "monotonic" timers is that they don't leap
backwards, and preferably don't leap forwards. Whether they are
absolute is of no consequence. I would suggest that the API reflect
this, and that more specific time values be obtained using the proper
raw syscall wrapper (like time.clock_gettime) if required.
time.relative(strict=False)?

The ultimate use of the function name being established is for use in
timeouts and relative timings.

Where an option is present, it disallows fallbacks like
CLOCK_MONOTONIC and other weaker forms:
* time.hardware(fallback=True) -> reflects the source of the timing
impeccably. alerts users to possible affinity issues
* time.monotonic_raw() -> a bit linux specific...
* time.relative(strict=False) -> matches the use case. a warning
could be put regarding hardware timings
* time.monotonic(raw_only=False) -> closest to the existing
implementation. the keyword name i think is better.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

guido at python

Mar 14, 2012, 9:22 AM

Post #23 of 67 (825 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

I have a totally different observation. Presumably the primary use
case for these timers is to measure real time intervals for the
purpose of profiling various operations. For this purpose we want them
to be as "steady" as possible: tick at a constant rate, don't jump
forward or backward. (And they shouldn't invoke the concept of "CPU"
time -- we already have time.clock() for that, besides it's often
wrong, e.g. you might be measuring some sort of I/O performance.) If
this means that a second as measured by time.time() is sometimes not
the same as a second measured by this timer (due to time.time()
occasionally jumping due to clock adjustments), that's fine with me.
If this means it's unreliable inside a VM, well, it seems that's a
property of the underlying OS timer, and there's not much we can do
about it except letting a knowledgeable user override the timer user.
As for names, I like Jeffrey's idea of having "steady" in the name.

--Guido

On Wed, Mar 14, 2012 at 9:11 AM, Matt Joiner <anacrolix [at] gmail> wrote:
> I have some observations regarding this:
>
> Victor's existing time.monotonic and time.wallclock make use of
> QueryPerformanceCounter, and CLOCK_MONOTONIC_RAW as possible. Both of
> these are hardware-based counters, their monotonicity is just a
> convenient property of the timer sources. Furthermore, time values can
> actually vary depending on the processor the call is served on.
> time.hardware()? time.monotonic_raw()?
>
> There are bug reports on Linux that CLOCK_MONOTONIC isn't always
> monotonic. This is why CLOCK_MONOTONIC_RAW was created. There's also
> the issue of time leaps (forward), which also isn't a problem with the
> latter form. time.monotonic(raw_only=False)?
>
> The real value of "monotonic" timers is that they don't leap
> backwards, and preferably don't leap forwards. Whether they are
> absolute is of no consequence. I would suggest that the API reflect
> this, and that more specific time values be obtained using the proper
> raw syscall wrapper (like time.clock_gettime) if required.
> time.relative(strict=False)?
>
> The ultimate use of the function name being established is for use in
> timeouts and relative timings.
>
> Where an option is present, it disallows fallbacks like
> CLOCK_MONOTONIC and other weaker forms:
> �* time.hardware(fallback=True) -> reflects the source of the timing
> impeccably. alerts users to possible affinity issues
> �* time.monotonic_raw() -> a bit linux specific...
> �* time.relative(strict=False) -> matches the use case. a warning
> could be put regarding hardware timings
> �* time.monotonic(raw_only=False) -> closest to the existing
> implementation. the keyword name i think is better.

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

kristjan at ccpgames

Mar 14, 2012, 9:42 AM

Post #24 of 67 (822 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

Yes, the intended use is relative timings and timeouts.
I think we are complicating things far too much.
1) Do we really need a fallback on windows? Will QPC ever fail?
2) is it a problem for the intended use if we cannot absolutely guarantee that time won't ever tick backwards?

IMHO, we shouldn't compilicate the api, and make whatever best try we can in C. On windows I would do this (pseudocode)

Static last_time = 0
If (QPC_works) time = QueryPerformanceCounter() else time = GetSystemTimeAsFileTime()
if (time > last_time) last_time=time else time = last_time
return time

in other words:
1) use QPC. If the api indicates that it isn't available (does this ever happen in real life?) use a fallback of system time
2) enforce monotonicity with a static. QPC, if the OS doesn"t is buggy (calls cpu counter rather than timer chip) can cause jitter because it is called on different cores.

No options in the api. No nothing. We simply provide the best api possible and some hardware/software combos might be less accurate.

K

-----Original Message-----
From: python-dev-bounces+kristjan=ccpgames.com [at] python [mailto:python-dev-bounces+kristjan=ccpgames.com [at] python] On Behalf Of Matt Joiner
Sent: 14. mars 2012 09:12
To: Antoine Pitrou; Victor Stinner; Guido van Rossum
Cc: python-dev [at] python
Subject: Re: [Python-Dev] Drop the new time.wallclock() function?

I have some observations regarding this:

Victor's existing time.monotonic and time.wallclock make use of QueryPerformanceCounter, and CLOCK_MONOTONIC_RAW as possible. Both of these are hardware-based counters, their monotonicity is just a convenient property of the timer sources. Furthermore, time values can actually vary depending on the processor the call is served on.
time.hardware()? time.monotonic_raw()?

There are bug reports on Linux that CLOCK_MONOTONIC isn't always monotonic. This is why CLOCK_MONOTONIC_RAW was created. There's also the issue of time leaps (forward), which also isn't a problem with the latter form. time.monotonic(raw_only=False)?

The real value of "monotonic" timers is that they don't leap backwards, and preferably don't leap forwards. Whether they are absolute is of no consequence. I would suggest that the API reflect this, and that more specific time values be obtained using the proper raw syscall wrapper (like time.clock_gettime) if required.
time.relative(strict=False)?

The ultimate use of the function name being established is for use in timeouts and relative timings.

Where an option is present, it disallows fallbacks like CLOCK_MONOTONIC and other weaker forms:
* time.hardware(fallback=True) -> reflects the source of the timing impeccably. alerts users to possible affinity issues
* time.monotonic_raw() -> a bit linux specific...
* time.relative(strict=False) -> matches the use case. a warning could be put regarding hardware timings
* time.monotonic(raw_only=False) -> closest to the existing implementation. the keyword name i think is better.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

nadeem.vawda at gmail

Mar 14, 2012, 9:47 AM

Post #25 of 67 (820 views)
Permalink

Re: Drop the new time.wallclock() function? [In reply to]

A summary of the discussion so far, as I've understood it:

- We should have *one* monotonic/steady timer function, using the
sources described in Victor's original post.

- By default, it should fall back to time.time if a better source is
not available, but there should be a flag that can disable this
fallback for users who really *need* a monotonic/steady time source.

- Proposed names for the function:
* monotonic
* steady_clock
* wallclock
* realtime

- Proposed names for the flag controlling fallback behavior:
* strict (=False)
* fallback (=True)
* monotonic (=False)

For the function name, I think monotonic() and steady_clock() convey
the purpose of the function much better than the other two; the term
"wallclock" is actively misleading, and "realtime" seems ambiguous.

For the flag name, I'm -1 on "monotonic" -- it sounds like a flag to
decide whether to use a monotonic time source always or never, while
it actually decides between "always" and "sometimes". I think "strict"
is nicer than "fallback", but I'm fine with either one.

Cheers,
Nadeem
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

1 2 3

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads