Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

[RFC] PEP 418: Add monotonic time, performance counter and process time functions

 

 

First page Previous page 1 2 Next page Last page  View All Python dev RSS feed   Index | Next | Previous | View Threaded


victor.stinner at gmail

Apr 15, 2012, 8:15 AM

Post #1 of 37 (611 views)
Permalink
[RFC] PEP 418: Add monotonic time, performance counter and process time functions

Hi,

Here is a simplified version of the first draft of the PEP 418. The
full version can be read online.
http://www.python.org/dev/peps/pep-0418/

The implementation of the PEP can be found in this issue:
http://bugs.python.org/issue14428

I post a simplified version for readability and to focus on changes
introduced by the PEP. Removed sections: Existing Functions,
Deprecated Function, Glossary, Hardware clocks, Operating system time
functions, System Standby, Links.

---

PEP: 418
Title: Add monotonic time, performance counter and process time functions
Version: f2bb3f74298a
Last-Modified: 2012-04-15 17:06:07 +0200 (Sun, 15 Apr 2012)
Author: Cameron Simpson <cs [at] zip>, Jim Jewett
<jimjjewett [at] gmail>, Victor Stinner <victor.stinner [at] gmail>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 26-March-2012
Python-Version: 3.3

Abstract
========

This PEP proposes to add ``time.get_clock_info(name)``,
``time.monotonic()``, ``time.perf_counter()`` and
``time.process_time()`` functions to Python 3.3.

Rationale
=========

If a program uses the system time to schedule events or to implement
a timeout, it will not run events at the right moment or stop the
timeout too early or too late when the system time is set manually or
adjusted automatically by NTP. A monotonic clock should be used
instead to not be affected by system time updates:
``time.monotonic()``.

To measure the performance of a function, ``time.clock()`` can be used
but it is very different on Windows and on Unix. On Windows,
``time.clock()`` includes time elapsed during sleep, whereas it does
not on Unix. ``time.clock()`` precision is very good on Windows, but
very bad on Unix. The new ``time.perf_counter()`` function should be
used instead to always get the most precise performance counter with a
portable behaviour (ex: include time spend during sleep).

To measure CPU time, Python does not provide directly a portable
function. ``time.clock()`` can be used on Unix, but it has a bad
precision. ``resource.getrusage()`` can also be used on Unix, but it
requires to get fields of a structure and compute the sum of time
spent in kernel space and user space. The new ``time.process_time()``
function acts as a portable counter that always measures CPU time
(doesn't include time elapsed during sleep) and has the best available
precision.

Each operating system implements clocks and performance counters
differently, and it is useful to know exactly which function is used
and some properties of the clock like its resolution and its
precision. The new ``time.get_clock_info()`` function gives access to
all available information of each Python time function.

New functions:

* ``time.monotonic()``: timeout and scheduling, not affected by system
clock updates
* ``time.perf_counter()``: benchmarking, most precise clock for short
period
* ``time.process_time()``: profiling, CPU time of the process

Users of new functions:

* time.monotonic(): concurrent.futures, multiprocessing, queue, subprocess,
telnet and threading modules to implement timeout
* time.perf_counter(): trace and timeit modules, pybench program
* time.process_time(): profile module
* time.get_clock_info(): pybench program to display information about the
timer like the precision or the resolution

The ``time.clock()`` function is deprecated because it is not
portable: it behaves differently depending on the operating system.
``time.perf_counter()`` or ``time.process_time()`` should be used
instead, depending on your requirements. ``time.clock()`` is marked as
deprecated but is not planned for removal.


Python functions
================

New Functions
-------------

time.get_clock_info(name)
^^^^^^^^^^^^^^^^^^^^^^^^^

Get information on the specified clock. Supported clock names:

* ``"clock"``: ``time.clock()``
* ``"monotonic"``: ``time.monotonic()``
* ``"perf_counter"``: ``time.perf_counter()``
* ``"process_time"``: ``time.process_time()``
* ``"time"``: ``time.time()``

Return a dictionary with the following keys:

* Mandatory keys:

* ``"implementation"`` (str): name of the underlying operating system
function. Examples: ``"QueryPerformanceCounter()"``,
``"clock_gettime(CLOCK_REALTIME)"``.
* ``"resolution"`` (float): resolution in seconds of the clock.
* ``"is_monotonic"`` (bool): True if the clock cannot go backward.

* Optional keys:

* ``"precision"`` (float): precision in seconds of the clock
reported by the operating system.
* ``"is_adjusted"`` (bool): True if the clock is adjusted (e.g. by a
NTP daemon).


time.monotonic()
^^^^^^^^^^^^^^^^

Monotonic clock, i.e. cannot go backward. It is not affected by system
clock updates. The reference point of the returned value is
undefined, so that only the difference between the results of
consecutive calls is valid and is a number of seconds.

On Windows versions older than Vista, ``time.monotonic()`` detects
``GetTickCount()`` integer overflow (32 bits, roll-over after 49.7
days): it increases a delta by 2\ :sup:`32` each time than an overflow
is detected. The delta is stored in the process-local state and so
the value of ``time.monotonic()`` may be different in two Python
processes running for more than 49 days. On more recent versions of
Windows and on other operating systems, ``time.monotonic()`` is
system-wide.

Availability: Windows, Mac OS X, Unix, Solaris. Not available on
GNU/Hurd.

Pseudo-code [#pseudo]_::

if os.name == 'nt':
# GetTickCount64() requires Windows Vista, Server 2008 or later
if hasattr(time, '_GetTickCount64'):
def monotonic():
return _time.GetTickCount64() * 1e-3
else:
def monotonic():
ticks = _time.GetTickCount()
if ticks < monotonic.last:
# Integer overflow detected
monotonic.delta += 2**32
monotonic.last = ticks
return (ticks + monotonic.delta) * 1e-3
monotonic.last = 0
monotonic.delta = 0

elif os.name == 'mac':
def monotonic():
if monotonic.factor is None:
factor = _time.mach_timebase_info()
monotonic.factor = timebase[0] / timebase[1]
return _time.mach_absolute_time() * monotonic.factor
monotonic.factor = None

elif hasattr(time, "clock_gettime") and hasattr(time, "CLOCK_HIGHRES"):
def monotonic():
return time.clock_gettime(time.CLOCK_HIGHRES)

elif hasattr(time, "clock_gettime") and hasattr(time, "CLOCK_MONOTONIC"):
def monotonic():
return time.clock_gettime(time.CLOCK_MONOTONIC)


On Windows, ``QueryPerformanceCounter()`` is not used even though it
has a better precision than ``GetTickCount()``. It is not reliable
and has too many issues.


time.perf_counter()
^^^^^^^^^^^^^^^^^^^

Performance counter with the highest available precision to measure a
duration. It does include time elapsed during sleep and is
system-wide. The reference point of the returned value is undefined,
so that only the difference between the results of consecutive calls
is valid and is a number of seconds.

Pseudo-code::

def perf_counter():
if perf_counter.use_performance_counter:
if perf_counter.performance_frequency is None:
try:
perf_counter.performance_frequency =
_time.QueryPerformanceFrequency()
except OSError:
# QueryPerformanceFrequency() fails if the installed
# hardware does not support a high-resolution performance
# counter
perf_counter.use_performance_counter = False
else:
return _time.QueryPerformanceCounter() /
perf_counter.performance_frequency
else:
return _time.QueryPerformanceCounter() /
perf_counter.performance_frequency
if perf_counter.use_monotonic:
# The monotonic clock is preferred over the system time
try:
return time.monotonic()
except OSError:
perf_counter.use_monotonic = False
return time.time()
perf_counter.use_performance_counter = (os.name == 'nt')
if perf_counter.use_performance_counter:
perf_counter.performance_frequency = None
perf_counter.use_monotonic = hasattr(time, 'monotonic')


time.process_time()
^^^^^^^^^^^^^^^^^^^

Sum of the system and user CPU time of the current process. It does
not include time elapsed during sleep. It is process-wide by
definition. The reference point of the returned value is undefined,
so that only the difference between the results of consecutive calls
is valid.

It is available on all platforms.

Pseudo-code [#pseudo]_::

if os.name == 'nt':
def process_time():
handle = win32process.GetCurrentProcess()
process_times = win32process.GetProcessTimes(handle)
return (process_times['UserTime'] +
process_times['KernelTime']) * 1e-7
else:
import os
try:
import resource
except ImportError:
has_resource = False
else:
has_resource = True

def process_time():
if process_time.use_process_cputime:
try:
return time.clock_gettime(time.CLOCK_PROCESS_CPUTIME_ID)
except OSError:
process_time.use_process_cputime = False
if process_time.use_getrusage:
try:
usage = resource.getrusage(resource.RUSAGE_SELF)
return usage[0] + usage[1]
except OSError:
process_time.use_getrusage = False
if process_time.use_times:
try:
times = os.times()
return times[0] + times[1]
except OSError:
process_time.use_getrusage = False
return _time.clock()
process_time.use_process_cputime = (
hasattr(time, 'clock_gettime')
and hasattr(time, 'CLOCK_PROCESS_CPUTIME_ID'))
process_time.use_getrusage = has_resource
# On OS/2, only the 5th field of os.times() is set, others are zeros
process_time.use_times = (hasattr(os, 'times') and os.name != 'os2')


Alternatives: API design
========================

Other names for time.monotonic()
--------------------------------

* time.counter()
* time.metronomic()
* time.seconds()
* time.steady(): "steady" is ambiguous: it means different things to
different people. For example, on Linux, CLOCK_MONOTONIC is
adjusted. If we uses the real time as the reference clock, we may
say that CLOCK_MONOTONIC is steady. But CLOCK_MONOTONIC gets
suspended on system suspend, whereas real time includes any time
spent in suspend.
* time.timeout_clock()
* time.wallclock(): time.monotonic() is not the system time aka the
"wall clock", but a monotonic clock with an unspecified starting
point.

The name "time.try_monotonic()" was also proposed for an older
proposition of time.monotonic() which was falling back to the system
time when no monotonic clock was available.

Other names for time.perf_counter()
-----------------------------------

* time.hires()
* time.highres()
* time.timer()

Only expose operating system clocks
-----------------------------------

To not have to define high-level clocks, which is a difficult task, a
simpler approach is to only expose operating system clocks.
time.clock_gettime() and related clock identifiers were already added
to Python 3.3 for example.


time.monotonic(): Fallback to system time
-----------------------------------------

If no monotonic clock is available, time.monotonic() falls back to the
system time.

Issues:

* It is hard to define correctly such function in the documentation:
is it monotonic? Is it steady? Is it adjusted?
* Some user want to decide what to do when no monotonic clock is
available: use another clock, display an error, or do something
else?

Different APIs were proposed to define such function.

One function with a flag: time.monotonic(fallback=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* time.monotonic(fallback=True) falls back to the system time if no
monotonic clock is available or if the monotonic clock failed.
* time.monotonic(fallback=False) raises OSError if monotonic clock
fails and NotImplementedError if the system does not provide a
monotonic clock

A keyword argument that gets passed as a constant in the caller is
usually poor API.

Raising NotImplementedError for a function is something uncommon in
Python and should be avoided.


One time.monotonic() function, no flag
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

time.monotonic() returns (time: float, is_monotonic: bool).

An alternative is to use a function attribute:
time.monotonic.is_monotonic. The attribute value would be None before
the first call to time.monotonic().


Choosing the clock from a list of constraints
---------------------------------------------

The PEP as proposed offers a few new clocks, but their guarentees
are deliberately loose in order to offer useful clocks on different
platforms. This inherently embeds policy in the calls, and the
caller must thus choose a policy.

The "choose a clock" approach suggests an additional API to let
callers implement their own policy if necessary
by making most platform clocks available and letting the caller pick
amongst them.
The PEP's suggested clocks are still expected to be available for the common
simple use cases.

To do this two facilities are needed:
an enumeration of clocks, and metadata on the clocks to enable the user to
evaluate their suitability.

The primary interface is a function make simple choices easy:
the caller can use ``time.get_clock(*flags)`` with some combination of flags.
This include at least:

* time.MONOTONIC: clock cannot go backward
* time.STEADY: clock rate is steady
* time.ADJUSTED: clock may be adjusted, for example by NTP
* time.HIGHRES: clock with the highest precision

It returns a clock object with a .now() method returning the current time.
The clock object is annotated with metadata describing the clock feature set;
its .flags field will contain at least all the requested flags.

time.get_clock() returns None if no matching clock is found and so calls can
be chained using the or operator. Example of a simple policy decision::

T = get_clock(MONOTONIC) or get_clock(STEADY) or get_clock()
t = T.now()

The available clocks always at least include a wrapper for ``time.time()``,
so a final call with no flags can always be used to obtain a working clock.

Example of flags of system clocks:

* QueryPerformanceCounter: MONOTONIC | HIGHRES
* GetTickCount: MONOTONIC | STEADY
* CLOCK_MONOTONIC: MONOTONIC | STEADY (or only MONOTONIC on Linux)
* CLOCK_MONOTONIC_RAW: MONOTONIC | STEADY
* gettimeofday(): (no flag)

The clock objects contain other metadata including the clock flags
with additional feature flags above those listed above, the name
of the underlying OS facility, and clock precisions.

``time.get_clock()`` still chooses a single clock; an enumeration
facility is also required.
The most obvious method is to offer ``time.get_clocks()`` with the
same signature as ``time.get_clock()``, but returning a sequence
of all clocks matching the requested flags.
Requesting no flags would thus enumerate all available clocks,
allowing the caller to make an arbitrary choice amongst them based
on their metadata.

Example partial implementation:
`clockutils.py <http://hg.python.org/peps/file/tip/pep-0418/clockutils.py>`_.

Working around operating system bugs?
-------------------------------------

Should Python ensure manually that a monotonic clock is truly
monotonic by computing the maximum with the clock value and the
previous value?

Since it's relatively straightforward to cache the last value returned
using a static variable, it might be interesting to use this to make
sure that the values returned are indeed monotonic.

* Virtual machines provide less reliable clocks.
* QueryPerformanceCounter() has known bugs (only one is not fixed yet)

Python may only work around a specific known operating system bug:
`KB274323`_ contains a code example to workaround the bug (use
GetTickCount() to detect QueryPerformanceCounter() leap).

Issues of a hacked monotonic function:

* if the clock is accidentally set forward by an hour and then back
again, you wouldn't have a useful clock for an hour
* the cache is not shared between processes so different processes
wouldn't see the same clock value
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


victor.stinner at gmail

Apr 15, 2012, 8:18 AM

Post #2 of 37 (587 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

> Here is a simplified version of the first draft of the PEP 418. The
> full version can be read online.
> http://www.python.org/dev/peps/pep-0418/

FYI there is no time.thread_time() function. It would only be
available on Windows and Linux. It does not use seconds but CPU
cycles. No module or program of the Python source code need such
function, whereas all other functions added by the PEP already have
users in the Python source code, see the Rationale section. For Linux,
CLOCK_THREAD_CPUTIME_ID is already available in Python 3.3. For
Windows, you can get GetThreadTimes() using ctypes or win32process.

> time.process_time()
> ^^^^^^^^^^^^^^^^^^^
>
> Pseudo-code [#pseudo]_::
>
>    if os.name == 'nt':
>        def process_time():
>            handle = win32process.GetCurrentProcess()
>            process_times = win32process.GetProcessTimes(handle)
>            return (process_times['UserTime'] +
> process_times['KernelTime']) * 1e-7
>    else:
>        import os
>        ...
>
>        def process_time():
>            ...
>            return _time.clock()

Is the C clock() function available on all platforms? timemodule.c
checks for HAVE_CLOCK, but test_time checks that time.clock() is
defined and does not fail since the changeset 4de05cbf5ad1, Dec 06
1996. If clock() is not available on all platforms,
time.process_time() documentation should be fixed.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


mal at egenix

Apr 15, 2012, 8:36 AM

Post #3 of 37 (586 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

Victor Stinner wrote:
> Hi,
>
> Here is a simplified version of the first draft of the PEP 418. The
> full version can be read online.
> http://www.python.org/dev/peps/pep-0418/
>
> The implementation of the PEP can be found in this issue:
> http://bugs.python.org/issue14428
>
> I post a simplified version for readability and to focus on changes
> introduced by the PEP. Removed sections: Existing Functions,
> Deprecated Function, Glossary, Hardware clocks, Operating system time
> functions, System Standby, Links.

Looks good.

I'd suggest to also include a tool or API to determine the
real resolution of a time function (as opposed to the advertised
one). See pybench's clockres.py helper as example. You often
find large differences between the advertised resolution and
the available one, e.g. while process timers often advertise
very good resolution, they are in fact often only updated
at very coarse rates.

E.g. compare the results of clockres.py on Linux:

Clock resolution of various timer implementations:
time.clock: 10000.000us
time.time: 0.954us
systimes.processtime: 999.000us

and FreeBSD:

Clock resolution of various timer implementations:
time.clock: 7812.500us
time.time: 1.907us
systimes.processtime: 1.000us

and Mac OS X:

Clock resolution of various timer implementations:
time.clock: 1.000us
time.time: 0.954us
systimes.processtime: 1.000us

Regarding changing pybench:
pybench has to stay backwards incompatible with
earlier releases to make it possible to compare timings.
You can add support for new timers to pybench, but please leave
the existing timers and defaults in place.

> ---
>
> PEP: 418
> Title: Add monotonic time, performance counter and process time functions
> Version: f2bb3f74298a
> Last-Modified: 2012-04-15 17:06:07 +0200 (Sun, 15 Apr 2012)
> Author: Cameron Simpson <cs [at] zip>, Jim Jewett
> <jimjjewett [at] gmail>, Victor Stinner <victor.stinner [at] gmail>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 26-March-2012
> Python-Version: 3.3
>
> Abstract
> ========
>
> This PEP proposes to add ``time.get_clock_info(name)``,
> ``time.monotonic()``, ``time.perf_counter()`` and
> ``time.process_time()`` functions to Python 3.3.
>
> Rationale
> =========
>
> If a program uses the system time to schedule events or to implement
> a timeout, it will not run events at the right moment or stop the
> timeout too early or too late when the system time is set manually or
> adjusted automatically by NTP. A monotonic clock should be used
> instead to not be affected by system time updates:
> ``time.monotonic()``.
>
> To measure the performance of a function, ``time.clock()`` can be used
> but it is very different on Windows and on Unix. On Windows,
> ``time.clock()`` includes time elapsed during sleep, whereas it does
> not on Unix. ``time.clock()`` precision is very good on Windows, but
> very bad on Unix. The new ``time.perf_counter()`` function should be
> used instead to always get the most precise performance counter with a
> portable behaviour (ex: include time spend during sleep).
>
> To measure CPU time, Python does not provide directly a portable
> function. ``time.clock()`` can be used on Unix, but it has a bad
> precision. ``resource.getrusage()`` can also be used on Unix, but it
> requires to get fields of a structure and compute the sum of time
> spent in kernel space and user space. The new ``time.process_time()``
> function acts as a portable counter that always measures CPU time
> (doesn't include time elapsed during sleep) and has the best available
> precision.
>
> Each operating system implements clocks and performance counters
> differently, and it is useful to know exactly which function is used
> and some properties of the clock like its resolution and its
> precision. The new ``time.get_clock_info()`` function gives access to
> all available information of each Python time function.
>
> New functions:
>
> * ``time.monotonic()``: timeout and scheduling, not affected by system
> clock updates
> * ``time.perf_counter()``: benchmarking, most precise clock for short
> period
> * ``time.process_time()``: profiling, CPU time of the process
>
> Users of new functions:
>
> * time.monotonic(): concurrent.futures, multiprocessing, queue, subprocess,
> telnet and threading modules to implement timeout
> * time.perf_counter(): trace and timeit modules, pybench program
> * time.process_time(): profile module
> * time.get_clock_info(): pybench program to display information about the
> timer like the precision or the resolution
>
> The ``time.clock()`` function is deprecated because it is not
> portable: it behaves differently depending on the operating system.
> ``time.perf_counter()`` or ``time.process_time()`` should be used
> instead, depending on your requirements. ``time.clock()`` is marked as
> deprecated but is not planned for removal.
>
>
> Python functions
> ================
>
> New Functions
> -------------
>
> time.get_clock_info(name)
> ^^^^^^^^^^^^^^^^^^^^^^^^^
>
> Get information on the specified clock. Supported clock names:
>
> * ``"clock"``: ``time.clock()``
> * ``"monotonic"``: ``time.monotonic()``
> * ``"perf_counter"``: ``time.perf_counter()``
> * ``"process_time"``: ``time.process_time()``
> * ``"time"``: ``time.time()``
>
> Return a dictionary with the following keys:
>
> * Mandatory keys:
>
> * ``"implementation"`` (str): name of the underlying operating system
> function. Examples: ``"QueryPerformanceCounter()"``,
> ``"clock_gettime(CLOCK_REALTIME)"``.
> * ``"resolution"`` (float): resolution in seconds of the clock.
> * ``"is_monotonic"`` (bool): True if the clock cannot go backward.
>
> * Optional keys:
>
> * ``"precision"`` (float): precision in seconds of the clock
> reported by the operating system.
> * ``"is_adjusted"`` (bool): True if the clock is adjusted (e.g. by a
> NTP daemon).
>
>
> time.monotonic()
> ^^^^^^^^^^^^^^^^
>
> Monotonic clock, i.e. cannot go backward. It is not affected by system
> clock updates. The reference point of the returned value is
> undefined, so that only the difference between the results of
> consecutive calls is valid and is a number of seconds.
>
> On Windows versions older than Vista, ``time.monotonic()`` detects
> ``GetTickCount()`` integer overflow (32 bits, roll-over after 49.7
> days): it increases a delta by 2\ :sup:`32` each time than an overflow
> is detected. The delta is stored in the process-local state and so
> the value of ``time.monotonic()`` may be different in two Python
> processes running for more than 49 days. On more recent versions of
> Windows and on other operating systems, ``time.monotonic()`` is
> system-wide.
>
> Availability: Windows, Mac OS X, Unix, Solaris. Not available on
> GNU/Hurd.
>
> Pseudo-code [#pseudo]_::
>
> if os.name == 'nt':
> # GetTickCount64() requires Windows Vista, Server 2008 or later
> if hasattr(time, '_GetTickCount64'):
> def monotonic():
> return _time.GetTickCount64() * 1e-3
> else:
> def monotonic():
> ticks = _time.GetTickCount()
> if ticks < monotonic.last:
> # Integer overflow detected
> monotonic.delta += 2**32
> monotonic.last = ticks
> return (ticks + monotonic.delta) * 1e-3
> monotonic.last = 0
> monotonic.delta = 0
>
> elif os.name == 'mac':
> def monotonic():
> if monotonic.factor is None:
> factor = _time.mach_timebase_info()
> monotonic.factor = timebase[0] / timebase[1]
> return _time.mach_absolute_time() * monotonic.factor
> monotonic.factor = None
>
> elif hasattr(time, "clock_gettime") and hasattr(time, "CLOCK_HIGHRES"):
> def monotonic():
> return time.clock_gettime(time.CLOCK_HIGHRES)
>
> elif hasattr(time, "clock_gettime") and hasattr(time, "CLOCK_MONOTONIC"):
> def monotonic():
> return time.clock_gettime(time.CLOCK_MONOTONIC)
>
>
> On Windows, ``QueryPerformanceCounter()`` is not used even though it
> has a better precision than ``GetTickCount()``. It is not reliable
> and has too many issues.
>
>
> time.perf_counter()
> ^^^^^^^^^^^^^^^^^^^
>
> Performance counter with the highest available precision to measure a
> duration. It does include time elapsed during sleep and is
> system-wide. The reference point of the returned value is undefined,
> so that only the difference between the results of consecutive calls
> is valid and is a number of seconds.
>
> Pseudo-code::
>
> def perf_counter():
> if perf_counter.use_performance_counter:
> if perf_counter.performance_frequency is None:
> try:
> perf_counter.performance_frequency =
> _time.QueryPerformanceFrequency()
> except OSError:
> # QueryPerformanceFrequency() fails if the installed
> # hardware does not support a high-resolution performance
> # counter
> perf_counter.use_performance_counter = False
> else:
> return _time.QueryPerformanceCounter() /
> perf_counter.performance_frequency
> else:
> return _time.QueryPerformanceCounter() /
> perf_counter.performance_frequency
> if perf_counter.use_monotonic:
> # The monotonic clock is preferred over the system time
> try:
> return time.monotonic()
> except OSError:
> perf_counter.use_monotonic = False
> return time.time()
> perf_counter.use_performance_counter = (os.name == 'nt')
> if perf_counter.use_performance_counter:
> perf_counter.performance_frequency = None
> perf_counter.use_monotonic = hasattr(time, 'monotonic')
>
>
> time.process_time()
> ^^^^^^^^^^^^^^^^^^^
>
> Sum of the system and user CPU time of the current process. It does
> not include time elapsed during sleep. It is process-wide by
> definition. The reference point of the returned value is undefined,
> so that only the difference between the results of consecutive calls
> is valid.
>
> It is available on all platforms.
>
> Pseudo-code [#pseudo]_::
>
> if os.name == 'nt':
> def process_time():
> handle = win32process.GetCurrentProcess()
> process_times = win32process.GetProcessTimes(handle)
> return (process_times['UserTime'] +
> process_times['KernelTime']) * 1e-7
> else:
> import os
> try:
> import resource
> except ImportError:
> has_resource = False
> else:
> has_resource = True
>
> def process_time():
> if process_time.use_process_cputime:
> try:
> return time.clock_gettime(time.CLOCK_PROCESS_CPUTIME_ID)
> except OSError:
> process_time.use_process_cputime = False
> if process_time.use_getrusage:
> try:
> usage = resource.getrusage(resource.RUSAGE_SELF)
> return usage[0] + usage[1]
> except OSError:
> process_time.use_getrusage = False
> if process_time.use_times:
> try:
> times = os.times()
> return times[0] + times[1]
> except OSError:
> process_time.use_getrusage = False
> return _time.clock()
> process_time.use_process_cputime = (
> hasattr(time, 'clock_gettime')
> and hasattr(time, 'CLOCK_PROCESS_CPUTIME_ID'))
> process_time.use_getrusage = has_resource
> # On OS/2, only the 5th field of os.times() is set, others are zeros
> process_time.use_times = (hasattr(os, 'times') and os.name != 'os2')
>
>
> Alternatives: API design
> ========================
>
> Other names for time.monotonic()
> --------------------------------
>
> * time.counter()
> * time.metronomic()
> * time.seconds()
> * time.steady(): "steady" is ambiguous: it means different things to
> different people. For example, on Linux, CLOCK_MONOTONIC is
> adjusted. If we uses the real time as the reference clock, we may
> say that CLOCK_MONOTONIC is steady. But CLOCK_MONOTONIC gets
> suspended on system suspend, whereas real time includes any time
> spent in suspend.
> * time.timeout_clock()
> * time.wallclock(): time.monotonic() is not the system time aka the
> "wall clock", but a monotonic clock with an unspecified starting
> point.
>
> The name "time.try_monotonic()" was also proposed for an older
> proposition of time.monotonic() which was falling back to the system
> time when no monotonic clock was available.
>
> Other names for time.perf_counter()
> -----------------------------------
>
> * time.hires()
> * time.highres()
> * time.timer()
>
> Only expose operating system clocks
> -----------------------------------
>
> To not have to define high-level clocks, which is a difficult task, a
> simpler approach is to only expose operating system clocks.
> time.clock_gettime() and related clock identifiers were already added
> to Python 3.3 for example.
>
>
> time.monotonic(): Fallback to system time
> -----------------------------------------
>
> If no monotonic clock is available, time.monotonic() falls back to the
> system time.
>
> Issues:
>
> * It is hard to define correctly such function in the documentation:
> is it monotonic? Is it steady? Is it adjusted?
> * Some user want to decide what to do when no monotonic clock is
> available: use another clock, display an error, or do something
> else?
>
> Different APIs were proposed to define such function.
>
> One function with a flag: time.monotonic(fallback=True)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> * time.monotonic(fallback=True) falls back to the system time if no
> monotonic clock is available or if the monotonic clock failed.
> * time.monotonic(fallback=False) raises OSError if monotonic clock
> fails and NotImplementedError if the system does not provide a
> monotonic clock
>
> A keyword argument that gets passed as a constant in the caller is
> usually poor API.
>
> Raising NotImplementedError for a function is something uncommon in
> Python and should be avoided.
>
>
> One time.monotonic() function, no flag
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> time.monotonic() returns (time: float, is_monotonic: bool).
>
> An alternative is to use a function attribute:
> time.monotonic.is_monotonic. The attribute value would be None before
> the first call to time.monotonic().
>
>
> Choosing the clock from a list of constraints
> ---------------------------------------------
>
> The PEP as proposed offers a few new clocks, but their guarentees
> are deliberately loose in order to offer useful clocks on different
> platforms. This inherently embeds policy in the calls, and the
> caller must thus choose a policy.
>
> The "choose a clock" approach suggests an additional API to let
> callers implement their own policy if necessary
> by making most platform clocks available and letting the caller pick
> amongst them.
> The PEP's suggested clocks are still expected to be available for the common
> simple use cases.
>
> To do this two facilities are needed:
> an enumeration of clocks, and metadata on the clocks to enable the user to
> evaluate their suitability.
>
> The primary interface is a function make simple choices easy:
> the caller can use ``time.get_clock(*flags)`` with some combination of flags.
> This include at least:
>
> * time.MONOTONIC: clock cannot go backward
> * time.STEADY: clock rate is steady
> * time.ADJUSTED: clock may be adjusted, for example by NTP
> * time.HIGHRES: clock with the highest precision
>
> It returns a clock object with a .now() method returning the current time.
> The clock object is annotated with metadata describing the clock feature set;
> its .flags field will contain at least all the requested flags.
>
> time.get_clock() returns None if no matching clock is found and so calls can
> be chained using the or operator. Example of a simple policy decision::
>
> T = get_clock(MONOTONIC) or get_clock(STEADY) or get_clock()
> t = T.now()
>
> The available clocks always at least include a wrapper for ``time.time()``,
> so a final call with no flags can always be used to obtain a working clock.
>
> Example of flags of system clocks:
>
> * QueryPerformanceCounter: MONOTONIC | HIGHRES
> * GetTickCount: MONOTONIC | STEADY
> * CLOCK_MONOTONIC: MONOTONIC | STEADY (or only MONOTONIC on Linux)
> * CLOCK_MONOTONIC_RAW: MONOTONIC | STEADY
> * gettimeofday(): (no flag)
>
> The clock objects contain other metadata including the clock flags
> with additional feature flags above those listed above, the name
> of the underlying OS facility, and clock precisions.
>
> ``time.get_clock()`` still chooses a single clock; an enumeration
> facility is also required.
> The most obvious method is to offer ``time.get_clocks()`` with the
> same signature as ``time.get_clock()``, but returning a sequence
> of all clocks matching the requested flags.
> Requesting no flags would thus enumerate all available clocks,
> allowing the caller to make an arbitrary choice amongst them based
> on their metadata.
>
> Example partial implementation:
> `clockutils.py <http://hg.python.org/peps/file/tip/pep-0418/clockutils.py>`_.
>
> Working around operating system bugs?
> -------------------------------------
>
> Should Python ensure manually that a monotonic clock is truly
> monotonic by computing the maximum with the clock value and the
> previous value?
>
> Since it's relatively straightforward to cache the last value returned
> using a static variable, it might be interesting to use this to make
> sure that the values returned are indeed monotonic.
>
> * Virtual machines provide less reliable clocks.
> * QueryPerformanceCounter() has known bugs (only one is not fixed yet)
>
> Python may only work around a specific known operating system bug:
> `KB274323`_ contains a code example to workaround the bug (use
> GetTickCount() to detect QueryPerformanceCounter() leap).
>
> Issues of a hacked monotonic function:
>
> * if the clock is accidentally set forward by an hour and then back
> again, you wouldn't have a useful clock for an hour
> * the cache is not shared between processes so different processes
> wouldn't see the same clock value
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/mal%40egenix.com

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Apr 15 2012)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
2012-04-28: PythonCamp 2012, Cologne, Germany 13 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


victor.stinner at gmail

Apr 15, 2012, 10:40 AM

Post #4 of 37 (577 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

2012/4/15 M.-A. Lemburg <mal [at] egenix>:
> I'd suggest to also include a tool or API to determine the
> real resolution of a time function (as opposed to the advertised
> one). See pybench's clockres.py helper as example.

The PEP includes such tool, but I forgot to mention it in the PEP:
http://hg.python.org/peps/file/tip/pep-0418/clock_precision.py

It is based on clockres.py from pybench. I used this tools to fill the
"Precision in Python" column of the different tables. The "Precision"
is the precision announced by the OS, whereas the "Precision in
Python" is the effictive precision in Python.

The full PEP includes results of different benchmarks: performance of
hardware clocks and performance of the different OS time functions.

> E.g. compare the results of clockres.py on Linux:
>
> Clock resolution of various timer implementations:
> time.clock:            10000.000us
> time.time:                 0.954us
> systimes.processtime:    999.000us
>
> and FreeBSD:
>
> Clock resolution of various timer implementations:
> time.clock:             7812.500us
> time.time:                 1.907us
> systimes.processtime:      1.000us

Cool, I found similar numbers :-)

> and Mac OS X:
>
> Clock resolution of various timer implementations:
> time.clock:                1.000us
> time.time:                 0.954us
> systimes.processtime:      1.000us

I will include these numbers on Mac OS X to the PEP.

> Regarding changing pybench:
> pybench has to stay backwards incompatible with
> earlier releases to make it possible to compare timings.
> You can add support for new timers to pybench, but please leave
> the existing timers and defaults in place.

I suppose that you are talking about this change:

-# Choose platform default timer
-if sys.platform[:3] == 'win':
- # On WinXP this has 2.5ms resolution
- TIMER_PLATFORM_DEFAULT = TIMER_TIME_CLOCK
-else:
- # On Linux this has 1ms resolution
- TIMER_PLATFORM_DEFAULT = TIMER_TIME_TIME
+TIMER_PLATFORM_DEFAULT = TIMER_TIME_PERF_COUNTER

from http://bugs.python.org/file25202/perf_counter_process_time.patch

It does not change the OS clock on Windows, only on Unix:
CLOCK_REALTIME (gettimeofday() for Python 3.2 and earlier) is replaced
with CLOCK_MONOTONIC. This change should only give a different result
if the system time is changed during the benchmark.

I'm ok to keep the default timer if you consider the change as incompatible.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


victor.stinner at gmail

Apr 15, 2012, 4:25 PM

Post #5 of 37 (577 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

> time.perf_counter()
> ^^^^^^^^^^^^^^^^^^^
>
> Performance counter with the highest available precision to measure a
> duration.  It does include time elapsed during sleep and is
> system-wide.  The reference point of the returned value is undefined,
> so that only the difference between the results of consecutive calls
> is valid and is a number of seconds.

It's maybe time for bikeshedding! Glyph wrote me in private:
"IMHO, perf_counter should be performance_counter() or
high_precision(); the abbreviation is silly :)"

The time module has other abbreviated names. I don't have a preference
between time.perf_counter() or time.performance_counter().

Solaris provides CLOCK_HIGHRES, "the nonadjustable, high-resolution
clock." If we map CLOCK_xxx names to functions name, we have:

* CLOCK_MONOTONIC: time.monotonic()
* CLOCK_HIGHRES: time.highres()

(whereas Windows provides QueryPerformanceCounter -> performance_counter)

I suppose that most people don't care that "resolution" and
"precision" are different things.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


anacrolix at gmail

Apr 16, 2012, 2:49 AM

Post #6 of 37 (584 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

This is becoming the Manhattan Project of bike sheds.


victor.stinner at gmail

Apr 16, 2012, 3:38 AM

Post #7 of 37 (573 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

2012/4/16 Matt Joiner <anacrolix [at] gmail>:
> This is becoming the Manhattan Project of bike sheds.

FreeBSD FAQ contains an entry "Why should I care what color the
bikeshed is?" which mention a "sleep(1) should take fractional second
arguments" saga in 1999.
http://www.freebsd.org/doc/en/books/faq/misc.html#BIKESHED-PAINTING

Bikeshedding is maybe a common issue with the discussion around time
function? :-)

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


phd at phdru

Apr 16, 2012, 3:46 AM

Post #8 of 37 (576 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

On Mon, Apr 16, 2012 at 12:38:41PM +0200, Victor Stinner <victor.stinner [at] gmail> wrote:
> Bikeshedding is maybe a common issue with the discussion around time
> function? :-)

Perhaps because everyone of us lives in a different Time-Space
Continuum? ;-)

Oleg.
--
Oleg Broytman http://phdru.name/ phd [at] phdru
Programmers don't die, they just GOSUB without RETURN.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


solipsis at pitrou

Apr 16, 2012, 4:16 AM

Post #9 of 37 (569 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

On Mon, 16 Apr 2012 01:25:42 +0200
Victor Stinner <victor.stinner [at] gmail> wrote:
>
> I suppose that most people don't care that "resolution" and
> "precision" are different things.

Don't they? Actually, they don't care about resolution since they
receive a Python float.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


victor.stinner at gmail

Apr 16, 2012, 4:22 PM

Post #10 of 37 (579 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

> Here is a simplified version of the first draft of the PEP 418. The
> full version can be read online.
> http://www.python.org/dev/peps/pep-0418/

I wrote an implementation in pure Python using ctypes for Python < 3.3:

https://bitbucket.org/haypo/misc/src/tip/python/pep418.py

I tested it on Linux, OpenBSD, FreeBSD and Windows. It's more a
proof-of-concept to test the PEP than a library written to be reused
by programs.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


sumerc at gmail

Apr 17, 2012, 12:23 AM

Post #11 of 37 (562 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

On Sun, Apr 15, 2012 at 6:18 PM, Victor Stinner <victor.stinner [at] gmail>wrote:

> > Here is a simplified version of the first draft of the PEP 418. The
> > full version can be read online.
> > http://www.python.org/dev/peps/pep-0418/
>
> FYI there is no time.thread_time() function. It would only be
> available on Windows and Linux. It does not use seconds but CPU
> cycles. No module or program of the Python source code need such
> function,


Just FYI: in MACOSx, you can use thread_info() to get that information.
Also you can get that information in Solaris,too. In yappi profiler I use
all of these approaches together to have an OS independent thread_times()
functionality. Here is the relevant code:
http://bitbucket.org/sumerc/yappi/src/7c7dc11e8728/timing.c<https://bitbucket.org/sumerc/yappi/src/7c7dc11e8728/timing.c>

I also think that you are right about Python really not have any use case
for this functionality, ...


--
Sümer Cip


victor.stinner at gmail

Apr 17, 2012, 5:06 PM

Post #12 of 37 (570 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

> Here is a simplified version of the first draft of the PEP 418. The
> full version can be read online.
> http://www.python.org/dev/peps/pep-0418/
>
> The implementation of the PEP can be found in this issue:
> http://bugs.python.org/issue14428

The PEP is now fully ready: I just finished the implementation.

It looks like people, who complained on older versions of the PEP,
don't have new complain. Am I wrong? Everybody agree with the PEP 418?

I created http://hg.python.org/features/pep418/ repository for the
implementation. I tested it on Linux 3.3, FreeBSD 8.2, OpenBSD 5.0 and
Windows Seven. The implementation is now waiting your review!

There is also the toy implementation in pure Python for Python < 3.3:
https://bitbucket.org/haypo/misc/src/tip/python/pep418.py

Antoine asked "Is there a designated dictator for this PEP?". Nobody
answered. Maybe Guido van Rossum?

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


guido at python

Apr 17, 2012, 5:50 PM

Post #13 of 37 (575 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

I'll do it. Give me a few days (tomorrow is fully booked with horrible
meetings).

On Tue, Apr 17, 2012 at 5:06 PM, Victor Stinner
<victor.stinner [at] gmail> wrote:
>> Here is a simplified version of the first draft of the PEP 418. The
>> full version can be read online.
>> http://www.python.org/dev/peps/pep-0418/
>>
>> The implementation of the PEP can be found in this issue:
>> http://bugs.python.org/issue14428
>
> The PEP is now fully ready: I just finished the implementation.
>
> It looks like people, who complained on older versions of the PEP,
> don't have new complain. Am I wrong? Everybody agree with the PEP 418?
>
> I created http://hg.python.org/features/pep418/ repository for the
> implementation. I tested it on Linux 3.3, FreeBSD 8.2, OpenBSD 5.0 and
> Windows Seven. The implementation is now waiting your review!
>
> There is also the toy implementation in pure Python for Python < 3.3:
> https://bitbucket.org/haypo/misc/src/tip/python/pep418.py
>
> Antoine asked "Is there a designated dictator for this PEP?". Nobody
> answered. Maybe Guido van Rossum?
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org



--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


victor.stinner at gmail

Apr 20, 2012, 5:38 PM

Post #14 of 37 (554 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

2012/4/15 Victor Stinner <victor.stinner [at] gmail>:
>> Here is a simplified version of the first draft of the PEP 418. The
>> full version can be read online.
>> http://www.python.org/dev/peps/pep-0418/
>
> FYI there is no time.thread_time() function. It would only be
> available on Windows and Linux. It does not use seconds but CPU
> cycles. No module or program of the Python source code need such
> function, whereas all other functions added by the PEP already have
> users in the Python source code, see the Rationale section. For Linux,
> CLOCK_THREAD_CPUTIME_ID is already available in Python 3.3. For
> Windows, you can get GetThreadTimes() using ctypes or win32process.

"Explicit is better than implicit" ! I listed the limitations of the
PEP directly in the PEP:
-------------
Limitations:

* The behaviour of clocks after a system suspend is not defined in the
documentation of new functions. The behaviour depends on the
operating system: see the `Monotonic Clocks`_ section below. Some
recent operating systems provide two clocks, one including time
elapsed during system suspsend, one not including this time. Most
operating systems only provide one kind of clock.
* time.monotonic() and time.perf_counter() may or may not be adjusted.
For example, ``CLOCK_MONOTONIC`` is slewed on Linux, whereas
``GetTickCount()`` is not adjusted on Windows.
``time.get_clock_info('monotonic')['is_adjusted']`` can be used to check
if the monotonic clock is adjusted or not.
* No time.thread_time() function is proposed by this PEP because it is
not needed by Python standard library nor a common asked feature.
Such function would only be available on Windows and Linux. On
Linux, it is possible use use
``time.clock_gettime(CLOCK_THREAD_CPUTIME_ID)``. On Windows, ctypes or
another module can be used to call the ``GetThreadTimes()``
function.
-------------

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


victor.stinner at gmail

Apr 23, 2012, 4:30 PM

Post #15 of 37 (526 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

> Here is a simplified version of the first draft of the PEP 418. The
> full version can be read online.
> http://www.python.org/dev/peps/pep-0418/

Thanks to everyone who helped me to work on this PEP!

I integrated last comments. There is no more open question. (Or did I
miss something?)

I didn't know that it would be so hard to add a such simple function
as time.monotonic()!?

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


guido at python

Apr 27, 2012, 3:38 PM

Post #16 of 37 (523 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

Hi Victor,

I read most of the PEP and I think it is ready for acceptance! Thanks
for your patience in shepherding this through such a difficult and
long discussion. Also thanks to the many other contributors,
especially those who ended up as co-authors. We will have an awesome
new set of time APIs! Now let the implementation roll...

--Guido

On Mon, Apr 23, 2012 at 4:30 PM, Victor Stinner
<victor.stinner [at] gmail> wrote:
>> Here is a simplified version of the first draft of the PEP 418. The
>> full version can be read online.
>> http://www.python.org/dev/peps/pep-0418/
>
> Thanks to everyone who helped me to work on this PEP!
>
> I integrated last comments. There is no more open question. (Or did I
> miss something?)
>
> I didn't know that it would be so hard to add a such simple function
> as time.monotonic()!?
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org



--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


steve at pearwood

Apr 27, 2012, 5:50 PM

Post #17 of 37 (515 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

Some issues with the PEP 418:


1) time.clock is deprecated, but also supported by get_clock_info. Why bother
supporting it if you don't want people to use it?


2) get_clock_info returns a dict. Why not a namedtuple?


3) The dict returned by get_clock_info includes an optional key,
"is_adjusted". Why is it optional?


4) The section on mach_absolute_time states:

According to the documentation (Technical Q&A QA1398), mach_timebase_info()
is always equal to one and never fails, even if the function may fail
according to its prototype.

I've read the linked technical note and I can't see anything about it always
being equal to one. I don't think your description is accurate.


5) In the glossary, you mark some terms in angle brackets <> but there is no
definition for them:

<nanosecond>
<strictly monotonic>
<clock monotonic> (which I think should be <monotonic clock> instead)


6) A stylistic suggestion: the glossary entries for Accuracy and Precision
should each say "Contrast <the other>" and link to the Wikipedia article.


7) There is a mismatch in tenses between "Adjusted" and "Resetting" in the
glossary. Suggest something like this instead:

Adjusted: Reset to the correct time. This may be done either
with a <Step> or by <Slewing>.


8) The glossary defines steady as high stability and "relatively high accuracy
and precision". But surely that is not correct -- a clock that ticks every
once per second (low precision) is still steady.


9) The perf_counter pseudocode seems a bit unusual (unPythonic?) to me. Rather
than checking flags at call-time, could you not use different function
definitions at compile time?


10) The "Alternatives" section should list arguments made for and against the
alternative APIs, not just list them.


Thanks for your excellent work Victor!



--
Steven
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


victor.stinner at gmail

Apr 27, 2012, 6:23 PM

Post #18 of 37 (514 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

> 1) time.clock is deprecated, but also supported by get_clock_info. Why
> bother supporting it if you don't want people to use it?

It will not be removed before Python 4, the function is still used by
Python < 3.3.

> 2) get_clock_info returns a dict. Why not a namedtuple?

I don't like the tuple API. I prefer a dict over a (named)tuple
because there is an optional key, and we migh add other optional keys
later.

> 3) The dict returned by get_clock_info includes an optional key,
> "is_adjusted". Why is it optional?

The value is not know for some platforms for some clock. I don't know
if process/thread time can be set for example. Sometimes the value is
hardcoded, sometimes the flag comes from the OS (ex: on Windows).

> 4) The section on mach_absolute_time states:
>
>   According to the documentation (Technical Q&A QA1398),
> mach_timebase_info()
>   is always equal to one and never fails, even if the function may fail
>   according to its prototype.
>
> I've read the linked technical note and I can't see anything about it always
> being equal to one. I don't think your description is accurate.

I don't remember where it does come from. I removed the sentence.

> 9) The perf_counter pseudocode seems a bit unusual (unPythonic?) to me.
> Rather than checking flags at call-time, could you not use different
> function definitions at compile time?

It's just a pseudo-code. I prefer to avoid duplication of code. The
pseudo-code is based on the C implemenation which use #ifdef;

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


guido at python

Apr 27, 2012, 8:40 PM

Post #19 of 37 (524 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

On Fri, Apr 27, 2012 at 5:50 PM, Steven D'Aprano <steve [at] pearwood> wrote:
> Some issues with the PEP 418:
>
>
> 1) time.clock is deprecated, but also supported by get_clock_info. Why
> bother supporting it if you don't want people to use it?

I see the deprecation of clock() as mostly symbolic -- it's used way
too much to do anything about it (as the PEP acknowledges). So I think
it's reasonable we should return info about it.

> 2) get_clock_info returns a dict. Why not a namedtuple?

Future flexibility. And there's no need for it to be a *tuple*.

> 3) The dict returned by get_clock_info includes an optional key,
> "is_adjusted". Why is it optional?

I wondered that myself, but I suspect it means "we don't know".

> 4) The section on mach_absolute_time states:
>
> According to the documentation (Technical Q&A QA1398),
> mach_timebase_info()
> is always equal to one and never fails, even if the function may fail
> according to its prototype.
>
> I've read the linked technical note and I can't see anything about it always
> being equal to one. I don't think your description is accurate.

Ok, you & Victor will have to figure that one out.

> 5) In the glossary, you mark some terms in angle brackets <> but there is no
> definition for them:
>
> <nanosecond>
> <strictly monotonic>
> <clock monotonic> (which I think should be <monotonic clock> instead)
>
>
> 6) A stylistic suggestion: the glossary entries for Accuracy and Precision
> should each say "Contrast <the other>" and link to the Wikipedia article.
>
>
> 7) There is a mismatch in tenses between "Adjusted" and "Resetting" in the
> glossary. Suggest something like this instead:
>
> Adjusted: Reset to the correct time. This may be done either
> with a <Step> or by <Slewing>.
>
>
> 8) The glossary defines steady as high stability and "relatively high
> accuracy
> and precision". But surely that is not correct -- a clock that ticks every
> once per second (low precision) is still steady.
>
>
> 9) The perf_counter pseudocode seems a bit unusual (unPythonic?) to me.
> Rather than checking flags at call-time, could you not use different
> function definitions at compile time?
>
>
> 10) The "Alternatives" section should list arguments made for and against
> the alternative APIs, not just list them.
>
>
> Thanks for your excellent work Victor!

Surely those are all very minor quibbles. I have one myself: at some
point it says:

On Linux, it is possible to use time.clock_gettime(CLOCK_THREAD_CPUTIME_ID).

But the PEP doesn't define a function by that name. Is it an editing
glitch? (Some of the pseudo code also uses this.)

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


victor.stinner at gmail

Apr 28, 2012, 12:40 AM

Post #20 of 37 (522 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

> Surely those are all very minor quibbles. I have one myself: at some
> point it says:
>
>    On Linux, it is possible to use time.clock_gettime(CLOCK_THREAD_CPUTIME_ID).
>
> But the PEP doesn't define a function by that name. Is it an editing
> glitch? (Some of the pseudo code also uses this.)

It is this function:
http://docs.python.org/dev/library/time.html#time.clock_gettime

It's just a binding of the C function clock_gettime(). Should the PEP
describe all functions used by the PEP?

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


victor.stinner at gmail

Apr 28, 2012, 1:48 AM

Post #21 of 37 (518 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

> 3) The dict returned by get_clock_info includes an optional key,
> "is_adjusted". Why is it optional?

More complete answer.

Rules used to fill the is_adjusted flag:

- System clock: is_adjusted=1 because the clock can be set manually
by the system administrator, except on Windows: is_adjusted is 0 if
GetSystemTimeAdjustment() returns isTimeAdjustmentDisabled=1
- Process time: is_adjusted=0 because I don't know an OS where the
process time can be modified
- Monotonic clocks: is_adjusted=0 on Windows, Mac OS X and Solaris,
is_adjusted=1 on Linux, it is not set otherwise. We may also set
is_adjusted to 0 on other OSes and so the key would not be optional
anymore.

Said differently, is_adjusted is only 1 for system clocks and for the
monotonic clock on Linux.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


guido at python

Apr 28, 2012, 7:02 AM

Post #22 of 37 (507 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

On Sat, Apr 28, 2012 at 12:40 AM, Victor Stinner
<victor.stinner [at] gmail> wrote:
>> Surely those are all very minor quibbles. I have one myself: at some
>> point it says:
>>
>> On Linux, it is possible to use time.clock_gettime(CLOCK_THREAD_CPUTIME_ID).
>>
>> But the PEP doesn't define a function by that name. Is it an editing
>> glitch? (Some of the pseudo code also uses this.)
>
> It is this function:
> http://docs.python.org/dev/library/time.html#time.clock_gettime
>
> It's just a binding of the C function clock_gettime(). Should the PEP
> describe all functions used by the PEP?

Oh, now I'm confused. Se in 3.3 we're adding a bunch of other new
functions to the time module that aren't described by the PEP? Aren't
those functions redundant? Or did I miss some part of the conversation
where this was discussed? What's *their* history?

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


solipsis at pitrou

Apr 28, 2012, 7:51 AM

Post #23 of 37 (509 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

On Sat, 28 Apr 2012 07:02:13 -0700
Guido van Rossum <guido [at] python> wrote:
> >
> > It is this function:
> > http://docs.python.org/dev/library/time.html#time.clock_gettime
> >
> > It's just a binding of the C function clock_gettime(). Should the PEP
> > describe all functions used by the PEP?
>
> Oh, now I'm confused. Se in 3.3 we're adding a bunch of other new
> functions to the time module that aren't described by the PEP? Aren't
> those functions redundant? Or did I miss some part of the conversation
> where this was discussed? What's *their* history?

time.clock_gettime() (and the related constants
CLOCK_{REALTIME,MONOTONIC, etc.}) is a thin wrapper around the
corresponding POSIX function, it's there for people who want low-level
control over their choice of APIs:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/clock_gettime.html

As a thin wrapper, adding it to the time module was pretty much
uncontroversial, I think. The PEP proposes cross-platform
functions with consistent semantics, which is where a discussion was
needed.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


guido at python

Apr 28, 2012, 8:57 AM

Post #24 of 37 (512 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

On Sat, Apr 28, 2012 at 7:51 AM, Antoine Pitrou <solipsis [at] pitrou> wrote:
> On Sat, 28 Apr 2012 07:02:13 -0700
> Guido van Rossum <guido [at] python> wrote:
>> >
>> > It is this function:
>> > http://docs.python.org/dev/library/time.html#time.clock_gettime
>> >
>> > It's just a binding of the C function clock_gettime(). Should the PEP
>> > describe all functions used by the PEP?
>>
>> Oh, now I'm confused. Se in 3.3 we're adding a bunch of other new
>> functions to the time module that aren't described by the PEP? Aren't
>> those functions redundant? Or did I miss some part of the conversation
>> where this was discussed? What's *their* history?
>
> time.clock_gettime() (and the related constants
> CLOCK_{REALTIME,MONOTONIC, etc.}) is a thin wrapper around the
> corresponding POSIX function, it's there for people who want low-level
> control over their choice of APIs:
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/clock_gettime.html
>
> As a thin wrapper, adding it to the time module was pretty much
> uncontroversial, I think. The PEP proposes cross-platform
> functions with consistent semantics, which is where a discussion was
> needed.

True, but does this mean clock_gettime and friends only exist on
POSIX? Shouldn't they be in the os or posix module then? I guess I'm
fine with either place but I don't know if enough thought was put into
the decision. Up until now the time module had only cross-platform
functions (even if clock()'s semantics vary widely).

--
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


victor.stinner at gmail

Apr 28, 2012, 1:32 PM

Post #25 of 37 (508 views)
Permalink
Re: [RFC] PEP 418: Add monotonic time, performance counter and process time functions [In reply to]

>> As a thin wrapper, adding it to the time module was pretty much
>> uncontroversial, I think. The PEP proposes cross-platform
>> functions with consistent semantics, which is where a discussion was
>> needed.
>
> True, but does this mean clock_gettime and friends only exist on
> POSIX? Shouldn't they be in the os or posix module then? I guess I'm
> fine with either place but I don't know if enough thought was put into
> the decision. Up until now the time module had only cross-platform
> functions (even if clock()'s semantics vary widely).

The os module is big enough. Low level networks functions are not in
the os module, but in the socket module.

Not all functions of the time module are always available. For
example, time.tzset() is not available on all platforms. Another
example, the new time.monotonic() is not available on all platforms
;-)

Oh, I forgot to mention that time.clock_*() functions are not always
available in the doc.

Victor
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

First page Previous page 1 2 Next page Last page  View All Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.