Mailing List Archive: Python: Bugs

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple*

Index | Next | Previous | View Threaded

report at bugs

May 4, 2012, 7:37 AM

Post #1 of 15 (103 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple*

New submission from Serhiy Storchaka <storchaka [at] gmail>:

In function convertsimple() in Python/getargs.c possible converting out of float range value from double to float.

Fortunately, 'f' format character is not used in CPython source code. But it can be used in the extensions.

Tests will be later.

----------
components: Interpreter Core
files: getargs_float_overflow.patch
keywords: patch
messages: 159937
nosy: storchaka
priority: normal
severity: normal
status: open
title: Overflow in parsing 'float' parameters in PyArg_ParseTuple*
versions: Python 2.7, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file25454/getargs_float_overflow.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 4, 2012, 7:50 AM

Post #2 of 15 (86 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Mark Dickinson <dickinsm [at] gmail> added the comment:

I don't think this change should be made for 2.7 or 3.2, since it has potential to break existing code.

Though technically, conversion of an out-of-range double to a float gives undefined behaviour (C99 6.3.1.5p2), I'm willing to bet that most current compilers just happily return +-infinity in these cases, and there's probably code out there that would break if we changed this.

So for 2.7 or 3.2, we could just return +-inf here instead. Though even that isn't quite right if you're picky about corner cases, since there are some values *just* outside [-FLOAT_MAX, FLOAT_MAX] that should still round to +-FLOAT_MAX under round-to-nearest.

I suggest leaving this alone for 2.7 and 3.2

For 3.3, it's not clear to me whether it's better to return +-inf or to raise here.

----------
nosy: +mark.dickinson

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 4, 2012, 7:50 AM

Post #3 of 15 (91 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Changes by Mark Dickinson <dickinsm [at] gmail>:

----------
nosy: +skrah

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 4, 2012, 8:04 AM

Post #4 of 15 (87 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Mark Dickinson <dickinsm [at] gmail> added the comment:

I was just remembering that I was *sure* I'd seen the double->float avoiding undefined behaviour issue mentioned on a C mailing list not so long ago. Turns out that there was a good reason for me remembering that...

https://groups.google.com/group/comp.lang.c/browse_thread/thread/5d93cc742025b298

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 4, 2012, 8:09 AM

Post #5 of 15 (86 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Changes by Mark Dickinson <dickinsm [at] gmail>:

----------
assignee: -> mark.dickinson

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 4, 2012, 10:56 AM

Post #6 of 15 (82 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Serhiy Storchaka <storchaka [at] gmail> added the comment:

I also thought about ±∞. But PyLong_AsDouble() raises OverflowError for
out-of-range value. And it checks strictly out of ±DBL_MAX.

Because float(10**1000) returns no float('inf'), but raises an
exception, I think that returning ±∞ will be wrong.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 4, 2012, 11:12 AM

Post #7 of 15 (81 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Mark Dickinson <dickinsm [at] gmail> added the comment:

> And it checks strictly out of ±DBL_MAX.

Nope. Values just larger than DBL_MAX won't raise OverflowError here.

> Because float(10**1000) returns no float('inf'), but raises an
> exception, I think that returning ±∞ will be wrong.

Possibly. But there's also the fact that 3.2 already returns inf here; we'd need a pretty good reason to break that. Like I said, I'm not sure which the right way to go here is.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 4, 2012, 12:29 PM

Post #8 of 15 (80 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Serhiy Storchaka <storchaka [at] gmail> added the comment:

Here is a patch with tests.

> Nope. Values just larger than DBL_MAX won't raise OverflowError here.

Isn't that a little bit. But values that rounded to DBL_MAX can raise
OverflowError. In any case it's too difficult to achieve strict behavior
in this corner case.

> Possibly. But there's also the fact that 3.2 already returns inf here; we'd need a pretty good reason to break that.

In the end, we may add the environment variable
PYTHONDONTRAISEEXCEPTIONIFFLOATOVERFLOWS to control this behaviour.

> Like I said, I'm not sure which the right way to go here is.

Take a look at the tests and may be you'll see the system.

----------
Added file: http://bugs.python.org/file25455/getargs_float_overflow_2.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________

Attachments:

getargs_float_overflow_2.patch (6.45 KB)

report at bugs

May 4, 2012, 12:31 PM

Post #9 of 15 (84 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Changes by Serhiy Storchaka <storchaka [at] gmail>:

----------
type: -> enhancement
versions: -Python 2.7, Python 3.2

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 4, 2012, 12:39 PM

Post #10 of 15 (80 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Mark Dickinson <dickinsm [at] gmail> added the comment:

> But values that rounded to DBL_MAX can raise
> OverflowError. In any case it's too difficult to achieve strict behavior
> in this corner case.

Well, PyLong_AsDouble *does* achieve strict behaviour in this corner case :-). Integers less than 0.5 * (sys.float_info.max + 2**1024) in absolute value give finite results; integers greater than or equal to that bound produce an OverflowError.

> Take a look at the tests and may be you'll see the system.

I don't see how looking at the tests helps with making a decision about breaking backwards compatibility or not. :-)

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 5, 2012, 10:41 AM

Post #11 of 15 (78 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Serhiy Storchaka <storchaka [at] gmail> added the comment:

No one integer produces infinity in 'double' parameter parsing.

But the 'float' parameter parsing can produce infinity, and it can raise
an exception. To be consistent, we need or produce infinity on double
overflow (in this case, we must explicitly produce infinity on float
overflow), or to raise an exception on float overflow.

There is also a third option -- deprecate the 'float' parameter parsing.
Leave the responsibility for the proper overflow handling on the user.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 5, 2012, 1:25 PM

Post #12 of 15 (80 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Stefan Krah <stefan-usenet [at] bytereef> added the comment:

The proposal makes sense at first glance, but I agree with Mark that it is
not clear what should be done. For example, all arrays in Python silently
convert to inf:

>>> from numpy import array
>>> x = array([1,2,3], 'f')
>>> x
array([ 1., 2., 3.], dtype=float32)
>>> x[0] = 10**100
>>> x
array([ inf, 2., 3.], dtype=float32)

Same for array.array and memoryview. I would not be surprised if users rely
on this behavior. Anyway, silently converting to infinity is exactly what
I'd expect (also for double BTW).

Regarding undefined behavior: I only know compilers that convert to infinity
without signaling overflow. The tests for the new memoryview implementation
should include this case (I think!), and I ran the tests with all compilers
that I've access to.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 7, 2012, 3:15 AM

Post #13 of 15 (76 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Mark Dickinson <dickinsm [at] gmail> added the comment:

Closing as "won't fix"; Python is sadly far from consistent about returning infinity versus raising OverflowError, in a wide variety of situations. For example, compare:

* float(Decimal('1e310')) with float(Fraction('1e310')), or

* struct.pack('f', 1e100) with struct.pack('<f', 1e100), or

* 1e160 * 1e160 with 1e160 ** 2.

Given this, I think it's far from clear what the right answer for the getargs 'f' code is, and in this situation I think we should just stick with the status quo.

----------
resolution: -> wont fix
status: open -> closed

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 7, 2012, 3:35 AM

Post #14 of 15 (70 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Stefan Krah <stefan-usenet [at] bytereef> added the comment:

I agree. Fixing all this would probably require a PEP. It looks like the
original plan was to provide a facility to turn off the Overflow exception:

http://mail.python.org/pipermail/python-dev/2000-May/003990.html

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

May 7, 2012, 5:31 AM

Post #15 of 15 (72 views)
Permalink

[issue14722] Overflow in parsing 'float' parameters in PyArg_ParseTuple* [In reply to]

Serhiy Storchaka <storchaka [at] gmail> added the comment:

May be proposed tests (except for the overflow) would be helpful? Right now 'f' and 'd' parsing code is not covered by tests.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue14722>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads