Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Bugs

[issue18702] Report skipped tests as skipped

 

 

Python bugs RSS feed   Index | Next | Previous | View Threaded


report at bugs

Aug 10, 2013, 4:09 AM

Post #1 of 6 (25 views)
Permalink
[issue18702] Report skipped tests as skipped

New submission from Serhiy Storchaka:

Some tests in Python testsuite are silently skipped if requirements is not satisfied. The proposed patch adds explicit "skipUnless()" and "raise SkipTest()" so that these tests now reported as skipped.

I.e. the code like

if not condition:
def test_foo(self):
...

is replaced by

@unittest.skipUnless(condition, "requires foo")
def test_foo(self):
...

and the code like

def test_foo(self):
...
if not condition:
return
...

is replaced by

def test_foo(self):
...
if not condition:
raise unittest.SkipTest("requires foo")
...

----------
components: Tests
files: skip_tests.patch
keywords: patch
messages: 194790
nosy: ezio.melotti, michael.foord, pitrou, serhiy.storchaka
priority: normal
severity: normal
stage: patch review
status: open
title: Report skipped tests as skipped
type: enhancement
versions: Python 3.3, Python 3.4
Added file: http://bugs.python.org/file31213/skip_tests.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue18702>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 11, 2013, 11:22 AM

Post #2 of 6 (20 views)
Permalink
[issue18702] Report skipped tests as skipped [In reply to]

Terry J. Reedy added the comment:

The patch applies cleanly on my 3.4 Win 7, fresh debug build. Somewhat fortuitously, as it turns out, I have not downloaded the files needed for ssl support. For each modified file, I ran
python_d -m test -v test_xxx

test test_nntplib crashed -- Traceback (most recent call last):
File "F:\Python\dev\py34\lib\test\regrtest.py", line 1298, in runtest_inner
the_module = importlib.import_module(abstest)
File "F:\Python\dev\py34\lib\importlib\__init__.py", line 93, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1613, in _gcd_import
File "<frozen importlib._bootstrap>", line 1594, in _find_and_load
File "<frozen importlib._bootstrap>", line 1561, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 607, in _check_name_wrapper
File "<frozen importlib._bootstrap>", line 1056, in load_module
File "<frozen importlib._bootstrap>", line 926, in load_module
File "<frozen importlib._bootstrap>", line 274, in _call_with_frames_removed
File "F:\Python\dev\py34\lib\test\test_nntplib.py", line 305, in <module>
class NetworkedNNTP_SSLTests(NetworkedNNTPTests):
File "F:\Python\dev\py34\lib\test\test_nntplib.py", line 315, in NetworkedNNTP_SSLTests
NNTP_CLASS = nntplib.NNTP_SSL
AttributeError: 'module' object has no attribute 'NNTP_SSL'

I do not understand the frozen importlib stuff, but nntplib._have_ssl is False, so the line asking for the non-existent attribute should not be executed.

The problem is that changing a guard from 'if hasattr' to the decorator is a *semantic change* in that it allows execution of the guarded statement. Skips only skip function calling. For a function statement, creating a function object from the compiled code object and binding a name is trivial. Not calling the function is the important part. For a class with code other than function definitions, the difference is crucial, as the above shows.

The general solution is to put class code inside a function so it is only executed later, as part of the test process, rather than during inport.
@classmethod
def setUpClass(cls):
cls.NNTP_CLASS = nntplib.NNTP_SSL

In this case, you could instead guard it with 'if _have_ssl', but since this assignment is the only thing being tested by this test case, it should be inside a method anyway.

I suggest that you review your patch for other changed classes that might have class-level code that could fail with the change to the skip decorator.

Here is similar problem: trying to subclass a class that does not exist.
test test_socketserver crashed
File "F:\Python\dev\py34\lib\test\test_socketserver.py", line 50, in <module>
socketserver.UnixStreamServer):
AttributeError: 'module' object has no attribute 'UnixStreamServer'

class ForkingUnixStreamServer(socketserver.ForkingMixIn,
socketserver.UnixStreamServer): pass

I think you have to go back to 'if HAVE_UNIX_SOCKETS:' for this. Or re-factor somehow.

---
I cannot help wondering whether test_math.xxx.test_exceptions should still be (normally) disabled. I sent Mark Dickinson a separate note.

----------
nosy: +terry.reedy

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue18702>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 11, 2013, 2:42 PM

Post #3 of 6 (20 views)
Permalink
[issue18702] Report skipped tests as skipped [In reply to]

Serhiy Storchaka added the comment:

Thank you Ezio and Terry for review. Here is updated patch. Problems with class initialization solved. I have made a lot of other changes especially in test_os.py and test_posix.py.

----------
Added file: http://bugs.python.org/file31233/skip_tests_2.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue18702>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 11, 2013, 3:34 PM

Post #4 of 6 (20 views)
Permalink
[issue18702] Report skipped tests as skipped [In reply to]

Terry J. Reedy added the comment:

Applies and runs (with text_posix entirely skipped).

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue18702>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 11, 2013, 9:57 PM

Post #5 of 6 (19 views)
Permalink
[issue18702] Report skipped tests as skipped [In reply to]

Vajrasky Kok added the comment:

I read the patch. It looks good.

Anyway, I have two questions:

1. I saw 3 variants of writing description for skipping test descriptor:

requires os.mkfifo

test needs socket.inet_pton()

needs os.read

Which one is the preferred way to go? "requires" or "test needs" or "needs"? Or it does not matter?

2. I saw 2 variants of writing the function name required for running the test.

test needs socket.inet_pton()

test needs socket.inet_pton

Which one is the preferred way to go? With or without parentheses? Or it does not matter?

----------
nosy: +vajrasky

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue18702>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 12, 2013, 2:29 AM

Post #6 of 6 (14 views)
Permalink
[issue18702] Report skipped tests as skipped [In reply to]

Serhiy Storchaka added the comment:

> Which one is the preferred way to go? "requires" or "test needs" or "needs"? Or it does not matter?

I used the wording which used in other skips in the same file or in similar skips in other files. If it matters I will correct messages.

> Which one is the preferred way to go? With or without parentheses? Or it does not matter?

Thank you for note. I will add missed parentheses to function names.

There are existing skip messages which uses function names without parentheses. I left them untouched (the patch already is a large enough).

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue18702>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Python bugs RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.