Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

unittest.TestSuite holding references to unittest.TestCase instances too long

 

 

Python dev RSS feed   Index | Next | Previous | View Threaded


matthewlmcclure at gmail

Aug 2, 2013, 4:51 AM

Post #1 of 13 (82 views)
Permalink
unittest.TestSuite holding references to unittest.TestCase instances too long

It seems unittest.TestSuite holds references to unittest.TestCase instances
after the test runs, until the test suite finishes. In a large suite, where
the TestCase instances consume memory during execution, that can lead to
exhausting all available memory and the OS killing the test process.

What do you think of a change like this?

$ hg diff
diff -r 3bd55ec317a7 Lib/unittest/suite.py
--- a/Lib/unittest/suite.py Thu Aug 01 23:57:21 2013 +0200
+++ b/Lib/unittest/suite.py Fri Aug 02 07:42:22 2013 -0400
@@ -90,7 +90,12 @@
if getattr(result, '_testRunEntered', False) is False:
result._testRunEntered = topLevel = True

- for test in self:
+ while True:
+ try:
+ test = self._tests.pop(0)
+ except IndexError:
+ break
+
if result.shouldStop:
break

See also the conversation on django-developers[1] that led me here.

[1]: https://groups.google.com/forum/#!topic/django-developers/XUMetDSGVT0

--
Matt McClure
http://matthewlmcclure.com
http://www.mapmyfitness.com/profile/matthewlmcclure


fuzzyman at voidspace

Aug 2, 2013, 8:13 AM

Post #2 of 13 (80 views)
Permalink
Re: unittest.TestSuite holding references to unittest.TestCase instances too long [In reply to]

Sent from my iPhone

On 2 Aug 2013, at 14:51, Matt McClure <matthewlmcclure [at] gmail> wrote:

> It seems unittest.TestSuite holds references to unittest.TestCase instances after the test runs, until the test suite finishes. In a large suite, where the TestCase instances consume memory during execution, that can lead to exhausting all available memory and the OS killing the test process.

Well individual tests not releasing resources could be seen as a bug in those tests.

That aside, there's an open bug for this with some discussion and a proposed fix:

http://bugs.python.org/issue11798

The agreed on approach just needs doing.

Michael


>
> What do you think of a change like this?
>
> $ hg diff
> diff -r 3bd55ec317a7 Lib/unittest/suite.py
> --- a/Lib/unittest/suite.py Thu Aug 01 23:57:21 2013 +0200
> +++ b/Lib/unittest/suite.py Fri Aug 02 07:42:22 2013 -0400
> @@ -90,7 +90,12 @@
> if getattr(result, '_testRunEntered', False) is False:
> result._testRunEntered = topLevel = True
>
> - for test in self:
> + while True:
> + try:
> + test = self._tests.pop(0)
> + except IndexError:
> + break
> +
> if result.shouldStop:
> break
>
> See also the conversation on django-developers[1] that led me here.
>
> [1]: https://groups.google.com/forum/#!topic/django-developers/XUMetDSGVT0
>
> --
> Matt McClure
> http://matthewlmcclure.com
> http://www.mapmyfitness.com/profile/matthewlmcclure
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk


solipsis at pitrou

Aug 2, 2013, 9:19 AM

Post #3 of 13 (77 views)
Permalink
Re: unittest.TestSuite holding references to unittest.TestCase instances too long [In reply to]

Le Fri, 2 Aug 2013 18:13:13 +0300,
Michael Foord <fuzzyman [at] voidspace> a écrit :
>
> On 2 Aug 2013, at 14:51, Matt McClure <matthewlmcclure [at] gmail>
> wrote:
>
> > It seems unittest.TestSuite holds references to unittest.TestCase
> > instances after the test runs, until the test suite finishes. In a
> > large suite, where the TestCase instances consume memory during
> > execution, that can lead to exhausting all available memory and the
> > OS killing the test process.
>
> Well individual tests not releasing resources could be seen as a bug
> in those tests.
>
> That aside, there's an open bug for this with some discussion and a
> proposed fix:
>
> http://bugs.python.org/issue11798
>
> The agreed on approach just needs doing.

The patch is basically ready for commit, except for a possible doc
addition, no?

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


fuzzyman at voidspace

Aug 2, 2013, 9:47 AM

Post #4 of 13 (78 views)
Permalink
Re: unittest.TestSuite holding references to unittest.TestCase instances too long [In reply to]

Sent from my iPhone

On 2 Aug 2013, at 19:19, Antoine Pitrou <solipsis [at] pitrou> wrote:

> Le Fri, 2 Aug 2013 18:13:13 +0300,
> Michael Foord <fuzzyman [at] voidspace> a écrit :
>>
>> On 2 Aug 2013, at 14:51, Matt McClure <matthewlmcclure [at] gmail>
>> wrote:
>>
>>> It seems unittest.TestSuite holds references to unittest.TestCase
>>> instances after the test runs, until the test suite finishes. In a
>>> large suite, where the TestCase instances consume memory during
>>> execution, that can lead to exhausting all available memory and the
>>> OS killing the test process.
>>
>> Well individual tests not releasing resources could be seen as a bug
>> in those tests.
>>
>> That aside, there's an open bug for this with some discussion and a
>> proposed fix:
>>
>> http://bugs.python.org/issue11798
>>
>> The agreed on approach just needs doing.
>
> The patch is basically ready for commit, except for a possible doc
> addition, no?

Looks to be the case, reading the patch it looks fine. I'm currently on holiday until Monday. If anyone is motivated to do the docs too and commit that would be great. Otherwise I'll get to it on my return.

Michael



>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev [at] python
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


matthewlmcclure at gmail

Aug 2, 2013, 10:27 AM

Post #5 of 13 (77 views)
Permalink
Re: unittest.TestSuite holding references to unittest.TestCase instances too long [In reply to]

On Fri, Aug 2, 2013 at 11:13 AM, Michael Foord <fuzzyman [at] voidspace>wrote:

> There's an open bug for this with some discussion and a proposed fix:
>
> http://bugs.python.org/issue11798
>
> The agreed on approach just needs doing.
>

Thanks for the link. I hadn't found that yet. I'll see if I can contribute
there.

--
Matt McClure
http://matthewlmcclure.com
http://www.mapmyfitness.com/profile/matthewlmcclure


matthewlmcclure at gmail

Aug 3, 2013, 7:27 AM

Post #6 of 13 (71 views)
Permalink
Re: unittest.TestSuite holding references to unittest.TestCase instances too long [In reply to]

Michael Foord <fuzzyman <at> voidspace.org.uk> writes:
> On 2 Aug 2013, at 19:19, Antoine Pitrou <solipsis <at> pitrou.net> wrote:
> > The patch is basically ready for commit, except for a possible doc
> > addition, no?
>
> Looks to be the case, reading the patch it looks fine. I'm currently on
holiday until Monday.
> If anyone is motivated to do the docs too and commit that would be great.
Otherwise I'll
> get to it on my return.

It looks like the patch is based on what will become 3.4. Would backporting
it to 2.7 be feasible? What's involved in doing so?

I took a crack at the docs.

# HG changeset patch
# User Matt McClure <matthewlmcclure [at] gmail>
# Date 1375538965 14400
# Node ID d748d70201929288c230862da4dbdba33d61ae9f
# Parent bf43956356ffe93e75ffdd5a7a8164fc68cf14ae
[11798] Document TestSuite.{__iter__, run} changes

diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -1470,15 +1470,24 @@

Tests grouped by a :class:`TestSuite` are always accessed by
iteration.
Subclasses can lazily provide tests by overriding :meth:`__iter__`.
Note
- that this method maybe called several times on a single suite
- (for example when counting tests or comparing for equality)
- so the tests returned must be the same for repeated iterations.
+ that this method may be called several times on a single suite (for
+ example when counting tests or comparing for equality) so the tests
+ returned by repeated iterations before :meth:`TestSuite.run` must be
the
+ same for each call iteration. After :meth:`TestSuite.run`, callers
should
+ not rely on the tests returned by this method unless the caller uses
a
+ subclass that overrides :meth:`TestSuite._removeTestAtIndex` to
preserve
+ test references.

.. versionchanged:: 3.2
In earlier versions the :class:`TestSuite` accessed tests
directly rather
than through iteration, so overriding :meth:`__iter__` wasn't
sufficient
for providing tests.

+ .. versionchanged:: 3.4
+ In earlier versions the :class:`TestSuite` held references to each
+ :class:`TestCase` after :meth:`TestSuite.run`. Subclasses can
restore
+ that behavior by overriding :meth:`TestSuite._removeTestAtIndex`.
+
In the typical usage of a :class:`TestSuite` object, the :meth:`run`
method
is invoked by a :class:`TestRunner` rather than by the end-user test
harness.

diff --git a/Lib/unittest/suite.py b/Lib/unittest/suite.py
--- a/Lib/unittest/suite.py
+++ b/Lib/unittest/suite.py
@@ -65,6 +65,7 @@
return result

def _removeTestAtIndex(self, index):
+ """Stop holding a reference to the TestCase at index."""
try:
self._tests[index] = None
except TypeError:

--
Matt McClure
http://matthewlmcclure.com
http://www.mapmyfitness.com/profile/matthewlmcclure


rdmurray at bitdance

Aug 3, 2013, 9:07 AM

Post #7 of 13 (71 views)
Permalink
Re: unittest.TestSuite holding references to unittest.TestCase instances too long [In reply to]

On Sat, 03 Aug 2013 10:27:30 -0400, Matt McClure <matthewlmcclure [at] gmail> wrote:
> Michael Foord <fuzzyman <at> voidspace.org.uk> writes:
> > On 2 Aug 2013, at 19:19, Antoine Pitrou <solipsis <at> pitrou.net> wrote:
> > > The patch is basically ready for commit, except for a possible doc
> > > addition, no?
> >
> > Looks to be the case, reading the patch it looks fine. I'm currently on
> holiday until Monday.
> > If anyone is motivated to do the docs too and commit that would be great.
> Otherwise I'll
> > get to it on my return.
>
> It looks like the patch is based on what will become 3.4. Would backporting
> it to 2.7 be feasible? What's involved in doing so?

That depends on how likely Michale thinks it is that it might break
things.

> I took a crack at the docs.

Thanks. Please post your patch to the issue, it will get lost here.

--David
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


matthewlmcclure at gmail

Aug 3, 2013, 11:01 AM

Post #8 of 13 (72 views)
Permalink
Re: unittest.TestSuite holding references to unittest.TestCase instances too long [In reply to]

On Aug 3, 2013, at 12:07 PM, "R. David Murray" <rdmurray [at] bitdance> wrote:

> Thanks. Please post your patch to the issue, it will get lost here.

I'm trying to register, but I'm not receiving a confirmation email to complete the registration.

--
http://matthewlmcclure.com
http://about.mapmyfitness.com

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


fuzzyman at voidspace

Aug 3, 2013, 12:27 PM

Post #9 of 13 (71 views)
Permalink
Re: unittest.TestSuite holding references to unittest.TestCase instances too long [In reply to]

Sent from my iPhone

On 3 Aug 2013, at 19:07, "R. David Murray" <rdmurray [at] bitdance> wrote:

> On Sat, 03 Aug 2013 10:27:30 -0400, Matt McClure <matthewlmcclure [at] gmail> wrote:
>> Michael Foord <fuzzyman <at> voidspace.org.uk> writes:
>>> On 2 Aug 2013, at 19:19, Antoine Pitrou <solipsis <at> pitrou.net> wrote:
>>>> The patch is basically ready for commit, except for a possible doc
>>>> addition, no?
>>>
>>> Looks to be the case, reading the patch it looks fine. I'm currently on
>> holiday until Monday.
>>> If anyone is motivated to do the docs too and commit that would be great.
>> Otherwise I'll
>>> get to it on my return.
>>
>> It looks like the patch is based on what will become 3.4. Would backporting
>> it to 2.7 be feasible? What's involved in doing so?
>
> That depends on how likely Michale thinks it is that it might break
> things.
>

It smells to me like a new feature rather than a bugfix, and it's a moderately big change. I don't think it can be backported to 2.7 other than through unittest2.

Michael


>> I took a crack at the docs.
>
> Thanks. Please post your patch to the issue, it will get lost here.
>
> --David
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


matthewlmcclure at gmail

Aug 5, 2013, 2:43 PM

Post #10 of 13 (55 views)
Permalink
Re: unittest.TestSuite holding references to unittest.TestCase instances too long [In reply to]

On Sat, Aug 3, 2013 at 3:27 PM, Michael Foord <fuzzyman [at] voidspace>wrote:

> It smells to me like a new feature rather than a bugfix, and it's a
> moderately big change. I don't think it can be backported to 2.7 other than
> through unittest2.
>

Is http://hg.python.org/unittest2 the place to backport to unittest2?

--
Matt McClure
http://matthewlmcclure.com
http://www.mapmyfitness.com/profile/matthewlmcclure


fuzzyman at voidspace

Aug 6, 2013, 1:25 AM

Post #11 of 13 (51 views)
Permalink
Re: unittest.TestSuite holding references to unittest.TestCase instances too long [In reply to]

On 6 Aug 2013, at 00:43, Matt McClure <matthewlmcclure [at] gmail> wrote:

> On Sat, Aug 3, 2013 at 3:27 PM, Michael Foord <fuzzyman [at] voidspace> wrote:
> It smells to me like a new feature rather than a bugfix, and it's a moderately big change. I don't think it can be backported to 2.7 other than through unittest2.
>
> Is http://hg.python.org/unittest2 the place to backport to unittest2?
>

It is, but... unittest itself has changed so extensively since the last release of unittest2 that I'm not sure whether a completely fresh start for unittest2 might be needed. (Although I intend to do another bugfix release of this version as well.)

Making unittest2 will involve:

* Taking the Python 3 unittest and porting code plus tests to run on python 2
* Run the new plus old tests (removing duplications) to ensure no functionality was lost (for example string handling will be wildly different so some tests may have been removed in Python 3 that are still relevant to Python 2)
* Fix the failing tests and decide whether new features that depend on later versions of Python (particularly around improvements to the inspect module and stack frames) even *can* be backported
* unittest2 classes all need to inherit from the unittest versions (and do some appropriate super calls because of the extra base class) - plus there are some tests for the differences
* there is a setuptools compatible test runner and the unit2 script also additional to vanilla unittest
* documentation updates - new features and differences
* stuff I've forgotten

So it's a pretty big job, but not insurmountable :-) A version that targets Python 3.2 would also be useful - it *may* be possible to do this in a single codebase. The current approach is to have two codebases (unittest2 and unittest2py3k). The reason for this is that it's rather easier to generate a Python 3 backport by applying a few patches than it is to generate the Python 2 version - so a Python 3 backport can be much simpler. It makes life harder for projects that use unittest2 though as which project they need as test runner depends on whether they are being run on Python 2 or Python 3 - so a single codebase (or single project anyway) would be better.

All the best,

Michael Foord

> --
> Matt McClure
> http://matthewlmcclure.com
> http://www.mapmyfitness.com/profile/matthewlmcclure


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing
http://www.sqlite.org/different.html





_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


matthewlmcclure at gmail

Aug 6, 2013, 11:05 AM

Post #12 of 13 (46 views)
Permalink
Re: unittest.TestSuite holding references to unittest.TestCase instances too long [In reply to]

Hi Michael,

On Tue, Aug 6, 2013 at 4:25 AM, Michael Foord <fuzzyman [at] voidspace>wrote:

> unittest itself has changed so extensively since the last release of
> unittest2 that I'm not sure whether a completely fresh start for unittest2
> might be needed. (Although I intend to do another bugfix release of this
> version as well.)
>
> Making unittest2 will involve:
>
> * Taking the Python 3 unittest and porting code plus tests to run
> on python 2
> [ ... ]
>

I took a different approach and simply applied the patch of diffs[1] from
the Python 3 issue to the unittest2 tip.

There was a small amount of renaming "unittest" to "unittest2" required,
but other than that, the patch applied pretty cleanly, and seems to pass
the unit tests and avoid the ever-increasing memory problem in my private
test suite.

Do you think it's sufficient to port just this feature? Or if not, what am
I missing that requires resyncing more of unittest2 with the changes from
Python 3?

Is Google Code[2] still the right place for unittest2 issues? I found that
via PyPI[3].

It looks like there have been a lot of commits in the unittest2 repository
since the last PyPI release (2010-07-12 -- 0.5.1). Would you plan to do
another PyPI release of unittest2 with this feature? Or would you recommend
using unittest2 from the repository to get it? Or am I missing a more
recent packaged release somewhere else?

[1]:
https://bitbucket.org/matthewlmcclure/unittest2/compare/issue11798-tip..issue11798-base#diff
[2]: https://code.google.com/p/unittest-ext/issues/detail?id=76&sort=-id
[3]: https://pypi.python.org/pypi/unittest2

--
Matt McClure
http://matthewlmcclure.com
http://www.mapmyfitness.com/profile/matthewlmcclure


fuzzyman at voidspace

Aug 6, 2013, 2:08 PM

Post #13 of 13 (47 views)
Permalink
Re: unittest.TestSuite holding references to unittest.TestCase instances too long [In reply to]

On 6 Aug 2013, at 21:05, Matt McClure <matthewlmcclure [at] gmail> wrote:

> Hi Michael,
>
> On Tue, Aug 6, 2013 at 4:25 AM, Michael Foord <fuzzyman [at] voidspace> wrote:
> unittest itself has changed so extensively since the last release of unittest2 that I'm not sure whether a completely fresh start for unittest2 might be needed. (Although I intend to do another bugfix release of this version as well.)
>
> Making unittest2 will involve:
>
> * Taking the Python 3 unittest and porting code plus tests to run on python 2
> [ ... ]
>
> I took a different approach and simply applied the patch of diffs[1] from the Python 3 issue to the unittest2 tip.
>
> There was a small amount of renaming "unittest" to "unittest2" required, but other than that, the patch applied pretty cleanly, and seems to pass the unit tests and avoid the ever-increasing memory problem in my private test suite.
>
> Do you think it's sufficient to port just this feature? Or if not, what am I missing that requires resyncing more of unittest2 with the changes from Python 3?
>
> Is Google Code[2] still the right place for unittest2 issues? I found that via PyPI[3].
>
> It looks like there have been a lot of commits in the unittest2 repository since the last PyPI release (2010-07-12 -- 0.5.1). Would you plan to do another PyPI release of unittest2 with this feature? Or would you recommend using unittest2 from the repository to get it? Or am I missing a more recent packaged release somewhere else?
>


I plan to do a bugfix release which fixes bugs in unittest2 since the PyPI release. (Foolishly I don't think I tagged so I need to work out which revision corresponds to the released version.)

I will also do a new release with *all* new features. I won't do a release with just this new feature no.

For unittest2 specific issues, yes google code is still the correct issue tracker.

Michael

> [1]: https://bitbucket.org/matthewlmcclure/unittest2/compare/issue11798-tip..issue11798-base#diff
> [2]: https://code.google.com/p/unittest-ext/issues/detail?id=76&sort=-id
> [3]: https://pypi.python.org/pypi/unittest2
>
> --
> Matt McClure
> http://matthewlmcclure.com
> http://www.mapmyfitness.com/profile/matthewlmcclure


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing
http://www.sqlite.org/different.html





_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.