Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Zope: Dev

Testrunner option for running tests in random order?!?

  

 
Zope dev RSS feed   Index | Next | Previous | View Threaded


ct at gocept

Oct 2, 2009, 5:00 AM

Post #1 of 16 (2063 views)
Permalink
Testrunner option for running tests in random order?!?
Hi,

I *thought* the test runner had a feature to run tests in randomized
order, but I can't find it. Is that something that Zope 2 had and that
was never ported to zope.testing?

I couldn't find in the history either.

/me is confused.

--
Christian Theune · ct [at] gocept
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


jonathan.ballet at securactive

Oct 8, 2009, 6:28 AM

Post #2 of 16 (1968 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
Hi Christian,

On Fri, Oct 2, 2009 at 2:00 PM, Christian Theune <ct [at] gocept> wrote:
> Hi,
>
> I *thought* the test runner had a feature to run tests in randomized
> order, but I can't find it. Is that something that Zope 2 had and that
> was never ported to zope.testing?
>
> I couldn't find in the history either.
>
> /me is confused.

I was about to send the same mail today, but I found out your mail before.

I didn't find anything relative to random tests in Zope 2 (but I'm not
familiar at all with Zope 2, so I'm pretty sure I could have missed
something), and I ended up writing a new feature for zope.testing's
testrunner.

I made a branch 'shuffle-tests', available on the Mercurial repository
at http://bitbucket.org/multani/zope.testing-shuffle/ , which adds two
new options:
* --shuffle to randomize the order of the tests in a layer (layers
are not yet randomized, I didn't look at it yet);
* --shuffle-seed XXX, to initialize the RNG with a specific value,
which allows to reproduce a specific randomization of tests (inspired
by this thread http://lists.idyll.org/pipermail/testing-in-python/2009-April/001812.html
)

It's not yet ready-to-commit: it still misses tests (tricky to write)
and more documentation.
However, feedbacks are welcome!

Cheers,

Jonathan
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


ct at gocept

Oct 9, 2009, 5:33 AM

Post #3 of 16 (1966 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
Hi,

On 10/08/2009 03:28 PM, Jonathan Ballet wrote:
> Hi Christian,
>
> On Fri, Oct 2, 2009 at 2:00 PM, Christian Theune <ct [at] gocept> wrote:
>> Hi,
>>
>> I *thought* the test runner had a feature to run tests in randomized
>> order, but I can't find it. Is that something that Zope 2 had and that
>> was never ported to zope.testing?
>>
>> I couldn't find in the history either.
>>
>> /me is confused.
>
> I was about to send the same mail today, but I found out your mail before.
>
> I didn't find anything relative to random tests in Zope 2 (but I'm not
> familiar at all with Zope 2, so I'm pretty sure I could have missed
> something), and I ended up writing a new feature for zope.testing's
> testrunner.
>
> I made a branch 'shuffle-tests', available on the Mercurial repository
> at http://bitbucket.org/multani/zope.testing-shuffle/ , which adds two
> new options:
>
> * --shuffle to randomize the order of the tests in a layer (layers
> are not yet randomized, I didn't look at it yet);
> * --shuffle-seed XXX, to initialize the RNG with a specific value,
> which allows to reproduce a specific randomization of tests (inspired
> by this thread http://lists.idyll.org/pipermail/testing-in-python/2009-April/001812.html
> )

Sounds right. It could even be folded into a single option:

--shuffle[=XXX]

> It's not yet ready-to-commit: it still misses tests (tricky to write)
> and more documentation.

That's what working branches are for. You do have commit rights, don't you?

It's not that tricky to test: seed the RNG from outside the tests.

> However, feedbacks are welcome!

Thanks for the work!

Christian

--
Christian Theune · ct [at] gocept
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development

_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


jonathan.ballet at securactive

Oct 9, 2009, 5:51 AM

Post #4 of 16 (1964 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
Hey,

On Fri, Oct 9, 2009 at 2:33 PM, Christian Theune <ct [at] gocept> wrote:
>>   * --shuffle to randomize the order of the tests in a layer (layers
>> are not yet randomized, I didn't look at it yet);
>>   * --shuffle-seed XXX, to initialize the RNG with a specific value,
>> which allows to reproduce a specific randomization of tests (inspired
>> by this thread http://lists.idyll.org/pipermail/testing-in-python/2009-April/001812.html
>> )
>
> Sounds right. It could even be folded into a single option:
>
> --shuffle[=XXX]

I'm not sure how to do that, considering those use cases:

* the user doesn't want to shuffle its tests and so, doesn't set --shuffle
* the user want to shuffle its tests, without specifying an order,
and use --shuffle
* the user want to shuffle its tests in specific order, and use --shuffle=42

If I want this option to have an optional value, I think I must set a
default value to the option. But then, how to make the distinction
between the two first cases? I might miss something I don't know in
option parsing, but I didn't find anything...

>> It's not yet ready-to-commit: it still misses tests (tricky to write)
>> and more documentation.
>
> That's what working branches are for. You do have commit rights, don't you?

I don't (personally) have commit rights, but my boss (sdouche) has.
I'll see with him.


Thanks for your advices!

Jonathan
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


benji at zope

Oct 9, 2009, 6:30 AM

Post #5 of 16 (1959 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
On Fri, Oct 9, 2009 at 8:51 AM, Jonathan Ballet
<jonathan.ballet [at] securactive> wrote:
> If I want this option to have an optional value, I think I must set a
> default value to the option. But then, how to make the distinction
> between the two first cases? I might miss something I don't know in
> option parsing, but I didn't find anything...

I'd try the same approach as default options in Python functions. Make
the default None (which isn't a valid seed) and then you can tell the
difference.
--
Benji York
Senior Software Engineer
Zope Corporation
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


jonathan.ballet at securactive

Oct 9, 2009, 7:28 AM

Post #6 of 16 (1955 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
Hi,

On Fri, Oct 9, 2009 at 3:30 PM, Benji York <benji [at] zope> wrote:
> On Fri, Oct 9, 2009 at 8:51 AM, Jonathan Ballet
> <jonathan.ballet [at] securactive> wrote:
>> If I want this option to have an optional value, I think I must set a
>> default value to the option. But then, how to make the distinction
>> between the two first cases? I might miss something I don't know in
>> option parsing, but I didn't find anything...
>
> I'd try the same approach as default options in Python functions.  Make
> the default None (which isn't a valid seed) and then you can tell the
> difference.

That's what I originally though, but then, how can I make the
difference between 'the user has not set --shuffle' (the option's
value is then None), and 'the user has set the --shuffle option, but
without specifying a seed' (then the option's value is None again)?

Jonathan
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


marius at gedmin

Oct 9, 2009, 11:18 PM

Post #7 of 16 (1960 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
On Fri, Oct 09, 2009 at 02:33:02PM +0200, Christian Theune wrote:
> On 10/08/2009 03:28 PM, Jonathan Ballet wrote:
> > I made a branch 'shuffle-tests', available on the Mercurial repository
> > at http://bitbucket.org/multani/zope.testing-shuffle/ , which adds two
> > new options:
> >
> > * --shuffle to randomize the order of the tests in a layer (layers
> > are not yet randomized, I didn't look at it yet);
> > * --shuffle-seed XXX, to initialize the RNG with a specific value,
> > which allows to reproduce a specific randomization of tests (inspired
> > by this thread http://lists.idyll.org/pipermail/testing-in-python/2009-April/001812.html
> > )
>
> Sounds right. It could even be folded into a single option:
>
> --shuffle[=XXX]

I doubt that. Ordinarily with the GNU long option style these two are
equivalent:

--shuffle=42
--shuffle 42

but there's no way to tell the second variant from a seed-less --shuffle
followed by an non-option argument.

Marius Gedminas
--
http://pov.lt/ -- Zope 3 consulting and development
Attachments: signature.asc (0.19 KB)


sdouche at gmail

Oct 16, 2009, 9:59 AM

Post #8 of 16 (1887 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
On Fri, Oct 9, 2009 at 14:33, Christian Theune <ct [at] gocept> wrote:

Hi Christian.

> That's what working branches are for.

Done :
http://svn.zope.org/zope.testing/branches/sdouche-shuffle/



--
Sebastien Douche <sdouche [at] gmail>
Twitter: http://bit.ly/afkrK (agile, python, open source)
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


jonathan.ballet at securactive

Oct 19, 2009, 2:57 AM

Post #9 of 16 (1815 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
Hello there,

as I mentioned by Sebastien, the patch has been commited into the
sdouche-shuffle branch. It now has tests and a bit of documentation
too.

Regarding CLI arguments, I'm not sure this could be changed like
Christian proposed. If you know how to do it, I'm listening, 'cause I
don't know! :)

I have one more thing I would like to discuss: how should this feature
behave with the --repeat feature?
Currently, the shuffling is done before the repeat feature. This means
that tests will *not* get reordered for each loop (just once, at the
very beginning of tests launching).
I know how to do it, so that for each loop, tests will get a different
order (basically, I would propose to wrap the __iter__() method of the
TestSuite used to launch tests, and to shuffle here):

class ShuffledTestSuite(unittest.TestSuite):
def __iter__(self):
tests = list(super(Shuffled, self))
seed = long(time.time() * 256)
shuffler = random.Random()
shuffler.shuffle(tests)
return iter(tests)

However, using this means we would loose the seed printing (which
seems very convenient to me). I'm not sure if we could get it and
print it for each loop...


Reviews, comments and feedbacks are welcome!
Cheers,

Jonathan
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


ct at gocept

Nov 3, 2009, 2:07 AM

Post #10 of 16 (1568 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
On 10/19/2009 11:57 AM, Jonathan Ballet wrote:
> Hello there,
>
> as I mentioned by Sebastien, the patch has been commited into the
> sdouche-shuffle branch. It now has tests and a bit of documentation
> too.
>
> Regarding CLI arguments, I'm not sure this could be changed like
> Christian proposed. If you know how to do it, I'm listening, 'cause I
> don't know! :)
>
> I have one more thing I would like to discuss: how should this feature
> behave with the --repeat feature?
> Currently, the shuffling is done before the repeat feature. This means
> that tests will *not* get reordered for each loop (just once, at the
> very beginning of tests launching).
> I know how to do it, so that for each loop, tests will get a different
> order (basically, I would propose to wrap the __iter__() method of the
> TestSuite used to launch tests, and to shuffle here):
>
> class ShuffledTestSuite(unittest.TestSuite):
> def __iter__(self):
> tests = list(super(Shuffled, self))
> seed = long(time.time() * 256)
> shuffler = random.Random()
> shuffler.shuffle(tests)
> return iter(tests)
>
> However, using this means we would loose the seed printing (which
> seems very convenient to me). I'm not sure if we could get it and
> print it for each loop...

I'd say leave it as is for now. I think it's more important to be able
to seed (for the sake of reproduction, so btw: do you log the seed if
you generated one?) externally than to reorder during each repeated run.

Also, we can add that later on in case you figure out a way how to do it. ;)

Christia

--
Christian Theune · ct [at] gocept
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development

_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


jonathan.ballet at securactive

Nov 3, 2009, 2:21 AM

Post #11 of 16 (1578 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
Hello Christian,

On Tue, Nov 3, 2009 at 11:07 AM, Christian Theune <ct [at] gocept> wrote:
> [...] btw: do you log the seed if
> you generated one?)

Yep, the seed is displayed at the end of tests, so that it can be
reused to reproduce a failing test.

$ ./bin/test --shuffle -cvvv
Running tests at level 1
Running zope.testing.testrunner.layer.UnitTests tests:
Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
...
Ran 257 tests with 0 failures and 0 errors in 21.908 seconds.
Tests were shuffled using seed number 321854366611

I already find it *very* useful ...

Cheers,

Jonathan
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


jonathan.ballet at securactive

Nov 23, 2009, 1:27 AM

Post #12 of 16 (1290 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
Hello,

On Tue, Nov 3, 2009 at 11:21 AM, Jonathan Ballet
<jonathan.ballet [at] securactive> wrote:
> Hello Christian,
>
> On Tue, Nov 3, 2009 at 11:07 AM, Christian Theune <ct [at] gocept> wrote:
>> [...] btw: do you log the seed if
>> you generated one?)
>
> Yep, the seed is displayed at the end of tests, so that it can be
> reused to reproduce a failing test.
>
>    $ ./bin/test --shuffle -cvvv
>    Running tests at level 1
>    Running zope.testing.testrunner.layer.UnitTests tests:
>      Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
>      ...
>      Ran 257 tests with 0 failures and 0 errors in 21.908 seconds.
>    Tests were shuffled using seed number 321854366611

Any thoughts on this feature? Did someone try it?
Is there any possibilities to merge this into trunk and to release a
new version with this?

Cheers,

Jonathan
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


ct at gocept

Nov 23, 2009, 11:04 AM

Post #13 of 16 (1287 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
Hi there,

On 11/23/2009 10:27 AM, Jonathan Ballet wrote:
> Hello,
>
> On Tue, Nov 3, 2009 at 11:21 AM, Jonathan Ballet
> <jonathan.ballet [at] securactive> wrote:
>> Hello Christian,
>>
>> On Tue, Nov 3, 2009 at 11:07 AM, Christian Theune<ct [at] gocept> wrote:
>>> [...] btw: do you log the seed if
>>> you generated one?)
>>
>> Yep, the seed is displayed at the end of tests, so that it can be
>> reused to reproduce a failing test.
>>
>> $ ./bin/test --shuffle -cvvv
>> Running tests at level 1
>> Running zope.testing.testrunner.layer.UnitTests tests:
>> Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
>> ...
>> Ran 257 tests with 0 failures and 0 errors in 21.908 seconds.
>> Tests were shuffled using seed number 321854366611
>
> Any thoughts on this feature? Did someone try it?
> Is there any possibilities to merge this into trunk and to release a
> new version with this?

It's still on my todo list and I'll get to it at some point ... maybe
even during this week. ;)

Christian

--
Christian Theune · ct [at] gocept
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development

_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


ct at gocept

Dec 18, 2009, 4:51 AM

Post #14 of 16 (1026 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/16/2009 06:59 PM, Sebastien Douche wrote:
| On Fri, Oct 9, 2009 at 14:33, Christian Theune<ct [at] gocept> wrote:
|
| Hi Christian.
|
|> That's what working branches are for.
|
| Done :
| http://svn.zope.org/zope.testing/branches/sdouche-shuffle/

Reviewed and merged.

I made some minor textual changes, otherwise that code was fine.

Christian

- --
Christian Theune · ct [at] gocept
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksrerUACgkQdUt9X/gknwIUXwCgqnT4K11BB0H8R/1uKNOM3toz
uX4An3XKwn+SDAXfnGFm77IyFlAFxJf/
=9VoN
-----END PGP SIGNATURE-----
Attachments: smime.p7s (3.74 KB)


jonathan.ballet at securactive

Dec 18, 2009, 5:49 AM

Post #15 of 16 (1032 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
Hi,

On Fri, Dec 18, 2009 at 1:51 PM, Christian Theune <ct [at] gocept> wrote:
>
> Reviewed and merged.
>
> I made some minor textual changes, otherwise that code was fine.

Thank you Christian!
BTW, the Pypi page for zope.testing is a bit broken currently (the
HTML has not been generated), can you fix this too?

Regards,

Jonathan
_______________________________________________
Zope-Dev maillist - Zope-Dev [at] zope
https://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope )


ct at gocept

Dec 23, 2009, 2:08 AM

Post #16 of 16 (952 views)
Permalink
Re: Testrunner option for running tests in random order?!? [In reply to]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

On 12/18/2009 02:49 PM, Jonathan Ballet wrote:
| Hi,
|
| On Fri, Dec 18, 2009 at 1:51 PM, Christian Theune<ct [at] gocept> wrote:
|>
|> Reviewed and merged.
|>
|> I made some minor textual changes, otherwise that code was fine.
|
| Thank you Christian!
| BTW, the Pypi page for zope.testing is a bit broken currently (the
| HTML has not been generated), can you fix this too?

Gnah. I stared at this for a while and didn't find any fault. Turns out
it probably was a glitch when registering that version with PyPI. I
re-registered the 3.8.6 package and the page shows fine now.

Christian

- --
Christian Theune · ct [at] gocept
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksx7CYACgkQdUt9X/gknwJ4QACgh6dH8M4a1/3uNLAc16PGhi5n
gBQAoJnZCmAFLWttyhmQt1LfkTp7z5ct
=G0//
-----END PGP SIGNATURE-----
Attachments: smime.p7s (3.74 KB)

Zope dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.