Mailing List Archive: Lucene: Java-Dev

Re: (LUCENE-1844) Speed up junit tests

Index | Next | Previous | View Threaded

lucene at mikemccandless

Nov 25, 2009, 2:07 PM

Post #1 of 9 (527 views)
Permalink

Re: (LUCENE-1844) Speed up junit tests

Is the only reason to upgrade to junit 4, to get the parallelization
possibility (which isn't sounding very compelling!)?

Ie, making our unit tests lean is fully independent of junit 4?

Mike

On Wed, Nov 25, 2009 at 4:17 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
>> junit 4 parallelization is still in its infancy. I think the docs for it
>> are just in the changes file that it was first released with. That
>> version had severe bugs that made it almost unusable - I think thats
>> mostly fixed in a newer release. There is also a much better impl of one
>> of the key classes (I think they call it computer) written by someone
>> else that will eventually go into the code base I think (written by the
>> guy(s) that I think found/fixed the initial buggy-ness) - essentially, I
>> think its still unbaked.
>
> There is another problem. Parallelization would only work with tests, that
> do not change gloabl defaults. E.g. LocalizedTestCase changes the default
> locale. If another test would run in Paralale, it would break.
>
> Son only isolated tests can run in parallel. This LocalizedTestCase cannot
> solved in another way. The same would have been in 2.9 with the
> TokenStream.useOnlyNewAPI switch, but this is now longer the case for 3.1.
>
> Uwe
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
> For additional commands, e-mail: java-dev-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

erickerickson at gmail

Nov 25, 2009, 5:29 PM

Post #2 of 9 (494 views)
Permalink

Re: (LUCENE-1844) Speed up junit tests [In reply to]

IMHO there are other reasons to upgrade to junit4 besides
parallelization, there are some nice new capabilities. I
suppose the analogous question is "why upgrade to
Lucene 2.9?"

Especially since it's not a matter of upgrading. Junit3 tests run
just fine under junit4. I've tested after removing the
junit3 jar from lib, no problem. It even seems to run
slightly faster, which makes me wonder...

So really, we have the best of both worlds. No work involved in
using Junit4 with the current tests, but the ability to use the
new features of Junit4. Although I'm sure there'll be
*something* that bites us, I have great faith in Murphy.

Kinda reminds me of the Lucene drop-in replacement policy
<G>.......

But on the topic of parallelization: I'm not at all sure
it's worth the effort. As far as I can tell, it really only gets significant
gains when you have more cores to run with. It's not at all clear
to me how much time we spend doing I/O in the tests... very little
I suspect (although I confess I don't know for sure). And if we're
CPU bound anyway, parallelization doesn't help. Anybody know
for sure?

And say we did all the work to parallelize all the tests. And say that
instead of taking 25 minutes on my 3 year old MacBook Pro, we
got it down to 10 minutes. Who cares? 10 minutes is still too long
according to the eXtreme Programming (XP) folks, and I sympathize
with their point of view. Even though I did spend some time trying
to trim some time.

The XP approach to unit testing is to run it almost every time you
change a line of code. OK, I'm exaggerating, but not by too
much with the die-hard XP folks. Even at 10 minutes, we can't
do that.

So, I think the value for Lucene/SOLR comes *not* from running the
tests 15 times an hour. I think the real value comes from not
letting errors hide for days/weeks/months/releases. So I'm quite willing
to let the automated builds catch the unit test failures in unexpected
places in those instances where I don't run all of the tests before
a patch is committed. As long as we fix them as soon as they're
found.

OK, I'm rambling. I'm off for Thanksgiving, and my daughter is
at her in-laws until tomorrow (they're visiting from CA). So sue
me <G>.

Best
Erick

On Wed, Nov 25, 2009 at 5:07 PM, Michael McCandless <
lucene [at] mikemccandless> wrote:

> Is the only reason to upgrade to junit 4, to get the parallelization
> possibility (which isn't sounding very compelling!)?
>
> Ie, making our unit tests lean is fully independent of junit 4?
>
> Mike
>
> On Wed, Nov 25, 2009 at 4:17 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
> >> junit 4 parallelization is still in its infancy. I think the docs for it
> >> are just in the changes file that it was first released with. That
> >> version had severe bugs that made it almost unusable - I think thats
> >> mostly fixed in a newer release. There is also a much better impl of one
> >> of the key classes (I think they call it computer) written by someone
> >> else that will eventually go into the code base I think (written by the
> >> guy(s) that I think found/fixed the initial buggy-ness) - essentially, I
> >> think its still unbaked.
> >
> > There is another problem. Parallelization would only work with tests,
> that
> > do not change gloabl defaults. E.g. LocalizedTestCase changes the default
> > locale. If another test would run in Paralale, it would break.
> >
> > Son only isolated tests can run in parallel. This LocalizedTestCase
> cannot
> > solved in another way. The same would have been in 2.9 with the
> > TokenStream.useOnlyNewAPI switch, but this is now longer the case for
> 3.1.
> >
> > Uwe
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
> > For additional commands, e-mail: java-dev-help [at] lucene
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
> For additional commands, e-mail: java-dev-help [at] lucene
>
>

lucene at mikemccandless

Nov 26, 2009, 2:14 AM

Post #3 of 9 (489 views)
Permalink

Re: (LUCENE-1844) Speed up junit tests [In reply to]

On Wed, Nov 25, 2009 at 8:29 PM, Erick Erickson <erickerickson [at] gmail> wrote:
> IMHO there are other reasons to upgrade to junit4 besides
> parallelization, there are some nice new capabilities. I
> suppose the analogous question is "why upgrade to
> Lucene 2.9?"

OK then I think we should do the upgrade, and separately think about
how to take advantage of parallelization... I applied the patch for
1084, and also carried it to the back-compat branch, and upgraded the
junit jar in the lib dir to 4.7, and all tests run fine. I also
separately tested 2037, and it ran fine as well.

> Especially since it's not a matter of upgrading. Junit3 tests run
> just fine under junit4. I've tested after removing the
> junit3 jar from lib, no problem. It even seems to run
> slightly faster, which makes me wonder...
>
> So really, we have the best of both worlds. No work involved in
> using Junit4 with the current tests, but the ability to use the
> new features of Junit4. Although I'm sure there'll be
> *something* that bites us, I have great faith in Murphy.
>
> Kinda reminds me of the Lucene drop-in replacement policy
> <G>.......

Heh ;)

> But on the topic of parallelization: I'm not at all sure
> it's worth the effort. As far as I can tell, it really only gets significant
> gains when you have more cores to run with. It's not at all clear
> to me how much time we spend doing I/O in the tests... very little
> I suspect (although I confess I don't know for sure). And if we're
> CPU bound anyway, parallelization doesn't help. Anybody know
> for sure?
>
> And say we did all the work to parallelize all the tests. And say that
> instead of taking 25 minutes on my 3 year old MacBook Pro, we
> got it down to 10 minutes. Who cares? 10 minutes is still too long
> according to the eXtreme Programming (XP) folks, and I sympathize
> with their point of view. Even though I did� spend some time trying
> to trim some time.

If you have enough cores, you can get decent speedups. I have a scary
python wrapper that runs the tests in parallel (runs 5 threads on an 8
core machine... any more threads seems not to help), by letting each
thread run a separate package, and sub-dividing the big (index,
search) packages w/ a temporary patch. Still it takes ~6.5 minutes to
run the tests on my machine these days, which is still too long for
interactive testing...

> The XP approach to unit testing is to run it almost every time you
> change a line of code. OK, I'm exaggerating, but not by too
> much with the die-hard XP folks. Even at 10 minutes, we can't
> do that.
>
> So,� I think the value for Lucene/SOLR comes *not* from running the
> tests 15 times an hour. I think the real value comes from not
> letting errors hide for days/weeks/months/releases. So I'm quite willing
> to let the automated builds catch the unit test failures in unexpected
> places in those instances where I don't run all of the tests before
> a patch is committed. As long as we fix them as soon as they're
> found.

Kinda like what's the point of a cell phone having better battery life
when it's already plenty for my daily usage yet not enough for 2 days
usage, ie, I still have to charge it up, nightly.... it's quantized.
Slightly faster tests won't really make a fundamental change in our
development approach.

But, still, while this speedup won't make the tests fully interactive,
it's still an important improvement. I run the unit tests an insane
number times, and bringing the time down even "just" by 2 minutes will
make a real difference.

> OK, I'm rambling. I'm off for Thanksgiving, and my daughter is
> at her in-laws until tomorrow (they're visiting from CA). So sue
> me <G>.

Happy Thanksgiving!!

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

lucene at mikemccandless

Nov 26, 2009, 2:24 AM

Post #4 of 9 (486 views)
Permalink

Re: (LUCENE-1844) Speed up junit tests [In reply to]

I still think there's value to faster tests, even if they don't become
so fast as to enable "fully interactive testing".

Plus, this is an ongoing goal with time, not a one-time event. As we
create tests we should generally try to maximize coverage and minimize
CPU cost, as long as the effort is smallish.

Mike

On Wed, Nov 25, 2009 at 9:32 PM, Erick Erickson <erickerickson [at] gmail> wrote:
> I posted a rather long diatribe outlining why I think speed-ups
> are a false goal for Lucene. Briefly, I'm convinced that as long
> as the tests are run when Hudson builds Lucene, 99% of the
> value of unit tests is realized. I suppose this implies that the
> hard-core committers agree that as long as failed tests
> are caught/corrected within a day things are fine.
>
> Although coming from a background where unit
> tests are not always required, my viewpoint may be
> suspect <G>.
>
> Erick [at] NotToBeConfusedWithHatcher
>
> On Wed, Nov 25, 2009 at 8:43 PM, Michael McCandless (JIRA)
> <jira [at] apache>wrote:
>
>>
>> � �[
>> https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782716#action_12782716]
>>
>> Michael McCandless commented on LUCENE-1844:
>> --------------------------------------------
>>
>> Will we also speed up back-compat tests?
>>
>> > Speed up junit tests
>> > --------------------
>> >
>> > � � � � � � � � Key: LUCENE-1844
>> > � � � � � � � � URL: https://issues.apache.org/jira/browse/LUCENE-1844
>> > � � � � � � Project: Lucene - Java
>> > � � � � �Issue Type: Improvement
>> > � � � � � �Reporter: Mark Miller
>> > � � � � Attachments: FastCnstScoreQTest.patch,
>> hi_junit_test_runtimes.png, LUCENE-1844.patch
>> >
>> >
>> > As Lucene grows, so does the number of JUnit tests. This is obviously a
>> good thing, but it comes with longer and longer test times. Now that we also
>> run back compat tests in a standard test run, this problem is essentially
>> doubled.
>> > There are some ways this may get better, including running parallel
>> tests. You will need the hardware to fully take advantage, but it should be
>> a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the
>> beginnings of something we might be able to count on soon. 4.6 was buggy,
>> and 4.7 still doesn't come with nice ant integration. Parallel tests will
>> come though.
>> > Beyond parallel testing, I think we also need to concentrate on keeping
>> our tests lean. We don't want to sacrifice coverage or quality, but I'm sure
>> there is plenty of fat to skim.
>> > I've started making a list of some of the longer tests - I think with
>> some work we can make our tests much faster - and then with parallelization,
>> I think we could see some really great gains.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
>> For additional commands, e-mail: java-dev-help [at] lucene
>>
>>
>
> I posted a rather long diatribe outlining why I think speed-ups
> are a false goal for Lucene. Briefly, I'm convinced that as long
> as the tests are run when Hudson builds Lucene, 99% of the
> value of unit tests is realized. I suppose this implies that the
> hard-core committers agree that as long as failed tests
> are caught/corrected within a day things are fine.
>
> Although coming from a background where unit
> tests are not always required, my viewpoint may be
> suspect <G>.
>
> Erick [at] NotToBeConfusedWithHatcher
>
> On Wed, Nov 25, 2009 at 8:43 PM, Michael McCandless (JIRA) <jira [at] apache> wrote:
>>
>> � �[ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782716#action_12782716 ]
>>
>> Michael McCandless commented on LUCENE-1844:
>> --------------------------------------------
>>
>> Will we also speed up back-compat tests?
>>
>> > Speed up junit tests
>> > --------------------
>> >
>> > � � � � � � � � Key: LUCENE-1844
>> > � � � � � � � � URL: https://issues.apache.org/jira/browse/LUCENE-1844
>> > � � � � � � Project: Lucene - Java
>> > � � � � �Issue Type: Improvement
>> > � � � � � �Reporter: Mark Miller
>> > � � � � Attachments: FastCnstScoreQTest.patch, hi_junit_test_runtimes.png, LUCENE-1844.patch
>> >
>> >
>> > As Lucene grows, so does the number of JUnit tests. This is obviously a good thing, but it comes with longer and longer test times. Now that we also run back compat tests in a standard test run, this problem is essentially doubled.
>> > There are some ways this may get better, including running parallel tests. You will need the hardware to fully take advantage, but it should be a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the beginnings of something we might be able to count on soon. 4.6 was buggy, and 4.7 still doesn't come with nice ant integration. Parallel tests will come though.
>> > Beyond parallel testing, I think we also need to concentrate on keeping our tests lean. We don't want to sacrifice coverage or quality, but I'm sure there is plenty of fat to skim.
>> > I've started making a list of some of the longer tests - I think with some work we can make our tests much faster - and then with parallelization, I think we could see some really great gains.
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
>> For additional commands, e-mail: java-dev-help [at] lucene
>>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

markrmiller at gmail

Nov 26, 2009, 6:02 AM

Post #5 of 9 (484 views)
Permalink

Re: (LUCENE-1844) Speed up junit tests [In reply to]

right - as soon as you have to start running the tests often enough,
any decent savings turns into less waiting and more work. Waiting for
tests to run is time that could be better spent elsewhere. And many of
us runthe tests *a lot* considering how long they take. And we will
only keep adding more and will continue to do so.

Also, many of us *are* on multicore and should be able to benifit from
it. I don't dev on anything less than 4 cores these days. It's a life
changer :) and cheap currently. I'd like 8.

- Mark

http://www.lucidimagination.com (mobile)

On Nov 26, 2009, at 5:24 AM, Michael McCandless <lucene [at] mikemccandless
> wrote:

> I still think there's value to faster tests, even if they don't become
> so fast as to enable "fully interactive testing".
>
> Plus, this is an ongoing goal with time, not a one-time event. As we
> create tests we should generally try to maximize coverage and minimize
> CPU cost, as long as the effort is smallish.
>
> Mike
>
> On Wed, Nov 25, 2009 at 9:32 PM, Erick Erickson <erickerickson [at] gmail
> > wrote:
>> I posted a rather long diatribe outlining why I think speed-ups
>> are a false goal for Lucene. Briefly, I'm convinced that as long
>> as the tests are run when Hudson builds Lucene, 99% of the
>> value of unit tests is realized. I suppose this implies that the
>> hard-core committers agree that as long as failed tests
>> are caught/corrected within a day things are fine.
>>
>> Although coming from a background where unit
>> tests are not always required, my viewpoint may be
>> suspect <G>.
>>
>> Erick [at] NotToBeConfusedWithHatcher
>>
>> On Wed, Nov 25, 2009 at 8:43 PM, Michael McCandless (JIRA)
>> <jira [at] apache>wrote:
>>
>>>
>>> [
>>> https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782716#action_12782716]
>>>
>>> Michael McCandless commented on LUCENE-1844:
>>> --------------------------------------------
>>>
>>> Will we also speed up back-compat tests?
>>>
>>>> Speed up junit tests
>>>> --------------------
>>>>
>>>> Key: LUCENE-1844
>>>> URL: https://issues.apache.org/jira/browse/LUCENE-1844
>>>> Project: Lucene - Java
>>>> Issue Type: Improvement
>>>> Reporter: Mark Miller
>>>> Attachments: FastCnstScoreQTest.patch,
>>> hi_junit_test_runtimes.png, LUCENE-1844.patch
>>>>
>>>>
>>>> As Lucene grows, so does the number of JUnit tests. This is
>>>> obviously a
>>> good thing, but it comes with longer and longer test times. Now
>>> that we also
>>> run back compat tests in a standard test run, this problem is
>>> essentially
>>> doubled.
>>>> There are some ways this may get better, including running parallel
>>> tests. You will need the hardware to fully take advantage, but it
>>> should be
>>> a nice gain. There is already an issue for this, and Junit 4.6,
>>> 4.7 have the
>>> beginnings of something we might be able to count on soon. 4.6 was
>>> buggy,
>>> and 4.7 still doesn't come with nice ant integration. Parallel
>>> tests will
>>> come though.
>>>> Beyond parallel testing, I think we also need to concentrate on
>>>> keeping
>>> our tests lean. We don't want to sacrifice coverage or quality,
>>> but I'm sure
>>> there is plenty of fat to skim.
>>>> I've started making a list of some of the longer tests - I think
>>>> with
>>> some work we can make our tests much faster - and then with
>>> parallelization,
>>> I think we could see some really great gains.
>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> -
>>> You can reply to this email to add a comment to the issue online.
>>>
>>>
>>> ---
>>> ------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
>>> For additional commands, e-mail: java-dev-help [at] lucene
>>>
>>>
>>
>> I posted a rather long diatribe outlining why I think speed-ups
>> are a false goal for Lucene. Briefly, I'm convinced that as long
>> as the tests are run when Hudson builds Lucene, 99% of the
>> value of unit tests is realized. I suppose this implies that the
>> hard-core committers agree that as long as failed tests
>> are caught/corrected within a day things are fine.
>>
>> Although coming from a background where unit
>> tests are not always required, my viewpoint may be
>> suspect <G>.
>>
>> Erick [at] NotToBeConfusedWithHatcher
>>
>> On Wed, Nov 25, 2009 at 8:43 PM, Michael McCandless (JIRA) <jira [at] apache
>> > wrote:
>>>
>>> [ https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782716#action_12782716
>>> ]
>>>
>>> Michael McCandless commented on LUCENE-1844:
>>> --------------------------------------------
>>>
>>> Will we also speed up back-compat tests?
>>>
>>>> Speed up junit tests
>>>> --------------------
>>>>
>>>> Key: LUCENE-1844
>>>> URL: https://issues.apache.org/jira/browse/LUCENE-1844
>>>> Project: Lucene - Java
>>>> Issue Type: Improvement
>>>> Reporter: Mark Miller
>>>> Attachments: FastCnstScoreQTest.patch,
>>>> hi_junit_test_runtimes.png, LUCENE-1844.patch
>>>>
>>>>
>>>> As Lucene grows, so does the number of JUnit tests. This is
>>>> obviously a good thing, but it comes with longer and longer test
>>>> times. Now that we also run back compat tests in a standard test
>>>> run, this problem is essentially doubled.
>>>> There are some ways this may get better, including running
>>>> parallel tests. You will need the hardware to fully take
>>>> advantage, but it should be a nice gain. There is already an
>>>> issue for this, and Junit 4.6, 4.7 have the beginnings of
>>>> something we might be able to count on soon. 4.6 was buggy, and
>>>> 4.7 still doesn't come with nice ant integration. Parallel tests
>>>> will come though.
>>>> Beyond parallel testing, I think we also need to concentrate on
>>>> keeping our tests lean. We don't want to sacrifice coverage or
>>>> quality, but I'm sure there is plenty of fat to skim.
>>>> I've started making a list of some of the longer tests - I think
>>>> with some work we can make our tests much faster - and then with
>>>> parallelization, I think we could see some really great gains.
>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> -
>>> You can reply to this email to add a comment to the issue online.
>>>
>>>
>>> ---
>>> ------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
>>> For additional commands, e-mail: java-dev-help [at] lucene
>>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
> For additional commands, e-mail: java-dev-help [at] lucene
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

erickerickson at gmail

Nov 26, 2009, 7:38 AM

Post #6 of 9 (479 views)
Permalink

Re: (LUCENE-1844) Speed up junit tests [In reply to]

Despite my long rambling, I agree that speeding things up is worthwhile.
Just
not a huge deal for some of us poor peons who are on dinky little 2-core
machines and feel inadequate even *talking* to people who have *real*
machines <G>...

Time to go get ready to eat Turkey....

Erick

On Thu, Nov 26, 2009 at 9:02 AM, Mark Miller <markrmiller [at] gmail> wrote:

> right - as soon as you have to start running the tests often enough, any
> decent savings turns into less waiting and more work. Waiting for tests to
> run is time that could be better spent elsewhere. And many of us runthe
> tests *a lot* considering how long they take. And we will only keep adding
> more and will continue to do so.
>
> Also, many of us *are* on multicore and should be able to benifit from it.
> I don't dev on anything less than 4 cores these days. It's a life changer :)
> and cheap currently. I'd like 8.
>
> - Mark
>
> http://www.lucidimagination.com (mobile)
>
>
> On Nov 26, 2009, at 5:24 AM, Michael McCandless <lucene [at] mikemccandless>
> wrote:
>
> I still think there's value to faster tests, even if they don't become
>> so fast as to enable "fully interactive testing".
>>
>> Plus, this is an ongoing goal with time, not a one-time event. As we
>> create tests we should generally try to maximize coverage and minimize
>> CPU cost, as long as the effort is smallish.
>>
>> Mike
>>
>> On Wed, Nov 25, 2009 at 9:32 PM, Erick Erickson <erickerickson [at] gmail>
>> wrote:
>>
>>> I posted a rather long diatribe outlining why I think speed-ups
>>> are a false goal for Lucene. Briefly, I'm convinced that as long
>>> as the tests are run when Hudson builds Lucene, 99% of the
>>> value of unit tests is realized. I suppose this implies that the
>>> hard-core committers agree that as long as failed tests
>>> are caught/corrected within a day things are fine.
>>>
>>> Although coming from a background where unit
>>> tests are not always required, my viewpoint may be
>>> suspect <G>.
>>>
>>> Erick [at] NotToBeConfusedWithHatcher
>>>
>>> On Wed, Nov 25, 2009 at 8:43 PM, Michael McCandless (JIRA)
>>> <jira [at] apache>wrote:
>>>
>>>
>>>> [
>>>>
>>>> https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782716#action_12782716
>>>> ]
>>>>
>>>> Michael McCandless commented on LUCENE-1844:
>>>> --------------------------------------------
>>>>
>>>> Will we also speed up back-compat tests?
>>>>
>>>> Speed up junit tests
>>>>> --------------------
>>>>>
>>>>> Key: LUCENE-1844
>>>>> URL: https://issues.apache.org/jira/browse/LUCENE-1844
>>>>> Project: Lucene - Java
>>>>> Issue Type: Improvement
>>>>> Reporter: Mark Miller
>>>>> Attachments: FastCnstScoreQTest.patch,
>>>>>
>>>> hi_junit_test_runtimes.png, LUCENE-1844.patch
>>>>
>>>>>
>>>>>
>>>>> As Lucene grows, so does the number of JUnit tests. This is obviously a
>>>>>
>>>> good thing, but it comes with longer and longer test times. Now that we
>>>> also
>>>> run back compat tests in a standard test run, this problem is
>>>> essentially
>>>> doubled.
>>>>
>>>>> There are some ways this may get better, including running parallel
>>>>>
>>>> tests. You will need the hardware to fully take advantage, but it should
>>>> be
>>>> a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have
>>>> the
>>>> beginnings of something we might be able to count on soon. 4.6 was
>>>> buggy,
>>>> and 4.7 still doesn't come with nice ant integration. Parallel tests
>>>> will
>>>> come though.
>>>>
>>>>> Beyond parallel testing, I think we also need to concentrate on keeping
>>>>>
>>>> our tests lean. We don't want to sacrifice coverage or quality, but I'm
>>>> sure
>>>> there is plenty of fat to skim.
>>>>
>>>>> I've started making a list of some of the longer tests - I think with
>>>>>
>>>> some work we can make our tests much faster - and then with
>>>> parallelization,
>>>> I think we could see some really great gains.
>>>>
>>>> --
>>>> This message is automatically generated by JIRA.
>>>> -
>>>> You can reply to this email to add a comment to the issue online.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
>>>> For additional commands, e-mail: java-dev-help [at] lucene
>>>>
>>>>
>>>>
>>> I posted a rather long diatribe outlining why I think speed-ups
>>> are a false goal for Lucene. Briefly, I'm convinced that as long
>>> as the tests are run when Hudson builds Lucene, 99% of the
>>> value of unit tests is realized. I suppose this implies that the
>>> hard-core committers agree that as long as failed tests
>>> are caught/corrected within a day things are fine.
>>>
>>> Although coming from a background where unit
>>> tests are not always required, my viewpoint may be
>>> suspect <G>.
>>>
>>> Erick [at] NotToBeConfusedWithHatcher
>>>
>>> On Wed, Nov 25, 2009 at 8:43 PM, Michael McCandless (JIRA) <
>>> jira [at] apache> wrote:
>>>
>>>>
>>>> [
>>>> https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782716#action_12782716
>>>> ]
>>>>
>>>> Michael McCandless commented on LUCENE-1844:
>>>> --------------------------------------------
>>>>
>>>> Will we also speed up back-compat tests?
>>>>
>>>> Speed up junit tests
>>>>> --------------------
>>>>>
>>>>> Key: LUCENE-1844
>>>>> URL: https://issues.apache.org/jira/browse/LUCENE-1844
>>>>> Project: Lucene - Java
>>>>> Issue Type: Improvement
>>>>> Reporter: Mark Miller
>>>>> Attachments: FastCnstScoreQTest.patch,
>>>>> hi_junit_test_runtimes.png, LUCENE-1844.patch
>>>>>
>>>>>
>>>>> As Lucene grows, so does the number of JUnit tests. This is obviously a
>>>>> good thing, but it comes with longer and longer test times. Now that we also
>>>>> run back compat tests in a standard test run, this problem is essentially
>>>>> doubled.
>>>>> There are some ways this may get better, including running parallel
>>>>> tests. You will need the hardware to fully take advantage, but it should be
>>>>> a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the
>>>>> beginnings of something we might be able to count on soon. 4.6 was buggy,
>>>>> and 4.7 still doesn't come with nice ant integration. Parallel tests will
>>>>> come though.
>>>>> Beyond parallel testing, I think we also need to concentrate on keeping
>>>>> our tests lean. We don't want to sacrifice coverage or quality, but I'm sure
>>>>> there is plenty of fat to skim.
>>>>> I've started making a list of some of the longer tests - I think with
>>>>> some work we can make our tests much faster - and then with parallelization,
>>>>> I think we could see some really great gains.
>>>>>
>>>>
>>>> --
>>>> This message is automatically generated by JIRA.
>>>> -
>>>> You can reply to this email to add a comment to the issue online.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
>>>> For additional commands, e-mail: java-dev-help [at] lucene
>>>>
>>>>
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
>> For additional commands, e-mail: java-dev-help [at] lucene
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
> For additional commands, e-mail: java-dev-help [at] lucene
>
>

erickerickson at gmail

Nov 27, 2009, 7:52 AM

Post #7 of 9 (455 views)
Permalink

Re: (LUCENE-1844) Speed up junit tests [In reply to]

But then I got to thinking..... I admit I've only scratched the
surface of the JUnit4 parallelization stuff. That said, it
seems like the real benefit comes from making use of
multiple cores, we don't get huge speedups just from
running multiple threads at once on a single core. Which
makes sense if you're not doing much in the way of I/O.

This notion was inspired by the "scary Python script"
comment.....

So what if we use Ant ForEach construct instead? Yet
again this is a fuzzy idea I'm throwing out without much
to back it up. Mostly I'm wondering if anyone's thought about
it before or can shoot it down before it takes wing. Or if
it is worth exploring.

Assuming we structure our test directories so there are only
directories at the root of the test area, could we persuade Ant
to fire off the tests N directories at a time in parallel?
N would default to 1 but could be passed in to the task, something
like -DmaxThreads=4. ForEach actually has a maxThreads
parameter..... In fact, we wouldn't even need to have only directories
at the test root, but the individual test files at the root would probably
be inefficiently run.

I suspect that keeping the test directories in balance would be
much less work that trying to parallelize using JUnit4, and be
much less fraught with gremlins. This assumes we get
sufficient isolation by Ant running separate threads, about
which I have absolutely NO information. Like I said, mostly
I'm wondering if anybody's gone down this path before and
has wisdom to offer.

Which *still* doesn't mean we shouldn't do whatever we can
to speed up individual tests, but looking that the timings there's
no obvious low-hanging fruit....

I wonder if we could somehow run the various directories in
time order, longest-to-shortest in the hope that all the threads
would finish up "close enough" to the same time. I haven't
thought about *how* to make this happen yet though....

Anyway, I'll be happy to pursue this if y'all think it has merit,
let me know and I'll open a JIRA and take it on. For the
benefit of those aforementioned *real* people with *real*
machines, who I'll rely upon to help test this notion....

Is the poor-mans version of this on a dual-core machine
just running "test-core" and "test-contrib" in two separate
windows?

Best
Erick

On Thu, Nov 26, 2009 at 10:38 AM, Erick Erickson <erickerickson [at] gmail>wrote:

> Despite my long rambling, I agree that speeding things up is worthwhile.
> Just
> not a huge deal for some of us poor peons who are on dinky little 2-core
> machines and feel inadequate even *talking* to people who have *real*
> machines <G>...
>
> Time to go get ready to eat Turkey....
>
> Erick
>
>
> On Thu, Nov 26, 2009 at 9:02 AM, Mark Miller <markrmiller [at] gmail>wrote:
>
>> right - as soon as you have to start running the tests often enough, any
>> decent savings turns into less waiting and more work. Waiting for tests to
>> run is time that could be better spent elsewhere. And many of us runthe
>> tests *a lot* considering how long they take. And we will only keep adding
>> more and will continue to do so.
>>
>> Also, many of us *are* on multicore and should be able to benifit from it.
>> I don't dev on anything less than 4 cores these days. It's a life changer :)
>> and cheap currently. I'd like 8.
>>
>> - Mark
>>
>> http://www.lucidimagination.com (mobile)
>>
>>
>> On Nov 26, 2009, at 5:24 AM, Michael McCandless <
>> lucene [at] mikemccandless> wrote:
>>
>> I still think there's value to faster tests, even if they don't become
>>> so fast as to enable "fully interactive testing".
>>>
>>> Plus, this is an ongoing goal with time, not a one-time event. As we
>>> create tests we should generally try to maximize coverage and minimize
>>> CPU cost, as long as the effort is smallish.
>>>
>>> Mike
>>>
>>> On Wed, Nov 25, 2009 at 9:32 PM, Erick Erickson <erickerickson [at] gmail>
>>> wrote:
>>>
>>>> I posted a rather long diatribe outlining why I think speed-ups
>>>> are a false goal for Lucene. Briefly, I'm convinced that as long
>>>> as the tests are run when Hudson builds Lucene, 99% of the
>>>> value of unit tests is realized. I suppose this implies that the
>>>> hard-core committers agree that as long as failed tests
>>>> are caught/corrected within a day things are fine.
>>>>
>>>> Although coming from a background where unit
>>>> tests are not always required, my viewpoint may be
>>>> suspect <G>.
>>>>
>>>> Erick [at] NotToBeConfusedWithHatcher
>>>>
>>>> On Wed, Nov 25, 2009 at 8:43 PM, Michael McCandless (JIRA)
>>>> <jira [at] apache>wrote:
>>>>
>>>>
>>>>> [
>>>>>
>>>>> https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782716#action_12782716
>>>>> ]
>>>>>
>>>>> Michael McCandless commented on LUCENE-1844:
>>>>> --------------------------------------------
>>>>>
>>>>> Will we also speed up back-compat tests?
>>>>>
>>>>> Speed up junit tests
>>>>>> --------------------
>>>>>>
>>>>>> Key: LUCENE-1844
>>>>>> URL: https://issues.apache.org/jira/browse/LUCENE-1844
>>>>>> Project: Lucene - Java
>>>>>> Issue Type: Improvement
>>>>>> Reporter: Mark Miller
>>>>>> Attachments: FastCnstScoreQTest.patch,
>>>>>>
>>>>> hi_junit_test_runtimes.png, LUCENE-1844.patch
>>>>>
>>>>>>
>>>>>>
>>>>>> As Lucene grows, so does the number of JUnit tests. This is obviously
>>>>>> a
>>>>>>
>>>>> good thing, but it comes with longer and longer test times. Now that we
>>>>> also
>>>>> run back compat tests in a standard test run, this problem is
>>>>> essentially
>>>>> doubled.
>>>>>
>>>>>> There are some ways this may get better, including running parallel
>>>>>>
>>>>> tests. You will need the hardware to fully take advantage, but it
>>>>> should be
>>>>> a nice gain. There is already an issue for this, and Junit 4.6, 4.7
>>>>> have the
>>>>> beginnings of something we might be able to count on soon. 4.6 was
>>>>> buggy,
>>>>> and 4.7 still doesn't come with nice ant integration. Parallel tests
>>>>> will
>>>>> come though.
>>>>>
>>>>>> Beyond parallel testing, I think we also need to concentrate on
>>>>>> keeping
>>>>>>
>>>>> our tests lean. We don't want to sacrifice coverage or quality, but I'm
>>>>> sure
>>>>> there is plenty of fat to skim.
>>>>>
>>>>>> I've started making a list of some of the longer tests - I think with
>>>>>>
>>>>> some work we can make our tests much faster - and then with
>>>>> parallelization,
>>>>> I think we could see some really great gains.
>>>>>
>>>>> --
>>>>> This message is automatically generated by JIRA.
>>>>> -
>>>>> You can reply to this email to add a comment to the issue online.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
>>>>> For additional commands, e-mail: java-dev-help [at] lucene
>>>>>
>>>>>
>>>>>
>>>> I posted a rather long diatribe outlining why I think speed-ups
>>>> are a false goal for Lucene. Briefly, I'm convinced that as long
>>>> as the tests are run when Hudson builds Lucene, 99% of the
>>>> value of unit tests is realized. I suppose this implies that the
>>>> hard-core committers agree that as long as failed tests
>>>> are caught/corrected within a day things are fine.
>>>>
>>>> Although coming from a background where unit
>>>> tests are not always required, my viewpoint may be
>>>> suspect <G>.
>>>>
>>>> Erick [at] NotToBeConfusedWithHatcher
>>>>
>>>> On Wed, Nov 25, 2009 at 8:43 PM, Michael McCandless (JIRA) <
>>>> jira [at] apache> wrote:
>>>>
>>>>>
>>>>> [
>>>>> https://issues.apache.org/jira/browse/LUCENE-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782716#action_12782716
>>>>> ]
>>>>>
>>>>> Michael McCandless commented on LUCENE-1844:
>>>>> --------------------------------------------
>>>>>
>>>>> Will we also speed up back-compat tests?
>>>>>
>>>>> Speed up junit tests
>>>>>> --------------------
>>>>>>
>>>>>> Key: LUCENE-1844
>>>>>> URL: https://issues.apache.org/jira/browse/LUCENE-1844
>>>>>> Project: Lucene - Java
>>>>>> Issue Type: Improvement
>>>>>> Reporter: Mark Miller
>>>>>> Attachments: FastCnstScoreQTest.patch,
>>>>>> hi_junit_test_runtimes.png, LUCENE-1844.patch
>>>>>>
>>>>>>
>>>>>> As Lucene grows, so does the number of JUnit tests. This is obviously
>>>>>> a good thing, but it comes with longer and longer test times. Now that we
>>>>>> also run back compat tests in a standard test run, this problem is
>>>>>> essentially doubled.
>>>>>> There are some ways this may get better, including running parallel
>>>>>> tests. You will need the hardware to fully take advantage, but it should be
>>>>>> a nice gain. There is already an issue for this, and Junit 4.6, 4.7 have the
>>>>>> beginnings of something we might be able to count on soon. 4.6 was buggy,
>>>>>> and 4.7 still doesn't come with nice ant integration. Parallel tests will
>>>>>> come though.
>>>>>> Beyond parallel testing, I think we also need to concentrate on
>>>>>> keeping our tests lean. We don't want to sacrifice coverage or quality, but
>>>>>> I'm sure there is plenty of fat to skim.
>>>>>> I've started making a list of some of the longer tests - I think with
>>>>>> some work we can make our tests much faster - and then with parallelization,
>>>>>> I think we could see some really great gains.
>>>>>>
>>>>>
>>>>> --
>>>>> This message is automatically generated by JIRA.
>>>>> -
>>>>> You can reply to this email to add a comment to the issue online.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
>>>>> For additional commands, e-mail: java-dev-help [at] lucene
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
>>> For additional commands, e-mail: java-dev-help [at] lucene
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
>> For additional commands, e-mail: java-dev-help [at] lucene
>>
>>
>

lucene at mikemccandless

Nov 27, 2009, 10:27 AM

Post #8 of 9 (453 views)
Permalink

Re: (LUCENE-1844) Speed up junit tests [In reply to]

On Fri, Nov 27, 2009 at 10:52 AM, Erick Erickson
<erickerickson [at] gmail> wrote:
> But then I got to thinking..... I admit I've only scratched the
> surface of the JUnit4 parallelization stuff. That said, it
> seems like the real benefit comes from making use of
> multiple cores, we don't get huge speedups just from
> running multiple threads at once on a single core. Which
> makes sense if you're not doing much in the way of I/O.

Right, it's the multi-core machines that gain the most from this.

> This notion was inspired by the "scary Python script"
> comment.....
>
> So what if we use Ant ForEach construct instead? Yet
> again this is a fuzzy idea I'm throwing out without much
> to back it up. Mostly I'm wondering if anyone's thought about
> it before or can shoot it down before it takes wing. Or if
> it is worth exploring.
>
> Assuming we structure our test directories so there are only
> directories at the root of the test area, could we persuade Ant
> to fire off the tests N directories at a time in parallel?
> N would default to 1 but could be passed in to the task, something
> like -DmaxThreads=4. ForEach actually has a maxThreads
> parameter..... In fact, we wouldn't even need to have only directories
> at the test root, but the individual test files at the root would probably
> be inefficiently run.
>
> I suspect that keeping the test directories in balance would be
> much less work that trying to parallelize using JUnit4, and be
> much less fraught with gremlins. This assumes we get
> sufficient isolation by Ant running separate threads, about
> which I have absolutely NO information. Like I said, mostly
> I'm wondering if anybody's gone down this path before and
> has wisdom to offer.

I think this rough idea is a good approach, though I don't know much
about ant's ForEach.

One thing the scary Python script does is divide up index & search
packages into 2 parts ("a" and "b"), by breaking up the tests
according to 1st letter. We might be able to take a similar approach,
so that we're not forced to unnaturally separate tests into subdirs?

The entire index or search package was too slow to run otherwise (ie,
I needed to throw concurrency at it).

> Which *still* doesn't mean we shouldn't do whatever we can
> to speed up individual tests, but looking that the timings there's
> no obvious low-hanging fruit....

Yup. It's definitely an ongoing thing too...

> I wonder if we could somehow run the various directories in
> time order, longest-to-shortest in the hope that all the threads
> would finish up "close enough" to the same time. I haven't
> thought about *how* to make this happen yet though....

This is very important -- I do the same thing in the python script.

Also, will ant's ForEach take a set of say 30 things to work on, and
take the # threads to use, and just pull from that queue of 30, in
order?

> Anyway, I'll be happy to pursue this if y'all think it has merit,
> let me know and I'll open a JIRA and take it on. For the
> benefit of those aforementioned *real* people with *real*
> machines, who I'll rely upon to help test this notion....
>
> Is the poor-mans version of this on a dual-core machine
> just running "test-core" and "test-contrib" in two separate
> windows?

I think you could, except, I think they share sub-tasks (eg,
"compile-core") so the two will sometimes stomp on each other.

The scary python script first uses a single thread to compile
everything, then runs N threads pulling from the queue. BUT: I apply
a temporary patch to the ant build files, so that the N threads do not
try to, eg, compile-core or jar-core, separately.

Also one thing I'd love to try is NOT forking the JVM for each test
(fork="no" in the junit task). I wonder how much time that'd buy...

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

erickerickson at gmail

Nov 27, 2009, 12:38 PM

Post #9 of 9 (448 views)
Permalink

Re: (LUCENE-1844) Speed up junit tests [In reply to]

<<<Also, will ant's ForEach take a set of say 30 things to work on, and
take the # threads to use, and just pull from that queue of 30, in
order?>>>

That's the implication I took from here:
http://ant-contrib.sourceforge.net/tasks/tasks/index.html

Ignorance is bliss, I didn't find the ForEach by looking at Ant
documentation, but by googling "ant parallel". Turns out this
is in Contrib. I don't even know if it's current.

Tell ya' what. I'll take a quick whack at it. I'm a believer
in prototyping if at all possible. So I'll create a really stupid
implementation of this with a hard-coded list of tests to run
and see what happens. If it works for me, I'll pass it along
to whoever wants to give it a spin and we'll get a clue whether
it provides enough of an improvement to pursue seriously.

I'll open a JIRA since at least Mike and I seem to be interested....

Erick

On Fri, Nov 27, 2009 at 1:27 PM, Michael McCandless <
lucene [at] mikemccandless> wrote:

> On Fri, Nov 27, 2009 at 10:52 AM, Erick Erickson
> <erickerickson [at] gmail> wrote:
> > But then I got to thinking..... I admit I've only scratched the
> > surface of the JUnit4 parallelization stuff. That said, it
> > seems like the real benefit comes from making use of
> > multiple cores, we don't get huge speedups just from
> > running multiple threads at once on a single core. Which
> > makes sense if you're not doing much in the way of I/O.
>
> Right, it's the multi-core machines that gain the most from this.
>
> > This notion was inspired by the "scary Python script"
> > comment.....
> >
> > So what if we use Ant ForEach construct instead? Yet
> > again this is a fuzzy idea I'm throwing out without much
> > to back it up. Mostly I'm wondering if anyone's thought about
> > it before or can shoot it down before it takes wing. Or if
> > it is worth exploring.
> >
> > Assuming we structure our test directories so there are only
> > directories at the root of the test area, could we persuade Ant
> > to fire off the tests N directories at a time in parallel?
> > N would default to 1 but could be passed in to the task, something
> > like -DmaxThreads=4. ForEach actually has a maxThreads
> > parameter..... In fact, we wouldn't even need to have only directories
> > at the test root, but the individual test files at the root would
> probably
> > be inefficiently run.
> >
> > I suspect that keeping the test directories in balance would be
> > much less work that trying to parallelize using JUnit4, and be
> > much less fraught with gremlins. This assumes we get
> > sufficient isolation by Ant running separate threads, about
> > which I have absolutely NO information. Like I said, mostly
> > I'm wondering if anybody's gone down this path before and
> > has wisdom to offer.
>
> I think this rough idea is a good approach, though I don't know much
> about ant's ForEach.
>
> One thing the scary Python script does is divide up index & search
> packages into 2 parts ("a" and "b"), by breaking up the tests
> according to 1st letter. We might be able to take a similar approach,
> so that we're not forced to unnaturally separate tests into subdirs?
>
> The entire index or search package was too slow to run otherwise (ie,
> I needed to throw concurrency at it).
>
> > Which *still* doesn't mean we shouldn't do whatever we can
> > to speed up individual tests, but looking that the timings there's
> > no obvious low-hanging fruit....
>
> Yup. It's definitely an ongoing thing too...
>
> > I wonder if we could somehow run the various directories in
> > time order, longest-to-shortest in the hope that all the threads
> > would finish up "close enough" to the same time. I haven't
> > thought about *how* to make this happen yet though....
>
> This is very important -- I do the same thing in the python script.
>
> Also, will ant's ForEach take a set of say 30 things to work on, and
> take the # threads to use, and just pull from that queue of 30, in
> order?
>
> > Anyway, I'll be happy to pursue this if y'all think it has merit,
> > let me know and I'll open a JIRA and take it on. For the
> > benefit of those aforementioned *real* people with *real*
> > machines, who I'll rely upon to help test this notion....
> >
> > Is the poor-mans version of this on a dual-core machine
> > just running "test-core" and "test-contrib" in two separate
> > windows?
>
> I think you could, except, I think they share sub-tasks (eg,
> "compile-core") so the two will sometimes stomp on each other.
>
> The scary python script first uses a single thread to compile
> everything, then runs N threads pulling from the queue. BUT: I apply
> a temporary patch to the ant build files, so that the N threads do not
> try to, eg, compile-core or jar-core, separately.
>
> Also one thing I'd love to try is NOT forking the JVM for each test
> (fork="no" in the junit task). I wonder how much time that'd buy...
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
> For additional commands, e-mail: java-dev-help [at] lucene
>
>

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads