Mailing List Archive: Lucene: Java-User

Hanging with fixed thread pool in the IndexSearcher multithread code

Index | Next | Previous | View Threaded

bimargulies at gmail

Feb 19, 2012, 6:08 AM

Post #1 of 10 (351 views)
Permalink

Hanging with fixed thread pool in the IndexSearcher multithread code

3.5.0: I passed a fixed size executor service with one thread, and
then with two threads, to the IndexSearcher constructor.

It hung.

With three threads, it didn't work, but I got different results than
when I don't pass in an executor service at all.

Is this expected? Should the javadoc say something? (I can make a patch).

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

rcmuir at gmail

Feb 19, 2012, 7:39 AM

Post #2 of 10 (341 views)
Permalink

Re: Hanging with fixed thread pool in the IndexSearcher multithread code [In reply to]

On Sun, Feb 19, 2012 at 9:08 AM, Benson Margulies <bimargulies [at] gmail> wrote:
> 3.5.0: I passed a fixed size executor service with one thread, and
> then with two threads, to the IndexSearcher constructor.
>
> It hung.
>
> With three threads, it didn't work, but I got different results than
> when I don't pass in an executor service at all.
>
> Is this expected? Should the javadoc say something? (I can make a patch).
>

I'm not sure I understand the details here, but I don't like the sound
of 'different results': is it possible you can work this down into a
test case that can be attached to jira?

--
lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

bimargulies at gmail

Feb 19, 2012, 8:52 AM

Post #3 of 10 (340 views)
Permalink

Re: Hanging with fixed thread pool in the IndexSearcher multithread code [In reply to]

I should have been clearer; the hang I can make into a test case but I
wondered if is would just get closed as 'works as designed'. the
result discrepancy needs some investigation, I should not have
mentioned it yet.

On Feb 19, 2012, at 10:40 AM, Robert Muir <rcmuir [at] gmail> wrote:

> On Sun, Feb 19, 2012 at 9:08 AM, Benson Margulies <bimargulies [at] gmail> wrote:
>> 3.5.0: I passed a fixed size executor service with one thread, and
>> then with two threads, to the IndexSearcher constructor.
>>
>> It hung.
>>
>> With three threads, it didn't work, but I got different results than
>> when I don't pass in an executor service at all.
>>
>> Is this expected? Should the javadoc say something? (I can make a patch).
>>
>
> I'm not sure I understand the details here, but I don't like the sound
> of 'different results': is it possible you can work this down into a
> test case that can be attached to jira?
>
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

bimargulies at gmail

Feb 19, 2012, 8:55 AM

Post #4 of 10 (338 views)
Permalink

Re: Hanging with fixed thread pool in the IndexSearcher multithread code [In reply to]

and there was a dumb typo.

1 thread: hang
2 threads: hang
3 or more: no hang

On Feb 19, 2012, at 10:40 AM, Robert Muir <rcmuir [at] gmail> wrote:

> On Sun, Feb 19, 2012 at 9:08 AM, Benson Margulies <bimargulies [at] gmail> wrote:
>> 3.5.0: I passed a fixed size executor service with one thread, and
>> then with two threads, to the IndexSearcher constructor.
>>
>> It hung.
>>
>> With three threads, it didn't work, but I got different results than
>> when I don't pass in an executor service at all.
>>
>> Is this expected? Should the javadoc say something? (I can make a patch).
>>
>
> I'm not sure I understand the details here, but I don't like the sound
> of 'different results': is it possible you can work this down into a
> test case that can be attached to jira?
>
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

bimargulies at gmail

Feb 19, 2012, 11:48 AM

Post #5 of 10 (336 views)
Permalink

Re: Hanging with fixed thread pool in the IndexSearcher multithread code [In reply to]

Conveniently, all the 'wrong-result' problems disappeared when I
followed your advice about counting hits.

On Sun, Feb 19, 2012 at 10:39 AM, Robert Muir <rcmuir [at] gmail> wrote:
> On Sun, Feb 19, 2012 at 9:08 AM, Benson Margulies <bimargulies [at] gmail> wrote:
>> 3.5.0: I passed a fixed size executor service with one thread, and
>> then with two threads, to the IndexSearcher constructor.
>>
>> It hung.
>>
>> With three threads, it didn't work, but I got different results than
>> when I don't pass in an executor service at all.
>>
>> Is this expected? Should the javadoc say something? (I can make a patch).
>>
>
> I'm not sure I understand the details here, but I don't like the sound
> of 'different results': is it possible you can work this down into a
> test case that can be attached to jira?
>
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

bimargulies at gmail

Feb 19, 2012, 4:47 PM

Post #6 of 10 (336 views)
Permalink

Re: Hanging with fixed thread pool in the IndexSearcher multithread code [In reply to]

See https://issues.apache.org/jira/browse/LUCENE-3803 for an example
of the hang. I think this nets out to pilot error, but maybe Javadoc
could protect the next person from making the same mistake.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

uwe at thetaphi

Feb 19, 2012, 5:07 PM

Post #7 of 10 (340 views)
Permalink

RE: Hanging with fixed thread pool in the IndexSearcher multithread code [In reply to]

See my response. The problem is not in Lucene; its in general a problem of fixed thread pools that execute other callables from within a callable running at the moment in the same thread pool. Callables are simply waiting for each other.

Use a separate thread pool for Lucene (or whenever you execute new callables from within another running callable)

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe [at] thetaphi

> -----Original Message-----
> From: Benson Margulies [mailto:bimargulies [at] gmail]
> Sent: Monday, February 20, 2012 1:47 AM
> To: java-user [at] lucene
> Subject: Re: Hanging with fixed thread pool in the IndexSearcher multithread
> code
>
> See https://issues.apache.org/jira/browse/LUCENE-3803 for an example of the
> hang. I think this nets out to pilot error, but maybe Javadoc could protect the
> next person from making the same mistake.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

bimargulies at gmail

Feb 19, 2012, 6:01 PM

Post #8 of 10 (346 views)
Permalink

Re: Hanging with fixed thread pool in the IndexSearcher multithread code [In reply to]

On Sun, Feb 19, 2012 at 8:07 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
> See my response. The problem is not in Lucene; its in general a problem of fixed thread pools that execute other callables from within a callable running at the moment in the same thread pool. Callables are simply waiting for each other.
>
> Use a separate thread pool for Lucene (or whenever you execute new callables from within another running callable)

Right. There's nothing like coding a test case to cast one's stupid
errors into high relief. Sorry for all the noise.

>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe [at] thetaphi
>
>> -----Original Message-----
>> From: Benson Margulies [mailto:bimargulies [at] gmail]
>> Sent: Monday, February 20, 2012 1:47 AM
>> To: java-user [at] lucene
>> Subject: Re: Hanging with fixed thread pool in the IndexSearcher multithread
>> code
>>
>> See https://issues.apache.org/jira/browse/LUCENE-3803 for an example of the
>> hang. I think this nets out to pilot error, but maybe Javadoc could protect the
>> next person from making the same mistake.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

trejkaz at trypticon

Feb 19, 2012, 7:39 PM

Post #9 of 10 (335 views)
Permalink

Re: Hanging with fixed thread pool in the IndexSearcher multithread code [In reply to]

On Mon, Feb 20, 2012 at 12:07 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
> See my response. The problem is not in Lucene; its in general a problem of fixed
> thread pools that execute other callables from within a callable running at the
> moment in the same thread pool. Callables are simply waiting for each other.

What we do to get around this issue is to have a utility class which
you call to submit jobs to the executor, but instead of waiting after
submitting them, it starts calling get() starting from the end of the
list. So if there is no other thread available on the executor, the
main thread ends up doing all the work and then returns like normal.

The problem with this solution is that it requires all code in the
system to go through this utility to avoid the issue, and obviously
Lucene is one of those things which isn't written to defend against
this.

Java 7's solution seems to be ForkJoinPool but I gather there is no
simple way to use that with Lucene...

TX

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

bimargulies at gmail

Feb 20, 2012, 3:56 AM

Post #10 of 10 (333 views)
Permalink

Re: Hanging with fixed thread pool in the IndexSearcher multithread code [In reply to]

On Sun, Feb 19, 2012 at 10:39 PM, Trejkaz <trejkaz [at] trypticon> wrote:
> On Mon, Feb 20, 2012 at 12:07 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
>> See my response. The problem is not in Lucene; its in general a problem of fixed
>> thread pools that execute other callables from within a callable running at the
>> moment in the same thread pool. Callables are simply waiting for each other.
>
> What we do to get around this issue is to have a utility class which
> you call to submit jobs to the executor, but instead of waiting after
> submitting them, it starts calling get() starting from the end of the
> list. So if there is no other thread available on the executor, the
> main thread ends up doing all the work and then returns like normal.
>
> The problem with this solution is that it requires all code in the
> system to go through this utility to avoid the issue, and obviously
> Lucene is one of those things which isn't written to defend against
> this.
>
> Java 7's solution seems to be ForkJoinPool but I gather there is no
> simple way to use that with Lucene...

I take it that a pool which rejects too much work (instead of blocking
for a slot) is just as bad from a Lucene standpoint.

>
> TX
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads