Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Lucene: Java-User

Here a merge thread, there a merge thread ...

 

 

Lucene java-user RSS feed   Index | Next | Previous | View Threaded


bimargulies at gmail

Feb 19, 2012, 6:05 PM

Post #1 of 5 (843 views)
Permalink
Here a merge thread, there a merge thread ...

A long-running program of mine (which Uwe's read a model of) slowly
keeps adding merge threads. I count 22 at the moment. Each one shows
up, runs for a bit, and then goes to sleep for, seemingly ever. I
don't do anything explicit to control merging behavior.

They name themselves "Lucene Merge Thread #xxx" where xxx is a
non-contiguous but ever-growing number.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


lucene at mikemccandless

Feb 24, 2012, 7:59 AM

Post #2 of 5 (826 views)
Permalink
Re: Here a merge thread, there a merge thread ... [In reply to]

This is from ConcurrentMergeScheduler (the default MergeScheduler).

But, are you sure the threads are sleeping, not exiting? (They should
be exiting).

This merge scheduler starts a new thread when a merge is needed,
allows that thread to do another merge (if one is immediately
available), else the thread exits.

Mike McCandless

http://blog.mikemccandless.com

On Sun, Feb 19, 2012 at 9:05 PM, Benson Margulies <bimargulies [at] gmail> wrote:
> A long-running program of mine (which Uwe's read a model of) slowly
> keeps adding merge threads. I count 22 at the moment. Each one shows
> up, runs for a bit, and then goes to sleep for, seemingly ever. I
> don't do anything explicit to control merging behavior.
>
> They name themselves "Lucene Merge Thread #xxx" where xxx is a
> non-contiguous but ever-growing number.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


bimargulies at gmail

Feb 24, 2012, 10:03 AM

Post #3 of 5 (818 views)
Permalink
Re: Here a merge thread, there a merge thread ... [In reply to]

On Fri, Feb 24, 2012 at 10:59 AM, Michael McCandless
<lucene [at] mikemccandless> wrote:
> This is from ConcurrentMergeScheduler (the default MergeScheduler).
>
> But, are you sure the threads are sleeping, not exiting?  (They should
> be exiting).
>
> This merge scheduler starts a new thread when a merge is needed,
> allows that thread to do another merge (if one is immediately
> available), else the thread exits.

They seem to exit eventually, but not quite as soon as they arrive.


>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sun, Feb 19, 2012 at 9:05 PM, Benson Margulies <bimargulies [at] gmail> wrote:
>> A long-running program of mine (which Uwe's read a model of) slowly
>> keeps adding merge threads. I count 22 at the moment. Each one shows
>> up, runs for a bit, and then goes to sleep for, seemingly ever. I
>> don't do anything explicit to control merging behavior.
>>
>> They name themselves "Lucene Merge Thread #xxx" where xxx is a
>> non-contiguous but ever-growing number.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


goksron at gmail

Feb 25, 2012, 4:04 PM

Post #4 of 5 (817 views)
Permalink
Re: Here a merge thread, there a merge thread ... [In reply to]

Solr uses TieredMergeScheduler by default now. You might find this
works more smoothly.

On Fri, Feb 24, 2012 at 10:03 AM, Benson Margulies
<bimargulies [at] gmail> wrote:
> On Fri, Feb 24, 2012 at 10:59 AM, Michael McCandless
> <lucene [at] mikemccandless> wrote:
>> This is from ConcurrentMergeScheduler (the default MergeScheduler).
>>
>> But, are you sure the threads are sleeping, not exiting?  (They should
>> be exiting).
>>
>> This merge scheduler starts a new thread when a merge is needed,
>> allows that thread to do another merge (if one is immediately
>> available), else the thread exits.
>
> They seem to exit eventually, but not quite as soon as they arrive.
>
>
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Sun, Feb 19, 2012 at 9:05 PM, Benson Margulies <bimargulies [at] gmail> wrote:
>>> A long-running program of mine (which Uwe's read a model of) slowly
>>> keeps adding merge threads. I count 22 at the moment. Each one shows
>>> up, runs for a bit, and then goes to sleep for, seemingly ever. I
>>> don't do anything explicit to control merging behavior.
>>>
>>> They name themselves "Lucene Merge Thread #xxx" where xxx is a
>>> non-contiguous but ever-growing number.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>> For additional commands, e-mail: java-user-help [at] lucene
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>



--
Lance Norskog
goksron [at] gmail

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


uwe at thetaphi

Feb 25, 2012, 4:11 PM

Post #5 of 5 (824 views)
Permalink
RE: Here a merge thread, there a merge thread ... [In reply to]

Lance,

There is no TieredMergeScheduler. You somehow confuse MergeSchedule with MergePolicy. TieredMergePolicy is new, but has nothing to do with the problem here. The MergeScheduler executres the merges serial (SerialMergeScheduler) or parallel (ConcurrentMergeScheduler, the default since long time). The MergePolicy simply tells under which conditions and how segments are merged.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe [at] thetaphi

> -----Original Message-----
> From: Lance Norskog [mailto:goksron [at] gmail]
> Sent: Sunday, February 26, 2012 1:05 AM
> To: java-user [at] lucene
> Subject: Re: Here a merge thread, there a merge thread ...
>
> Solr uses TieredMergeScheduler by default now. You might find this works
> more smoothly.
>
> On Fri, Feb 24, 2012 at 10:03 AM, Benson Margulies
> <bimargulies [at] gmail> wrote:
> > On Fri, Feb 24, 2012 at 10:59 AM, Michael McCandless
> > <lucene [at] mikemccandless> wrote:
> >> This is from ConcurrentMergeScheduler (the default MergeScheduler).
> >>
> >> But, are you sure the threads are sleeping, not exiting? (They
> >> should be exiting).
> >>
> >> This merge scheduler starts a new thread when a merge is needed,
> >> allows that thread to do another merge (if one is immediately
> >> available), else the thread exits.
> >
> > They seem to exit eventually, but not quite as soon as they arrive.
> >
> >
> >>
> >> Mike McCandless
> >>
> >> http://blog.mikemccandless.com
> >>
> >> On Sun, Feb 19, 2012 at 9:05 PM, Benson Margulies
> <bimargulies [at] gmail> wrote:
> >>> A long-running program of mine (which Uwe's read a model of) slowly
> >>> keeps adding merge threads. I count 22 at the moment. Each one shows
> >>> up, runs for a bit, and then goes to sleep for, seemingly ever. I
> >>> don't do anything explicit to control merging behavior.
> >>>
> >>> They name themselves "Lucene Merge Thread #xxx" where xxx is a
> >>> non-contiguous but ever-growing number.
> >>>
> >>> --------------------------------------------------------------------
> >>> - To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> >>> For additional commands, e-mail: java-user-help [at] lucene
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> >> For additional commands, e-mail: java-user-help [at] lucene
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > For additional commands, e-mail: java-user-help [at] lucene
> >
>
>
>
> --
> Lance Norskog
> goksron [at] gmail
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Lucene java-user RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.