Mailing List Archive: Lucene: Java-User

CloseableThreadLocal problem

Index | Next | Previous | View Threaded

matthewb at labkey

Feb 29, 2012, 9:17 AM

Post #1 of 3 (136 views)
Permalink

CloseableThreadLocal problem

We tracked down a large memory leak (effectively a leak anyway) caused
by how Analyzer users CloseableThreadLocal.
CloseableThreadLocal.hardRefs holds references to Thread objects as
keys. The problem is that it only frees these references in the set()
method, and SnowballAnalyzer will only call set() when it is used by a
NEW thread.

The problem scenario is as follows:

The server experiences a spike in usage (say by robots or whatever)
and many threads are created and referenced by
CloseableThreadLocal.hardRefs. The server quiesces and lets many of
these threads expire normally. Now we have a smaller, but adequate
thread pool. So CloseableThreadLocal.set() may not be called by
SnowBallAnalyzer (via Analyzer) for a _long_ time. The purge code is
never called, and these threads along with their thread local storage
(lucene related or not) is never cleaned up.

I think calling the purge code in both get() and set() would have
avoided this problem, perhaps using WeakHashMap instead of HashMap may
also have helped (WeakHashMap purges on get() and set())

Our current work around is to not share SnowBallAnalyzer instances
among HTTP searcher threads. We open and close one on every request.

Thanks,
Matt

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Mar 1, 2012, 2:32 PM

Post #2 of 3 (120 views)
Permalink

Re: CloseableThreadLocal problem [In reply to]

Phew, tricky.

The problem is purging is potentially costly... it iterates all
entries in the map (threads that have called get) looking for dead
threads.

Can you open an issue...? We can iterate there. Thanks for raising this,

Mike McCandless

http://blog.mikemccandless.com

On Wed, Feb 29, 2012 at 12:17 PM, Matthew Bellew <matthewb [at] labkey> wrote:
> We tracked down a large memory leak (effectively a leak anyway) caused
> by how Analyzer users CloseableThreadLocal.
> CloseableThreadLocal.hardRefs holds references to Thread objects as
> keys. The problem is that it only frees these references in the set()
> method, and SnowballAnalyzer will only call set() when it is used by a
> NEW thread.
>
> The problem scenario is as follows:
>
> The server experiences a spike in usage (say by robots or whatever)
> and many threads are created and referenced by
> CloseableThreadLocal.hardRefs. The server quiesces and lets many of
> these threads expire normally. Now we have a smaller, but adequate
> thread pool. So CloseableThreadLocal.set() may not be called by
> SnowBallAnalyzer (via Analyzer) for a _long_ time. The purge code is
> never called, and these threads along with their thread local storage
> (lucene related or not) is never cleaned up.
>
> I think calling the purge code in both get() and set() would have
> avoided this problem, perhaps using WeakHashMap instead of HashMap may
> also have helped (WeakHashMap purges on get() and set())
>
> Our current work around is to not share SnowBallAnalyzer instances
> among HTTP searcher threads. We open and close one on every request.
>
> Thanks,
> Matt
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

matthewb at labkey

Mar 2, 2012, 3:56 PM

Post #3 of 3 (119 views)
Permalink

Re: CloseableThreadLocal problem [In reply to]

Thanks for your reply MIke, I create this bug.

https://issues.apache.org/jira/browse/LUCENE-3841

Matt

On Thu, Mar 1, 2012 at 2:32 PM, Michael McCandless
<lucene [at] mikemccandless> wrote:
> Phew, tricky.
>
> The problem is purging is potentially costly... it iterates all
> entries in the map (threads that have called get) looking for dead
> threads.
>
> Can you open an issue...? Ā We can iterate there. Ā Thanks for raising this,
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Wed, Feb 29, 2012 at 12:17 PM, Matthew Bellew <matthewb [at] labkey> wrote:
>> We tracked down a large memory leak (effectively a leak anyway) caused
>> by how Analyzer users CloseableThreadLocal.
>> CloseableThreadLocal.hardRefs holds references to Thread objects as
>> keys. Ā The problem is that it only frees these references in the set()
>> method, and SnowballAnalyzer will only call set() when it is used by a
>> NEW thread.
>>
>> The problem scenario is as follows:
>>
>> The server experiences a spike in usage (say by robots or whatever)
>> and many threads are created and referenced by
>> CloseableThreadLocal.hardRefs. Ā The server quiesces and lets many of
>> these threads expire normally. Ā Now we have a smaller, but adequate
>> thread pool. Ā So CloseableThreadLocal.set() may not be called by
>> SnowBallAnalyzer (via Analyzer) for a _long_ time. Ā The purge code is
>> never called, and these threads along with their thread local storage
>> (lucene related or not) is never cleaned up.
>>
>> I think calling the purge code in both get() and set() would have
>> avoided this problem, perhaps using WeakHashMap instead of HashMap may
>> also have helped (WeakHashMap purges on get() and set())
>>
>> Our current work around is to not share SnowBallAnalyzer instances
>> among HTTP searcher threads. Ā We open and close one on every request.
>>
>> Thanks,
>> Matt
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads