lucene at mikemccandless
Jun 2, 2012, 10:56 AM
Post #3 of 3
(150 views)
Permalink
|
It could be your index has an unusual number of unique terms.
If you can upgrade to the latest 3.x, the RAM used by the terms index
has been very substantially reduced...
If not, try setting the termInfosIndexDivisor to eg 2 or 3 ... this
will load 1/2 or 1/3 of the indexed terms into RAM, but may slow down
certain searches...
Mike McCandless
http://blog.mikemccandless.com
On Sat, Jun 2, 2012 at 12:02 PM, Jack Krupansky <jack [at] basetechnology> wrote:
> Have you tried a System.gc before opening the index? And check how much
> memory is available before the open. Does the open seem to be taking more
> memory that could reasonably be expected? How big is the index, how many
> docs, how many unique terms in that index?
>
> 1.8G doesn't sound like a lot. You may simply have to bite the bullet and
> move to an environment with more memory. Or, split the index and run on
> multiple machines/JVMs.
>
> Have many deletes been performed? How many segments are there? Maybe you can
> merge using a standalone program that doesn't have the rest of your app
> consuming memory.
>
> -- Jack Krupansky
>
> -----Original Message----- From: nishesh.gupta [at] emc
> Sent: Friday, June 01, 2012 7:53 PM
> To: java-user [at] lucene
> Subject: OOM during IndexReader open
>
>
> Hi,
>
> I am getting the following OOM consistently whenever the index is opened .
> Is it because now the index is holding too many terms ? Our application (
> that has Lucene 2.9.3 ) already has reached 1.8G of virtual memory.
> Is there a way I can avoid OOM during IndexReader open because of too many
> terms ?
>
>
> 2012-05-30 14:12:46,139 [MC:10.2.2.1-1332015615933-1264
> FS:parinfprdcl01k_655394_0] ERROR util - Context: MC: 10.2.2
> .1-1332015615933-1264 failed at the state LocalIndexOptimizeAndCompress,
> NumActiveThreads = 64. Error: java.lang.OutO fMemoryError: Java heap
> space % STACK:
> org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:90)
> org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:122)
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:652)
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:613)
> org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:106)
> org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683)
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:71)
> org.apache.lucene.index.IndexReader.open(IndexReader.java:482)
> org.apache.lucene.index.IndexReader.open(IndexReader.java:320)
> com.kazeon.search.indexingengine.context.MergerContext.deleteAgainstFiler(MergerContext.java:1357)
> com.kazeon.search.indexingengine.context.MergerContext.deleteAgainstFiler(MergerContext.java:656)
> com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIndexOptimizeAndCompressModifier.modifyState
> Aux(LocalIndexOptimizeAndCompressModifier.java:303)
> com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIndexOptimizeAndCompressModifier.mergeAllICs
> (LocalIndexOptimizeAndCompressModifier.java:175)
> com.kazeon.search.indexingengine.statemachine.modifiers.merger.LocalIndexOptimizeAndCompressModifier.modifyState
> (LocalIndexOptimizeAndCompressModifier.java:106)
> com.kazeon.util.scoreboard.WorkerThread.run(WorkerThread.java:31)
>
>
> thanks,
> Nishesh
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene
|
