Mailing List Archive: Lucene: Java-User

OOM with 2.9

Index | Next | Previous | View Threaded

eksdev at yahoo

Jul 12, 2009, 4:22 AM

Post #1 of 15 (1496 views)
Permalink

OOM with 2.9

Hi,
We just upgraded to 2.9 and noticed some (to me) not expected OOM.

We use MMapDirectory and after upgrade, on exactly the same Index/machine/jvm/params/setup... we cannot start index as mapping screams "No memory"

any explanation why this could be the case?

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

eksdev at yahoo

Jul 12, 2009, 4:23 AM

Post #2 of 15 (1461 views)
Permalink

Re: OOM with 2.9 [In reply to]

Stack trace

java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(Unknown Source)
at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(Unknown Source)
at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(Unknown Source)
at org.apache.lucene.store.MMapDirectory.openInput(Unknown Source)
at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(Unknown Source)
at org.apache.lucene.index.SegmentReader.get(Unknown Source)
at org.apache.lucene.index.SegmentReader.get(Unknown Source)
at org.apache.lucene.index.DirectoryReader.<init>(Unknown Source)
at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(Unknown Source)
at org.apache.lucene.index.DirectoryReader$1.doBody(Unknown Source)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(Unknown Source)
at org.apache.lucene.index.DirectoryReader.open(Unknown Source)
at org.apache.lucene.index.IndexReader.open(Unknown Source)
at org.apache.lucene.index.IndexReader.open(Unknown Source)
at com.qspi.su.search.IndexReaderFactory.getNewIndexReaderInstance(IndexReaderFactory.java:70)
at com.qspi.su.search.SearchStrategy.<init>(SearchStrategy.java:147)
at com.qspi.su.search.Searcher.<init>(Searcher.java:170)
at com.qspi.su.search.Searchers.<init>(Searchers.java:21)
at com.qspi.su.ipc.SUProtocolImpl.<init>(SUProtocolImpl.java:76)
at com.qspi.su.ipc.SUServer.run(SUServer.java:84)
at com.qspi.su.ipc.SUServer.main(SUServer.java:204)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

uwe at thetaphi

Jul 12, 2009, 4:39 AM

Post #3 of 15 (1458 views)
Permalink

RE: OOM with 2.9 [In reply to]

Is it a 32 or 64 bit JVM? If 32 bit, this error can happen, if the new
Lucene Core needs more real memory and so the MMaped address space is
already used. MMapDirectory should only be used with 64 bit machines.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe [at] thetaphi

> -----Original Message-----
> From: eks dev [mailto:eksdev [at] yahoo]
> Sent: Sunday, July 12, 2009 1:24 PM
> To: java-user [at] lucene
> Subject: Re: OOM with 2.9
>
>
> Stack trace
>
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(Unknown
> Source)
> at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(Unknown
> Source)
> at org.apache.lucene.store.MMapDirectory.openInput(Unknown Source)
> at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(Unknown
> Source)
> at org.apache.lucene.index.SegmentReader.get(Unknown Source)
> at org.apache.lucene.index.SegmentReader.get(Unknown Source)
> at org.apache.lucene.index.DirectoryReader.<init>(Unknown Source)
> at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(Unknown Source)
> at org.apache.lucene.index.DirectoryReader$1.doBody(Unknown Source)
> at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(Unknown
> Source)
> at org.apache.lucene.index.DirectoryReader.open(Unknown Source)
> at org.apache.lucene.index.IndexReader.open(Unknown Source)
> at org.apache.lucene.index.IndexReader.open(Unknown Source)
> at
> com.qspi.su.search.IndexReaderFactory.getNewIndexReaderInstance(IndexReade
> rFactory.java:70)
> at com.qspi.su.search.SearchStrategy.<init>(SearchStrategy.java:147)
> at com.qspi.su.search.Searcher.<init>(Searcher.java:170)
> at com.qspi.su.search.Searchers.<init>(Searchers.java:21)
> at com.qspi.su.ipc.SUProtocolImpl.<init>(SUProtocolImpl.java:76)
> at com.qspi.su.ipc.SUServer.run(SUServer.java:84)
> at com.qspi.su.ipc.SUServer.main(SUServer.java:204)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

eksdev at yahoo

Jul 12, 2009, 4:53 AM

Post #4 of 15 (1456 views)
Permalink

Re: OOM with 2.9 [In reply to]

Thanks Uwe,
yes it is on 32 bit in extremely tight memory situation. I just was curious what the reason is (on bigger machine it works). Lucene core requires more real memory? Can you please a bit elaborate, can I estimate it somehow? I do not remember seeing any changes that would cause core to use more memory (we do not sort, no norms overall... this failed just on opening IndexReader)...

cheers, eks

----- Original Message ----
> From: Uwe Schindler <uwe [at] thetaphi>
> To: java-user [at] lucene
> Sent: Sunday, 12 July, 2009 13:39:23
> Subject: RE: OOM with 2.9
>
> Is it a 32 or 64 bit JVM? If 32 bit, this error can happen, if the new
> Lucene Core needs more real memory and so the MMaped address space is
> already used. MMapDirectory should only be used with 64 bit machines.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe [at] thetaphi
>
> > -----Original Message-----
> > From: eks dev [mailto:eksdev [at] yahoo]
> > Sent: Sunday, July 12, 2009 1:24 PM
> > To: java-user [at] lucene
> > Subject: Re: OOM with 2.9
> >
> >
> > Stack trace
> >
> > java.io.IOException: Map failed
> > at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> > at org.apache.lucene.store.MMapDirectory$MMapIndexInput.(Unknown
> > Source)
> > at org.apache.lucene.store.MMapDirectory$MMapIndexInput.(Unknown
> > Source)
> > at org.apache.lucene.store.MMapDirectory.openInput(Unknown Source)
> > at org.apache.lucene.index.SegmentReader$CoreReaders.(Unknown
> > Source)
> > at org.apache.lucene.index.SegmentReader.get(Unknown Source)
> > at org.apache.lucene.index.SegmentReader.get(Unknown Source)
> > at org.apache.lucene.index.DirectoryReader.(Unknown Source)
> > at org.apache.lucene.index.ReadOnlyDirectoryReader.(Unknown Source)
> > at org.apache.lucene.index.DirectoryReader$1.doBody(Unknown Source)
> > at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(Unknown
> > Source)
> > at org.apache.lucene.index.DirectoryReader.open(Unknown Source)
> > at org.apache.lucene.index.IndexReader.open(Unknown Source)
> > at org.apache.lucene.index.IndexReader.open(Unknown Source)
> > at
> > com.qspi.su.search.IndexReaderFactory.getNewIndexReaderInstance(IndexReade
> > rFactory.java:70)
> > at com.qspi.su.search.SearchStrategy.(SearchStrategy.java:147)
> > at com.qspi.su.search.Searcher.(Searcher.java:170)
> > at com.qspi.su.search.Searchers.(Searchers.java:21)
> > at com.qspi.su.ipc.SUProtocolImpl.(SUProtocolImpl.java:76)
> > at com.qspi.su.ipc.SUServer.run(SUServer.java:84)
> > at com.qspi.su.ipc.SUServer.main(SUServer.java:204)
> > Caused by: java.lang.OutOfMemoryError: Map failed
> > at sun.nio.ch.FileChannelImpl.map0(Native Method)
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > For additional commands, e-mail: java-user-help [at] lucene
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

uwe at thetaphi

Jul 12, 2009, 5:10 AM

Post #5 of 15 (1464 views)
Permalink

RE: OOM with 2.9 [In reply to]

It is very hard to find out the exact problematic place in a 32 bit VM, as
the smallest change (even a bigger JAR file, more loaded classes, order in
opening files,...) can be the cause for this problem.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe [at] thetaphi

> -----Original Message-----
> From: eks dev [mailto:eksdev [at] yahoo]
> Sent: Sunday, July 12, 2009 1:54 PM
> To: java-user [at] lucene
> Subject: Re: OOM with 2.9
>
>
> Thanks Uwe,
> yes it is on 32 bit in extremely tight memory situation. I just was
> curious what the reason is (on bigger machine it works). Lucene core
> requires more real memory? Can you please a bit elaborate, can I estimate
> it somehow? I do not remember seeing any changes that would cause core to
> use more memory (we do not sort, no norms overall... this failed just on
> opening IndexReader)...
>
> cheers, eks
>
>
>
>
> ----- Original Message ----
> > From: Uwe Schindler <uwe [at] thetaphi>
> > To: java-user [at] lucene
> > Sent: Sunday, 12 July, 2009 13:39:23
> > Subject: RE: OOM with 2.9
> >
> > Is it a 32 or 64 bit JVM? If 32 bit, this error can happen, if the new
> > Lucene Core needs more real memory and so the MMaped address space is
> > already used. MMapDirectory should only be used with 64 bit machines.
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe [at] thetaphi
> >
> > > -----Original Message-----
> > > From: eks dev [mailto:eksdev [at] yahoo]
> > > Sent: Sunday, July 12, 2009 1:24 PM
> > > To: java-user [at] lucene
> > > Subject: Re: OOM with 2.9
> > >
> > >
> > > Stack trace
> > >
> > > java.io.IOException: Map failed
> > > at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> > > at org.apache.lucene.store.MMapDirectory$MMapIndexInput.(Unknown
> > > Source)
> > > at org.apache.lucene.store.MMapDirectory$MMapIndexInput.(Unknown
> > > Source)
> > > at org.apache.lucene.store.MMapDirectory.openInput(Unknown Source)
> > > at org.apache.lucene.index.SegmentReader$CoreReaders.(Unknown
> > > Source)
> > > at org.apache.lucene.index.SegmentReader.get(Unknown Source)
> > > at org.apache.lucene.index.SegmentReader.get(Unknown Source)
> > > at org.apache.lucene.index.DirectoryReader.(Unknown Source)
> > > at org.apache.lucene.index.ReadOnlyDirectoryReader.(Unknown Source)
> > > at org.apache.lucene.index.DirectoryReader$1.doBody(Unknown Source)
> > > at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(Unknown
> > > Source)
> > > at org.apache.lucene.index.DirectoryReader.open(Unknown Source)
> > > at org.apache.lucene.index.IndexReader.open(Unknown Source)
> > > at org.apache.lucene.index.IndexReader.open(Unknown Source)
> > > at
> > >
> com.qspi.su.search.IndexReaderFactory.getNewIndexReaderInstance(IndexReade
> > > rFactory.java:70)
> > > at com.qspi.su.search.SearchStrategy.(SearchStrategy.java:147)
> > > at com.qspi.su.search.Searcher.(Searcher.java:170)
> > > at com.qspi.su.search.Searchers.(Searchers.java:21)
> > > at com.qspi.su.ipc.SUProtocolImpl.(SUProtocolImpl.java:76)
> > > at com.qspi.su.ipc.SUServer.run(SUServer.java:84)
> > > at com.qspi.su.ipc.SUServer.main(SUServer.java:204)
> > > Caused by: java.lang.OutOfMemoryError: Map failed
> > > at sun.nio.ch.FileChannelImpl.map0(Native Method)
> > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > > For additional commands, e-mail: java-user-help [at] lucene
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > For additional commands, e-mail: java-user-help [at] lucene
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

uwe at thetaphi

Jul 12, 2009, 5:30 AM

Post #6 of 15 (1461 views)
Permalink

RE: OOM with 2.9 [In reply to]

Something other:

If you have a tight memory situation and really want to use MMap on 32bit,
you should think about using the new support for the Sun JVM hack to "close"
MMapped regions:
http://hudson.zones.apache.org/hudson/job/Lucene-trunk/javadoc/core/org/apac
he/lucene/store/MMapDirectory.html#setUseUnmap(boolean)

You can only use this, if you do not use the system property to switch the
directory implementation. You should instantiate the MMapDirectory directly
and then enable this new feature/hack. As you may be noticed,
FSDirectory.getDirectory()/IndexReader.open(using String/File index
locations) and the correspondingsystem property is deprecated.

A additional possibility is to use the new
org.apache.lucene.store.FileSwitchDirectory
(http://hudson.zones.apache.org/hudson/job/Lucene-trunk/javadoc/core/org/apa
che/lucene/store/FileSwitchDirectory.html) and only use MMap for file types,
that are senseful for mapping (opening small and often changing files like
the segments file with MMap is not senseful). MMap should only be used for
index files containing the term dicts and so on. E.g. it is an idea to
disable mmap for stored field files. Note: when using FileSwitchDirectory,
compound files should be disabled.

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

http://www.thetaphi.de

eMail: uwe [at] thetaphi

> -----Original Message-----

> From: eks dev [mailto:eksdev [at] yahoo]

> Sent: Sunday, July 12, 2009 1:54 PM

> To: java-user [at] lucene

> Subject: Re: OOM with 2.9

>

>

> Thanks Uwe,

> yes it is on 32 bit in extremely tight memory situation. I just was

> curious what the reason is (on bigger machine it works). Lucene core

> requires more real memory? Can you please a bit elaborate, can I estimate

> it somehow? I do not remember seeing any changes that would cause core to

> use more memory (we do not sort, no norms overall... this failed just on

> opening IndexReader)...

>

> cheers, eks

>

>

>

>

> ----- Original Message ----

> > From: Uwe Schindler <uwe [at] thetaphi>

> > To: java-user [at] lucene

> > Sent: Sunday, 12 July, 2009 13:39:23

> > Subject: RE: OOM with 2.9

> >

> > Is it a 32 or 64 bit JVM? If 32 bit, this error can happen, if the new

> > Lucene Core needs more real memory and so the MMaped address space is

> > already used. MMapDirectory should only be used with 64 bit machines.

> >

> > -----

> > Uwe Schindler

> > H.-H.-Meier-Allee 63, D-28213 Bremen

> > http://www.thetaphi.de

> > eMail: uwe [at] thetaphi

> >

> > > -----Original Message-----

> > > From: eks dev [mailto:eksdev [at] yahoo]

> > > Sent: Sunday, July 12, 2009 1:24 PM

> > > To: java-user [at] lucene

> > > Subject: Re: OOM with 2.9

> > >

> > >

> > > Stack trace

> > >

> > > java.io.IOException: Map failed

> > > at sun.nio.ch.FileChannelImpl.map(Unknown Source)

> > > at org.apache.lucene.store.MMapDirectory$MMapIndexInput.(Unknown

> > > Source)

> > > at org.apache.lucene.store.MMapDirectory$MMapIndexInput.(Unknown

> > > Source)

> > > at org.apache.lucene.store.MMapDirectory.openInput(Unknown Source)

> > > at org.apache.lucene.index.SegmentReader$CoreReaders.(Unknown

> > > Source)

> > > at org.apache.lucene.index.SegmentReader.get(Unknown Source)

> > > at org.apache.lucene.index.SegmentReader.get(Unknown Source)

> > > at org.apache.lucene.index.DirectoryReader.(Unknown Source)

> > > at org.apache.lucene.index.ReadOnlyDirectoryReader.(Unknown Source)

> > > at org.apache.lucene.index.DirectoryReader$1.doBody(Unknown Source)

> > > at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(Unknown

> > > Source)

> > > at org.apache.lucene.index.DirectoryReader.open(Unknown Source)

> > > at org.apache.lucene.index.IndexReader.open(Unknown Source)

> > > at org.apache.lucene.index.IndexReader.open(Unknown Source)

> > > at

> > >

> com.qspi.su.search.IndexReaderFactory.getNewIndexReaderInstance(IndexReade

> > > rFactory.java:70)

> > > at com.qspi.su.search.SearchStrategy.(SearchStrategy.java:147)

> > > at com.qspi.su.search.Searcher.(Searcher.java:170)

> > > at com.qspi.su.search.Searchers.(Searchers.java:21)

> > > at com.qspi.su.ipc.SUProtocolImpl.(SUProtocolImpl.java:76)

> > > at com.qspi.su.ipc.SUServer.run(SUServer.java:84)

> > > at com.qspi.su.ipc.SUServer.main(SUServer.java:204)

> > > Caused by: java.lang.OutOfMemoryError: Map failed

> > > at sun.nio.ch.FileChannelImpl.map0(Native Method)

> > >

> > >

> > >

> > >

> > > ---------------------------------------------------------------------

> > > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene

> > > For additional commands, e-mail: java-user-help [at] lucene

> >

> >

> >

> > ---------------------------------------------------------------------

> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene

> > For additional commands, e-mail: java-user-help [at] lucene

>

>

>

>

>

> ---------------------------------------------------------------------

> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene

> For additional commands, e-mail: java-user-help [at] lucene

eksdev at yahoo

Jul 12, 2009, 5:44 AM

Post #7 of 15 (1465 views)
Permalink

Re: OOM with 2.9 [In reply to]

> he/lucene/store/MMapDirectory.html#setUseUnmap(boolean)

I have tried it already, but it does not help in opening Index... mapping fails

>
> A additional possibility is to use the new
> org.apache.lucene.store.FileSwitchDirectory
this one is nice, will try it

Thanks for tips Uwe.

>
>
>
>
>
> -----
>
> Uwe Schindler
>
> H.-H.-Meier-Allee 63, D-28213 Bremen
>
> http://www.thetaphi.de
>
> eMail: uwe [at] thetaphi
>
>
>
>
>
> > -----Original Message-----
>
> > From: eks dev [mailto:eksdev [at] yahoo]
>
> > Sent: Sunday, July 12, 2009 1:54 PM
>
> > To: java-user [at] lucene
>
> > Subject: Re: OOM with 2.9
>
> >
>
> >
>
> > Thanks Uwe,
>
> > yes it is on 32 bit in extremely tight memory situation. I just was
>
> > curious what the reason is (on bigger machine it works). Lucene core
>
> > requires more real memory? Can you please a bit elaborate, can I estimate
>
> > it somehow? I do not remember seeing any changes that would cause core to
>
> > use more memory (we do not sort, no norms overall... this failed just on
>
> > opening IndexReader)...
>
> >
>
> > cheers, eks
>
> >
>
> >
>
> >
>
> >
>
> > ----- Original Message ----
>
> > > From: Uwe Schindler
>
> > > To: java-user [at] lucene
>
> > > Sent: Sunday, 12 July, 2009 13:39:23
>
> > > Subject: RE: OOM with 2.9
>
> > >
>
> > > Is it a 32 or 64 bit JVM? If 32 bit, this error can happen, if the new
>
> > > Lucene Core needs more real memory and so the MMaped address space is
>
> > > already used. MMapDirectory should only be used with 64 bit machines.
>
> > >
>
> > > -----
>
> > > Uwe Schindler
>
> > > H.-H.-Meier-Allee 63, D-28213 Bremen
>
> > > http://www.thetaphi.de
>
> > > eMail: uwe [at] thetaphi
>
> > >
>
> > > > -----Original Message-----
>
> > > > From: eks dev [mailto:eksdev [at] yahoo]
>
> > > > Sent: Sunday, July 12, 2009 1:24 PM
>
> > > > To: java-user [at] lucene
>
> > > > Subject: Re: OOM with 2.9
>
> > > >
>
> > > >
>
> > > > Stack trace
>
> > > >
>
> > > > java.io.IOException: Map failed
>
> > > > at sun.nio.ch.FileChannelImpl.map(Unknown Source)
>
> > > > at org.apache.lucene.store.MMapDirectory$MMapIndexInput.(Unknown
>
> > > > Source)
>
> > > > at org.apache.lucene.store.MMapDirectory$MMapIndexInput.(Unknown
>
> > > > Source)
>
> > > > at org.apache.lucene.store.MMapDirectory.openInput(Unknown Source)
>
> > > > at org.apache.lucene.index.SegmentReader$CoreReaders.(Unknown
>
> > > > Source)
>
> > > > at org.apache.lucene.index.SegmentReader.get(Unknown Source)
>
> > > > at org.apache.lucene.index.SegmentReader.get(Unknown Source)
>
> > > > at org.apache.lucene.index.DirectoryReader.(Unknown Source)
>
> > > > at org.apache.lucene.index.ReadOnlyDirectoryReader.(Unknown Source)
>
> > > > at org.apache.lucene.index.DirectoryReader$1.doBody(Unknown Source)
>
> > > > at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(Unknown
>
> > > > Source)
>
> > > > at org.apache.lucene.index.DirectoryReader.open(Unknown Source)
>
> > > > at org.apache.lucene.index.IndexReader.open(Unknown Source)
>
> > > > at org.apache.lucene.index.IndexReader.open(Unknown Source)
>
> > > > at
>
> > > >
>
> > com.qspi.su.search.IndexReaderFactory.getNewIndexReaderInstance(IndexReade
>
> > > > rFactory.java:70)
>
> > > > at com.qspi.su.search.SearchStrategy.(SearchStrategy.java:147)
>
> > > > at com.qspi.su.search.Searcher.(Searcher.java:170)
>
> > > > at com.qspi.su.search.Searchers.(Searchers.java:21)
>
> > > > at com.qspi.su.ipc.SUProtocolImpl.(SUProtocolImpl.java:76)
>
> > > > at com.qspi.su.ipc.SUServer.run(SUServer.java:84)
>
> > > > at com.qspi.su.ipc.SUServer.main(SUServer.java:204)
>
> > > > Caused by: java.lang.OutOfMemoryError: Map failed
>
> > > > at sun.nio.ch.FileChannelImpl.map0(Native Method)
>
> > > >
>
> > > >
>
> > > >
>
> > > >
>
> > > > ---------------------------------------------------------------------
>
> > > > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>
> > > > For additional commands, e-mail: java-user-help [at] lucene
>
> > >
>
> > >
>
> > >
>
> > > ---------------------------------------------------------------------
>
> > > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>
> > > For additional commands, e-mail: java-user-help [at] lucene
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > ---------------------------------------------------------------------
>
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>
> > For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

uwe at thetaphi

Jul 12, 2009, 6:01 AM

Post #8 of 15 (1455 views)
Permalink

RE: OOM with 2.9 [In reply to]

> > he/lucene/store/MMapDirectory.html#setUseUnmap(boolean)
>
>
> I have tried it already, but it does not help in opening Index... mapping
> fails

Various other hints: -Xmx and -Xms should be set to the same value, because
the mmapped region must map in complete into the available address space. If
it is fragmented because of resizing of Java Heap it could fail. Also the
address space for mmapping must be *outside* of -Xmx, because it is not part
of the Java heap (it is extra). So maybe you should make Xms/Xmx smaller.

Another problem with mmap (may be different on various operating systems):
it can be the case that you have already another JVM running in parallel
(with the old lucene version)? Maybe your operating system is not able to
map it twice to different addresses or the address space is filled by the
other JVM (normally every process should have 2 separate Gigabytes of
address space, but I am not sure).

> >
> > A additional possibility is to use the new
> > org.apache.lucene.store.FileSwitchDirectory
> this one is nice, will try it
>
> Thanks for tips Uwe.

Uwe

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

eksdev at yahoo

Jul 12, 2009, 7:51 AM

Post #9 of 15 (1465 views)
Permalink

Re: OOM with 2.9 [In reply to]

-Xms Xms were set to the same value
imo, the problem was to convince OS (Win XP) to map huge continuous block... there were no jvm processes running at the same time, just this one... but after killing some desktop processes and restarting machine it worked.

hmm,
MMapDirectory has support for chunking (Ineteger.MAX_VALUE) anyhow... maybe for such cases this threshold can become user settable. I will try to experiment with it (I am talking about MMapDirectory -> private final int MAX_BBUF = Integer.MAX_VALUE;)

if that makes sense or not is just another question, using MMAP in such tight memory setup where something like that has to be done is probably not really justified. On the other hand, if I have free process memory I would like to use it to the max, it sounds naive to expect, this memory can be allocated in one continuous block...

----- Original Message ----
> From: Uwe Schindler <uwe [at] thetaphi>
> To: java-user [at] lucene
> Sent: Sunday, 12 July, 2009 15:01:35
> Subject: RE: OOM with 2.9
>
> > > he/lucene/store/MMapDirectory.html#setUseUnmap(boolean)
> >
> >
> > I have tried it already, but it does not help in opening Index... mapping
> > fails
>
> Various other hints: -Xmx and -Xms should be set to the same value, because
> the mmapped region must map in complete into the available address space. If
> it is fragmented because of resizing of Java Heap it could fail. Also the
> address space for mmapping must be *outside* of -Xmx, because it is not part
> of the Java heap (it is extra). So maybe you should make Xms/Xmx smaller.
>
> Another problem with mmap (may be different on various operating systems):
> it can be the case that you have already another JVM running in parallel
> (with the old lucene version)? Maybe your operating system is not able to
> map it twice to different addresses or the address space is filled by the
> other JVM (normally every process should have 2 separate Gigabytes of
> address space, but I am not sure).
>
> > >
> > > A additional possibility is to use the new
> > > org.apache.lucene.store.FileSwitchDirectory
> > this one is nice, will try it
> >
> > Thanks for tips Uwe.
>
> Uwe
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Jul 13, 2009, 2:39 AM

Post #10 of 15 (1414 views)
Permalink

Re: OOM with 2.9 [In reply to]

I would not expect a 2.9 IndexReader to consume more RAM. Was this
definitely the case? (It wasn't just a matter of other processes
taking up RAM). If so, we should drill in to understand the root
cause / regression.

One thing you can do in 2.9 is IndexReader.setDisableFakeNorms(true),
to prevent allocation of the "fake" byte[maxDoc()] norms array, to
save RAM.

Mike

On Sun, Jul 12, 2009 at 7:22 AM, eks dev<eksdev [at] yahoo> wrote:
>
> Hi,
> We just upgraded to 2.9 and noticed some (to me) not expected OOM.
>
> We use MMapDirectory and after upgrade, on exactly the same Index/machine/jvm/params/setup... we cannot start index as mapping screams "No memory"
>
> any explanation why this could be the case?
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Jul 13, 2009, 2:40 AM

Post #11 of 15 (1418 views)
Permalink

Re: OOM with 2.9 [In reply to]

On Sun, Jul 12, 2009 at 10:51 AM, eks dev<eksdev [at] yahoo> wrote:

> MMapDirectory has support for chunking (Ineteger.MAX_VALUE) anyhow... maybe for such cases this threshold can become user settable. I will try to experiment with it �(I am talking about MMapDirectory -> private final int MAX_BBUF = Integer.MAX_VALUE;)

This chunking doesn't reduce virtual memory usage, because the buffers
are all mapped up front on opening the IndexInput.

It's only done because of a limitation of ByteBuffer (it uses int not
long to address the values).

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

uwe at thetaphi

Jul 13, 2009, 2:49 AM

Post #12 of 15 (1417 views)
Permalink

RE: OOM with 2.9 [In reply to]

> On Sun, Jul 12, 2009 at 10:51 AM, eks dev<eksdev [at] yahoo> wrote:
>
> > MMapDirectory has support for chunking (Ineteger.MAX_VALUE) anyhow...
> maybe for such cases this threshold can become user settable. I will try
> to experiment with it �(I am talking about MMapDirectory -> private final
> int MAX_BBUF = Integer.MAX_VALUE;)
>
> This chunking doesn't reduce virtual memory usage, because the buffers
> are all mapped up front on opening the IndexInput.
>
> It's only done because of a limitation of ByteBuffer (it uses int not
> long to address the values).

The idea of chunking is because of address space fragmentation. If the Java
VM allocates or frees memory (e.g. when expanding the heap from Xms to Xmx),
the address space of the process gets fragmented. If the index is very
large, the byte buffer must fit in complete as one fragment into the address
space, which may easily fail when you only have 32 bit.

If the file is mapped in different chunks, each chunk can be somewhere else
in address space (e.g., not only one 1 Gig junk is needed, there can be ten
100 Meg junks at different places).

Uwe

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Jul 13, 2009, 2:52 AM

Post #13 of 15 (1421 views)
Permalink

Re: OOM with 2.9 [In reply to]

Ahh good point. I agree it makes sense to make MMapDir's chunking
user-controllable. Can you open an issue?

Mike

On Mon, Jul 13, 2009 at 5:49 AM, Uwe Schindler<uwe [at] thetaphi> wrote:
>> On Sun, Jul 12, 2009 at 10:51 AM, eks dev<eksdev [at] yahoo> wrote:
>>
>> > MMapDirectory has support for chunking (Ineteger.MAX_VALUE) anyhow...
>> maybe for such cases this threshold can become user settable. I will try
>> to experiment with it �(I am talking about MMapDirectory -> private final
>> int MAX_BBUF = Integer.MAX_VALUE;)
>>
>> This chunking doesn't reduce virtual memory usage, because the buffers
>> are all mapped up front on opening the IndexInput.
>>
>> It's only done because of a limitation of ByteBuffer (it uses int not
>> long to address the values).
>
> The idea of chunking is because of address space fragmentation. If the Java
> VM allocates or frees memory (e.g. when expanding the heap from Xms to Xmx),
> the address space of the process gets fragmented. If the index is very
> large, the byte buffer must fit in complete as one fragment into the address
> space, which may easily fail when you only have 32 bit.
>
> If the file is mapped in different chunks, each chunk can be somewhere else
> in address space (e.g., not only one 1 Gig junk is needed, there can be ten
> 100 Meg junks at different places).
>
> Uwe
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

eksdev at yahoo

Jul 13, 2009, 4:11 AM

Post #14 of 15 (1416 views)
Permalink

Re: OOM with 2.9 [In reply to]

Hi Mike,

thanks for looking into it...

I am now positive, it was definitely a problem for OS to map() large continuous chunk of process memory... if I use this machine for a while as a desktop, eclipse,... I get the same problem again... but after cold restart, mapping succeeds.

The problem never happened with 2.4, but I guess this was a pure coincidence.

Cheers,
eks

----- Original Message ----
> From: Michael McCandless <lucene [at] mikemccandless>
> To: java-user [at] lucene
> Sent: Monday, 13 July, 2009 11:52:31
> Subject: Re: OOM with 2.9
>
> Ahh good point. I agree it makes sense to make MMapDir's chunking
> user-controllable. Can you open an issue?
>
> Mike
>
> On Mon, Jul 13, 2009 at 5:49 AM, Uwe Schindlerwrote:
> >> On Sun, Jul 12, 2009 at 10:51 AM, eks devwrote:
> >>
> >> > MMapDirectory has support for chunking (Ineteger.MAX_VALUE) anyhow...
> >> maybe for such cases this threshold can become user settable. I will try
> >> to experiment with it (I am talking about MMapDirectory -> private final
> >> int MAX_BBUF = Integer.MAX_VALUE;)
> >>
> >> This chunking doesn't reduce virtual memory usage, because the buffers
> >> are all mapped up front on opening the IndexInput.
> >>
> >> It's only done because of a limitation of ByteBuffer (it uses int not
> >> long to address the values).
> >
> > The idea of chunking is because of address space fragmentation. If the Java
> > VM allocates or frees memory (e.g. when expanding the heap from Xms to Xmx),
> > the address space of the process gets fragmented. If the index is very
> > large, the byte buffer must fit in complete as one fragment into the address
> > space, which may easily fail when you only have 32 bit.
> >
> > If the file is mapped in different chunks, each chunk can be somewhere else
> > in address space (e.g., not only one 1 Gig junk is needed, there can be ten
> > 100 Meg junks at different places).
> >
> > Uwe
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > For additional commands, e-mail: java-user-help [at] lucene
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

uwe at thetaphi

Jul 13, 2009, 7:17 AM

Post #15 of 15 (1411 views)
Permalink

RE: OOM with 2.9 [In reply to]

DONE.

> -----Original Message-----
> From: Michael McCandless [mailto:lucene [at] mikemccandless]
> Sent: Monday, July 13, 2009 11:53 AM
> To: java-user [at] lucene
> Subject: Re: OOM with 2.9
>
> Ahh good point. I agree it makes sense to make MMapDir's chunking
> user-controllable. Can you open an issue?
>
> Mike
>
> On Mon, Jul 13, 2009 at 5:49 AM, Uwe Schindler<uwe [at] thetaphi> wrote:
> >> On Sun, Jul 12, 2009 at 10:51 AM, eks dev<eksdev [at] yahoo> wrote:
> >>
> >> > MMapDirectory has support for chunking (Ineteger.MAX_VALUE) anyhow...
> >> maybe for such cases this threshold can become user settable. I will
> try
> >> to experiment with it �(I am talking about MMapDirectory -> private
> final
> >> int MAX_BBUF = Integer.MAX_VALUE;)
> >>
> >> This chunking doesn't reduce virtual memory usage, because the buffers
> >> are all mapped up front on opening the IndexInput.
> >>
> >> It's only done because of a limitation of ByteBuffer (it uses int not
> >> long to address the values).
> >
> > The idea of chunking is because of address space fragmentation. If the
> Java
> > VM allocates or frees memory (e.g. when expanding the heap from Xms to
> Xmx),
> > the address space of the process gets fragmented. If the index is very
> > large, the byte buffer must fit in complete as one fragment into the
> address
> > space, which may easily fail when you only have 32 bit.
> >
> > If the file is mapped in different chunks, each chunk can be somewhere
> else
> > in address space (e.g., not only one 1 Gig junk is needed, there can be
> ten
> > 100 Meg junks at different places).
> >
> > Uwe
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > For additional commands, e-mail: java-user-help [at] lucene
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads