Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Lucene: Java-User

OutOfMemoryError when opening the index ?

 

 

Lucene java-user RSS feed   Index | Next | Previous | View Threaded


teddyyyy123 at gmail

Jun 13, 2012, 2:15 PM

Post #1 of 2 (325 views)
Permalink
OutOfMemoryError when opening the index ?

I got the OutOfMemoryError when I tried to open an Lucene index.

it's very weird since this is only seen when I run this inside an Apache
PIG LATIN script on a particular hadoop cluster of ours,
and it runs fine on an old cluster; also it runs fine if I run the java
code outside of PIG.


as I understand, OutOfMemoryError is not out of heap, so does this mean
that mmap() failed?
I checked the ulimit of the user account running this, it seems to be all
unlimited


any clues?

Thanks
yang

Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:265)
at
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
at
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:86)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:115)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:93)
at
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:113)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:29)
at
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754)
at
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:462)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:308)
at
cgm.places.matcher.ListingSearcher.createIndex(ListingSearcher.java:45)
at
cgm.places.matcher.ListingSearcher.<init>(ListingSearcher.java:38)
at
cgm.places.matcher.SearchListingUdf.<init>(SearchListingUdf.java:34)
... 83 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
... 98 more


teddyyyy123 at gmail

Jun 13, 2012, 4:19 PM

Post #2 of 2 (317 views)
Permalink
Re: OutOfMemoryError when opening the index ? [In reply to]

ok, found it:

we are using Cloudera CDHu3u, they change the ulimit for child jobs.
but I still don't know how to change their default settings yet


On Wed, Jun 13, 2012 at 2:15 PM, Yang <teddyyyy123 [at] gmail> wrote:

> I got the OutOfMemoryError when I tried to open an Lucene index.
>
> it's very weird since this is only seen when I run this inside an Apache
> PIG LATIN script on a particular hadoop cluster of ours,
> and it runs fine on an old cluster; also it runs fine if I run the java
> code outside of PIG.
>
>
> as I understand, OutOfMemoryError is not out of heap, so does this mean
> that mmap() failed?
> I checked the ulimit of the user account running this, it seems to be all
> unlimited
>
>
> any clues?
>
> Thanks
> yang
>
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
> at
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:265)
> at
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
> at
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:86)
> at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:115)
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:93)
> at
> org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:113)
> at
> org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:29)
> at
> org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81)
> at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754)
> at
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:462)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:308)
> at
> cgm.places.matcher.ListingSearcher.createIndex(ListingSearcher.java:45)
> at
> cgm.places.matcher.ListingSearcher.<init>(ListingSearcher.java:38)
> at
> cgm.places.matcher.SearchListingUdf.<init>(SearchListingUdf.java:34)
> ... 83 more
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
> ... 98 more
>
>

Lucene java-user RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.