Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Lucene: Java-Dev

Too many open files

 

 

Lucene java-dev RSS feed   Index | Next | Previous | View Threaded


kalle at snigel

Mar 4, 2004, 1:36 PM

Post #1 of 3 (284 views)
Permalink
Too many open files

I'm getting this exception, and I can't explain it. It only occurs when
calling my method that retrieves the content from my index. I get the
same exception in 1.2 as in 1.3-final, been searching the web all over
the place but can't find anything else than the same problem described,
no solution.



It takes some 50-100 calls to the method until the exception in thrown.
Don't I close the IndexReader correct? Is there anything else I need to
close?

Greatful for any hints.


----> exception snippet

java.io.FileNotFoundException: /home/kalle/projects/snigel/egdelon_data/lucene/_2ur.fdx (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:200)
at org.apache.lucene.store.FSInputStream$Descriptor.<init>(FSDirectory.java:389)
at org.apache.lucene.store.FSInputStream.<init>(FSDirectory.java:418)
at org.apache.lucene.store.FSDirectory.openFile(FSDirectory.java:291)
at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:80)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:139)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:120)
at org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:122)
at org.apache.lucene.store.Lock$With.run(Lock.java:148)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:111)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:99)
at org.apache.lucene.search.IndexSearcher.<init>(IndexSearcher.java:75)
at se.snigel.egdelon.document.indexer.LuceneIndexer.getSegmentContents(LuceneIndexer.java:48)

row 48 >> s = new IndexSearcher(Egdelon.getSetting("lucene.path"));

----> full method

public String getSegmentContents(Segment segment)
throws IOException {
File path = new File(Egdelon.getSetting("lucene.path"));

if (!IndexReader.indexExists(path))
return null;

Searcher s = null;

try {
Analyzer a = new StandardAnalyzer();
Query q = QueryParser.parse(String.valueOf(segment.getController().getPrimaryKey().getPK()),
"segment.pk", a);
s = new IndexSearcher(Egdelon.getSetting("lucene.path"));

Hits hits = s.search(q);

if (hits.length() == 1)
return hits.doc(0).getField("contents").stringValue();

s.close();

return null;
} catch (ParseException e) {
e.printStackTrace();

if (s != null)
s.close();

return null;
}
}

--

karl
--

kalle


--

kalle

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe [at] jakarta
For additional commands, e-mail: lucene-dev-help [at] jakarta


dmitrys at earthlink

Mar 4, 2004, 3:32 PM

Post #2 of 3 (273 views)
Permalink
Re: Too many open files [In reply to]

You can tell how many file handles you are allowed to open by your OS
(looks like some flavor of Unix from the paths that you have included).
One way to reduce the number of files Lucene opens is to use compound
indexes (where each index segment uses a single file). Look for this
flag on IndexWriter object. You will have to optimize existing indexes
to convert them.

If you need to know how to tell how many files you are allowed to open
on Unix, the command is ulimit. Try ulimit -Ha and look for
"nofiles(descriptors)" - this works on Solaris.
There are usually two limits for each variable - hard and soft. Soft
limit you can control with the ulimit command before you start your
process. You can rase it as far as the hard limit. The hard limit is
given in the /etc/system file (again for Solaris) and is generally not
changeable except by root. You will find many references on the web that
say not to increase this parameter in /etc/system beyound 2000 or so,
but in reality any significant J2EE installation on Solaris will have
8000 to 9000 setting.

Good luck.
Dmitry.


karl wettin wrote:

>I'm getting this exception, and I can't explain it. It only occurs when
>calling my method that retrieves the content from my index. I get the
>same exception in 1.2 as in 1.3-final, been searching the web all over
>the place but can't find anything else than the same problem described,
>no solution.
>
>
>
>It takes some 50-100 calls to the method until the exception in thrown.
>Don't I close the IndexReader correct? Is there anything else I need to
>close?
>
>Greatful for any hints.
>
>
>----> exception snippet
>
>java.io.FileNotFoundException: /home/kalle/projects/snigel/egdelon_data/lucene/_2ur.fdx (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:200)
> at org.apache.lucene.store.FSInputStream$Descriptor.<init>(FSDirectory.java:389)
> at org.apache.lucene.store.FSInputStream.<init>(FSDirectory.java:418)
> at org.apache.lucene.store.FSDirectory.openFile(FSDirectory.java:291)
> at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:80)
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:139)
> at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:120)
> at org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:122)
> at org.apache.lucene.store.Lock$With.run(Lock.java:148)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:111)
> at org.apache.lucene.index.IndexReader.open(IndexReader.java:99)
> at org.apache.lucene.search.IndexSearcher.<init>(IndexSearcher.java:75)
> at se.snigel.egdelon.document.indexer.LuceneIndexer.getSegmentContents(LuceneIndexer.java:48)
>
>row 48 >> s = new IndexSearcher(Egdelon.getSetting("lucene.path"));
>
>----> full method
>
> public String getSegmentContents(Segment segment)
> throws IOException {
> File path = new File(Egdelon.getSetting("lucene.path"));
>
> if (!IndexReader.indexExists(path))
> return null;
>
> Searcher s = null;
>
> try {
> Analyzer a = new StandardAnalyzer();
> Query q = QueryParser.parse(String.valueOf(segment.getController().getPrimaryKey().getPK()),
> "segment.pk", a);
> s = new IndexSearcher(Egdelon.getSetting("lucene.path"));
>
> Hits hits = s.search(q);
>
> if (hits.length() == 1)
> return hits.doc(0).getField("contents").stringValue();
>
> s.close();
>
> return null;
> } catch (ParseException e) {
> e.printStackTrace();
>
> if (s != null)
> s.close();
>
> return null;
> }
> }
>
>
>



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe [at] jakarta
For additional commands, e-mail: lucene-dev-help [at] jakarta


kalle at snigel

Mar 10, 2004, 3:17 AM

Post #3 of 3 (273 views)
Permalink
Re: Too many open files [In reply to]

On Thu, 04 Mar 2004 15:32:53 -0700
Dmitry Serebrennikov <dmitrys [at] earthlink> wrote:

> karl wettin wrote:
>
> >I'm getting this exception, and I can't explain it. It only occurs
> >when calling my method that retrieves the content from my index. I
> >get the same exception in 1.2 as in 1.3-final, been searching the web
> >all over the place but can't find anything else than the same problem
> >described, no solution.
> >
> >
> >
> >It takes some 50-100 calls to the method until the exception in
> >thrown. Don't I close the IndexReader correct? Is there anything else
> >I need to close?
> >
> >Greatful for any hints.

> You can tell how many file handles you are allowed to open by your OS
> (looks like some flavor of Unix from the paths that you have
> included). One way to reduce the number of files Lucene opens is to
> use compound indexes (where each index segment uses a single file).
> Look for this flag on IndexWriter object. You will have to optimize
> existing indexes to convert them.

My file limit was set to "unlimited" when I got the problem, so I
started using .setUseCompoundFile(true) in order to resolve my problem.

Even though I only add new data (no delete) to the index, I started
getting the exception "Lock obtain timed out" after the change above.
Passing on "-DdisableLuceneLocks=true" to java "solved" this problem,
but it does not feel right. I only access Lucene from one JVM, and I
make sure I never write and delete at the same time. Do I have anyhting
to worry about?


--

karl

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe [at] jakarta
For additional commands, e-mail: lucene-dev-help [at] jakarta

Lucene java-dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.