Mailing List Archive: Lucene: Java-User

IndexWriter.close() no longer seems to close everything

Index | Next | Previous | View Threaded

daniel at nuix

Nov 8, 2009, 7:29 PM

Post #1 of 16 (1330 views)
Permalink

IndexWriter.close() no longer seems to close everything

Hi all.

We updated to Lucene 2.9, and now we find that after closing our text
index, it is not possible to rename the directory in which it resides
(we are actually renaming a directory further up the hierarchy.)

We discovered that the following files were still open by the process:

_0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf

We are calling IndexWriter.close() shortly before attempting to write
to the directory (a few lines of code earlier) so I suspect it could
be related to timing somehow if Lucene is perhaps still doing
something on a background thread at this time (though I was under the
impression that close() waited for merges and so forth to complete
before returning.)

Daniel

--
Daniel Noll Forensic and eDiscovery Software
Senior Developer The world's most advanced
Nuix email data analysis
http://nuix.com/ and eDiscovery software

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

john.wang at gmail

Nov 8, 2009, 7:41 PM

Post #2 of 16 (1294 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

I am seeing the samething, but only when IndexWriter.getReader is called at
a high rate.

from lsof, I see file handles growing.

-John

On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix> wrote:

> Hi all.
>
> We updated to Lucene 2.9, and now we find that after closing our text
> index, it is not possible to rename the directory in which it resides
> (we are actually renaming a directory further up the hierarchy.)
>
> We discovered that the following files were still open by the process:
>
> _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>
> We are calling IndexWriter.close() shortly before attempting to write
> to the directory (a few lines of code earlier) so I suspect it could
> be related to timing somehow if Lucene is perhaps still doing
> something on a background thread at this time (though I was under the
> impression that close() waited for merges and so forth to complete
> before returning.)
>
> Daniel
>
> --
> Daniel Noll Forensic and eDiscovery Software
> Senior Developer The world's most advanced
> Nuix email data analysis
> http://nuix.com/ and eDiscovery software
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

daniel at nuix

Nov 8, 2009, 8:17 PM

Post #3 of 16 (1307 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

On Mon, Nov 9, 2009 at 14:41, John Wang <john.wang [at] gmail> wrote:
> I am seeing the samething, but only when IndexWriter.getReader is called at
> a high rate.
>
> from lsof, I see file handles growing.

This hint turned out to help. :-)

Turns out we had an IndexReader hanging around from a previous index
state (before adding docs.) When the IndexWriter was opened, another
reader was opened, so even though we thought we were closing both, it
turned out there were two readers and one writer, and we were only
closing one of the readers.

Daniel

--
Daniel Noll Forensic and eDiscovery Software
Senior Developer The world's most advanced
Nuix email data analysis
http://nuix.com/ and eDiscovery software

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Nov 9, 2009, 2:43 AM

Post #4 of 16 (1282 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

Does this look like a real leak John? You're definitely closing every
reader you get back from getReader?

Mike

On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang [at] gmail> wrote:
> I am seeing the samething, but only when IndexWriter.getReader is called at
> a high rate.
>
> from lsof, I see file handles growing.
>
> -John
>
> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix> wrote:
>
>> Hi all.
>>
>> We updated to Lucene 2.9, and now we find that after closing our text
>> index, it is not possible to rename the directory in which it resides
>> (we are actually renaming a directory further up the hierarchy.)
>>
>> We discovered that the following files were still open by the process:
>>
>> �_0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>
>> We are calling IndexWriter.close() shortly before attempting to write
>> to the directory (a few lines of code earlier) so I suspect it could
>> be related to timing somehow if Lucene is perhaps still doing
>> something on a background thread at this time (though I was under the
>> impression that close() waited for merges and so forth to complete
>> before returning.)
>>
>> Daniel
>>
>> --
>> Daniel Noll � � � � � � � � � � � � � �Forensic and eDiscovery Software
>> Senior Developer � � � � � � � � � � � � � � �The world's most advanced
>> Nuix � � � � � � � � � � � � � � � � � � � � � � � �email data analysis
>> http://nuix.com/ � � � � � � � � � � � � � � � �and eDiscovery software
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

albertjuhe at gmail

Nov 11, 2009, 2:10 AM

Post #5 of 16 (1239 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

I don't know if it's the same problem but I think it's similar,

My problem is with the Indexsearcher. I've installed a web search engine
that uses Lucene, after a search I make a close operation like this:

private IndexSearcher searcher;

NIOFSDirectory directory = new NIOFSDirectory(new File(path));
this.searcher = new IndexSearcher(directory, true);

public void close() {
try {
System.out.println("Closing: "
+this.searcher.getIndexReader().directory().toString() );
this.searcher.getIndexReader().directory().close();
this.searcher.getIndexReader().close();
this.searcher.close();
} catch (IOException e) {
System.out.println(" caught a " + e.getClass() + "\n with
message: " + e.getMessage());
}
}
jboss don't close *.cfs index files, everytime that I make a new search a
new file is caught by jboss.
If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command, the
only way to free this files and close it is stopping jboss.

The problem is that after search many times, jboss go down and I have to
restart it.

Do you have the same problem with IndexSearch or is only with IndexWriter?

Albert Juhe
Learning Technologies
Universitat Oberta de Catalunya

Michael McCandless-2 wrote:
>
> Does this look like a real leak John? You're definitely closing every
> reader you get back from getReader?
>
> Mike
>
> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang [at] gmail> wrote:
>> I am seeing the samething, but only when IndexWriter.getReader is called
>> at
>> a high rate.
>>
>> from lsof, I see file handles growing.
>>
>> -John
>>
>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix> wrote:
>>
>>> Hi all.
>>>
>>> We updated to Lucene 2.9, and now we find that after closing our text
>>> index, it is not possible to rename the directory in which it resides
>>> (we are actually renaming a directory further up the hierarchy.)
>>>
>>> We discovered that the following files were still open by the process:
>>>
>>> _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>
>>> We are calling IndexWriter.close() shortly before attempting to write
>>> to the directory (a few lines of code earlier) so I suspect it could
>>> be related to timing somehow if Lucene is perhaps still doing
>>> something on a background thread at this time (though I was under the
>>> impression that close() waited for merges and so forth to complete
>>> before returning.)
>>>
>>> Daniel
>>>
>>> --
>>> Daniel Noll Forensic and eDiscovery Software
>>> Senior Developer The world's most advanced
>>> Nuix email data analysis
>>> http://nuix.com/ and eDiscovery software
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>> For additional commands, e-mail: java-user-help [at] lucene
>>>
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>
>

--
View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Nov 11, 2009, 2:15 AM

Post #6 of 16 (1244 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

Do you see your exception handler printing anything out?

You don't need to close the underlying IndexReader, just the
IndexSearcher (which will close the IndexReader, since it was the one
that had opened it).

Mike

On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <albertjuhe [at] gmail> wrote:
>
> I don't know if it's the same problem but I think it's similar,
>
> My problem is with the Indexsearcher. I've installed a web search engine
> that uses Lucene, after a search I make a close operation like this:
>
> private IndexSearcher searcher;
>
> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
> this.searcher = new IndexSearcher(directory, true);
>
> �public void close() {
> � � � �try {
> � � � � � �System.out.println("Closing: "
> +this.searcher.getIndexReader().directory().toString() );
> � � � � � �this.searcher.getIndexReader().directory().close();
> � � � � � �this.searcher.getIndexReader().close();
> � � � � � �this.searcher.close();
> � � � �} catch (IOException e) {
> � � � � � �System.out.println(" caught a " + e.getClass() + "\n with
> message: " + e.getMessage());
> � � � �}
> � �}
> jboss don't close *.cfs index files, everytime that I make a new search a
> new file is caught by jboss.
> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command, the
> only way to free this files and close it is stopping jboss.
>
> The problem is that after search many times, �jboss go down and I have to
> restart it.
>
> Do you have the same problem with IndexSearch or is only with IndexWriter?
>
> Albert Juhe
> Learning Technologies
> Universitat Oberta de Catalunya
>
>
> Michael McCandless-2 wrote:
>>
>> Does this look like a real leak John? �You're definitely closing every
>> reader you get back from getReader?
>>
>> Mike
>>
>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang [at] gmail> wrote:
>>> I am seeing the samething, but only when IndexWriter.getReader is called
>>> at
>>> a high rate.
>>>
>>> from lsof, I see file handles growing.
>>>
>>> -John
>>>
>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix> wrote:
>>>
>>>> Hi all.
>>>>
>>>> We updated to Lucene 2.9, and now we find that after closing our text
>>>> index, it is not possible to rename the directory in which it resides
>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>
>>>> We discovered that the following files were still open by the process:
>>>>
>>>> �_0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>
>>>> We are calling IndexWriter.close() shortly before attempting to write
>>>> to the directory (a few lines of code earlier) so I suspect it could
>>>> be related to timing somehow if Lucene is perhaps still doing
>>>> something on a background thread at this time (though I was under the
>>>> impression that close() waited for merges and so forth to complete
>>>> before returning.)
>>>>
>>>> Daniel
>>>>
>>>> --
>>>> Daniel Noll � � � � � � � � � � � � � �Forensic and eDiscovery Software
>>>> Senior Developer � � � � � � � � � � � � � � �The world's most advanced
>>>> Nuix � � � � � � � � � � � � � � � � � � � � � � � �email data analysis
>>>> http://nuix.com/ � � � � � � � � � � � � � � � �and eDiscovery software
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

albertjuhe at gmail

Nov 11, 2009, 2:54 AM

Post #7 of 16 (1248 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

I don't get any exception.

thank you Mike

Michael McCandless-2 wrote:
>
> Do you see your exception handler printing anything out?
>
> You don't need to close the underlying IndexReader, just the
> IndexSearcher (which will close the IndexReader, since it was the one
> that had opened it).
>
> Mike
>
> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <albertjuhe [at] gmail> wrote:
>>
>> I don't know if it's the same problem but I think it's similar,
>>
>> My problem is with the Indexsearcher. I've installed a web search engine
>> that uses Lucene, after a search I make a close operation like this:
>>
>> private IndexSearcher searcher;
>>
>> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
>> this.searcher = new IndexSearcher(directory, true);
>>
>> public void close() {
>> try {
>> System.out.println("Closing: "
>> +this.searcher.getIndexReader().directory().toString() );
>> this.searcher.getIndexReader().directory().close();
>> this.searcher.getIndexReader().close();
>> this.searcher.close();
>> } catch (IOException e) {
>> System.out.println(" caught a " + e.getClass() + "\n with
>> message: " + e.getMessage());
>> }
>> }
>> jboss don't close *.cfs index files, everytime that I make a new search a
>> new file is caught by jboss.
>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command, the
>> only way to free this files and close it is stopping jboss.
>>
>> The problem is that after search many times, jboss go down and I have to
>> restart it.
>>
>> Do you have the same problem with IndexSearch or is only with
>> IndexWriter?
>>
>> Albert Juhe
>> Learning Technologies
>> Universitat Oberta de Catalunya
>>
>>
>> Michael McCandless-2 wrote:
>>>
>>> Does this look like a real leak John? You're definitely closing every
>>> reader you get back from getReader?
>>>
>>> Mike
>>>
>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang [at] gmail> wrote:
>>>> I am seeing the samething, but only when IndexWriter.getReader is
>>>> called
>>>> at
>>>> a high rate.
>>>>
>>>> from lsof, I see file handles growing.
>>>>
>>>> -John
>>>>
>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix> wrote:
>>>>
>>>>> Hi all.
>>>>>
>>>>> We updated to Lucene 2.9, and now we find that after closing our text
>>>>> index, it is not possible to rename the directory in which it resides
>>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>>
>>>>> We discovered that the following files were still open by the process:
>>>>>
>>>>> _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>>
>>>>> We are calling IndexWriter.close() shortly before attempting to write
>>>>> to the directory (a few lines of code earlier) so I suspect it could
>>>>> be related to timing somehow if Lucene is perhaps still doing
>>>>> something on a background thread at this time (though I was under the
>>>>> impression that close() waited for merges and so forth to complete
>>>>> before returning.)
>>>>>
>>>>> Daniel
>>>>>
>>>>> --
>>>>> Daniel Noll Forensic and eDiscovery
>>>>> Software
>>>>> Senior Developer The world's most
>>>>> advanced
>>>>> Nuix email data
>>>>> analysis
>>>>> http://nuix.com/ and eDiscovery
>>>>> software
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>
>>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>> For additional commands, e-mail: java-user-help [at] lucene
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>
>

--
View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299405.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Nov 11, 2009, 2:59 AM

Post #8 of 16 (1235 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

Can you narrow the leak down to a small self-contained test?

Mike

On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe <albertjuhe [at] gmail> wrote:
>
> I don't get any exception.
>
> thank you Mike
>
>
> Michael McCandless-2 wrote:
>>
>> Do you see your exception handler printing anything out?
>>
>> You don't need to close the underlying IndexReader, just the
>> IndexSearcher (which will close the IndexReader, since it was the one
>> that had opened it).
>>
>> Mike
>>
>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <albertjuhe [at] gmail> wrote:
>>>
>>> I don't know if it's the same problem but I think it's similar,
>>>
>>> My problem is with the Indexsearcher. I've installed a web search engine
>>> that uses Lucene, after a search I make a close operation like this:
>>>
>>> private IndexSearcher searcher;
>>>
>>> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
>>> this.searcher = new IndexSearcher(directory, true);
>>>
>>> �public void close() {
>>> � � � �try {
>>> � � � � � �System.out.println("Closing: "
>>> +this.searcher.getIndexReader().directory().toString() );
>>> � � � � � �this.searcher.getIndexReader().directory().close();
>>> � � � � � �this.searcher.getIndexReader().close();
>>> � � � � � �this.searcher.close();
>>> � � � �} catch (IOException e) {
>>> � � � � � �System.out.println(" caught a " + e.getClass() + "\n with
>>> message: " + e.getMessage());
>>> � � � �}
>>> � �}
>>> jboss don't close *.cfs index files, everytime that I make a new search a
>>> new file is caught by jboss.
>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command, the
>>> only way to free this files and close it is stopping jboss.
>>>
>>> The problem is that after search many times, �jboss go down and I have to
>>> restart it.
>>>
>>> Do you have the same problem with IndexSearch or is only with
>>> IndexWriter?
>>>
>>> Albert Juhe
>>> Learning Technologies
>>> Universitat Oberta de Catalunya
>>>
>>>
>>> Michael McCandless-2 wrote:
>>>>
>>>> Does this look like a real leak John? �You're definitely closing every
>>>> reader you get back from getReader?
>>>>
>>>> Mike
>>>>
>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang [at] gmail> wrote:
>>>>> I am seeing the samething, but only when IndexWriter.getReader is
>>>>> called
>>>>> at
>>>>> a high rate.
>>>>>
>>>>> from lsof, I see file handles growing.
>>>>>
>>>>> -John
>>>>>
>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix> wrote:
>>>>>
>>>>>> Hi all.
>>>>>>
>>>>>> We updated to Lucene 2.9, and now we find that after closing our text
>>>>>> index, it is not possible to rename the directory in which it resides
>>>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>>>
>>>>>> We discovered that the following files were still open by the process:
>>>>>>
>>>>>> �_0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>>>
>>>>>> We are calling IndexWriter.close() shortly before attempting to write
>>>>>> to the directory (a few lines of code earlier) so I suspect it could
>>>>>> be related to timing somehow if Lucene is perhaps still doing
>>>>>> something on a background thread at this time (though I was under the
>>>>>> impression that close() waited for merges and so forth to complete
>>>>>> before returning.)
>>>>>>
>>>>>> Daniel
>>>>>>
>>>>>> --
>>>>>> Daniel Noll � � � � � � � � � � � � � �Forensic and eDiscovery
>>>>>> Software
>>>>>> Senior Developer � � � � � � � � � � � � � � �The world's most
>>>>>> advanced
>>>>>> Nuix � � � � � � � � � � � � � � � � � � � � � � � �email data
>>>>>> analysis
>>>>>> http://nuix.com/ � � � � � � � � � � � � � � � �and eDiscovery
>>>>>> software
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>> For additional commands, e-mail: java-user-help [at] lucene
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299405.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

albertjuhe at gmail

Nov 11, 2009, 3:41 AM

Post #9 of 16 (1246 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

I think that this is the best way to proceed.

thank you Mike

Michael McCandless-2 wrote:
>
> Can you narrow the leak down to a small self-contained test?
>
> Mike
>
> On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe <albertjuhe [at] gmail> wrote:
>>
>> I don't get any exception.
>>
>> thank you Mike
>>
>>
>> Michael McCandless-2 wrote:
>>>
>>> Do you see your exception handler printing anything out?
>>>
>>> You don't need to close the underlying IndexReader, just the
>>> IndexSearcher (which will close the IndexReader, since it was the one
>>> that had opened it).
>>>
>>> Mike
>>>
>>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <albertjuhe [at] gmail>
>>> wrote:
>>>>
>>>> I don't know if it's the same problem but I think it's similar,
>>>>
>>>> My problem is with the Indexsearcher. I've installed a web search
>>>> engine
>>>> that uses Lucene, after a search I make a close operation like this:
>>>>
>>>> private IndexSearcher searcher;
>>>>
>>>> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
>>>> this.searcher = new IndexSearcher(directory, true);
>>>>
>>>> public void close() {
>>>> try {
>>>> System.out.println("Closing: "
>>>> +this.searcher.getIndexReader().directory().toString() );
>>>> this.searcher.getIndexReader().directory().close();
>>>> this.searcher.getIndexReader().close();
>>>> this.searcher.close();
>>>> } catch (IOException e) {
>>>> System.out.println(" caught a " + e.getClass() + "\n with
>>>> message: " + e.getMessage());
>>>> }
>>>> }
>>>> jboss don't close *.cfs index files, everytime that I make a new search
>>>> a
>>>> new file is caught by jboss.
>>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command,
>>>> the
>>>> only way to free this files and close it is stopping jboss.
>>>>
>>>> The problem is that after search many times, jboss go down and I have
>>>> to
>>>> restart it.
>>>>
>>>> Do you have the same problem with IndexSearch or is only with
>>>> IndexWriter?
>>>>
>>>> Albert Juhe
>>>> Learning Technologies
>>>> Universitat Oberta de Catalunya
>>>>
>>>>
>>>> Michael McCandless-2 wrote:
>>>>>
>>>>> Does this look like a real leak John? You're definitely closing every
>>>>> reader you get back from getReader?
>>>>>
>>>>> Mike
>>>>>
>>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang [at] gmail>
>>>>> wrote:
>>>>>> I am seeing the samething, but only when IndexWriter.getReader is
>>>>>> called
>>>>>> at
>>>>>> a high rate.
>>>>>>
>>>>>> from lsof, I see file handles growing.
>>>>>>
>>>>>> -John
>>>>>>
>>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix> wrote:
>>>>>>
>>>>>>> Hi all.
>>>>>>>
>>>>>>> We updated to Lucene 2.9, and now we find that after closing our
>>>>>>> text
>>>>>>> index, it is not possible to rename the directory in which it
>>>>>>> resides
>>>>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>>>>
>>>>>>> We discovered that the following files were still open by the
>>>>>>> process:
>>>>>>>
>>>>>>> _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>>>>
>>>>>>> We are calling IndexWriter.close() shortly before attempting to
>>>>>>> write
>>>>>>> to the directory (a few lines of code earlier) so I suspect it could
>>>>>>> be related to timing somehow if Lucene is perhaps still doing
>>>>>>> something on a background thread at this time (though I was under
>>>>>>> the
>>>>>>> impression that close() waited for merges and so forth to complete
>>>>>>> before returning.)
>>>>>>>
>>>>>>> Daniel
>>>>>>>
>>>>>>> --
>>>>>>> Daniel Noll Forensic and eDiscovery
>>>>>>> Software
>>>>>>> Senior Developer The world's most
>>>>>>> advanced
>>>>>>> Nuix email data
>>>>>>> analysis
>>>>>>> http://nuix.com/ and eDiscovery
>>>>>>> software
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>> For additional commands, e-mail: java-user-help [at] lucene
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299405.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>
>

--
View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299941.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

john.wang at gmail

Nov 12, 2009, 8:24 AM

Post #10 of 16 (1210 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

Well, I have code in the finally block to call IndexReader.close for every
reader I get from IndexWriter.getReader.

On Mon, Nov 9, 2009 at 2:43 AM, Michael McCandless <
lucene [at] mikemccandless> wrote:

> Does this look like a real leak John? You're definitely closing every
> reader you get back from getReader?
>
> Mike
>
> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang [at] gmail> wrote:
> > I am seeing the samething, but only when IndexWriter.getReader is called
> at
> > a high rate.
> >
> > from lsof, I see file handles growing.
> >
> > -John
> >
> > On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix> wrote:
> >
> >> Hi all.
> >>
> >> We updated to Lucene 2.9, and now we find that after closing our text
> >> index, it is not possible to rename the directory in which it resides
> >> (we are actually renaming a directory further up the hierarchy.)
> >>
> >> We discovered that the following files were still open by the process:
> >>
> >> _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
> >>
> >> We are calling IndexWriter.close() shortly before attempting to write
> >> to the directory (a few lines of code earlier) so I suspect it could
> >> be related to timing somehow if Lucene is perhaps still doing
> >> something on a background thread at this time (though I was under the
> >> impression that close() waited for merges and so forth to complete
> >> before returning.)
> >>
> >> Daniel
> >>
> >> --
> >> Daniel Noll Forensic and eDiscovery Software
> >> Senior Developer The world's most advanced
> >> Nuix email data analysis
> >> http://nuix.com/ and eDiscovery software
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> >> For additional commands, e-mail: java-user-help [at] lucene
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

john.wang at gmail

Nov 12, 2009, 8:25 AM

Post #11 of 16 (1197 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

If you run the zoie test turned to nrt, you can replicate it rather easily:

While the test is running, do lsof on your process, e.g.

lsof -p <proc_id> | <indexdir> | wc

-John

On Thu, Nov 12, 2009 at 8:24 AM, John Wang <john.wang [at] gmail> wrote:

> Well, I have code in the finally block to call IndexReader.close for every
> reader I get from IndexWriter.getReader.
>
>
> On Mon, Nov 9, 2009 at 2:43 AM, Michael McCandless <
> lucene [at] mikemccandless> wrote:
>
>> Does this look like a real leak John? You're definitely closing every
>> reader you get back from getReader?
>>
>> Mike
>>
>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang [at] gmail> wrote:
>> > I am seeing the samething, but only when IndexWriter.getReader is called
>> at
>> > a high rate.
>> >
>> > from lsof, I see file handles growing.
>> >
>> > -John
>> >
>> > On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix> wrote:
>> >
>> >> Hi all.
>> >>
>> >> We updated to Lucene 2.9, and now we find that after closing our text
>> >> index, it is not possible to rename the directory in which it resides
>> >> (we are actually renaming a directory further up the hierarchy.)
>> >>
>> >> We discovered that the following files were still open by the process:
>> >>
>> >> _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>> >>
>> >> We are calling IndexWriter.close() shortly before attempting to write
>> >> to the directory (a few lines of code earlier) so I suspect it could
>> >> be related to timing somehow if Lucene is perhaps still doing
>> >> something on a background thread at this time (though I was under the
>> >> impression that close() waited for merges and so forth to complete
>> >> before returning.)
>> >>
>> >> Daniel
>> >>
>> >> --
>> >> Daniel Noll Forensic and eDiscovery Software
>> >> Senior Developer The world's most advanced
>> >> Nuix email data analysis
>> >> http://nuix.com/ and eDiscovery
>> software
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> >> For additional commands, e-mail: java-user-help [at] lucene
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>
>

jason.rutherglen at gmail

Nov 12, 2009, 10:15 AM

Post #12 of 16 (1205 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

If there's a bug you're seeing, it's helpful to open an issue and post
code reproducing it.

On Wed, Nov 11, 2009 at 3:41 AM, Albert Juhe <albertjuhe [at] gmail> wrote:
>
> I think that this is the best way to proceed.
>
> thank you Mike
>
>
>
> Michael McCandless-2 wrote:
>>
>> Can you narrow the leak down to a small self-contained test?
>>
>> Mike
>>
>> On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe <albertjuhe [at] gmail> wrote:
>>>
>>> I don't get any exception.
>>>
>>> thank you Mike
>>>
>>>
>>> Michael McCandless-2 wrote:
>>>>
>>>> Do you see your exception handler printing anything out?
>>>>
>>>> You don't need to close the underlying IndexReader, just the
>>>> IndexSearcher (which will close the IndexReader, since it was the one
>>>> that had opened it).
>>>>
>>>> Mike
>>>>
>>>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <albertjuhe [at] gmail>
>>>> wrote:
>>>>>
>>>>> I don't know if it's the same problem but I think it's similar,
>>>>>
>>>>> My problem is with the Indexsearcher. I've installed a web search
>>>>> engine
>>>>> that uses Lucene, after a search I make a close operation like this:
>>>>>
>>>>> private IndexSearcher searcher;
>>>>>
>>>>> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
>>>>> this.searcher = new IndexSearcher(directory, true);
>>>>>
>>>>> �public void close() {
>>>>> � � � �try {
>>>>> � � � � � �System.out.println("Closing: "
>>>>> +this.searcher.getIndexReader().directory().toString() );
>>>>> � � � � � �this.searcher.getIndexReader().directory().close();
>>>>> � � � � � �this.searcher.getIndexReader().close();
>>>>> � � � � � �this.searcher.close();
>>>>> � � � �} catch (IOException e) {
>>>>> � � � � � �System.out.println(" caught a " + e.getClass() + "\n with
>>>>> message: " + e.getMessage());
>>>>> � � � �}
>>>>> � �}
>>>>> jboss don't close *.cfs index files, everytime that I make a new search
>>>>> a
>>>>> new file is caught by jboss.
>>>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command,
>>>>> the
>>>>> only way to free this files and close it is stopping jboss.
>>>>>
>>>>> The problem is that after search many times, �jboss go down and I have
>>>>> to
>>>>> restart it.
>>>>>
>>>>> Do you have the same problem with IndexSearch or is only with
>>>>> IndexWriter?
>>>>>
>>>>> Albert Juhe
>>>>> Learning Technologies
>>>>> Universitat Oberta de Catalunya
>>>>>
>>>>>
>>>>> Michael McCandless-2 wrote:
>>>>>>
>>>>>> Does this look like a real leak John? �You're definitely closing every
>>>>>> reader you get back from getReader?
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang [at] gmail>
>>>>>> wrote:
>>>>>>> I am seeing the samething, but only when IndexWriter.getReader is
>>>>>>> called
>>>>>>> at
>>>>>>> a high rate.
>>>>>>>
>>>>>>> from lsof, I see file handles growing.
>>>>>>>
>>>>>>> -John
>>>>>>>
>>>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix> wrote:
>>>>>>>
>>>>>>>> Hi all.
>>>>>>>>
>>>>>>>> We updated to Lucene 2.9, and now we find that after closing our
>>>>>>>> text
>>>>>>>> index, it is not possible to rename the directory in which it
>>>>>>>> resides
>>>>>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>>>>>
>>>>>>>> We discovered that the following files were still open by the
>>>>>>>> process:
>>>>>>>>
>>>>>>>> �_0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>>>>>
>>>>>>>> We are calling IndexWriter.close() shortly before attempting to
>>>>>>>> write
>>>>>>>> to the directory (a few lines of code earlier) so I suspect it could
>>>>>>>> be related to timing somehow if Lucene is perhaps still doing
>>>>>>>> something on a background thread at this time (though I was under
>>>>>>>> the
>>>>>>>> impression that close() waited for merges and so forth to complete
>>>>>>>> before returning.)
>>>>>>>>
>>>>>>>> Daniel
>>>>>>>>
>>>>>>>> --
>>>>>>>> Daniel Noll � � � � � � � � � � � � � �Forensic and eDiscovery
>>>>>>>> Software
>>>>>>>> Senior Developer � � � � � � � � � � � � � � �The world's most
>>>>>>>> advanced
>>>>>>>> Nuix � � � � � � � � � � � � � � � � � � � � � � � �email data
>>>>>>>> analysis
>>>>>>>> http://nuix.com/ � � � � � � � � � � � � � � � �and eDiscovery
>>>>>>>> software
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299405.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>> For additional commands, e-mail: java-user-help [at] lucene
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299941.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

albertjuhe at gmail

Nov 13, 2009, 2:12 AM

Post #13 of 16 (1175 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

Hi,

About this problem I did a test yesterday, I did a downgrade, I changed
versión 2.9.1 to 2.4.1, and the problem has been solved, all the files are
closed corretly and JBOSS isn't unstable.

Another problem that we have observed is:

Sometimes, random success, when you try to make a serach the first time
don't return result but the second time yes (Lucene 2.9.1. BooleanQuery).
I get this error message:
java.lang.NullPointerException
at
org.apache.jsp.objects.buscar_005fresposta_jsp._jspService(buscar_005fresposta_jsp.java:631)
at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
at
org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at
org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
at java.lang.Thread.run(Thread.java:595)

I don't know whats the meanig of it, but buscar_resposta.jsp is the jsp that
makes a Search.

With 2.4.1 this problem don't happens.

Jason Rutherglen-2 wrote:
>
> If there's a bug you're seeing, it's helpful to open an issue and post
> code reproducing it.
>
> On Wed, Nov 11, 2009 at 3:41 AM, Albert Juhe <albertjuhe [at] gmail> wrote:
>>
>> I think that this is the best way to proceed.
>>
>> thank you Mike
>>
>>
>>
>> Michael McCandless-2 wrote:
>>>
>>> Can you narrow the leak down to a small self-contained test?
>>>
>>> Mike
>>>
>>> On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe <albertjuhe [at] gmail>
>>> wrote:
>>>>
>>>> I don't get any exception.
>>>>
>>>> thank you Mike
>>>>
>>>>
>>>> Michael McCandless-2 wrote:
>>>>>
>>>>> Do you see your exception handler printing anything out?
>>>>>
>>>>> You don't need to close the underlying IndexReader, just the
>>>>> IndexSearcher (which will close the IndexReader, since it was the one
>>>>> that had opened it).
>>>>>
>>>>> Mike
>>>>>
>>>>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <albertjuhe [at] gmail>
>>>>> wrote:
>>>>>>
>>>>>> I don't know if it's the same problem but I think it's similar,
>>>>>>
>>>>>> My problem is with the Indexsearcher. I've installed a web search
>>>>>> engine
>>>>>> that uses Lucene, after a search I make a close operation like this:
>>>>>>
>>>>>> private IndexSearcher searcher;
>>>>>>
>>>>>> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
>>>>>> this.searcher = new IndexSearcher(directory, true);
>>>>>>
>>>>>> public void close() {
>>>>>> try {
>>>>>> System.out.println("Closing: "
>>>>>> +this.searcher.getIndexReader().directory().toString() );
>>>>>> this.searcher.getIndexReader().directory().close();
>>>>>> this.searcher.getIndexReader().close();
>>>>>> this.searcher.close();
>>>>>> } catch (IOException e) {
>>>>>> System.out.println(" caught a " + e.getClass() + "\n with
>>>>>> message: " + e.getMessage());
>>>>>> }
>>>>>> }
>>>>>> jboss don't close *.cfs index files, everytime that I make a new
>>>>>> search
>>>>>> a
>>>>>> new file is caught by jboss.
>>>>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command,
>>>>>> the
>>>>>> only way to free this files and close it is stopping jboss.
>>>>>>
>>>>>> The problem is that after search many times, jboss go down and I
>>>>>> have
>>>>>> to
>>>>>> restart it.
>>>>>>
>>>>>> Do you have the same problem with IndexSearch or is only with
>>>>>> IndexWriter?
>>>>>>
>>>>>> Albert Juhe
>>>>>> Learning Technologies
>>>>>> Universitat Oberta de Catalunya
>>>>>>
>>>>>>
>>>>>> Michael McCandless-2 wrote:
>>>>>>>
>>>>>>> Does this look like a real leak John? You're definitely closing
>>>>>>> every
>>>>>>> reader you get back from getReader?
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang [at] gmail>
>>>>>>> wrote:
>>>>>>>> I am seeing the samething, but only when IndexWriter.getReader is
>>>>>>>> called
>>>>>>>> at
>>>>>>>> a high rate.
>>>>>>>>
>>>>>>>> from lsof, I see file handles growing.
>>>>>>>>
>>>>>>>> -John
>>>>>>>>
>>>>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all.
>>>>>>>>>
>>>>>>>>> We updated to Lucene 2.9, and now we find that after closing our
>>>>>>>>> text
>>>>>>>>> index, it is not possible to rename the directory in which it
>>>>>>>>> resides
>>>>>>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>>>>>>
>>>>>>>>> We discovered that the following files were still open by the
>>>>>>>>> process:
>>>>>>>>>
>>>>>>>>> _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>>>>>>
>>>>>>>>> We are calling IndexWriter.close() shortly before attempting to
>>>>>>>>> write
>>>>>>>>> to the directory (a few lines of code earlier) so I suspect it
>>>>>>>>> could
>>>>>>>>> be related to timing somehow if Lucene is perhaps still doing
>>>>>>>>> something on a background thread at this time (though I was under
>>>>>>>>> the
>>>>>>>>> impression that close() waited for merges and so forth to complete
>>>>>>>>> before returning.)
>>>>>>>>>
>>>>>>>>> Daniel
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Daniel Noll Forensic and eDiscovery
>>>>>>>>> Software
>>>>>>>>> Senior Developer The world's most
>>>>>>>>> advanced
>>>>>>>>> Nuix email data
>>>>>>>>> analysis
>>>>>>>>> http://nuix.com/ and eDiscovery
>>>>>>>>> software
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
>>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299405.html
>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>> For additional commands, e-mail: java-user-help [at] lucene
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299941.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>
>

--
View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26334253.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Nov 13, 2009, 2:33 AM

Post #14 of 16 (1179 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

Any luck narrowing this to a standalone test case that shows the problem?

That new exception appears to be inside the Java code created by the
app server compiling your JSP -- it's not very helpful since it
doesn't "enter" Lucene. Can you try to narrow this to a standalone
test case, too?

Thanks.

Mike

On Fri, Nov 13, 2009 at 5:12 AM, Albert Juhe <albertjuhe [at] gmail> wrote:
>
> Hi,
>
> About this problem I did a test yesterday, I did a downgrade, I changed
> versi�n 2.9.1 to 2.4.1, and the problem has been solved, all the files are
> closed corretly and JBOSS isn't unstable.
>
> Another problem that we have observed is:
>
> Sometimes, random success, when you try to make a serach the first time
> don't return result but the second time yes (Lucene 2.9.1. BooleanQuery).
> I get this error message:
> java.lang.NullPointerException
> � � � �at
> org.apache.jsp.objects.buscar_005fresposta_jsp._jspService(buscar_005fresposta_jsp.java:631)
> � � � �at
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> � � � �at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
> � � � �at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)
> � � � �at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
> � � � �at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
> � � � �at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
> � � � �at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
> � � � �at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
> � � � �at
> org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
> � � � �at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
> � � � �at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
> � � � �at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> � � � �at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
> � � � �at
> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
> � � � �at
> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
> � � � �at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
> � � � �at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
> � � � �at
> org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
> � � � �at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
> � � � �at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
> � � � �at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
> � � � �at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
> � � � �at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
> � � � �at
> org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
> � � � �at java.lang.Thread.run(Thread.java:595)
>
> I don't know whats the meanig of it, but buscar_resposta.jsp is the jsp that
> makes a Search.
>
> With 2.4.1 this problem don't happens.
>
>
> Jason Rutherglen-2 wrote:
>>
>> If there's a bug you're seeing, it's helpful to open an issue and post
>> code reproducing it.
>>
>> On Wed, Nov 11, 2009 at 3:41 AM, Albert Juhe <albertjuhe [at] gmail> wrote:
>>>
>>> I think that this is the best way to proceed.
>>>
>>> thank you Mike
>>>
>>>
>>>
>>> Michael McCandless-2 wrote:
>>>>
>>>> Can you narrow the leak down to a small self-contained test?
>>>>
>>>> Mike
>>>>
>>>> On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe <albertjuhe [at] gmail>
>>>> wrote:
>>>>>
>>>>> I don't get any exception.
>>>>>
>>>>> thank you Mike
>>>>>
>>>>>
>>>>> Michael McCandless-2 wrote:
>>>>>>
>>>>>> Do you see your exception handler printing anything out?
>>>>>>
>>>>>> You don't need to close the underlying IndexReader, just the
>>>>>> IndexSearcher (which will close the IndexReader, since it was the one
>>>>>> that had opened it).
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <albertjuhe [at] gmail>
>>>>>> wrote:
>>>>>>>
>>>>>>> I don't know if it's the same problem but I think it's similar,
>>>>>>>
>>>>>>> My problem is with the Indexsearcher. I've installed a web search
>>>>>>> engine
>>>>>>> that uses Lucene, after a search I make a close operation like this:
>>>>>>>
>>>>>>> private IndexSearcher searcher;
>>>>>>>
>>>>>>> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
>>>>>>> this.searcher = new IndexSearcher(directory, true);
>>>>>>>
>>>>>>> �public void close() {
>>>>>>> � � � �try {
>>>>>>> � � � � � �System.out.println("Closing: "
>>>>>>> +this.searcher.getIndexReader().directory().toString() );
>>>>>>> � � � � � �this.searcher.getIndexReader().directory().close();
>>>>>>> � � � � � �this.searcher.getIndexReader().close();
>>>>>>> � � � � � �this.searcher.close();
>>>>>>> � � � �} catch (IOException e) {
>>>>>>> � � � � � �System.out.println(" caught a " + e.getClass() + "\n with
>>>>>>> message: " + e.getMessage());
>>>>>>> � � � �}
>>>>>>> � �}
>>>>>>> jboss don't close *.cfs index files, everytime that I make a new
>>>>>>> search
>>>>>>> a
>>>>>>> new file is caught by jboss.
>>>>>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof command,
>>>>>>> the
>>>>>>> only way to free this files and close it is stopping jboss.
>>>>>>>
>>>>>>> The problem is that after search many times, �jboss go down and I
>>>>>>> have
>>>>>>> to
>>>>>>> restart it.
>>>>>>>
>>>>>>> Do you have the same problem with IndexSearch or is only with
>>>>>>> IndexWriter?
>>>>>>>
>>>>>>> Albert Juhe
>>>>>>> Learning Technologies
>>>>>>> Universitat Oberta de Catalunya
>>>>>>>
>>>>>>>
>>>>>>> Michael McCandless-2 wrote:
>>>>>>>>
>>>>>>>> Does this look like a real leak John? �You're definitely closing
>>>>>>>> every
>>>>>>>> reader you get back from getReader?
>>>>>>>>
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang [at] gmail>
>>>>>>>> wrote:
>>>>>>>>> I am seeing the samething, but only when IndexWriter.getReader is
>>>>>>>>> called
>>>>>>>>> at
>>>>>>>>> a high rate.
>>>>>>>>>
>>>>>>>>> from lsof, I see file handles growing.
>>>>>>>>>
>>>>>>>>> -John
>>>>>>>>>
>>>>>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all.
>>>>>>>>>>
>>>>>>>>>> We updated to Lucene 2.9, and now we find that after closing our
>>>>>>>>>> text
>>>>>>>>>> index, it is not possible to rename the directory in which it
>>>>>>>>>> resides
>>>>>>>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>>>>>>>
>>>>>>>>>> We discovered that the following files were still open by the
>>>>>>>>>> process:
>>>>>>>>>>
>>>>>>>>>> �_0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>>>>>>>
>>>>>>>>>> We are calling IndexWriter.close() shortly before attempting to
>>>>>>>>>> write
>>>>>>>>>> to the directory (a few lines of code earlier) so I suspect it
>>>>>>>>>> could
>>>>>>>>>> be related to timing somehow if Lucene is perhaps still doing
>>>>>>>>>> something on a background thread at this time (though I was under
>>>>>>>>>> the
>>>>>>>>>> impression that close() waited for merges and so forth to complete
>>>>>>>>>> before returning.)
>>>>>>>>>>
>>>>>>>>>> Daniel
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Daniel Noll � � � � � � � � � � � � � �Forensic and eDiscovery
>>>>>>>>>> Software
>>>>>>>>>> Senior Developer � � � � � � � � � � � � � � �The world's most
>>>>>>>>>> advanced
>>>>>>>>>> Nuix � � � � � � � � � � � � � � � � � � � � � � � �email data
>>>>>>>>>> analysis
>>>>>>>>>> http://nuix.com/ � � � � � � � � � � � � � � � �and eDiscovery
>>>>>>>>>> software
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
>>>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299405.html
>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299941.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>> For additional commands, e-mail: java-user-help [at] lucene
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26334253.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Nov 13, 2009, 3:39 AM

Post #15 of 16 (1172 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

Alas, I can't repro this problem ("leaking file descriptors with NRT"), either.

I've got a decent stress test setup -- start with a 5M Wikipedia
index, update (delete & add) @ 1000 docs/sec (using 2 threads), reopen
10X per second, searching at redline (using 9 threads), and the open
file descriptors look contained.

Are you using compound format or not? Do you hit any odd exceptions
in your test? Any other differences in how Zoie uses Lucene NRT that
you can think of...?

Mike

On Thu, Nov 12, 2009 at 11:24 AM, John Wang <john.wang [at] gmail> wrote:
> Well, I have code in the finally block to call IndexReader.close for every
> reader I get from IndexWriter.getReader.
>
> On Mon, Nov 9, 2009 at 2:43 AM, Michael McCandless <
> lucene [at] mikemccandless> wrote:
>
>> Does this look like a real leak John? �You're definitely closing every
>> reader you get back from getReader?
>>
>> Mike
>>
>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang [at] gmail> wrote:
>> > I am seeing the samething, but only when IndexWriter.getReader is called
>> at
>> > a high rate.
>> >
>> > from lsof, I see file handles growing.
>> >
>> > -John
>> >
>> > On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix> wrote:
>> >
>> >> Hi all.
>> >>
>> >> We updated to Lucene 2.9, and now we find that after closing our text
>> >> index, it is not possible to rename the directory in which it resides
>> >> (we are actually renaming a directory further up the hierarchy.)
>> >>
>> >> We discovered that the following files were still open by the process:
>> >>
>> >> �_0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>> >>
>> >> We are calling IndexWriter.close() shortly before attempting to write
>> >> to the directory (a few lines of code earlier) so I suspect it could
>> >> be related to timing somehow if Lucene is perhaps still doing
>> >> something on a background thread at this time (though I was under the
>> >> impression that close() waited for merges and so forth to complete
>> >> before returning.)
>> >>
>> >> Daniel
>> >>
>> >> --
>> >> Daniel Noll � � � � � � � � � � � � � �Forensic and eDiscovery Software
>> >> Senior Developer � � � � � � � � � � � � � � �The world's most advanced
>> >> Nuix � � � � � � � � � � � � � � � � � � � � � � � �email data analysis
>> >> http://nuix.com/ � � � � � � � � � � � � � � � �and eDiscovery software
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> >> For additional commands, e-mail: java-user-help [at] lucene
>> >>
>> >>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

albertjuhe at gmail

Nov 13, 2009, 3:46 AM

Post #16 of 16 (1179 views)
Permalink

Re: IndexWriter.close() no longer seems to close everything [In reply to]

I don't know how, because the problem is with JBOSS in a productions
environment, in a localhost this don't happens.

The JBOSS server is in a production environment, contains a lot of projects,
I don't know if Lucene enters in fight with othe libraries.

I don't have control of this computer, I send the war to the production
departament and they install it, I will try to recover more logs files.

Other part of log file:
2009-11-11 16:50:28,567 INFO [STDOUT] ************ START SEARCHING
BIBLIOGRAPHY ***************
2009-11-11 16:50:28,577 INFO [STDOUT] Query
MultiFieldQueryParser:tecnologia 1
2009-11-11 16:50:28,577 INFO [STDOUT] Query
MultiFieldQueryParser:tecnologia 1
2009-11-11 16:50:28,577 INFO [STDOUT] Query
MultiFieldQueryParser:tecnologia 1
2009-11-11 16:50:28,731 ERROR
[org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/webapps/uocle].[jsp]]
Servlet.service() for servlet jsp threw exception

The code that generates the error is:

The query is: MultiFieldQueryParser:tecnologia

BooleanQuery bqCodi = new BooleanQuery();

MultiFieldQueryParser mFilefQueryParser = new
MultiFieldQueryParser(this.fields,this.analyzer);
mFilefQueryParser.setDefaultOperator(QueryParser.AND_OPERATOR);
queryM = mFilefQueryParser.parse(this.query_text);
bqCodi.add(queryM, BooleanClause.Occur.SHOULD);

DuplicateFilter df = new DuplicateFilter("title");
df.setKeepMode(DuplicateFilter.KM_USE_FIRST_OCCURRENCE);
this.topDocs = getSearcher().search(bqCodi, df, MAX_TERMES_BIBLIOGRAFIA,
Sort.RELEVANCE);

I don't know if this information can be usefull.

When I changed the Lucene library to 2.4.1 with the same code, only I have
changed how I open Indexreader, in 2.9.1 I opened in readonly=true, the
applications runs perfect.

Michael McCandless-2 wrote:
>
> Any luck narrowing this to a standalone test case that shows the problem?
>
> That new exception appears to be inside the Java code created by the
> app server compiling your JSP -- it's not very helpful since it
> doesn't "enter" Lucene. Can you try to narrow this to a standalone
> test case, too?
>
> Thanks.
>
> Mike
>
> On Fri, Nov 13, 2009 at 5:12 AM, Albert Juhe <albertjuhe [at] gmail> wrote:
>>
>> Hi,
>>
>> About this problem I did a test yesterday, I did a downgrade, I changed
>> versión 2.9.1 to 2.4.1, and the problem has been solved, all the files
>> are
>> closed corretly and JBOSS isn't unstable.
>>
>> Another problem that we have observed is:
>>
>> Sometimes, random success, when you try to make a serach the first time
>> don't return result but the second time yes (Lucene 2.9.1. BooleanQuery).
>> I get this error message:
>> java.lang.NullPointerException
>> at
>> org.apache.jsp.objects.buscar_005fresposta_jsp._jspService(buscar_005fresposta_jsp.java:631)
>> at
>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
>> at
>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334)
>> at
>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
>> at
>> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>> at
>> org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
>> at
>> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
>> at
>> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
>> at
>> org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
>> at
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
>> at
>> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
>> at
>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
>> at
>> org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
>> at java.lang.Thread.run(Thread.java:595)
>>
>> I don't know whats the meanig of it, but buscar_resposta.jsp is the jsp
>> that
>> makes a Search.
>>
>> With 2.4.1 this problem don't happens.
>>
>>
>> Jason Rutherglen-2 wrote:
>>>
>>> If there's a bug you're seeing, it's helpful to open an issue and post
>>> code reproducing it.
>>>
>>> On Wed, Nov 11, 2009 at 3:41 AM, Albert Juhe <albertjuhe [at] gmail>
>>> wrote:
>>>>
>>>> I think that this is the best way to proceed.
>>>>
>>>> thank you Mike
>>>>
>>>>
>>>>
>>>> Michael McCandless-2 wrote:
>>>>>
>>>>> Can you narrow the leak down to a small self-contained test?
>>>>>
>>>>> Mike
>>>>>
>>>>> On Wed, Nov 11, 2009 at 5:54 AM, Albert Juhe <albertjuhe [at] gmail>
>>>>> wrote:
>>>>>>
>>>>>> I don't get any exception.
>>>>>>
>>>>>> thank you Mike
>>>>>>
>>>>>>
>>>>>> Michael McCandless-2 wrote:
>>>>>>>
>>>>>>> Do you see your exception handler printing anything out?
>>>>>>>
>>>>>>> You don't need to close the underlying IndexReader, just the
>>>>>>> IndexSearcher (which will close the IndexReader, since it was the
>>>>>>> one
>>>>>>> that had opened it).
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>> On Wed, Nov 11, 2009 at 5:10 AM, Albert Juhe <albertjuhe [at] gmail>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I don't know if it's the same problem but I think it's similar,
>>>>>>>>
>>>>>>>> My problem is with the Indexsearcher. I've installed a web search
>>>>>>>> engine
>>>>>>>> that uses Lucene, after a search I make a close operation like
>>>>>>>> this:
>>>>>>>>
>>>>>>>> private IndexSearcher searcher;
>>>>>>>>
>>>>>>>> NIOFSDirectory directory = new NIOFSDirectory(new File(path));
>>>>>>>> this.searcher = new IndexSearcher(directory, true);
>>>>>>>>
>>>>>>>> public void close() {
>>>>>>>> try {
>>>>>>>> System.out.println("Closing: "
>>>>>>>> +this.searcher.getIndexReader().directory().toString() );
>>>>>>>> this.searcher.getIndexReader().directory().close();
>>>>>>>> this.searcher.getIndexReader().close();
>>>>>>>> this.searcher.close();
>>>>>>>> } catch (IOException e) {
>>>>>>>> System.out.println(" caught a " + e.getClass() + "\n
>>>>>>>> with
>>>>>>>> message: " + e.getMessage());
>>>>>>>> }
>>>>>>>> }
>>>>>>>> jboss don't close *.cfs index files, everytime that I make a new
>>>>>>>> search
>>>>>>>> a
>>>>>>>> new file is caught by jboss.
>>>>>>>> If I make 5 searchs jboss caught 5 *.cfs, I see it with lsof
>>>>>>>> command,
>>>>>>>> the
>>>>>>>> only way to free this files and close it is stopping jboss.
>>>>>>>>
>>>>>>>> The problem is that after search many times, jboss go down and I
>>>>>>>> have
>>>>>>>> to
>>>>>>>> restart it.
>>>>>>>>
>>>>>>>> Do you have the same problem with IndexSearch or is only with
>>>>>>>> IndexWriter?
>>>>>>>>
>>>>>>>> Albert Juhe
>>>>>>>> Learning Technologies
>>>>>>>> Universitat Oberta de Catalunya
>>>>>>>>
>>>>>>>>
>>>>>>>> Michael McCandless-2 wrote:
>>>>>>>>>
>>>>>>>>> Does this look like a real leak John? You're definitely closing
>>>>>>>>> every
>>>>>>>>> reader you get back from getReader?
>>>>>>>>>
>>>>>>>>> Mike
>>>>>>>>>
>>>>>>>>> On Sun, Nov 8, 2009 at 10:41 PM, John Wang <john.wang [at] gmail>
>>>>>>>>> wrote:
>>>>>>>>>> I am seeing the samething, but only when IndexWriter.getReader is
>>>>>>>>>> called
>>>>>>>>>> at
>>>>>>>>>> a high rate.
>>>>>>>>>>
>>>>>>>>>> from lsof, I see file handles growing.
>>>>>>>>>>
>>>>>>>>>> -John
>>>>>>>>>>
>>>>>>>>>> On Sun, Nov 8, 2009 at 7:29 PM, Daniel Noll <daniel [at] nuix>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all.
>>>>>>>>>>>
>>>>>>>>>>> We updated to Lucene 2.9, and now we find that after closing our
>>>>>>>>>>> text
>>>>>>>>>>> index, it is not possible to rename the directory in which it
>>>>>>>>>>> resides
>>>>>>>>>>> (we are actually renaming a directory further up the hierarchy.)
>>>>>>>>>>>
>>>>>>>>>>> We discovered that the following files were still open by the
>>>>>>>>>>> process:
>>>>>>>>>>>
>>>>>>>>>>> _0.tis, _0.frq, _0.prx, _0.fdt, _0.fdx, _0.tvx, _0.tvd, _0.tvf
>>>>>>>>>>>
>>>>>>>>>>> We are calling IndexWriter.close() shortly before attempting to
>>>>>>>>>>> write
>>>>>>>>>>> to the directory (a few lines of code earlier) so I suspect it
>>>>>>>>>>> could
>>>>>>>>>>> be related to timing somehow if Lucene is perhaps still doing
>>>>>>>>>>> something on a background thread at this time (though I was
>>>>>>>>>>> under
>>>>>>>>>>> the
>>>>>>>>>>> impression that close() waited for merges and so forth to
>>>>>>>>>>> complete
>>>>>>>>>>> before returning.)
>>>>>>>>>>>
>>>>>>>>>>> Daniel
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Daniel Noll Forensic and eDiscovery
>>>>>>>>>>> Software
>>>>>>>>>>> Senior Developer The world's most
>>>>>>>>>>> advanced
>>>>>>>>>>> Nuix email data
>>>>>>>>>>> analysis
>>>>>>>>>>> http://nuix.com/ and eDiscovery
>>>>>>>>>>> software
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>> java-user-help [at] lucene
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26298910.html
>>>>>>>> Sent from the Lucene - Java Users mailing list archive at
>>>>>>>> Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299405.html
>>>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26299941.html
>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>>> For additional commands, e-mail: java-user-help [at] lucene
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>>> For additional commands, e-mail: java-user-help [at] lucene
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26334253.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>
>

--
View this message in context: http://old.nabble.com/IndexWriter.close%28%29-no-longer-seems-to-close-everything-tp26260801p26335402.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads