Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Lucene: Java-User

Cached Hits / closing IndexSearcher after add/delete w/IndexModifier

 

 

Lucene java-user RSS feed   Index | Next | Previous | View Threaded


yogieric.dev at gmail

Nov 29, 2006, 2:40 AM

Post #1 of 7 (1448 views)
Permalink
Cached Hits / closing IndexSearcher after add/delete w/IndexModifier

I'm using lucene as a backend for my webservices that provide add,
remove and search operations. When I add or remove documents via
IndexModifier, I believe I'm supposed to close the IndexSearcher I
use for query requests. However, I cache Hits and I believe the
javadocs indicate closing an IndexSearcher will invalidate the cached
Hits such that trying to retrieve a doc may throw an exception. So my
solution is to wrap the IndexSearcher in another class with a
finalizer that closes the IndexSearcher when all the Hits dereference
it. My question is whether my cached hits have a solid reference to
IndexSearcher such that I can rely on it being closed only after I
expire the Hits from the cache?

Also, should I keep one IndexModifier open for the life of my service/
application or should I open and close it when I get new requests to
add or remove documents? (I don't really have control over batching
unfortunately -- though I've certainly pointed it out.)

Thanks,
Eric


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


yonik at apache

Nov 29, 2006, 11:16 AM

Post #2 of 7 (1398 views)
Permalink
Re: Cached Hits / closing IndexSearcher after add/delete w/IndexModifier [In reply to]

On 11/29/06, Eric Brown <yogieric.dev [at] gmail> wrote:
> I'm using lucene as a backend for my webservices that provide add,
> remove and search operations. When I add or remove documents via
> IndexModifier, I believe I'm supposed to close the IndexSearcher I
> use for query requests. However, I cache Hits and I believe the
> javadocs indicate closing an IndexSearcher will invalidate the cached
> Hits such that trying to retrieve a doc may throw an exception. So my
> solution is to wrap the IndexSearcher in another class with a
> finalizer that closes the IndexSearcher when all the Hits dereference
> it. My question is whether my cached hits have a solid reference to
> IndexSearcher such that I can rely on it being closed only after I
> expire the Hits from the cache?

Yes, they do.

> Also, should I keep one IndexModifier open for the life of my service/
> application or should I open and close it when I get new requests to
> add or remove documents? (I don't really have control over batching
> unfortunately -- though I've certainly pointed it out.)

IndexModifier currently has very low performance with mixed adds and deletes.
You can keep the same one over the lifetime of the app though.

If you haven't seen it, another alternative that might fit your needs is Solr.

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


yogieric.dev at gmail

Nov 29, 2006, 12:32 PM

Post #3 of 7 (1400 views)
Permalink
Re: Cached Hits / closing IndexSearcher after add/delete w/IndexModifier [In reply to]

Hi Yonik,

On Nov 29, 2006, at 11:16 AM, Yonik Seeley wrote:
> On 11/29/06, Eric Brown <yogieric.dev [at] gmail> wrote:
[snip]
>> Also, should I keep one IndexModifier open for the life of my
>> service/
>> application or should I open and close it when I get new requests to
>> add or remove documents? (I don't really have control over batching
>> unfortunately -- though I've certainly pointed it out.)
>
> IndexModifier currently has very low performance with mixed adds
> and deletes.
> You can keep the same one over the lifetime of the app though.

Assuming I call flush() after every operation (they won't be that
frequent), if I don't call close() when my application shuts down,
will I run into locking issues when I restart my application?

> If you haven't seen it, another alternative that might fit your
> needs is Solr.

I just took a look. It looks very applicable and I'll look at it
seriously for subsequent roll-outs of our site. For the short-term
I'm too tight for time to change direction.

Thanks!
Eric


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


yonik at apache

Nov 29, 2006, 2:02 PM

Post #4 of 7 (1414 views)
Permalink
Re: Cached Hits / closing IndexSearcher after add/delete w/IndexModifier [In reply to]

On 11/29/06, Eric Brown <yogieric.dev [at] gmail> wrote:
> > IndexModifier currently has very low performance with mixed adds
> > and deletes.
> > You can keep the same one over the lifetime of the app though.
>
> Assuming I call flush() after every operation (they won't be that
> frequent), if I don't call close() when my application shuts down,
> will I run into locking issues when I restart my application?

IndexWriter has a finalizer, bug finalizers aren't guaranteed to be run AFAIK.
There is a JVM shutdown hook you could use to clean up though.
You could also use native locks (in the current lucene devel version).

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


yonik at apache

Nov 29, 2006, 2:06 PM

Post #5 of 7 (1420 views)
Permalink
Re: Cached Hits / closing IndexSearcher after add/delete w/IndexModifier [In reply to]

On 11/29/06, Yonik Seeley <yonik [at] apache> wrote:
> On 11/29/06, Eric Brown <yogieric.dev [at] gmail> wrote:
> > > IndexModifier currently has very low performance with mixed adds
> > > and deletes.
> > > You can keep the same one over the lifetime of the app though.
> >
> > Assuming I call flush() after every operation (they won't be that
> > frequent), if I don't call close() when my application shuts down,
> > will I run into locking issues when I restart my application?
>
> IndexWriter has a finalizer, bug finalizers aren't guaranteed to be run AFAIK.
> There is a JVM shutdown hook you could use to clean up though.
> You could also use native locks (in the current lucene devel version).

System.runFinalizersOnExit(true) is also an option.

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


markrmiller at gmail

Nov 29, 2006, 2:56 PM

Post #6 of 7 (1404 views)
Permalink
Re: Cached Hits / closing IndexSearcher after add/delete w/IndexModifier [In reply to]

If you haven't seen the following option you might want to check it out. It uses reference counting to keep track of writers/searchers/etc(I think solr does as well).
Has worked great for everything I have thrown at it other than need some tweaks to its multisearcher support. Also is a good
base for adding the ability to 'warm' searchers.

The oversimplified gist is: When a writer is closed, new searchers are opened for the index.


BEGIN QUOTE:

You can download the source here:

http://www.blizzy.de/lucene/lucene-indexaccess-0.1.0.zip


Using LuceneIndexAccessor is incredibly simple:


Directory directory = ...
Analyzer analyzer = ...

// somewhere near program start
IIndexAccessProvider accessProvider =
new IndexAccessProvider(directory, analyzer);
ILuceneIndexAccessor accessor = new LuceneIndexAccessor(accessProvider);
accessor.open();


IndexWriter writer = null;
try {
writer = accessor.getWriter();

// use writer...
} catch (IOException e) {
// ...
} finally {
accessor.release(write);
}


// somewhere near program exit
accessor.close();



Eric Brown wrote:
> Hi Yonik,
>
> On Nov 29, 2006, at 11:16 AM, Yonik Seeley wrote:
>> On 11/29/06, Eric Brown <yogieric.dev [at] gmail> wrote:
> [snip]
>>> Also, should I keep one IndexModifier open for the life of my service/
>>> application or should I open and close it when I get new requests to
>>> add or remove documents? (I don't really have control over batching
>>> unfortunately -- though I've certainly pointed it out.)
>>
>> IndexModifier currently has very low performance with mixed adds and
>> deletes.
>> You can keep the same one over the lifetime of the app though.
>
> Assuming I call flush() after every operation (they won't be that
> frequent), if I don't call close() when my application shuts down,
> will I run into locking issues when I restart my application?
>
>> If you haven't seen it, another alternative that might fit your needs
>> is Solr.
>
> I just took a look. It looks very applicable and I'll look at it
> seriously for subsequent roll-outs of our site. For the short-term I'm
> too tight for time to change direction.
>
> Thanks!
> Eric
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


yonik at apache

Nov 29, 2006, 5:13 PM

Post #7 of 7 (1403 views)
Permalink
Re: Cached Hits / closing IndexSearcher after add/delete w/IndexModifier [In reply to]

On 11/29/06, Mark Miller <markrmiller [at] gmail> wrote:
> If you haven't seen the following option you might want to check it out. It uses reference counting to keep track of writers/searchers/etc(I think solr does as well).

Right, Solr uses reference counting... relying on GC will get out
out-of-filehandle exceptions.
http://svn.apache.org/viewvc/incubator/solr/trunk/src/java/org/apache/solr/util/RefCounted.java?view=markup


-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Lucene java-user RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.