Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Lucene: Java-User

Clarification about IndexWriter.deleteDocuments and flush.

 

 

Lucene java-user RSS feed   Index | Next | Previous | View Threaded


cambazz at gmail

Jan 21, 2008, 6:30 AM

Post #1 of 3 (4832 views)
Permalink
Clarification about IndexWriter.deleteDocuments and flush.

Hello,

When we delete documents from index - will it autoflush when count of
deleted documents reach a certain value. I am controlling my own flush
operation, and I have disabled autoflush by:

writer.setMaxBufferedDocs(IndexWriter.DISABLE_AUTO_FLUSH);


But I have taken a peek at the IndexWriter and the DocumentWriter source
codes and from what I understand delete will flush the index when it reaches
a certain threshold.
But I am not sure if that will happen when the autoflush is disabled.

Also when deleted documents trigger a flush, that will be a cumulative flush
that flushes all the addDocs as well? is this correct?

Best,

-C.B.


lucene at mikemccandless

Jan 21, 2008, 6:47 AM

Post #2 of 3 (4489 views)
Permalink
Re: Clarification about IndexWriter.deleteDocuments and flush. [In reply to]

Cam Bazz wrote:

> Hello,
>
> When we delete documents from index - will it autoflush when count of
> deleted documents reach a certain value. I am controlling my own flush
> operation, and I have disabled autoflush by:
>
> writer.setMaxBufferedDocs(IndexWriter.DISABLE_AUTO_FLUSH);

By default (in 2.3) the writer only flushes when RAM usage hit 16
MB. It should not flush due to # buffered docs nor # buffered delete
terms, and, that method call shouldn't be necessary since it's
already the default.

>
> But I have taken a peek at the IndexWriter and the DocumentWriter
> source
> codes and from what I understand delete will flush the index when
> it reaches
> a certain threshold.
> But I am not sure if that will happen when the autoflush is disabled.

Right, if you disable it (as above), it won't flush by count but
rather by RAM.

> Also when deleted documents trigger a flush, that will be a
> cumulative flush
> that flushes all the addDocs as well? is this correct?

Yes whenever a flush occurs (due to any trigger) both buffered docs
and buffered deletes are flushed.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


cambazz at gmail

Jan 21, 2008, 7:00 AM

Post #3 of 3 (4477 views)
Permalink
Re: Clarification about IndexWriter.deleteDocuments and flush. [In reply to]

Thanks Michael,

> Right, if you disable it (as above), it won't flush by count but
> rather by RAM.

I had made a test case monitoring ram usage and never flushing manually -
(with disabled autoflush)
and I think it wont flush itself when it reaches a certain buffered ram.

Having read the source to lucene made makes me think of extensions. Nice
code.

Best,


On Jan 21, 2008 4:47 PM, Michael McCandless <lucene [at] mikemccandless>
wrote:

>
> Cam Bazz wrote:
>
> > Hello,
> >
> > When we delete documents from index - will it autoflush when count of
> > deleted documents reach a certain value. I am controlling my own flush
> > operation, and I have disabled autoflush by:
> >
> > writer.setMaxBufferedDocs(IndexWriter.DISABLE_AUTO_FLUSH);
>
> By default (in 2.3) the writer only flushes when RAM usage hit 16
> MB. It should not flush due to # buffered docs nor # buffered delete
> terms, and, that method call shouldn't be necessary since it's
> already the default.
>
> >
> > But I have taken a peek at the IndexWriter and the DocumentWriter
> > source
> > codes and from what I understand delete will flush the index when
> > it reaches
> > a certain threshold.
> > But I am not sure if that will happen when the autoflush is disabled.
>
> Right, if you disable it (as above), it won't flush by count but
> rather by RAM.
>
> > Also when deleted documents trigger a flush, that will be a
> > cumulative flush
> > that flushes all the addDocs as well? is this correct?
>
> Yes whenever a flush occurs (due to any trigger) both buffered docs
> and buffered deletes are flushed.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

Lucene java-user RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.