Mailing List Archive: Lucene: Java-User

Index files not deleted after optimization

Index | Next | Previous | View Threaded

emailgane at yahoo

Nov 2, 2009, 3:54 AM

Post #1 of 7 (1107 views)
Permalink

Index files not deleted after optimization

Hello all,

I am using Lucene 2.4.1 and My app is running inside Tomcat.

In Windows, after database optimization, the old db files are not getting deleted. I enabled the info stream and found the below entries. I used ProcessExplorer from SysInternals to view the lock file, but old db files are not locked by any process. Even rename of those files are successful. If i stop the tomcat, old db files are deleted. I could see only the optimized segment files. The same code works fine in Linux. I am facing this issue in Windows XP and 2003.

IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.frq": java.io.IOException: Cannot delete <FolderName>\_0.frq>; Will re-try later.
IFD [FLUSH]: delete "_0.prx"
IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.prx": java.io.IOException: Cannot delete <FolderName>\_0.prx; Will re-try later.
IFD [FLUSH]: delete "_0.tis"
IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.tis": java.io.IOException: Cannot delete <FolderName>\_0.tis; Will re-try later.

Regards
Ganesh

Send instant messages to your online friends http://in.messenger.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Nov 2, 2009, 4:33 AM

Post #2 of 7 (1061 views)
Permalink

Re: Index files not deleted after optimization [In reply to]

Something must still have these file handles open at the time the
optimization completed.

EG do you have a reader open on this index?

Mike

On Mon, Nov 2, 2009 at 6:54 AM, Ganesh <emailgane [at] yahoo> wrote:
> Hello all,
>
> I am using Lucene 2.4.1 and My app is running inside Tomcat.
>
> In Windows, after database optimization, the old db files are not getting deleted. I enabled the info stream and found the below entries. I used ProcessExplorer from SysInternals to view the lock file, but old db files are not locked by any process. Even rename of those files are successful. If i stop the tomcat, old db files are deleted. I could see only the optimized segment files. The same code works fine in Linux. I am facing this issue in Windows XP and 2003.
>
> IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.frq": java.io.IOException: Cannot delete <FolderName>\_0.frq>; Will re-try later.
> IFD [FLUSH]: delete "_0.prx"
> IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.prx": java.io.IOException: Cannot delete <FolderName>\_0.prx; Will re-try later.
> IFD [FLUSH]: delete "_0.tis"
> IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.tis": java.io.IOException: Cannot delete <FolderName>\_0.tis; Will re-try later.
>
> Regards
> Ganesh
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

emailgane at yahoo

Nov 3, 2009, 1:17 AM

Post #3 of 7 (1045 views)
Permalink

Re: Index files not deleted after optimization [In reply to]

I am reopening the reader and closing the old one. I am not facing this issue in Linux otherwise Linux will show <filename>(deleted) under /proc/pid/fd/.

searcher.getIndexReader().close();
searcher.close();

Regards
Ganesh

----- Original Message -----
From: "Michael McCandless" <lucene [at] mikemccandless>
To: <java-user [at] lucene>
Sent: Monday, November 02, 2009 6:03 PM
Subject: Re: Index files not deleted after optimization

Something must still have these file handles open at the time the
optimization completed.

EG do you have a reader open on this index?

Mike

On Mon, Nov 2, 2009 at 6:54 AM, Ganesh <emailgane [at] yahoo> wrote:
> Hello all,
>
> I am using Lucene 2.4.1 and My app is running inside Tomcat.
>
> In Windows, after database optimization, the old db files are not getting deleted. I enabled the info stream and found the below entries. I used ProcessExplorer from SysInternals to view the lock file, but old db files are not locked by any process. Even rename of those files are successful. If i stop the tomcat, old db files are deleted. I could see only the optimized segment files. The same code works fine in Linux. I am facing this issue in Windows XP and 2003.
>
> IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.frq": java.io.IOException: Cannot delete <FolderName>\_0.frq>; Will re-try later.
> IFD [FLUSH]: delete "_0.prx"
> IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.prx": java.io.IOException: Cannot delete <FolderName>\_0.prx; Will re-try later.
> IFD [FLUSH]: delete "_0.tis"
> IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.tis": java.io.IOException: Cannot delete <FolderName>\_0.tis; Will re-try later.
>
> Regards
> Ganesh
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Send instant messages to your online friends http://in.messenger.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Nov 3, 2009, 1:52 AM

Post #4 of 7 (1046 views)
Permalink

Re: Index files not deleted after optimization [In reply to]

It depends on the relative timing.

If the old IndexReader is still open when the optimize completes then
the files it has open cannot be deleted.

But, if that IndexReader hadn't been reopened in a while, it's
possible it did not in fact have the files of the just-merged segments
open (in which case the delete would succeed).

Mike

On Tue, Nov 3, 2009 at 4:17 AM, Ganesh <emailgane [at] yahoo> wrote:
> I am reopening the reader and closing the old one. I am not facing this issue in Linux otherwise Linux will show <filename>(deleted) under /proc/pid/fd/.
>
> searcher.getIndexReader().close();
> searcher.close();
>
> Regards
> Ganesh
>
> ----- Original Message -----
> From: "Michael McCandless" <lucene [at] mikemccandless>
> To: <java-user [at] lucene>
> Sent: Monday, November 02, 2009 6:03 PM
> Subject: Re: Index files not deleted after optimization
>
>
> Something must still have these file handles open at the time the
> optimization completed.
>
> EG do you have a reader open on this index?
>
> Mike
>
> On Mon, Nov 2, 2009 at 6:54 AM, Ganesh <emailgane [at] yahoo> wrote:
>> Hello all,
>>
>> I am using Lucene 2.4.1 and My app is running inside Tomcat.
>>
>> In Windows, after database optimization, the old db files are not getting deleted. I enabled the info stream and found the below entries. I used ProcessExplorer from SysInternals to view the lock file, but old db files are not locked by any process. Even rename of those files are successful. If i stop the tomcat, old db files are deleted. I could see only the optimized segment files. The same code works fine in Linux. I am facing this issue in Windows XP and 2003.
>>
>> IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.frq": java.io.IOException: Cannot delete <FolderName>\_0.frq>; Will re-try later.
>> IFD [FLUSH]: delete "_0.prx"
>> IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.prx": java.io.IOException: Cannot delete <FolderName>\_0.prx; Will re-try later.
>> IFD [FLUSH]: delete "_0.tis"
>> IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.tis": java.io.IOException: Cannot delete <FolderName>\_0.tis; Will re-try later.
>>
>> Regards
>> Ganesh
>>
>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

emailgane at yahoo

Nov 3, 2009, 6:45 AM

Post #5 of 7 (1043 views)
Permalink

Re: Index files not deleted after optimization [In reply to]

My IndexReader and Searcher is open all the time. I am reopening it at constant interval.

Below are the code sequence.

1. DB optimize
2. Close writer
3. Open writer
4. Reopen new reader
5. Close old reader
6. Close old searcher.

>>If the old IndexReader is still open when the optimize completes then the files it has open cannot be deleted.
If reader should not be opened while doing optimize then User cannot perform search during the time period.

Regards
Ganesh

----- Original Message -----
From: "Michael McCandless" <lucene [at] mikemccandless>
To: <java-user [at] lucene>
Sent: Tuesday, November 03, 2009 3:22 PM
Subject: Re: Index files not deleted after optimization

It depends on the relative timing.

If the old IndexReader is still open when the optimize completes then
the files it has open cannot be deleted.

But, if that IndexReader hadn't been reopened in a while, it's
possible it did not in fact have the files of the just-merged segments
open (in which case the delete would succeed).

Mike

On Tue, Nov 3, 2009 at 4:17 AM, Ganesh <emailgane [at] yahoo> wrote:
> I am reopening the reader and closing the old one. I am not facing this issue in Linux otherwise Linux will show <filename>(deleted) under /proc/pid/fd/.
>
> searcher.getIndexReader().close();
> searcher.close();
>
> Regards
> Ganesh
>
> ----- Original Message -----
> From: "Michael McCandless" <lucene [at] mikemccandless>
> To: <java-user [at] lucene>
> Sent: Monday, November 02, 2009 6:03 PM
> Subject: Re: Index files not deleted after optimization
>
>
> Something must still have these file handles open at the time the
> optimization completed.
>
> EG do you have a reader open on this index?
>
> Mike
>
> On Mon, Nov 2, 2009 at 6:54 AM, Ganesh <emailgane [at] yahoo> wrote:
>> Hello all,
>>
>> I am using Lucene 2.4.1 and My app is running inside Tomcat.
>>
>> In Windows, after database optimization, the old db files are not getting deleted. I enabled the info stream and found the below entries. I used ProcessExplorer from SysInternals to view the lock file, but old db files are not locked by any process. Even rename of those files are successful. If i stop the tomcat, old db files are deleted. I could see only the optimized segment files. The same code works fine in Linux. I am facing this issue in Windows XP and 2003.
>>
>> IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.frq": java.io.IOException: Cannot delete <FolderName>\_0.frq>; Will re-try later.
>> IFD [FLUSH]: delete "_0.prx"
>> IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.prx": java.io.IOException: Cannot delete <FolderName>\_0.prx; Will re-try later.
>> IFD [FLUSH]: delete "_0.tis"
>> IFD [FLUSH]: IndexFileDeleter: unable to remove file "_0.tis": java.io.IOException: Cannot delete <FolderName>\_0.tis; Will re-try later.
>>
>> Regards
>> Ganesh
>>
>> Send instant messages to your online friends http://in.messenger.yahoo.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Send instant messages to your online friends http://in.messenger.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Nov 3, 2009, 7:35 AM

Post #6 of 7 (1042 views)
Permalink

Re: Index files not deleted after optimization [In reply to]

On Tue, Nov 3, 2009 at 9:45 AM, Ganesh <emailgane [at] yahoo> wrote:
> My IndexReader and Searcher is open all the time. I am reopening it at constant interval.
>
> Below are the code sequence.
>
> 1. DB optimize
> 2. Close writer
> 3. Open writer
> 4. Reopen new reader
> 5. Close old reader
> 6. Close old searcher.

Assuming there is an implicit:

0. Open a reader/searcher

Then it's expected after step 6 that you have leftover files, because
the writers in step 2 and 3 were unable to delete the files since the
reader from step 0 still had them open. Once you open a new writer,
or commit any changes with the existing writer, the files will be
deleted.

>>>If the old IndexReader is still open when the optimize completes then the files it has open cannot be deleted.
> If reader should not be opened while doing optimize then User cannot perform search during the time period.

It's fine to leave a reader open... it just means the old files will
persist until the writer is able to delete them.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

emailgane at yahoo

Nov 5, 2009, 11:22 PM

Post #7 of 7 (1010 views)
Permalink

Re: Index files not deleted after optimization [In reply to]

Thanks Michael.

Below order helped to solve the issue,

1. DB optimize
2. DB commit
3. Reopen new reader
4. Close old reader
5. Close writer
6. Open writer

Regards
Ganesh

----- Original Message -----
From: "Michael McCandless" <lucene [at] mikemccandless>
To: <java-user [at] lucene>
Sent: Tuesday, November 03, 2009 9:05 PM
Subject: Re: Index files not deleted after optimization

> On Tue, Nov 3, 2009 at 9:45 AM, Ganesh <emailgane [at] yahoo> wrote:
>> My IndexReader and Searcher is open all the time. I am reopening it at constant interval.
>>
>> Below are the code sequence.
>>
>> 1. DB optimize
>> 2. Close writer
>> 3. Open writer
>> 4. Reopen new reader
>> 5. Close old reader
>> 6. Close old searcher.
>
> Assuming there is an implicit:
>
> 0. Open a reader/searcher
>
> Then it's expected after step 6 that you have leftover files, because
> the writers in step 2 and 3 were unable to delete the files since the
> reader from step 0 still had them open. Once you open a new writer,
> or commit any changes with the existing writer, the files will be
> deleted.
>
>>>>If the old IndexReader is still open when the optimize completes then the files it has open cannot be deleted.
>> If reader should not be opened while doing optimize then User cannot perform search during the time period.
>
> It's fine to leave a reader open... it just means the old files will
> persist until the writer is able to delete them.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
Send instant messages to your online friends http://in.messenger.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads