Mailing List Archive: Lucene: Java-User

running a lucene indexing app as a windows service on xp, crashing

Index | Next | Previous | View Threaded

MModrall at glgroup

Aug 4, 2006, 2:14 PM

Post #1 of 9 (676 views)
Permalink

running a lucene indexing app as a windows service on xp, crashing

Hi...

Not sure what's the right group for this question. We have a java
program running with the tanuki Windows Service Wrapper on XP. This
program is using Lucene to do a fair amount of indexing, creating and
deleting scads of files in its working directory.

The Lucene code is crashing under circumstances that seem pretty lame.
At periodic intervals, lucene tries to File.renameTo(newfile).
Sometimes this fails, so Lucene implemented some fall-back code to
manually copy the contents of the file from old to new. Our problem is
that sometimes *this* fails too, then the whole thing blows up.

Specifically, the exception comes when Lucene is trying to create a
FileOutputStream(newFile) where the newFile doesn't exist yet. The
error message reported is Access Denied, yet Lucene has already created
and destroyed hundreds of files in the directory already so it really
can't be a permissions thing on the directory.

This is semi-reproducible, but the thing that seems to trigger the error
is having a separate Windows Explorer window up, looking at the
directory and hitting Refresh frequently. It seems like if Windows
Explorer is refreshing at the exact moment File.renameTo() and new
FileOutputStream() run, you get an Access Denied exception.

I don't know if this weak behavior stems from the JVM's implementation
or a weakness in the filesystem, but it's a bit of a nuisance. If we
never use explorer to look at the directory, it may work but it seems
weak to have to tell people "Don't look at that".

Any idea what the root cause is and how to work around it?

thanks

-Mark

This e-mail message, and any attachments, is intended only for the use of the individual or entity identified in the alias address of this message and may contain information that is confidential, privileged and subject to legal restrictions and penalties regarding its unauthorized disclosure and use. Any unauthorized review, copying, disclosure, use or distribution is strictly prohibited. If you have received this e-mail message in error, please notify the sender immediately by reply e-mail and delete this message, and any attachments, from your system. Thank you.

lucene at mikemccandless

Aug 4, 2006, 2:28 PM

Post #2 of 9 (670 views)
Permalink

Re: running a lucene indexing app as a windows service on xp, crashing [In reply to]

> The Lucene code is crashing under circumstances that seem pretty lame.
> At periodic intervals, lucene tries to File.renameTo(newfile).
> Sometimes this fails, so Lucene implemented some fall-back code to
> manually copy the contents of the file from old to new. Our problem is
> that sometimes *this* fails too, then the whole thing blows up.

I agree Lucene should not be affected by things like having Windows
Explorer windows up!

Can you post the traceback you're seeing? In particular, what's failing
about the fallback approach?

Is it in the FSDirectory's renameFile method? There is this comment in
there:

// Rename the old file to the new one. Unfortunately, the renameTo()
// method does not work reliably under some JVMs. Therefore, if the
// rename fails, we manually rename by copying the old file to the
new one

and correspondingly this comment in the original commit for this (back
on March 1 2003):

- Added a manual renaming of files in case the java.io.File's renameTo(File)
call fails. It has been reported that this happens on Windows JVMs.
Contributed by: Matt Tucker

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

MModrall at glgroup

Aug 4, 2006, 2:45 PM

Post #3 of 9 (666 views)
Permalink

RE: running a lucene indexing app as a windows service on xp, crashing [In reply to]

Hi Michael...

Here's the traceback:
[Indexer.java 652] buildFullIndex: Error building full index
java.io.IOException: Cannot rename
D:\indexbuild1\contact_index\deleteable.new to
D:\indexbuild1\contact_index\deletable at
org.apache.lucene.store.FSDirectory.renameFile(FSDirectory.java:294) at
org.apache.lucene.index.IndexWriter.writeDeleteableFiles(IndexWriter.jav
a:827) at

org.apache.lucene.index.IndexWriter.deleteSegments(IndexWriter.java:771)
at
org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:732)
at
org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:686)
at
org.apache.lucene.index.IndexWriter.maybeMergeSegments(IndexWriter.java:
674)at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:479)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:462)
at
com.glgroup.glissando.Indexer.indexAllDocuments(Indexer.java:546) at
com.glgroup.glissando.Indexer.buildFullIndex(Indexer.java:646) at
com.glgroup.glissando.Indexer.buildIndex(Indexer.java:728) at
com.glgroup.glissando.Indexer.buildIndices(Indexer.java:845) at

com.glgroup.glissando.IndexingService.stepService(IndexingService.java:3
9) at
com.glgroup.util.threading.ServiceThread.run(ServiceThread.java:60)
Caused by: java.io.FileNotFoundException:
D:\indexbuild1\contact_index\deletable (Access is denied) at
java.io.FileOutputStream.open(Native Method) at
java.io.FileOutputStream.<init>(Unknown Source) at
java.io.FileOutputStream.<init>(Unknown Source) at
org.apache.lucene.store.FSDirectory.renameFile(FSDirectory.java:278)
... 13 more

So you can see, yes it's exactly in that spot. Given the
circumstances we've seen, it seems like File.renameTo() is working most
of the time but whatever causes it to fail at this specific point in
time also causes the
New FileOutputStream (nu);
to blow up 2 lines later.

As I said, the trigger seems to be Windows Explorer doing a
refresh at the same moment that Lucene is trying to do one of these
swaps. I don't know if this is a weakness in the jvm implementation of
the file classes or what, but I've found complaints/bugs about the java
file interactions on windows coming up to 2004.

-Mark

This e-mail message, and any attachments, is intended only for the use of the individual or entity identified in the alias address of this message and may contain information that is confidential, privileged and subject to legal restrictions and penalties regarding its unauthorized disclosure and use. Any unauthorized review, copying, disclosure, use or distribution is strictly prohibited. If you have received this e-mail message in error, please notify the sender immediately by reply e-mail and delete this message, and any attachments, from your system. Thank you.

-----Original Message-----

From: Michael McCandless [mailto:lucene [at] mikemccandless]
Sent: Friday, August 04, 2006 5:28 PM
To: java-user [at] lucene
Subject: Re: running a lucene indexing app as a windows service on xp,
crashing

> The Lucene code is crashing under circumstances that seem pretty lame.
> At periodic intervals, lucene tries to File.renameTo(newfile).
> Sometimes this fails, so Lucene implemented some fall-back code to
> manually copy the contents of the file from old to new. Our problem
is
> that sometimes *this* fails too, then the whole thing blows up.

I agree Lucene should not be affected by things like having Windows
Explorer windows up!

Can you post the traceback you're seeing? In particular, what's failing

about the fallback approach?

Is it in the FSDirectory's renameFile method? There is this comment in
there:

// Rename the old file to the new one. Unfortunately, the
renameTo()
// method does not work reliably under some JVMs. Therefore, if
the
// rename fails, we manually rename by copying the old file to the
new one

and correspondingly this comment in the original commit for this (back
on March 1 2003):

- Added a manual renaming of files in case the java.io.File's
renameTo(File)
call fails. It has been reported that this happens on Windows JVMs.
Contributed by: Matt Tucker

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

eksdev at yahoo

Aug 4, 2006, 11:15 PM

Post #4 of 9 (665 views)
Permalink

Re: running a lucene indexing app as a windows service on xp, crashing [In reply to]

This is windows/jvm issue . Have a look at how ant is dealing with it, maybe we could give it a try with something like that (I have not noticed ant having problems).

We are not able to reproduce this in our environment systematically, so it would be great if you could patch your lucene with this trick and let us know how it goes.
delete/rename->System.gc()->Thread.sleap()->delete/rename Again

/**
* Accommodate Windows bug encountered in both Sun and IBM JDKs.
* Others possible. If the delete does not work, call System.gc(),
* wait a little and try again.
*/
private boolean delete(File f) {
if (!f.delete()) {
System.gc();
try {
Thread.sleep(DELETE_RETRY_SLEEP_MILLIS);
} catch (InterruptedException ex) {
// Ignore Exception
}
if (!f.delete()) {
if (deleteOnExit) {
int level = quiet ? Project.MSG_VERBOSE : Project.MSG_INFO;
log("Failed to delete " + f + ", calling deleteOnExit."
+ " This attempts to delete the file when the Ant jvm"
+ " has exited and might not succeed.", level);
f.deleteOnExit();
return true;
}
return false;
}
}
return true;
}

----- Original Message ----
From: Michael McCandless <lucene [at] mikemccandless>
To: java-user [at] lucene
Sent: Friday, 4 August, 2006 11:28:02 PM
Subject: Re: running a lucene indexing app as a windows service on xp, crashing

> The Lucene code is crashing under circumstances that seem pretty lame.
> At periodic intervals, lucene tries to File.renameTo(newfile).
> Sometimes this fails, so Lucene implemented some fall-back code to
> manually copy the contents of the file from old to new. Our problem is
> that sometimes *this* fails too, then the whole thing blows up.

I agree Lucene should not be affected by things like having Windows
Explorer windows up!

Can you post the traceback you're seeing? In particular, what's failing
about the fallback approach?

Is it in the FSDirectory's renameFile method? There is this comment in
there:

// Rename the old file to the new one. Unfortunately, the renameTo()
// method does not work reliably under some JVMs. Therefore, if the
// rename fails, we manually rename by copying the old file to the
new one

and correspondingly this comment in the original commit for this (back
on March 1 2003):

- Added a manual renaming of files in case the java.io.File's renameTo(File)
call fails. It has been reported that this happens on Windows JVMs.
Contributed by: Matt Tucker

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Aug 5, 2006, 1:33 PM

Post #5 of 9 (656 views)
Permalink

Re: running a lucene indexing app as a windows service on xp, crashing [In reply to]

> This is windows/jvm issue . Have a look at how ant is dealing with it, maybe we could give it a try with something like that (I have not noticed ant having problems).

Indeed it seems like Windows somehow believes the "deletable" file must
be still held open, given that the File.renameTo and also the "fallback"
open failed.

I tried a quick test to reproduce this but could not. I just modified
the demo IndexFiles command line tool to add documents forever, then
used Windows Explorer to refresh like crazy... I tested with Sun JDK
1.4.2 on Windows XP Pro SP 2.

Which JDK / version, and Windows XP version, are you using?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

eksdev at yahoo

Aug 6, 2006, 12:50 AM

Post #6 of 9 (645 views)
Permalink

Re: running a lucene indexing app as a windows service on xp, crashing [In reply to]

XP Proffesionall / win 2003 Server, we had this issue on JVMs 1.5/1.6.

It seams it this happens "not so often" on 1.6/Win2003, but we have this in production only for 2 weeks.

We have single update machine that builds index in batch and replicates to many Index readers, so at least customers are not affected by this annoyance, they get their updates only a few hours later ...

Anyhow, I would be really interested to have this problem solved somehow, my logic was if ant never had this problem (not that I know), let us try their approach. What is annoyance for me personally is that I am not able to reproduce this problem reliably enough... happens only in production once in a month (daily updates, ca 40Mio documents)

What Store you use, MMAP or FS? IndexWriter cannot function with MMAP on our machines (yet another WIN problem), Reader works like a charm

On a side note, Latest 1.6 binaries look stable (as far as our Lucene based app is concerned), brings nice speed-up for free (up to 30% on 64 bit platforms!)

----- Original Message ----
From: Michael McCandless <lucene [at] mikemccandless>
To: java-user [at] lucene
Sent: Saturday, 5 August, 2006 10:33:31 PM
Subject: Re: running a lucene indexing app as a windows service on xp, crashing

> This is windows/jvm issue . Have a look at how ant is dealing with it, maybe we could give it a try with something like that (I have not noticed ant having problems).

Indeed it seems like Windows somehow believes the "deletable" file must
be still held open, given that the File.renameTo and also the "fallback"
open failed.

I tried a quick test to reproduce this but could not. I just modified
the demo IndexFiles command line tool to add documents forever, then
used Windows Explorer to refresh like crazy... I tested with Sun JDK
1.4.2 on Windows XP Pro SP 2.

Which JDK / version, and Windows XP version, are you using?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Aug 6, 2006, 6:02 PM

Post #7 of 9 (646 views)
Permalink

Re: running a lucene indexing app as a windows service on xp, crashing [In reply to]

> XP Proffesionall / win 2003 Server, we had this issue on JVMs 1.5/1.6.
>
> It seams it this happens "not so often" on 1.6/Win2003, but we have this in production only for 2 weeks.
>
> We have single update machine that builds index in batch and replicates to many Index readers, so at least customers are not affected by this annoyance, they get their updates only a few hours later ...

Have you also noticed that opening Windows explorer to the index output
directory causes this error? Are you seeing that same traceback as the
original poster (Mark)?

> Anyhow, I would be really interested to have this problem solved somehow, my logic was if ant never had this problem (not that I know), let us try their approach. What is annoyance for me personally is that I am not able to reproduce this problem reliably enough... happens only in production once in a month (daily updates, ca 40Mio documents)
>
> What Store you use, MMAP or FS? IndexWriter cannot function with MMAP on our machines (yet another WIN problem), Reader works like a charm

I'm just testing with FSDirectory right now. I also tested on Sun JDK 1.5:

Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_07-b03)

and refreshed the directory in Windows Explorer constantly, but no
Access Denied exceptions. I'm just running with this diff on the demo
IndexFiles.java:

===================================================================
--- src/demo/org/apache/lucene/demo/IndexFiles.java (revision 425202)
+++ src/demo/org/apache/lucene/demo/IndexFiles.java (working copy)
@@ -54,7 +54,9 @@
try {
IndexWriter writer = new IndexWriter(INDEX_DIR, new
StandardAnalyzer(), true);
System.out.println("Indexing to directory '" +INDEX_DIR+ "'...");
- indexDocs(writer, docDir);
+ for(int i=0;i<10000;i++) {
+ indexDocs(writer, docDir);
+ }
System.out.println("Optimizing...");
writer.optimize();
writer.close();

Mark could you try this modified IndexFiles.java and see if you can
reproduce the issue on your box? If so then I think there must be some
environment / versioning issue that tickles this JVM issue.

> On a side note, Latest 1.6 binaries look stable (as far as our Lucene based app is concerned), brings nice speed-up for free (up to 30% on 64 bit platforms!)

Fantastic! Progress is progress ...

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

MModrall at glgroup

Aug 7, 2006, 5:45 AM

Post #8 of 9 (631 views)
Permalink

RE: running a lucene indexing app as a windows service on xp, crashing [In reply to]

Hi Mike...

Sorry I didn't respond over the weekend; I wasn't checking work
email.

A few more pieces of information about our circumstances: the
indexing system is running Windows Server 2003 sp1, the directories that
the indexer is using are shares (helped reproduce; we had a couple of
people using explorer to watch the directory), the corpus being indexed
at the time had something like 190K rows and indexing took about 1.42
hours (so plenty of time to try). Another person here suggested the
fact that our explorer windows were set to "Details" might be relevant
since explorer needs to touch each file to get the details.

When we had 2 users hitting refresh on the window as frequently
as they could, the crash took only about 3 minutes to reproduce.

The traceback we saw was in the fallback code (when renameTo
didn't work) on the line opening the new output file (which didn't
exist). That made me wonder if the problem wasn't with the single
destination file but with access to the root directory. The indexer
cranks out scads of those little temp files; if Explorer is in the
middle of a details refresh, maybe it has some kind of lock on directory
modification and a long list would increase the chance that the java
code may be trying that code at the same time.

I'll ask the guy who's running the indexer if he'd be open to
trying the patch.

Thanks
_mark

This e-mail message, and any attachments, is intended only for the use of the individual or entity identified in the alias address of this message and may contain information that is confidential, privileged and subject to legal restrictions and penalties regarding its unauthorized disclosure and use. Any unauthorized review, copying, disclosure, use or distribution is strictly prohibited. If you have received this e-mail message in error, please notify the sender immediately by reply e-mail and delete this message, and any attachments, from your system. Thank you.

-----Original Message-----

From: Michael McCandless [mailto:lucene [at] mikemccandless]
Sent: Saturday, August 05, 2006 4:34 PM
To: java-user [at] lucene
Subject: Re: running a lucene indexing app as a windows service on xp,
crashing

> This is windows/jvm issue . Have a look at how ant is dealing with it,
maybe we could give it a try with something like that (I have not
noticed ant having problems).

Indeed it seems like Windows somehow believes the "deletable" file must
be still held open, given that the File.renameTo and also the "fallback"

open failed.

I tried a quick test to reproduce this but could not. I just modified
the demo IndexFiles command line tool to add documents forever, then
used Windows Explorer to refresh like crazy... I tested with Sun JDK
1.4.2 on Windows XP Pro SP 2.

Which JDK / version, and Windows XP version, are you using?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

MModrall at glgroup

Aug 7, 2006, 5:46 AM

Post #9 of 9 (639 views)
Permalink

RE: running a lucene indexing app as a windows service on xp, crashing [In reply to]

Oh, sorry forgot - jdk 1.5.0_06

This e-mail message, and any attachments, is intended only for the use of the individual or entity identified in the alias address of this message and may contain information that is confidential, privileged and subject to legal restrictions and penalties regarding its unauthorized disclosure and use. Any unauthorized review, copying, disclosure, use or distribution is strictly prohibited. If you have received this e-mail message in error, please notify the sender immediately by reply e-mail and delete this message, and any attachments, from your system. Thank you.

-----Original Message-----

From: Mark Modrall [mailto:MModrall [at] glgroup]
Sent: Monday, August 07, 2006 8:45 AM
To: java-user [at] lucene
Subject: RE: running a lucene indexing app as a windows service on xp,
crashing

Hi Mike...

Sorry I didn't respond over the weekend; I wasn't checking work
email.

A few more pieces of information about our circumstances: the
indexing system is running Windows Server 2003 sp1, the directories that
the indexer is using are shares (helped reproduce; we had a couple of
people using explorer to watch the directory), the corpus being indexed
at the time had something like 190K rows and indexing took about 1.42
hours (so plenty of time to try). Another person here suggested the
fact that our explorer windows were set to "Details" might be relevant
since explorer needs to touch each file to get the details.

When we had 2 users hitting refresh on the window as frequently
as they could, the crash took only about 3 minutes to reproduce.

The traceback we saw was in the fallback code (when renameTo
didn't work) on the line opening the new output file (which didn't
exist). That made me wonder if the problem wasn't with the single
destination file but with access to the root directory. The indexer
cranks out scads of those little temp files; if Explorer is in the
middle of a details refresh, maybe it has some kind of lock on directory
modification and a long list would increase the chance that the java
code may be trying that code at the same time.

I'll ask the guy who's running the indexer if he'd be open to
trying the patch.

Thanks
_mark

This e-mail message, and any attachments, is intended only for the use
of the individual or entity identified in the alias address of this
message and may contain information that is confidential, privileged and
subject to legal restrictions and penalties regarding its unauthorized
disclosure and use. Any unauthorized review, copying, disclosure, use or
distribution is strictly prohibited. If you have received this e-mail
message in error, please notify the sender immediately by reply e-mail
and delete this message, and any attachments, from your system. Thank
you.

-----Original Message-----

From: Michael McCandless [mailto:lucene [at] mikemccandless]
Sent: Saturday, August 05, 2006 4:34 PM
To: java-user [at] lucene
Subject: Re: running a lucene indexing app as a windows service on xp,
crashing

> This is windows/jvm issue . Have a look at how ant is dealing with it,
maybe we could give it a try with something like that (I have not
noticed ant having problems).

Indeed it seems like Windows somehow believes the "deletable" file must
be still held open, given that the File.renameTo and also the "fallback"

open failed.

I tried a quick test to reproduce this but could not. I just modified
the demo IndexFiles command line tool to add documents forever, then
used Windows Explorer to refresh like crazy... I tested with Sun JDK
1.4.2 on Windows XP Pro SP 2.

Which JDK / version, and Windows XP version, are you using?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com