Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Lucene: Java-User

IndexWriters and write locks

 

 

Lucene java-user RSS feed   Index | Next | Previous | View Threaded


pulkitsinghal at gmail

Nov 10, 2010, 6:38 AM

Post #1 of 11 (2027 views)
Permalink
IndexWriters and write locks

Hello,

1) On Windows, I often shut down my application server (which has active
IndexWriters open) using the ctrl+c keys.
2) I inspect my directories on the file system I see that the write.lock
file is still there.
3) I start the app server again, and do some operations that would require
IndexWriters to write to the same directories again and it works!

I don't understand why I do not run into any exceptions?
I mean there is already a lock file present which should prevent the
IndexWriters from getting access to the directories ... no?
I should be happy but I'm not because other folks are able to get exceptions
when they bounce their servers an I'm unable to reproduce the problem and I
can't help them.

Any clues? Anyone?

Thank You,
- Pulkit


lucene at mikemccandless

Nov 10, 2010, 8:21 AM

Post #2 of 11 (1987 views)
Permalink
Re: IndexWriters and write locks [In reply to]

Likely you are using NativeFSLockFactory?

In which case, a leftover lock file does not mean the index is in fact
locked, since the OS will [correctly] release the lock on process
exit.

Mike

On Wed, Nov 10, 2010 at 9:38 AM, Pulkit Singhal <pulkitsinghal [at] gmail> wrote:
> Hello,
>
> 1) On Windows, I often shut down my application server (which has active
> IndexWriters open) using the ctrl+c keys.
> 2) I inspect my directories on the file system I see that the write.lock
> file is still there.
> 3) I start the app server again, and do some operations that would require
> IndexWriters to write to the same directories again and it works!
>
> I don't understand why I do not run into any exceptions?
> I mean there is already a lock file present which should prevent the
> IndexWriters from getting access to the directories ... no?
> I should be happy but I'm not because other folks are able to get exceptions
> when they bounce their servers an I'm unable to reproduce the problem and I
> can't help them.
>
> Any clues? Anyone?
>
> Thank You,
> - Pulkit
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


pulkitsinghal at gmail

Nov 10, 2010, 9:36 AM

Post #3 of 11 (1991 views)
Permalink
Re: IndexWriters and write locks [In reply to]

I do not actually take the trouble to specify what Lock Factory to use,
hmmm.

Are you suggesting that because I'm using FSDirectory.open() in my code, I
get a locking scheme that works ... while on other machine for other folks,
they get one that runs into issues and throws
java.nio.channels.OverlappingFileLockException?

- Pulkit

On Wed, Nov 10, 2010 at 11:21 AM, Michael McCandless <
lucene [at] mikemccandless> wrote:

> Likely you are using NativeFSLockFactory?
>
> In which case, a leftover lock file does not mean the index is in fact
> locked, since the OS will [correctly] release the lock on process
> exit.
>
> Mike
>
> On Wed, Nov 10, 2010 at 9:38 AM, Pulkit Singhal <pulkitsinghal [at] gmail>
> wrote:
> > Hello,
> >
> > 1) On Windows, I often shut down my application server (which has active
> > IndexWriters open) using the ctrl+c keys.
> > 2) I inspect my directories on the file system I see that the write.lock
> > file is still there.
> > 3) I start the app server again, and do some operations that would
> require
> > IndexWriters to write to the same directories again and it works!
> >
> > I don't understand why I do not run into any exceptions?
> > I mean there is already a lock file present which should prevent the
> > IndexWriters from getting access to the directories ... no?
> > I should be happy but I'm not because other folks are able to get
> exceptions
> > when they bounce their servers an I'm unable to reproduce the problem and
> I
> > can't help them.
> >
> > Any clues? Anyone?
> >
> > Thank You,
> > - Pulkit
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>


uwe at thetaphi

Nov 10, 2010, 9:57 AM

Post #4 of 11 (1990 views)
Permalink
RE: IndexWriters and write locks [In reply to]

This is because Lucene uses Native Filesystem Locks. The lock file itself is just a placeholder which is not cleaned up on Ctrl-C. The lock is not the file itself, its *on* the file.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe [at] thetaphi

> -----Original Message-----
> From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> Sent: Wednesday, November 10, 2010 3:38 PM
> To: java-user [at] lucene
> Subject: IndexWriters and write locks
>
> Hello,
>
> 1) On Windows, I often shut down my application server (which has active
> IndexWriters open) using the ctrl+c keys.
> 2) I inspect my directories on the file system I see that the write.lock file is still
> there.
> 3) I start the app server again, and do some operations that would require
> IndexWriters to write to the same directories again and it works!
>
> I don't understand why I do not run into any exceptions?
> I mean there is already a lock file present which should prevent the
> IndexWriters from getting access to the directories ... no?
> I should be happy but I'm not because other folks are able to get exceptions
> when they bounce their servers an I'm unable to reproduce the problem and I
> can't help them.
>
> Any clues? Anyone?
>
> Thank You,
> - Pulkit


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


pulkitsinghal at gmail

Nov 10, 2010, 10:57 AM

Post #5 of 11 (2001 views)
Permalink
Re: IndexWriters and write locks [In reply to]

Thanks Uwe, that helps explain why the lock file is still there.

The last piece of the puzzle is why someone may see exceptions such as the
following from time to time:

java.nio.channels.OverlappingFileLockException
at
sun.nio.ch.FileChannelImpl$SharedFileLockTable.checkList(FileChannelImpl.java:1176)
at
sun.nio.ch.FileChannelImpl$SharedFileLockTable.add(FileChannelImpl.java:1078)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:878)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
at
org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:236)
at org.apache.lucene.store.Lock.obtain(Lock.java:72)
at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1041)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:864)

I suppose this means that the OS itself hasn't released the lock even after
I shut down my application server and restarted it.
Am I right?

Or is there something else that can possibly be the culprit (in anyone's
experience) that I can investigate?

- Pulkit

On Wed, Nov 10, 2010 at 12:57 PM, Uwe Schindler <uwe [at] thetaphi> wrote:

> This is because Lucene uses Native Filesystem Locks. The lock file itself
> is just a placeholder which is not cleaned up on Ctrl-C. The lock is not the
> file itself, its *on* the file.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe [at] thetaphi
>
> > -----Original Message-----
> > From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> > Sent: Wednesday, November 10, 2010 3:38 PM
> > To: java-user [at] lucene
> > Subject: IndexWriters and write locks
> >
> > Hello,
> >
> > 1) On Windows, I often shut down my application server (which has active
> > IndexWriters open) using the ctrl+c keys.
> > 2) I inspect my directories on the file system I see that the write.lock
> file is still
> > there.
> > 3) I start the app server again, and do some operations that would
> require
> > IndexWriters to write to the same directories again and it works!
> >
> > I don't understand why I do not run into any exceptions?
> > I mean there is already a lock file present which should prevent the
> > IndexWriters from getting access to the directories ... no?
> > I should be happy but I'm not because other folks are able to get
> exceptions
> > when they bounce their servers an I'm unable to reproduce the problem and
> I
> > can't help them.
> >
> > Any clues? Anyone?
> >
> > Thank You,
> > - Pulkit
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>


uwe at thetaphi

Nov 10, 2010, 11:40 AM

Post #6 of 11 (1993 views)
Permalink
RE: IndexWriters and write locks [In reply to]

Are you using NFS as filesystem? NFS is incompatible to lucene :-)

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe [at] thetaphi


> -----Original Message-----
> From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> Sent: Wednesday, November 10, 2010 7:57 PM
> To: java-user [at] lucene
> Subject: Re: IndexWriters and write locks
>
> Thanks Uwe, that helps explain why the lock file is still there.
>
> The last piece of the puzzle is why someone may see exceptions such as the
> following from time to time:
>
> java.nio.channels.OverlappingFileLockException
> at
> sun.nio.ch.FileChannelImpl$SharedFileLockTable.checkList(FileChannelImpl.java
> :1176)
> at
> sun.nio.ch.FileChannelImpl$SharedFileLockTable.add(FileChannelImpl.java:107
> 8)
> at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:878)
> at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
> at
> org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:236)
> at org.apache.lucene.store.Lock.obtain(Lock.java:72)
> at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1041)
> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:864)
>
> I suppose this means that the OS itself hasn't released the lock even after I shut
> down my application server and restarted it.
> Am I right?
>
> Or is there something else that can possibly be the culprit (in anyone's
> experience) that I can investigate?
>
> - Pulkit
>
> On Wed, Nov 10, 2010 at 12:57 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
>
> > This is because Lucene uses Native Filesystem Locks. The lock file
> > itself is just a placeholder which is not cleaned up on Ctrl-C. The
> > lock is not the file itself, its *on* the file.
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe [at] thetaphi
> >
> > > -----Original Message-----
> > > From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> > > Sent: Wednesday, November 10, 2010 3:38 PM
> > > To: java-user [at] lucene
> > > Subject: IndexWriters and write locks
> > >
> > > Hello,
> > >
> > > 1) On Windows, I often shut down my application server (which has
> > > active IndexWriters open) using the ctrl+c keys.
> > > 2) I inspect my directories on the file system I see that the
> > > write.lock
> > file is still
> > > there.
> > > 3) I start the app server again, and do some operations that would
> > require
> > > IndexWriters to write to the same directories again and it works!
> > >
> > > I don't understand why I do not run into any exceptions?
> > > I mean there is already a lock file present which should prevent the
> > > IndexWriters from getting access to the directories ... no?
> > > I should be happy but I'm not because other folks are able to get
> > exceptions
> > > when they bounce their servers an I'm unable to reproduce the
> > > problem and
> > I
> > > can't help them.
> > >
> > > Any clues? Anyone?
> > >
> > > Thank You,
> > > - Pulkit
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > For additional commands, e-mail: java-user-help [at] lucene
> >
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


pulkitsinghal at gmail

Nov 10, 2010, 11:55 AM

Post #7 of 11 (1993 views)
Permalink
Re: IndexWriters and write locks [In reply to]

You know that really confuses me. I've heard that stated a few times and
every time I just felt that it couldn't possibly be right. Maybe it was
meant in some very specific manner because otherwise aren't all Windows OSs
off-limits to Lucene then?

On Wed, Nov 10, 2010 at 2:40 PM, Uwe Schindler <uwe [at] thetaphi> wrote:

> Are you using NFS as filesystem? NFS is incompatible to lucene :-)
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe [at] thetaphi
>
>
> > -----Original Message-----
> > From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> > Sent: Wednesday, November 10, 2010 7:57 PM
> > To: java-user [at] lucene
> > Subject: Re: IndexWriters and write locks
> >
> > Thanks Uwe, that helps explain why the lock file is still there.
> >
> > The last piece of the puzzle is why someone may see exceptions such as
> the
> > following from time to time:
> >
> > java.nio.channels.OverlappingFileLockException
> > at
> >
> sun.nio.ch.FileChannelImpl$SharedFileLockTable.checkList(FileChannelImpl.java
> > :1176)
> > at
> >
> sun.nio.ch.FileChannelImpl$SharedFileLockTable.add(FileChannelImpl.java:107
> > 8)
> > at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:878)
> > at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
> > at
> > org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:236)
> > at org.apache.lucene.store.Lock.obtain(Lock.java:72)
> > at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1041)
> > at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:864)
> >
> > I suppose this means that the OS itself hasn't released the lock even
> after I shut
> > down my application server and restarted it.
> > Am I right?
> >
> > Or is there something else that can possibly be the culprit (in anyone's
> > experience) that I can investigate?
> >
> > - Pulkit
> >
> > On Wed, Nov 10, 2010 at 12:57 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
> >
> > > This is because Lucene uses Native Filesystem Locks. The lock file
> > > itself is just a placeholder which is not cleaned up on Ctrl-C. The
> > > lock is not the file itself, its *on* the file.
> > >
> > > -----
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > http://www.thetaphi.de
> > > eMail: uwe [at] thetaphi
> > >
> > > > -----Original Message-----
> > > > From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> > > > Sent: Wednesday, November 10, 2010 3:38 PM
> > > > To: java-user [at] lucene
> > > > Subject: IndexWriters and write locks
> > > >
> > > > Hello,
> > > >
> > > > 1) On Windows, I often shut down my application server (which has
> > > > active IndexWriters open) using the ctrl+c keys.
> > > > 2) I inspect my directories on the file system I see that the
> > > > write.lock
> > > file is still
> > > > there.
> > > > 3) I start the app server again, and do some operations that would
> > > require
> > > > IndexWriters to write to the same directories again and it works!
> > > >
> > > > I don't understand why I do not run into any exceptions?
> > > > I mean there is already a lock file present which should prevent the
> > > > IndexWriters from getting access to the directories ... no?
> > > > I should be happy but I'm not because other folks are able to get
> > > exceptions
> > > > when they bounce their servers an I'm unable to reproduce the
> > > > problem and
> > > I
> > > > can't help them.
> > > >
> > > > Any clues? Anyone?
> > > >
> > > > Thank You,
> > > > - Pulkit
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > > For additional commands, e-mail: java-user-help [at] lucene
> > >
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>


sarowe at syr

Nov 10, 2010, 12:01 PM

Post #8 of 11 (1993 views)
Permalink
RE: IndexWriters and write locks [In reply to]

NFS[1] != NTFS[2]

[1] NFS: <http://en.wikipedia.org/wiki/Network_File_System_%28protocol%29>
[2] NTFS: <http://en.wikipedia.org/wiki/NTFS>

> -----Original Message-----
> From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> Sent: Wednesday, November 10, 2010 2:55 PM
> To: java-user [at] lucene
> Subject: Re: IndexWriters and write locks
>
> You know that really confuses me. I've heard that stated a few times and
> every time I just felt that it couldn't possibly be right. Maybe it was
> meant in some very specific manner because otherwise aren't all Windows
> OSs
> off-limits to Lucene then?
>
> On Wed, Nov 10, 2010 at 2:40 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
>
> > Are you using NFS as filesystem? NFS is incompatible to lucene :-)
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe [at] thetaphi
> >
> >
> > > -----Original Message-----
> > > From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> > > Sent: Wednesday, November 10, 2010 7:57 PM
> > > To: java-user [at] lucene
> > > Subject: Re: IndexWriters and write locks
> > >
> > > Thanks Uwe, that helps explain why the lock file is still there.
> > >
> > > The last piece of the puzzle is why someone may see exceptions such as
> > the
> > > following from time to time:
> > >
> > > java.nio.channels.OverlappingFileLockException
> > > at
> > >
> >
> sun.nio.ch.FileChannelImpl$SharedFileLockTable.checkList(FileChannelImpl.j
> ava
> > > :1176)
> > > at
> > >
> >
> sun.nio.ch.FileChannelImpl$SharedFileLockTable.add(FileChannelImpl.java:10
> 7
> > > 8)
> > > at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:878)
> > > at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
> > > at
> > >
> org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:236)
> > > at org.apache.lucene.store.Lock.obtain(Lock.java:72)
> > > at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1041)
> > > at
> org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:864)
> > >
> > > I suppose this means that the OS itself hasn't released the lock even
> > after I shut
> > > down my application server and restarted it.
> > > Am I right?
> > >
> > > Or is there something else that can possibly be the culprit (in
> anyone's
> > > experience) that I can investigate?
> > >
> > > - Pulkit
> > >
> > > On Wed, Nov 10, 2010 at 12:57 PM, Uwe Schindler <uwe [at] thetaphi>
> wrote:
> > >
> > > > This is because Lucene uses Native Filesystem Locks. The lock file
> > > > itself is just a placeholder which is not cleaned up on Ctrl-C. The
> > > > lock is not the file itself, its *on* the file.
> > > >
> > > > -----
> > > > Uwe Schindler
> > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > http://www.thetaphi.de
> > > > eMail: uwe [at] thetaphi
> > > >
> > > > > -----Original Message-----
> > > > > From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> > > > > Sent: Wednesday, November 10, 2010 3:38 PM
> > > > > To: java-user [at] lucene
> > > > > Subject: IndexWriters and write locks
> > > > >
> > > > > Hello,
> > > > >
> > > > > 1) On Windows, I often shut down my application server (which has
> > > > > active IndexWriters open) using the ctrl+c keys.
> > > > > 2) I inspect my directories on the file system I see that the
> > > > > write.lock
> > > > file is still
> > > > > there.
> > > > > 3) I start the app server again, and do some operations that would
> > > > require
> > > > > IndexWriters to write to the same directories again and it works!
> > > > >
> > > > > I don't understand why I do not run into any exceptions?
> > > > > I mean there is already a lock file present which should prevent
> the
> > > > > IndexWriters from getting access to the directories ... no?
> > > > > I should be happy but I'm not because other folks are able to get
> > > > exceptions
> > > > > when they bounce their servers an I'm unable to reproduce the
> > > > > problem and
> > > > I
> > > > > can't help them.
> > > > >
> > > > > Any clues? Anyone?
> > > > >
> > > > > Thank You,
> > > > > - Pulkit
> > > >
> > > >
> > > > --------------------------------------------------------------------
> -
> > > > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > > > For additional commands, e-mail: java-user-help [at] lucene
> > > >
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > For additional commands, e-mail: java-user-help [at] lucene
> >
> >


uwe at thetaphi

Nov 10, 2010, 12:02 PM

Post #9 of 11 (1990 views)
Permalink
RE: IndexWriters and write locks [In reply to]

Windows does not use NFS natively, it’s a network file system for UNIX O/S.

But can you confirm that you are working on a local filesystem even on windows (NTFS?). Not something like samba shares?

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe [at] thetaphi


> -----Original Message-----
> From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> Sent: Wednesday, November 10, 2010 8:55 PM
> To: java-user [at] lucene
> Subject: Re: IndexWriters and write locks
>
> You know that really confuses me. I've heard that stated a few times and every
> time I just felt that it couldn't possibly be right. Maybe it was meant in some
> very specific manner because otherwise aren't all Windows OSs off-limits to
> Lucene then?
>
> On Wed, Nov 10, 2010 at 2:40 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
>
> > Are you using NFS as filesystem? NFS is incompatible to lucene :-)
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe [at] thetaphi
> >
> >
> > > -----Original Message-----
> > > From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> > > Sent: Wednesday, November 10, 2010 7:57 PM
> > > To: java-user [at] lucene
> > > Subject: Re: IndexWriters and write locks
> > >
> > > Thanks Uwe, that helps explain why the lock file is still there.
> > >
> > > The last piece of the puzzle is why someone may see exceptions such
> > > as
> > the
> > > following from time to time:
> > >
> > > java.nio.channels.OverlappingFileLockException
> > > at
> > >
> > sun.nio.ch.FileChannelImpl$SharedFileLockTable.checkList(FileChannelIm
> > pl.java
> > > :1176)
> > > at
> > >
> > sun.nio.ch.FileChannelImpl$SharedFileLockTable.add(FileChannelImpl.jav
> > a:107
> > > 8)
> > > at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:878)
> > > at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
> > > at
> > > org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:236)
> > > at org.apache.lucene.store.Lock.obtain(Lock.java:72)
> > > at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1041)
> > > at
> > > org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:864)
> > >
> > > I suppose this means that the OS itself hasn't released the lock
> > > even
> > after I shut
> > > down my application server and restarted it.
> > > Am I right?
> > >
> > > Or is there something else that can possibly be the culprit (in
> > > anyone's
> > > experience) that I can investigate?
> > >
> > > - Pulkit
> > >
> > > On Wed, Nov 10, 2010 at 12:57 PM, Uwe Schindler <uwe [at] thetaphi>
> wrote:
> > >
> > > > This is because Lucene uses Native Filesystem Locks. The lock file
> > > > itself is just a placeholder which is not cleaned up on Ctrl-C.
> > > > The lock is not the file itself, its *on* the file.
> > > >
> > > > -----
> > > > Uwe Schindler
> > > > H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de
> > > > eMail: uwe [at] thetaphi
> > > >
> > > > > -----Original Message-----
> > > > > From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> > > > > Sent: Wednesday, November 10, 2010 3:38 PM
> > > > > To: java-user [at] lucene
> > > > > Subject: IndexWriters and write locks
> > > > >
> > > > > Hello,
> > > > >
> > > > > 1) On Windows, I often shut down my application server (which
> > > > > has active IndexWriters open) using the ctrl+c keys.
> > > > > 2) I inspect my directories on the file system I see that the
> > > > > write.lock
> > > > file is still
> > > > > there.
> > > > > 3) I start the app server again, and do some operations that
> > > > > would
> > > > require
> > > > > IndexWriters to write to the same directories again and it works!
> > > > >
> > > > > I don't understand why I do not run into any exceptions?
> > > > > I mean there is already a lock file present which should prevent
> > > > > the IndexWriters from getting access to the directories ... no?
> > > > > I should be happy but I'm not because other folks are able to
> > > > > get
> > > > exceptions
> > > > > when they bounce their servers an I'm unable to reproduce the
> > > > > problem and
> > > > I
> > > > > can't help them.
> > > > >
> > > > > Any clues? Anyone?
> > > > >
> > > > > Thank You,
> > > > > - Pulkit
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --- To unsubscribe, e-mail:
> > > > java-user-unsubscribe [at] lucene
> > > > For additional commands, e-mail: java-user-help [at] lucene
> > > >
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > For additional commands, e-mail: java-user-help [at] lucene
> >
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene


pulkitsinghal at gmail

Nov 10, 2010, 12:13 PM

Post #10 of 11 (1987 views)
Permalink
Re: IndexWriters and write locks [In reply to]

Ah exactly the kind of wake-up call that I was looking for! Thank You :)

On Wed, Nov 10, 2010 at 3:01 PM, Steven A Rowe <sarowe [at] syr> wrote:

> NFS[1] != NTFS[2]
>
> [1] NFS: <http://en.wikipedia.org/wiki/Network_File_System_%28protocol%29>
> [2] NTFS: <http://en.wikipedia.org/wiki/NTFS>
>
> > -----Original Message-----
> > From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> > Sent: Wednesday, November 10, 2010 2:55 PM
> > To: java-user [at] lucene
> > Subject: Re: IndexWriters and write locks
> >
> > You know that really confuses me. I've heard that stated a few times and
> > every time I just felt that it couldn't possibly be right. Maybe it was
> > meant in some very specific manner because otherwise aren't all Windows
> > OSs
> > off-limits to Lucene then?
> >
> > On Wed, Nov 10, 2010 at 2:40 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
> >
> > > Are you using NFS as filesystem? NFS is incompatible to lucene :-)
> > >
> > > -----
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > http://www.thetaphi.de
> > > eMail: uwe [at] thetaphi
> > >
> > >
> > > > -----Original Message-----
> > > > From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> > > > Sent: Wednesday, November 10, 2010 7:57 PM
> > > > To: java-user [at] lucene
> > > > Subject: Re: IndexWriters and write locks
> > > >
> > > > Thanks Uwe, that helps explain why the lock file is still there.
> > > >
> > > > The last piece of the puzzle is why someone may see exceptions such
> as
> > > the
> > > > following from time to time:
> > > >
> > > > java.nio.channels.OverlappingFileLockException
> > > > at
> > > >
> > >
> >
> sun.nio.ch.FileChannelImpl$SharedFileLockTable.checkList(FileChannelImpl.j
> > ava
> > > > :1176)
> > > > at
> > > >
> > >
> >
> sun.nio.ch.FileChannelImpl$SharedFileLockTable.add(FileChannelImpl.java:10
> > 7
> > > > 8)
> > > > at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:878)
> > > > at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
> > > > at
> > > >
> > org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:236)
> > > > at org.apache.lucene.store.Lock.obtain(Lock.java:72)
> > > > at
> org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1041)
> > > > at
> > org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:864)
> > > >
> > > > I suppose this means that the OS itself hasn't released the lock even
> > > after I shut
> > > > down my application server and restarted it.
> > > > Am I right?
> > > >
> > > > Or is there something else that can possibly be the culprit (in
> > anyone's
> > > > experience) that I can investigate?
> > > >
> > > > - Pulkit
> > > >
> > > > On Wed, Nov 10, 2010 at 12:57 PM, Uwe Schindler <uwe [at] thetaphi>
> > wrote:
> > > >
> > > > > This is because Lucene uses Native Filesystem Locks. The lock file
> > > > > itself is just a placeholder which is not cleaned up on Ctrl-C. The
> > > > > lock is not the file itself, its *on* the file.
> > > > >
> > > > > -----
> > > > > Uwe Schindler
> > > > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > > > http://www.thetaphi.de
> > > > > eMail: uwe [at] thetaphi
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Pulkit Singhal [mailto:pulkitsinghal [at] gmail]
> > > > > > Sent: Wednesday, November 10, 2010 3:38 PM
> > > > > > To: java-user [at] lucene
> > > > > > Subject: IndexWriters and write locks
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > 1) On Windows, I often shut down my application server (which has
> > > > > > active IndexWriters open) using the ctrl+c keys.
> > > > > > 2) I inspect my directories on the file system I see that the
> > > > > > write.lock
> > > > > file is still
> > > > > > there.
> > > > > > 3) I start the app server again, and do some operations that
> would
> > > > > require
> > > > > > IndexWriters to write to the same directories again and it works!
> > > > > >
> > > > > > I don't understand why I do not run into any exceptions?
> > > > > > I mean there is already a lock file present which should prevent
> > the
> > > > > > IndexWriters from getting access to the directories ... no?
> > > > > > I should be happy but I'm not because other folks are able to get
> > > > > exceptions
> > > > > > when they bounce their servers an I'm unable to reproduce the
> > > > > > problem and
> > > > > I
> > > > > > can't help them.
> > > > > >
> > > > > > Any clues? Anyone?
> > > > > >
> > > > > > Thank You,
> > > > > > - Pulkit
> > > > >
> > > > >
> > > > >
> --------------------------------------------------------------------
> > -
> > > > > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > > > > For additional commands, e-mail: java-user-help [at] lucene
> > > > >
> > > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > > For additional commands, e-mail: java-user-help [at] lucene
> > >
> > >
>


lucene at mikemccandless

Nov 11, 2010, 10:43 AM

Post #11 of 11 (1968 views)
Permalink
Re: IndexWriters and write locks [In reply to]

On Wed, Nov 10, 2010 at 2:40 PM, Uwe Schindler <uwe [at] thetaphi> wrote:
> Are you using NFS as filesystem? NFS is incompatible to lucene :-)

Actually Lucene can work over NFS, except, it's not a well tested combination.

However, because NFS does not protect open files from deletion, you
must use an app-specific deletion policy to protect commit points
still in use.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Lucene java-user RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.