Mailing List Archive: Lucene: Java-Dev

[jira] Commented: (LUCENE-771) Change default write lock file location to index directory (not java.io.tmpdir)

Index | Next | Previous | View Threaded

jira at apache

Jan 12, 2007, 12:49 PM

Post #1 of 5 (498 views)
Permalink

[jira] Commented: (LUCENE-771) Change default write lock file location to index directory (not java.io.tmpdir)

[ https://issues.apache.org/jira/browse/LUCENE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464350 ]

Yonik Seeley commented on LUCENE-771:
-------------------------------------

Sounds good, I agree with all the changes you outlined.

> Change default write lock file location to index directory (not java.io.tmpdir)
> -------------------------------------------------------------------------------
>
> Key: LUCENE-771
> URL: https://issues.apache.org/jira/browse/LUCENE-771
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 2.1
> Reporter: Michael McCandless
> Assigned To: Michael McCandless
> Priority: Minor
> Fix For: 2.1
>
>
> Now that readers are read-only, we no longer need to store lock files
> in a different global lock directory than the index directory. This
> has been a source of confusion and caused problems to users in the
> past.
> Furthermore, once the write lock is stored in the index directory, it
> no longer needs the big digest prefix that was previously required
> to make sure lock files in the global lock directory, from different
> indexes, did not conflict.
> This way, all files related to an index will appear in a single
> directory. And you can easily list that directory to see if a
> "write.lock" is present to check whether a writer is open on the
> index.
> Note that this change just affects how FSDirectory creates its default
> lockFactory if no lockFactory was specified. It is still possible
> (just no longer the default) to pick a different directory to store
> your lock files by pre-instantiating your own LockFactory.
> As part of this I would like to remove LOCK_DIR and the no-argument
> constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
> think we should have the notion of a global default lock directory
> anymore. This is actually an API change. However, neither
> SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
> so I think this API removal is allowed?
> Finally I want to deprecate (but not yet remove, because this has been
> in the API for many releases) the static LOCK_DIR that's in
> FSDirectory. But it's now entirely unused.
> See here for discussion leading to this:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43940

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

jira at apache

Jan 12, 2007, 4:44 PM

Post #2 of 5 (433 views)
Permalink

[jira] Commented: (LUCENE-771) Change default write lock file location to index directory (not java.io.tmpdir) [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464385 ]

Doron Cohen commented on LUCENE-771:
------------------------------------

I have a question on this change - though I didn't look at the code yet - we don't support backwards compatability with regard to existing lock files, right? Specifically, if there exist an older version index with locks in tmp dir, and the Lucene lib is replaced for one client at a time, while other clients are running, there would be a problem, possible corruption. So who ever is migrating from an index before this change to an index with this change should make sure that all clients are upgraded at the time. At least all clients that may attempt to write.

> Change default write lock file location to index directory (not java.io.tmpdir)
> -------------------------------------------------------------------------------
>
> Key: LUCENE-771
> URL: https://issues.apache.org/jira/browse/LUCENE-771
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 2.1
> Reporter: Michael McCandless
> Assigned To: Michael McCandless
> Priority: Minor
> Fix For: 2.1
>
>
> Now that readers are read-only, we no longer need to store lock files
> in a different global lock directory than the index directory. This
> has been a source of confusion and caused problems to users in the
> past.
> Furthermore, once the write lock is stored in the index directory, it
> no longer needs the big digest prefix that was previously required
> to make sure lock files in the global lock directory, from different
> indexes, did not conflict.
> This way, all files related to an index will appear in a single
> directory. And you can easily list that directory to see if a
> "write.lock" is present to check whether a writer is open on the
> index.
> Note that this change just affects how FSDirectory creates its default
> lockFactory if no lockFactory was specified. It is still possible
> (just no longer the default) to pick a different directory to store
> your lock files by pre-instantiating your own LockFactory.
> As part of this I would like to remove LOCK_DIR and the no-argument
> constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
> think we should have the notion of a global default lock directory
> anymore. This is actually an API change. However, neither
> SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
> so I think this API removal is allowed?
> Finally I want to deprecate (but not yet remove, because this has been
> in the API for many releases) the static LOCK_DIR that's in
> FSDirectory. But it's now entirely unused.
> See here for discussion leading to this:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43940

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

jira at apache

Jan 12, 2007, 6:01 PM

Post #3 of 5 (436 views)
Permalink

[jira] Commented: (LUCENE-771) Change default write lock file location to index directory (not java.io.tmpdir) [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464393 ]

Michael McCandless commented on LUCENE-771:
-------------------------------------------

Yes, that is true.

But there are also quite a few other changes in 2.1 (eg the file format changes) that would also generally necessitate that all places that will share the given index, upgrade to 2.1 at once.

> Change default write lock file location to index directory (not java.io.tmpdir)
> -------------------------------------------------------------------------------
>
> Key: LUCENE-771
> URL: https://issues.apache.org/jira/browse/LUCENE-771
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 2.1
> Reporter: Michael McCandless
> Assigned To: Michael McCandless
> Priority: Minor
> Fix For: 2.1
>
>
> Now that readers are read-only, we no longer need to store lock files
> in a different global lock directory than the index directory. This
> has been a source of confusion and caused problems to users in the
> past.
> Furthermore, once the write lock is stored in the index directory, it
> no longer needs the big digest prefix that was previously required
> to make sure lock files in the global lock directory, from different
> indexes, did not conflict.
> This way, all files related to an index will appear in a single
> directory. And you can easily list that directory to see if a
> "write.lock" is present to check whether a writer is open on the
> index.
> Note that this change just affects how FSDirectory creates its default
> lockFactory if no lockFactory was specified. It is still possible
> (just no longer the default) to pick a different directory to store
> your lock files by pre-instantiating your own LockFactory.
> As part of this I would like to remove LOCK_DIR and the no-argument
> constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
> think we should have the notion of a global default lock directory
> anymore. This is actually an API change. However, neither
> SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
> so I think this API removal is allowed?
> Finally I want to deprecate (but not yet remove, because this has been
> in the API for many releases) the static LOCK_DIR that's in
> FSDirectory. But it's now entirely unused.
> See here for discussion leading to this:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43940

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

jira at apache

Jan 12, 2007, 6:08 PM

Post #4 of 5 (434 views)
Permalink

[jira] Commented: (LUCENE-771) Change default write lock file location to index directory (not java.io.tmpdir) [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464395 ]

Doron Cohen commented on LUCENE-771:
------------------------------------

Is that true? I thought that for previous format changes, the combination of - { (1) point-in-time index reading by readers (2) backwards compatibility (3) locks } - allowed not to require this.

> Change default write lock file location to index directory (not java.io.tmpdir)
> -------------------------------------------------------------------------------
>
> Key: LUCENE-771
> URL: https://issues.apache.org/jira/browse/LUCENE-771
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 2.1
> Reporter: Michael McCandless
> Assigned To: Michael McCandless
> Priority: Minor
> Fix For: 2.1
>
>
> Now that readers are read-only, we no longer need to store lock files
> in a different global lock directory than the index directory. This
> has been a source of confusion and caused problems to users in the
> past.
> Furthermore, once the write lock is stored in the index directory, it
> no longer needs the big digest prefix that was previously required
> to make sure lock files in the global lock directory, from different
> indexes, did not conflict.
> This way, all files related to an index will appear in a single
> directory. And you can easily list that directory to see if a
> "write.lock" is present to check whether a writer is open on the
> index.
> Note that this change just affects how FSDirectory creates its default
> lockFactory if no lockFactory was specified. It is still possible
> (just no longer the default) to pick a different directory to store
> your lock files by pre-instantiating your own LockFactory.
> As part of this I would like to remove LOCK_DIR and the no-argument
> constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
> think we should have the notion of a global default lock directory
> anymore. This is actually an API change. However, neither
> SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
> so I think this API removal is allowed?
> Finally I want to deprecate (but not yet remove, because this has been
> in the API for many releases) the static LOCK_DIR that's in
> FSDirectory. But it's now entirely unused.
> See here for discussion leading to this:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43940

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

jira at apache

Jan 12, 2007, 7:36 PM

Post #5 of 5 (437 views)
Permalink

[jira] Commented: (LUCENE-771) Change default write lock file location to index directory (not java.io.tmpdir) [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12464409 ]

Michael McCandless commented on LUCENE-771:
-------------------------------------------

You're right, backwards compatibility will allow a 2.1 client to
read/write to an older index, but as soon as a 2.1 client writes to
the index then all pre-2.1 readers won't be able to read it. You're
right that if these readers are not reopened then "point in time"
would allow them to keep running pre-2.1.

So I think you could individually update each reader to 2.1, but then
(after readers are updated), all writers would need to go to 2.1 at
once, to make sure they all "agree" that the write lock is now in the
index.

> Change default write lock file location to index directory (not java.io.tmpdir)
> -------------------------------------------------------------------------------
>
> Key: LUCENE-771
> URL: https://issues.apache.org/jira/browse/LUCENE-771
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: 2.1
> Reporter: Michael McCandless
> Assigned To: Michael McCandless
> Priority: Minor
> Fix For: 2.1
>
>
> Now that readers are read-only, we no longer need to store lock files
> in a different global lock directory than the index directory. This
> has been a source of confusion and caused problems to users in the
> past.
> Furthermore, once the write lock is stored in the index directory, it
> no longer needs the big digest prefix that was previously required
> to make sure lock files in the global lock directory, from different
> indexes, did not conflict.
> This way, all files related to an index will appear in a single
> directory. And you can easily list that directory to see if a
> "write.lock" is present to check whether a writer is open on the
> index.
> Note that this change just affects how FSDirectory creates its default
> lockFactory if no lockFactory was specified. It is still possible
> (just no longer the default) to pick a different directory to store
> your lock files by pre-instantiating your own LockFactory.
> As part of this I would like to remove LOCK_DIR and the no-argument
> constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
> think we should have the notion of a global default lock directory
> anymore. This is actually an API change. However, neither
> SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
> so I think this API removal is allowed?
> Finally I want to deprecate (but not yet remove, because this has been
> in the API for many releases) the static LOCK_DIR that's in
> FSDirectory. But it's now entirely unused.
> See here for discussion leading to this:
> http://www.gossamer-threads.com/lists/lucene/java-dev/43940

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com