Mailing List Archive: Lucene: Java-Dev

[jira] Updated: (LUCENE-1726) IndexWriter.readerPool create new segmentReader outside of sync block

Index | Next | Previous | View Threaded

jira at apache

Jul 2, 2009, 2:55 PM

Post #1 of 8 (341 views)
Permalink

[jira] Updated: (LUCENE-1726) IndexWriter.readerPool create new segmentReader outside of sync block

[ https://issues.apache.org/jira/browse/LUCENE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated LUCENE-1726:
-------------------------------------

Attachment: LUCENE-1726.patch

We don't block accessing readers in the IW.readerPool when a new
segmentReader is being warmed/instantiated. This is important
when new segmentReaders on large new segments are being accessed
for the first time. Otherwise today IW.getReader may wait while
the new SR is being created.

* IW.readerPool map values are now of type MapValue

* We synchronize on the MapValue in methods that access the SR

* synchronize for the entire readerPool.get method is removed.

* All tests pass

> IndexWriter.readerPool create new segmentReader outside of sync block
> ---------------------------------------------------------------------
>
> Key: LUCENE-1726
> URL: https://issues.apache.org/jira/browse/LUCENE-1726
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Priority: Trivial
> Fix For: 2.9
>
> Attachments: LUCENE-1726.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I think we will want to do something like what field cache does
> with CreationPlaceholder for IndexWriter.readerPool. Otherwise
> we have the (I think somewhat problematic) issue of all other
> readerPool.get* methods waiting for an SR to warm.
> It would be good to implement this for 2.9.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

jira at apache

Jul 6, 2009, 8:04 AM

Post #2 of 8 (283 views)
Permalink

[jira] Updated: (LUCENE-1726) IndexWriter.readerPool create new segmentReader outside of sync block [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-1726:
---------------------------------------

Fix Version/s: (was: 2.9)
3.1

> IndexWriter.readerPool create new segmentReader outside of sync block
> ---------------------------------------------------------------------
>
> Key: LUCENE-1726
> URL: https://issues.apache.org/jira/browse/LUCENE-1726
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Assignee: Michael McCandless
> Priority: Trivial
> Fix For: 3.1
>
> Attachments: LUCENE-1726.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I think we will want to do something like what field cache does
> with CreationPlaceholder for IndexWriter.readerPool. Otherwise
> we have the (I think somewhat problematic) issue of all other
> readerPool.get* methods waiting for an SR to warm.
> It would be good to implement this for 2.9.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

jira at apache

Jul 6, 2009, 11:55 AM

Post #3 of 8 (282 views)
Permalink

[jira] Updated: (LUCENE-1726) IndexWriter.readerPool create new segmentReader outside of sync block [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated LUCENE-1726:
-------------------------------------

Attachment: LUCENE-1726.patch

* New SRMapValue is strongly typed

* All tests pass

{quote}I think there is a thread hazard here, in particular a
risk that one thread decrefs a reader just as another thread is
trying to get it, and the reader in fact gets closed while the
other thread has an mv.reader != null and illegally increfs
that. I think you'll have to do the sr.incRef inside the
synchronized(this), but I don't think that entirely resolves
it.{quote}

Are you referring to a decref on a reader outside of IW? The
asserts we have did help in catching synchronization errors.
It's unclear to me how to recreate the use case described such
that it breaks things. We need a test case that fails with the
current patch?

> IndexWriter.readerPool create new segmentReader outside of sync block
> ---------------------------------------------------------------------
>
> Key: LUCENE-1726
> URL: https://issues.apache.org/jira/browse/LUCENE-1726
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Assignee: Michael McCandless
> Priority: Trivial
> Fix For: 3.1
>
> Attachments: LUCENE-1726.patch, LUCENE-1726.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I think we will want to do something like what field cache does
> with CreationPlaceholder for IndexWriter.readerPool. Otherwise
> we have the (I think somewhat problematic) issue of all other
> readerPool.get* methods waiting for an SR to warm.
> It would be good to implement this for 2.9.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

jira at apache

Jul 7, 2009, 5:05 PM

Post #4 of 8 (275 views)
Permalink

[jira] Updated: (LUCENE-1726) IndexWriter.readerPool create new segmentReader outside of sync block [In reply to]

jira at apache

Jul 7, 2009, 5:31 PM

Post #5 of 8 (274 views)
Permalink

[jira] Updated: (LUCENE-1726) IndexWriter.readerPool create new segmentReader outside of sync block [In reply to]

jira at apache

Jul 8, 2009, 9:47 AM

Post #6 of 8 (270 views)
Permalink

[jira] Updated: (LUCENE-1726) IndexWriter.readerPool create new segmentReader outside of sync block [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Rutherglen updated LUCENE-1726:
-------------------------------------

Attachment: LUCENE-1726.trunk.test.patch

I tried the test on trunk and get the same error. They're all
docstore related files so maybe extra doc stores are being
opened?

{code}
[junit] MockRAMDirectory: cannot close: there are still open files: {_s4.fdt=2, _g2.fdx=2, _s4.fdx=2, _g2.tvf=2, _dw.fdx=2, _g2.tvd=2, _g2.tvx=2, _ks.tvf=2, _n9.tvx=2, _ks.tvx=2, _n9.fdx=2, _ks.fdx=2, _dw.cfx=1, _n9.tvf=2, _cp.cfx=1, _s4.tvf=2, _dw.tvx=2, _87.fdx=2, _fr.tvx=2, _87.tvf=2, _fr.tvd=2, _87.fdt=2, _ks.tvd=2, _s4.tvd=2, _dw.tvd=2, _n9.fdt=2, _g2.fdt=2, _87.tvd=2, _fr.fdt=2, _dw.fdt=2, _dj.cfx=1, _s4.tvx=2, _ks.fdt=2, _n9.tvd=2, _fr.tvf=2, _fr.fdx=2, _dw.tvf=2, _87.tvx=2}
[junit] java.lang.RuntimeException: MockRAMDirectory: cannot close: there are still open files: {_s4.fdt=2, _g2.fdx=2, _s4.fdx=2, _g2.tvf=2, _dw.fdx=2, _g2.tvd=2, _g2.tvx=2, _ks.tvf=2, _n9.tvx=2, _ks.tvx=2, _n9.fdx=2, _ks.fdx=2, _dw.cfx=1, _n9.tvf=2, _cp.cfx=1, _s4.tvf=2, _dw.tvx=2, _87.fdx=2, _fr.tvx=2, _87.tvf=2, _fr.tvd=2, _87.fdt=2, _ks.tvd=2, _s4.tvd=2, _dw.tvd=2, _n9.fdt=2, _g2.fdt=2, _87.tvd=2, _fr.fdt=2, _dw.fdt=2, _dj.cfx=1, _s4.tvx=2, _ks.fdt=2, _n9.tvd=2, _fr.tvf=2, _fr.fdx=2, _dw.tvf=2, _87.tvx=2}
[junit] at org.apache.lucene.store.MockRAMDirectory.close(MockRAMDirectory.java:278)
[junit] at org.apache.lucene.index.Test1726.testIndexing(Test1726.java:48)
[junit] at org.apache.lucene.util.LuceneTestCase.runTest(LuceneTestCase.java:88)
{code}

> IndexWriter.readerPool create new segmentReader outside of sync block
> ---------------------------------------------------------------------
>
> Key: LUCENE-1726
> URL: https://issues.apache.org/jira/browse/LUCENE-1726
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Assignee: Michael McCandless
> Priority: Trivial
> Fix For: 3.1
>
> Attachments: LUCENE-1726.patch, LUCENE-1726.patch, LUCENE-1726.patch, LUCENE-1726.patch, LUCENE-1726.trunk.test.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I think we will want to do something like what field cache does
> with CreationPlaceholder for IndexWriter.readerPool. Otherwise
> we have the (I think somewhat problematic) issue of all other
> readerPool.get* methods waiting for an SR to warm.
> It would be good to implement this for 2.9.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

jira at apache

Jul 8, 2009, 1:37 PM

Post #7 of 8 (269 views)
Permalink

[jira] Updated: (LUCENE-1726) IndexWriter.readerPool create new segmentReader outside of sync block [In reply to]

jira at apache

Jul 9, 2009, 4:25 AM

Post #8 of 8 (261 views)
Permalink

[jira] Updated: (LUCENE-1726) IndexWriter.readerPool create new segmentReader outside of sync block [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-1726:
---------------------------------------

Attachment: LUCENE-1726.patch

Attached new patch (the patch is worse than it looks, because many
things moved into the CoreReaders class):

* Moved more stuff into CoreReaders (fieldInfos, dir, segment, etc.)
and moved methods down as well (ctor, openDocStores, decRef).

* Made members final when possible, else synchronized access to
getting them (to avoid running amok of JMM).

{quote}
Right however wouldn't it be somewhat cleaner to sync on core
for all clone operations given we don't want those to occur
(external to IW) at the same time? Ultimately we want core to be
the controller of it's resources rather than the SR being cloned?
{quote}

In fact, I'm not sure why cloning/reopening a segment needs to be
synchronized at all.

Sure it'd be weird for an app to send multiple threads down,
attempting to reopen/clone the same SR or core at once, but from
Lucene's standpoint there's nothing "wrong" with doing so, I think?

(Though, DirectoryReader does need its sync when its transferring the
write lock due to reopen on a reader w/ pending changes).

{quote}
I ran the test with the SRMapValue sync code, (4 threads) with
the sync on SR.core in openDocStore for 10 minutes, 2 core
Windows XML laptop Java 6.14 and no errors. Then same with 2
threads for 5 minutes and no errors. I'll keep on running it to
see if we can get an error.
{quote}

You could try inserting a testPoint (see the other testPoints in
IndexWriter) after the SRMapValue is pulled from the hash but before
we check if its reader is null, and then modify the threads in your
test to randomly yield on that testPoint (by subclassing IW)? Ie
"exacerbate" the path that exposes the hazard.

{quote}
I'm still a little confused as to why we're going to see the bug
if readerPool.get is syncing on the SRMapValue. I guess there's
a slight possibility of the error, and perhaps a more randomized
test would produce it.
{quote}

The hazard exists because there's a time when no synchronization is
held. Ie, you retrieve SRMapValue from the hash while sync'd on
ReaderPool. You then leave that sync entirely (this is where hazard
comes in), and next you sync on the SRMapValue. Another thread can
sneak in and close the SRMapValue.reader during the time that no sync
is held.

> IndexWriter.readerPool create new segmentReader outside of sync block
> ---------------------------------------------------------------------
>
> Key: LUCENE-1726
> URL: https://issues.apache.org/jira/browse/LUCENE-1726
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Affects Versions: 2.4.1
> Reporter: Jason Rutherglen
> Assignee: Michael McCandless
> Priority: Trivial
> Fix For: 3.1
>
> Attachments: LUCENE-1726.patch, LUCENE-1726.patch, LUCENE-1726.patch, LUCENE-1726.patch, LUCENE-1726.patch, LUCENE-1726.patch, LUCENE-1726.trunk.test.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I think we will want to do something like what field cache does
> with CreationPlaceholder for IndexWriter.readerPool. Otherwise
> we have the (I think somewhat problematic) issue of all other
> readerPool.get* methods waiting for an SR to warm.
> It would be good to implement this for 2.9.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com