Mailing List Archive: Lucene: Java-Dev

[jira] Commented: (LUCENE-431) RAMInputStream and RAMOutputStream without further buffering

Index | Next | Previous | View Threaded

jira at apache

Mar 27, 2007, 2:48 AM

Post #1 of 6 (1167 views)
Permalink

[jira] Commented: (LUCENE-431) RAMInputStream and RAMOutputStream without further buffering

[ https://issues.apache.org/jira/browse/LUCENE-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12484383 ]

Michael McCandless commented on LUCENE-431:
-------------------------------------------

Michael, I wasn't able to cleanly apply this patch on the current trunk. I get this:

patch -p0 < lucene-431.patch
patching file src/java/org/apache/lucene/store/RAMInputStream.java
Hunk #2 FAILED at 21.
1 out of 2 hunks FAILED -- saving rejects to file src/java/org/apache/lucene/store/RAMInputStream.java.rej
patching file src/java/org/apache/lucene/store/RAMOutputStream.java
Hunk #1 FAILED at 21.
1 out of 3 hunks FAILED -- saving rejects to file src/java/org/apache/lucene/store/RAMOutputStream.java.rej
patching file src/test/org/apache/lucene/store/MockRAMOutputStream.java

I'd like to test this net performance gain with LUCENE-843. I think fixing this plus doing LUCENE-843 should make indexing into a RAMDirectory quite a bit faster.

> RAMInputStream and RAMOutputStream without further buffering
> ------------------------------------------------------------
>
> Key: LUCENE-431
> URL: https://issues.apache.org/jira/browse/LUCENE-431
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: CVS Nightly - Specify date in submission
> Environment: Operating System: other
> Platform: Other
> Reporter: Paul Elschot
> Assigned To: Michael Busch
> Priority: Minor
> Attachments: lucene-431.patch, RAMInputStream.java
>
>
> From java-dev, Doug's reply of 12 Sep 2005
> on Delaying buffer allocation in BufferedIndexInput:
>
> Paul Elschot wrote:
> ...
> > I noticed that RAMIndexInput extends BufferedIndexInput.
> > It has all data in buffers already, so why is there another
> > layer of buffering?
>
> No good reason: it's historical.
>
> To avoid this either: (a) the BufferedIndexInput API would need to be
> modified to permit subclasses to supply the buffer; or (b)
> RAMInputStream could subclass IndexInput directly, using its own
> buffers. The latter would probably be simpler.
>
> End of quote.
>
> I made version (b) of RAMInputStream.
> Using this RAMInputStream, TestTermVectorsReader failed as the only
> failing test.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Mar 27, 2007, 11:35 AM

Post #2 of 6 (1100 views)
Permalink

[jira] Commented: (LUCENE-431) RAMInputStream and RAMOutputStream without further buffering [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12484535 ]

Doug Cutting commented on LUCENE-431:
-------------------------------------

> I'd like to test this net performance gain with LUCENE-843.

Yes, it would be great to see how much each improves things individually as well as combined.

> RAMInputStream and RAMOutputStream without further buffering
> ------------------------------------------------------------
>
> Key: LUCENE-431
> URL: https://issues.apache.org/jira/browse/LUCENE-431
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: CVS Nightly - Specify date in submission
> Environment: Operating System: other
> Platform: Other
> Reporter: Paul Elschot
> Assigned To: Michael Busch
> Priority: Minor
> Attachments: lucene-431.patch, lucene-431.zip, RAMInputStream.java
>
>
> From java-dev, Doug's reply of 12 Sep 2005
> on Delaying buffer allocation in BufferedIndexInput:
>
> Paul Elschot wrote:
> ...
> > I noticed that RAMIndexInput extends BufferedIndexInput.
> > It has all data in buffers already, so why is there another
> > layer of buffering?
>
> No good reason: it's historical.
>
> To avoid this either: (a) the BufferedIndexInput API would need to be
> modified to permit subclasses to supply the buffer; or (b)
> RAMInputStream could subclass IndexInput directly, using its own
> buffers. The latter would probably be simpler.
>
> End of quote.
>
> I made version (b) of RAMInputStream.
> Using this RAMInputStream, TestTermVectorsReader failed as the only
> failing test.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Mar 27, 2007, 11:55 AM

Post #3 of 6 (1103 views)
Permalink

[jira] Commented: (LUCENE-431) RAMInputStream and RAMOutputStream without further buffering [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12484542 ]

Michael McCandless commented on LUCENE-431:
-------------------------------------------

>> I'd like to test this net performance gain with LUCENE-843.
>
> Yes, it would be great to see how much each improves things individually as well as combined.

Will do!

> RAMInputStream and RAMOutputStream without further buffering
> ------------------------------------------------------------
>
> Key: LUCENE-431
> URL: https://issues.apache.org/jira/browse/LUCENE-431
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: CVS Nightly - Specify date in submission
> Environment: Operating System: other
> Platform: Other
> Reporter: Paul Elschot
> Assigned To: Michael Busch
> Priority: Minor
> Attachments: lucene-431.patch, lucene-431.zip, RAMInputStream.java
>
>
> From java-dev, Doug's reply of 12 Sep 2005
> on Delaying buffer allocation in BufferedIndexInput:
>
> Paul Elschot wrote:
> ...
> > I noticed that RAMIndexInput extends BufferedIndexInput.
> > It has all data in buffers already, so why is there another
> > layer of buffering?
>
> No good reason: it's historical.
>
> To avoid this either: (a) the BufferedIndexInput API would need to be
> modified to permit subclasses to supply the buffer; or (b)
> RAMInputStream could subclass IndexInput directly, using its own
> buffers. The latter would probably be simpler.
>
> End of quote.
>
> I made version (b) of RAMInputStream.
> Using this RAMInputStream, TestTermVectorsReader failed as the only
> failing test.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Mar 27, 2007, 11:57 AM

Post #4 of 6 (1105 views)
Permalink

[jira] Commented: (LUCENE-431) RAMInputStream and RAMOutputStream without further buffering [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12484546 ]

Michael McCandless commented on LUCENE-431:
-------------------------------------------

Michael, the patch problem seems to be something on my end, which I can't yet explain.

When I take your zip (thanks!), unzip into a fresh trunk checkout, run 'svn diff', take the output to another fresh trunk checkout, and try to apply that patch, I get the same error. Somehow my version of patch (2.5.4 on Debian) cannot handle the output of 'svn diff'. Spooky!

> RAMInputStream and RAMOutputStream without further buffering
> ------------------------------------------------------------
>
> Key: LUCENE-431
> URL: https://issues.apache.org/jira/browse/LUCENE-431
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: CVS Nightly - Specify date in submission
> Environment: Operating System: other
> Platform: Other
> Reporter: Paul Elschot
> Assigned To: Michael Busch
> Priority: Minor
> Attachments: lucene-431.patch, lucene-431.zip, RAMInputStream.java
>
>
> From java-dev, Doug's reply of 12 Sep 2005
> on Delaying buffer allocation in BufferedIndexInput:
>
> Paul Elschot wrote:
> ...
> > I noticed that RAMIndexInput extends BufferedIndexInput.
> > It has all data in buffers already, so why is there another
> > layer of buffering?
>
> No good reason: it's historical.
>
> To avoid this either: (a) the BufferedIndexInput API would need to be
> modified to permit subclasses to supply the buffer; or (b)
> RAMInputStream could subclass IndexInput directly, using its own
> buffers. The latter would probably be simpler.
>
> End of quote.
>
> I made version (b) of RAMInputStream.
> Using this RAMInputStream, TestTermVectorsReader failed as the only
> failing test.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Mar 27, 2007, 3:41 PM

Post #5 of 6 (1102 views)
Permalink

[jira] Commented: (LUCENE-431) RAMInputStream and RAMOutputStream without further buffering [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12484632 ]

Joe Shaw commented on LUCENE-431:
---------------------------------

Michael: mysterious patch failures like that are usually caused by problems with line endings. Try running dos2unix on the patch and then apply it.

> RAMInputStream and RAMOutputStream without further buffering
> ------------------------------------------------------------
>
> Key: LUCENE-431
> URL: https://issues.apache.org/jira/browse/LUCENE-431
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: CVS Nightly - Specify date in submission
> Environment: Operating System: other
> Platform: Other
> Reporter: Paul Elschot
> Assigned To: Michael Busch
> Priority: Minor
> Attachments: lucene-431.patch, lucene-431.zip, RAMInputStream.java
>
>
> From java-dev, Doug's reply of 12 Sep 2005
> on Delaying buffer allocation in BufferedIndexInput:
>
> Paul Elschot wrote:
> ...
> > I noticed that RAMIndexInput extends BufferedIndexInput.
> > It has all data in buffers already, so why is there another
> > layer of buffering?
>
> No good reason: it's historical.
>
> To avoid this either: (a) the BufferedIndexInput API would need to be
> modified to permit subclasses to supply the buffer; or (b)
> RAMInputStream could subclass IndexInput directly, using its own
> buffers. The latter would probably be simpler.
>
> End of quote.
>
> I made version (b) of RAMInputStream.
> Using this RAMInputStream, TestTermVectorsReader failed as the only
> failing test.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Mar 27, 2007, 3:45 PM

Post #6 of 6 (1095 views)
Permalink

[jira] Commented: (LUCENE-431) RAMInputStream and RAMOutputStream without further buffering [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12484634 ]

Michael McCandless commented on LUCENE-431:
-------------------------------------------

Thanks for the advice :) Alas, I had already tried that on the original patch and it gives the same error. I remain baffled!

> RAMInputStream and RAMOutputStream without further buffering
> ------------------------------------------------------------
>
> Key: LUCENE-431
> URL: https://issues.apache.org/jira/browse/LUCENE-431
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Store
> Affects Versions: CVS Nightly - Specify date in submission
> Environment: Operating System: other
> Platform: Other
> Reporter: Paul Elschot
> Assigned To: Michael Busch
> Priority: Minor
> Attachments: lucene-431.patch, lucene-431.zip, RAMInputStream.java
>
>
> From java-dev, Doug's reply of 12 Sep 2005
> on Delaying buffer allocation in BufferedIndexInput:
>
> Paul Elschot wrote:
> ...
> > I noticed that RAMIndexInput extends BufferedIndexInput.
> > It has all data in buffers already, so why is there another
> > layer of buffering?
>
> No good reason: it's historical.
>
> To avoid this either: (a) the BufferedIndexInput API would need to be
> modified to permit subclasses to supply the buffer; or (b)
> RAMInputStream could subclass IndexInput directly, using its own
> buffers. The latter would probably be simpler.
>
> End of quote.
>
> I made version (b) of RAMInputStream.
> Using this RAMInputStream, TestTermVectorsReader failed as the only
> failing test.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads