Mailing List Archive: Lucene: Java-Dev

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher

Index | Next | Previous | View Threaded

jira at apache

Nov 6, 2009, 3:44 PM

Post #1 of 10 (412 views)
Permalink

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774492#action_12774492 ]

Simon Willnauer commented on LUCENE-2041:
-----------------------------------------

Hey Joey,
good to see you coming up with patches, nice that you catch up on that TODO. You added support for parallel DocFreq which is good! Did you think about using an interface from the new concurrent utils instead of subclassing thread? I would like to see people using the new features which make code slightly more readable too. You would be able to simply throw the exception in the Callable#call() method and retrieve it once it has terminated.
you could also use a Pooled Executor to run those threads so you don't have to spawn them each time you call docFreq on the multi searcher.

one more thing, your patch is hard to read as you formated the source code. please try to do not format it - that would make it way easier to read and see what has been added / changed.

> Complete parallelizaton of ParallelMultiSearcher
> ------------------------------------------------
>
> Key: LUCENE-2041
> URL: https://issues.apache.org/jira/browse/LUCENE-2041
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.9
> Reporter: Joey Surls
> Priority: Minor
> Fix For: 3.0
>
> Attachments: LUCENE-2041.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> ParallelMultiSearcher is parallel only for the method signatures of 'search'.
> Part of a query process calls the method docFreq(). There was a TODO comment to parallelize this. Parallelizing this method actually increases the performance of a query on multiple indexes, especially remotely.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Nov 6, 2009, 3:54 PM

Post #2 of 10 (400 views)
Permalink

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774498#action_12774498 ]

Joey Surls commented on LUCENE-2041:
------------------------------------

I agree with you on the Pooled Executor and concurrent utils, I was actually trying to copy the semantics of the search methods in place so someone could quickly grasp the code.

Ya, I thought that's why the patch was full was from the formatted code. Sorry about that. I'll look for that in the future. I noticed the link in the wiki for the formatting under Helpful Resources doesn't work anymore.

http://wiki.apache.org/lucene-java/HowToContribute

> Complete parallelizaton of ParallelMultiSearcher
> ------------------------------------------------
>
> Key: LUCENE-2041
> URL: https://issues.apache.org/jira/browse/LUCENE-2041
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.9
> Reporter: Joey Surls
> Assignee: Simon Willnauer
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2041.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> ParallelMultiSearcher is parallel only for the method signatures of 'search'.
> Part of a query process calls the method docFreq(). There was a TODO comment to parallelize this. Parallelizing this method actually increases the performance of a query on multiple indexes, especially remotely.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Nov 6, 2009, 4:09 PM

Post #3 of 10 (400 views)
Permalink

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774501#action_12774501 ]

Uwe Schindler commented on LUCENE-2041:
---------------------------------------

bq. Ya, I thought that's why the patch was full was from the formatted code. Sorry about that. I'll look for that in the future. I noticed the link in the wiki for the formatting under Helpful Resources doesn't work anymore.

Fixed that and uploaded a new Eclipse Galieo style also containing our way for Java 5 Generics.

> Complete parallelizaton of ParallelMultiSearcher
> ------------------------------------------------
>
> Key: LUCENE-2041
> URL: https://issues.apache.org/jira/browse/LUCENE-2041
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.9
> Reporter: Joey Surls
> Assignee: Simon Willnauer
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2041.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> ParallelMultiSearcher is parallel only for the method signatures of 'search'.
> Part of a query process calls the method docFreq(). There was a TODO comment to parallelize this. Parallelizing this method actually increases the performance of a query on multiple indexes, especially remotely.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Nov 9, 2009, 6:00 AM

Post #4 of 10 (375 views)
Permalink

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774974#action_12774974 ]

Simon Willnauer commented on LUCENE-2041:
-----------------------------------------

thanks uwe!

> Complete parallelizaton of ParallelMultiSearcher
> ------------------------------------------------
>
> Key: LUCENE-2041
> URL: https://issues.apache.org/jira/browse/LUCENE-2041
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.9
> Reporter: Joey Surls
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2041.patch, LUCENE_2041.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> ParallelMultiSearcher is parallel only for the method signatures of 'search'.
> Part of a query process calls the method docFreq(). There was a TODO comment to parallelize this. Parallelizing this method actually increases the performance of a query on multiple indexes, especially remotely.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Nov 9, 2009, 6:08 AM

Post #5 of 10 (374 views)
Permalink

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774976#action_12774976 ]

Uwe Schindler commented on LUCENE-2041:
---------------------------------------

String.isEmpty() is Java 6 only, changed this to length() == 0.

> Complete parallelizaton of ParallelMultiSearcher
> ------------------------------------------------
>
> Key: LUCENE-2041
> URL: https://issues.apache.org/jira/browse/LUCENE-2041
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.9
> Reporter: Joey Surls
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2041.patch, LUCENE_2041.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> ParallelMultiSearcher is parallel only for the method signatures of 'search'.
> Part of a query process calls the method docFreq(). There was a TODO comment to parallelize this. Parallelizing this method actually increases the performance of a query on multiple indexes, especially remotely.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Nov 9, 2009, 6:22 AM

Post #6 of 10 (374 views)
Permalink

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774982#action_12774982 ]

Simon Willnauer commented on LUCENE-2041:
-----------------------------------------

hehe - my patch was quicker but you comment appears first :)

> Complete parallelizaton of ParallelMultiSearcher
> ------------------------------------------------
>
> Key: LUCENE-2041
> URL: https://issues.apache.org/jira/browse/LUCENE-2041
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.9
> Reporter: Joey Surls
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2041.patch, LUCENE-2041.patch, LUCENE-2041.patch, LUCENE_2041.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> ParallelMultiSearcher is parallel only for the method signatures of 'search'.
> Part of a query process calls the method docFreq(). There was a TODO comment to parallelize this. Parallelizing this method actually increases the performance of a query on multiple indexes, especially remotely.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Nov 9, 2009, 6:28 AM

Post #7 of 10 (373 views)
Permalink

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774983#action_12774983 ]

Uwe Schindler commented on LUCENE-2041:
---------------------------------------

I also changed some formatting, we require whitespace after control statements like for, while, catch,... and before {

> Complete parallelizaton of ParallelMultiSearcher
> ------------------------------------------------
>
> Key: LUCENE-2041
> URL: https://issues.apache.org/jira/browse/LUCENE-2041
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.9
> Reporter: Joey Surls
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2041.patch, LUCENE-2041.patch, LUCENE-2041.patch, LUCENE_2041.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> ParallelMultiSearcher is parallel only for the method signatures of 'search'.
> Part of a query process calls the method docFreq(). There was a TODO comment to parallelize this. Parallelizing this method actually increases the performance of a query on multiple indexes, especially remotely.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Nov 9, 2009, 6:30 AM

Post #8 of 10 (373 views)
Permalink

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774987#action_12774987 ]

Uwe Schindler commented on LUCENE-2041:
---------------------------------------

I will also readd support for MaxScore in the fielded case (it is still supported by Searchable.search(SortField))!!!

> Complete parallelizaton of ParallelMultiSearcher
> ------------------------------------------------
>
> Key: LUCENE-2041
> URL: https://issues.apache.org/jira/browse/LUCENE-2041
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.9
> Reporter: Joey Surls
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2041.patch, LUCENE-2041.patch, LUCENE-2041.patch, LUCENE_2041.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> ParallelMultiSearcher is parallel only for the method signatures of 'search'.
> Part of a query process calls the method docFreq(). There was a TODO comment to parallelize this. Parallelizing this method actually increases the performance of a query on multiple indexes, especially remotely.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Nov 10, 2009, 8:19 AM

Post #9 of 10 (344 views)
Permalink

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775927#action_12775927 ]

Uwe Schindler commented on LUCENE-2041:
---------------------------------------

Oh I didn't see your patch. But its not so different, I also did lots of code cleanup.

I also used ReentrantLocks. I added a new DummyLock to utils, that is a replacement for ReentrantLock, but does simply nothing. By this, the synchronized around the HitQueue in MultiSearcher can be ommitted.

> Complete parallelizaton of ParallelMultiSearcher
> ------------------------------------------------
>
> Key: LUCENE-2041
> URL: https://issues.apache.org/jira/browse/LUCENE-2041
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.9
> Reporter: Joey Surls
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.0
>
> Attachments: LUCENE-2041-refactor.patch, LUCENE-2041-refactor.patch, LUCENE-2041.patch, LUCENE-2041.patch, LUCENE-2041.patch, LUCENE-2041.patch, LUCENE-2041.patch, LUCENE_2041.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> ParallelMultiSearcher is parallel only for the method signatures of 'search'.
> Part of a query process calls the method docFreq(). There was a TODO comment to parallelize this. Parallelizing this method actually increases the performance of a query on multiple indexes, especially remotely.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

jira at apache

Nov 10, 2009, 9:13 AM

Post #10 of 10 (344 views)
Permalink

[jira] Commented: (LUCENE-2041) Complete parallelizaton of ParallelMultiSearcher [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775945#action_12775945 ]

Simon Willnauer commented on LUCENE-2041:
-----------------------------------------

+1 go for it - seems to be quite pretty compared to the current trunk version.

:)

> Complete parallelizaton of ParallelMultiSearcher
> ------------------------------------------------
>
> Key: LUCENE-2041
> URL: https://issues.apache.org/jira/browse/LUCENE-2041
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.9
> Reporter: Joey Surls
> Assignee: Uwe Schindler
> Priority: Minor
> Fix For: 3.0
>
> Attachments: LUCENE-2041-final.patch, LUCENE-2041-refactor.patch, LUCENE-2041-refactor.patch, LUCENE-2041.patch, LUCENE-2041.patch, LUCENE-2041.patch, LUCENE-2041.patch, LUCENE-2041.patch, LUCENE_2041.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> ParallelMultiSearcher is parallel only for the method signatures of 'search'.
> Part of a query process calls the method docFreq(). There was a TODO comment to parallelize this. Parallelizing this method actually increases the performance of a query on multiple indexes, especially remotely.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads