Mailing List Archive: Lucene: Java-Dev

[jira] Commented: (LUCENE-1997) Explore performance of multi-PQ vs single-PQ sorting API

1 2 3 4 5

View All

Index | Next | Previous | View Threaded

markrmiller at gmail

Nov 3, 2009, 12:32 PM

Post #101 of 110 (157 views)
Permalink

Re: [jira] Commented: (LUCENE-1997) Explore performance of multi-PQ vs single-PQ sorting API [In reply to]

Your obviously too stuck on single examples. We have to consider
everyone in lucene land.

I'm against 2 Apis. A custom search is advanced - it's not worth the
baggage of maintaining two APIs or be limited by the APIs and back
compat when moving forward.

If the advantage of the second API is just going to be it's simpler,
I'm not for it currently.

- Mark

http://www.lucidimagination.com (mobile)

On Nov 3, 2009, at 10:51 AM, "Jake Mannix (JIRA)" <jira[at]apache.org>
wrote:

>
> [ https://issues.apache.org/jira/browse/LUCENE-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773114#action_12773114
> ]
>
> Jake Mannix commented on LUCENE-1997:
> -------------------------------------
>
> bq. Since each approach has distinct advantages, why not offer both
> ("simple" and "expert") comparator extensions APIs?
>
> +1 from me on this one, as long as the simpler one is around. I'll
> bet we'll find that we regret keeping the "expert" one by 3.2 or so
> though, but I'll take any compromise which gets the simpler API in
> there.
>
> bq. Don't forget that this is multiplied by however many queries are
> currently in flight.
>
> Sure, so if you're running with 100 queries per second on a single
> shard (pretty fast!), with 100 segments, and you want to do sorting
> by value on the top 1000 values (how far down the long tail of
> extreme cases are we at now? Do librarians hit their search servers
> with 100 QPS and have indices poorly built with hundreds of segments
> and can't take downtime to *ever* optimize?), we're now talking
> about 40MB.
>
> *Forty megabytes*. On a beefy machine which is supposed to be
> handling 100QPS across an index big enough to need 100 segments.
> How much heap would such a machine already be allocating? 4GB? 6?
> More?
>
> We're talking about less than 1% of the heap is being used by the
> multiPQ approach in comparison to singlePQ.
>
>> Explore performance of multi-PQ vs single-PQ sorting API
>> --------------------------------------------------------
>>
>> Key: LUCENE-1997
>> URL: https://issues.apache.org/jira/browse/LUCENE-1997
>> Project: Lucene - Java
>> Issue Type: Improvement
>> Components: Search
>> Affects Versions: 2.9
>> Reporter: Michael McCandless
>> Assignee: Michael McCandless
>> Attachments: LUCENE-1997.patch, LUCENE-1997.patch,
>> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch,
>> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch,
>> LUCENE-1997.patch
>>
>>
>> Spinoff from recent "lucene 2.9 sorting algorithm" thread on java-
>> dev,
>> where a simpler (non-segment-based) comparator API is proposed that
>> gathers results into multiple PQs (one per segment) and then merges
>> them in the end.
>> I started from John's multi-PQ code and worked it into
>> contrib/benchmark so that we could run perf tests. Then I generified
>> the Python script I use for running search benchmarks (in
>> contrib/benchmark/sortBench.py).
>> The script first creates indexes with 1M docs (based on
>> SortableSingleDocSource, and based on wikipedia, if available). Then
>> it runs various combinations:
>> * Index with 20 balanced segments vs index with the "normal" log
>> segment size
>> * Queries with different numbers of hits (only for wikipedia index)
>> * Different top N
>> * Different sorts (by title, for wikipedia, and by random string,
>> random int, and country for the random index)
>> For each test, 7 search rounds are run and the best QPS is kept. The
>> script runs singlePQ then multiPQ, and records the resulting best QPS
>> for each and produces table (in Jira format) as output.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-dev-help[at]lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

markrmiller at gmail

Nov 3, 2009, 12:40 PM

Post #102 of 110 (157 views)
Permalink

Re: [jira] Commented: (LUCENE-1997) Explore performance of multi-PQ vs single-PQ sorting API [In reply to]

Not *ever* being able to optimize is a common case, without jumping a
lot of hoops. There are many systems that need to be on nearly 24/7 -
an optimize on a large index can take many hours - usually an unknown
number. Linkedin and it's use cases are not the only consumers of
lucene.

- Mark

http://www.lucidimagination.com (mobile)

On Nov 3, 2009, at 10:51 AM, "Jake Mannix (JIRA)" <jira[at]apache.org>
wrote:

>
> [ https://issues.apache.org/jira/browse/LUCENE-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773114#action_12773114
> ]
>
> Jake Mannix commented on LUCENE-1997:
> -------------------------------------
>
> bq. Since each approach has distinct advantages, why not offer both
> ("simple" and "expert") comparator extensions APIs?
>
> +1 from me on this one, as long as the simpler one is around. I'll
> bet we'll find that we regret keeping the "expert" one by 3.2 or so
> though, but I'll take any compromise which gets the simpler API in
> there.
>
> bq. Don't forget that this is multiplied by however many queries are
> currently in flight.
>
> Sure, so if you're running with 100 queries per second on a single
> shard (pretty fast!), with 100 segments, and you want to do sorting
> by value on the top 1000 values (how far down the long tail of
> extreme cases are we at now? Do librarians hit their search servers
> with 100 QPS and have indices poorly built with hundreds of segments
> and can't take downtime to *ever* optimize?), we're now talking
> about 40MB.
>
> *Forty megabytes*. On a beefy machine which is supposed to be
> handling 100QPS across an index big enough to need 100 segments.
> How much heap would such a machine already be allocating? 4GB? 6?
> More?
>
> We're talking about less than 1% of the heap is being used by the
> multiPQ approach in comparison to singlePQ.
>
>> Explore performance of multi-PQ vs single-PQ sorting API
>> --------------------------------------------------------
>>
>> Key: LUCENE-1997
>> URL: https://issues.apache.org/jira/browse/LUCENE-1997
>> Project: Lucene - Java
>> Issue Type: Improvement
>> Components: Search
>> Affects Versions: 2.9
>> Reporter: Michael McCandless
>> Assignee: Michael McCandless
>> Attachments: LUCENE-1997.patch, LUCENE-1997.patch,
>> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch,
>> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch,
>> LUCENE-1997.patch
>>
>>
>> Spinoff from recent "lucene 2.9 sorting algorithm" thread on java-
>> dev,
>> where a simpler (non-segment-based) comparator API is proposed that
>> gathers results into multiple PQs (one per segment) and then merges
>> them in the end.
>> I started from John's multi-PQ code and worked it into
>> contrib/benchmark so that we could run perf tests. Then I generified
>> the Python script I use for running search benchmarks (in
>> contrib/benchmark/sortBench.py).
>> The script first creates indexes with 1M docs (based on
>> SortableSingleDocSource, and based on wikipedia, if available). Then
>> it runs various combinations:
>> * Index with 20 balanced segments vs index with the "normal" log
>> segment size
>> * Queries with different numbers of hits (only for wikipedia index)
>> * Different top N
>> * Different sorts (by title, for wikipedia, and by random string,
>> random int, and country for the random index)
>> For each test, 7 search rounds are run and the best QPS is kept. The
>> script runs singlePQ then multiPQ, and records the resulting best QPS
>> for each and produces table (in Jira format) as output.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-dev-help[at]lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

jake.mannix at gmail

Nov 3, 2009, 12:42 PM

Post #103 of 110 (157 views)
Permalink

Re: [jira] Commented: (LUCENE-1997) Explore performance of multi-PQ vs single-PQ sorting API [In reply to]

Mark, I'm not stuck on single examples, I'm thinking about all of lucene
land: what tiny fraction of people need the combined intersection of

a) many many segments

AND

b) deep paging

AND

c) high QPS

AND

e) can't handle another 40MB of RAM usage.

Only people in the intersection of all of those bitsets would possibly have
a problem with the memory requirements of multiPQ.

On Tue, Nov 3, 2009 at 12:32 PM, Mark Miller <markrmiller[at]gmail.com> wrote:

> Your obviously too stuck on single examples. We have to consider everyone
> in lucene land.
>
> I'm against 2 Apis. A custom search is advanced - it's not worth the
> baggage of maintaining two APIs or be limited by the APIs and back compat
> when moving forward.
>
> If the advantage of the second API is just going to be it's simpler, I'm
> not for it currently.
>
> - Mark
>
> http://www.lucidimagination.com (mobile)
>
> On Nov 3, 2009, at 10:51 AM, "Jake Mannix (JIRA)" <jira[at]apache.org> wrote:
>
>
>> [
>> https://issues.apache.org/jira/browse/LUCENE-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773114#action_12773114
>> ]
>>
>> Jake Mannix commented on LUCENE-1997:
>> -------------------------------------
>>
>> bq. Since each approach has distinct advantages, why not offer both
>> ("simple" and "expert") comparator extensions APIs?
>>
>> +1 from me on this one, as long as the simpler one is around. I'll bet
>> we'll find that we regret keeping the "expert" one by 3.2 or so though, but
>> I'll take any compromise which gets the simpler API in there.
>>
>> bq. Don't forget that this is multiplied by however many queries are
>> currently in flight.
>>
>> Sure, so if you're running with 100 queries per second on a single shard
>> (pretty fast!), with 100 segments, and you want to do sorting by value on
>> the top 1000 values (how far down the long tail of extreme cases are we at
>> now? Do librarians hit their search servers with 100 QPS and have indices
>> poorly built with hundreds of segments and can't take downtime to *ever*
>> optimize?), we're now talking about 40MB.
>>
>> *Forty megabytes*. On a beefy machine which is supposed to be handling
>> 100QPS across an index big enough to need 100 segments. How much heap would
>> such a machine already be allocating? 4GB? 6? More?
>>
>> We're talking about less than 1% of the heap is being used by the multiPQ
>> approach in comparison to singlePQ.
>>
>> Explore performance of multi-PQ vs single-PQ sorting API
>>> --------------------------------------------------------
>>>
>>> Key: LUCENE-1997
>>> URL: https://issues.apache.org/jira/browse/LUCENE-1997
>>> Project: Lucene - Java
>>> Issue Type: Improvement
>>> Components: Search
>>> Affects Versions: 2.9
>>> Reporter: Michael McCandless
>>> Assignee: Michael McCandless
>>> Attachments: LUCENE-1997.patch, LUCENE-1997.patch,
>>> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch,
>>> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch
>>>
>>>
>>> Spinoff from recent "lucene 2.9 sorting algorithm" thread on java-dev,
>>> where a simpler (non-segment-based) comparator API is proposed that
>>> gathers results into multiple PQs (one per segment) and then merges
>>> them in the end.
>>> I started from John's multi-PQ code and worked it into
>>> contrib/benchmark so that we could run perf tests. Then I generified
>>> the Python script I use for running search benchmarks (in
>>> contrib/benchmark/sortBench.py).
>>> The script first creates indexes with 1M docs (based on
>>> SortableSingleDocSource, and based on wikipedia, if available). Then
>>> it runs various combinations:
>>> * Index with 20 balanced segments vs index with the "normal" log
>>> segment size
>>> * Queries with different numbers of hits (only for wikipedia index)
>>> * Different top N
>>> * Different sorts (by title, for wikipedia, and by random string,
>>> random int, and country for the random index)
>>> For each test, 7 search rounds are run and the best QPS is kept. The
>>> script runs singlePQ then multiPQ, and records the resulting best QPS
>>> for each and produces table (in Jira format) as output.
>>>
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
>> For additional commands, e-mail: java-dev-help[at]lucene.apache.org
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-dev-help[at]lucene.apache.org
>
>

jake.mannix at gmail

Nov 3, 2009, 12:52 PM

Post #104 of 110 (157 views)
Permalink

Re: [jira] Commented: (LUCENE-1997) Explore performance of multi-PQ vs single-PQ sorting API [In reply to]

There are really not that many hoops you need to jump through to be able to
periodically optimize down to 10 segments or so. I've used lucene at plenty
of other places before LinkedIn, and rarely (since 2.3's indexing speed blew
through the roof) have I had to worry about setting the merge factor too
high, and even when I do, you simply index into another directory while
you're optimizing the original (which is now kept read-only while keeping an
in-memory delete set). It's not that hard, and it does wonders for your
performance.

Sure, plenty of lucene installations can't optimize, and while many of those
could do with some much-needed refactoring to allow them the possibility of
doing that (otherwise, you get what happened at my old company before I
worked there - there was never any optimize, high merge factor, and commit
after every document [ouch!], and eventually query latency went through the
roof and the system just fell over), I understand that not everyone is going
to do that.

But even in these installations, I'm still saying that you've narrowed the
field down to a very tiny number if you add up all the requirements for
multiPQ to be painful for them (seriously: when is 40MB going to hurt a
system that's designed to handle 100QPS per box? Or when does 4MB hurt one
designed to handle 10QPS?)

-jake

On Tue, Nov 3, 2009 at 12:40 PM, Mark Miller <markrmiller[at]gmail.com> wrote:

> Not *ever* being able to optimize is a common case, without jumping a lot
> of hoops. There are many systems that need to be on nearly 24/7 - an
> optimize on a large index can take many hours - usually an unknown number.
> Linkedin and it's use cases are not the only consumers of lucene.
>
>
> - Mark
>
> http://www.lucidimagination.com (mobile)
>
> On Nov 3, 2009, at 10:51 AM, "Jake Mannix (JIRA)" <jira[at]apache.org> wrote:
>
>
>> [
>> https://issues.apache.org/jira/browse/LUCENE-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773114#action_12773114
>> ]
>>
>> Jake Mannix commented on LUCENE-1997:
>> -------------------------------------
>>
>> bq. Since each approach has distinct advantages, why not offer both
>> ("simple" and "expert") comparator extensions APIs?
>>
>> +1 from me on this one, as long as the simpler one is around. I'll bet
>> we'll find that we regret keeping the "expert" one by 3.2 or so though, but
>> I'll take any compromise which gets the simpler API in there.
>>
>> bq. Don't forget that this is multiplied by however many queries are
>> currently in flight.
>>
>> Sure, so if you're running with 100 queries per second on a single shard
>> (pretty fast!), with 100 segments, and you want to do sorting by value on
>> the top 1000 values (how far down the long tail of extreme cases are we at
>> now? Do librarians hit their search servers with 100 QPS and have indices
>> poorly built with hundreds of segments and can't take downtime to *ever*
>> optimize?), we're now talking about 40MB.
>>
>> *Forty megabytes*. On a beefy machine which is supposed to be handling
>> 100QPS across an index big enough to need 100 segments. How much heap would
>> such a machine already be allocating? 4GB? 6? More?
>>
>> We're talking about less than 1% of the heap is being used by the multiPQ
>> approach in comparison to singlePQ.
>>
>> Explore performance of multi-PQ vs single-PQ sorting API
>>> --------------------------------------------------------
>>>
>>> Key: LUCENE-1997
>>> URL: https://issues.apache.org/jira/browse/LUCENE-1997
>>> Project: Lucene - Java
>>> Issue Type: Improvement
>>> Components: Search
>>> Affects Versions: 2.9
>>> Reporter: Michael McCandless
>>> Assignee: Michael McCandless
>>> Attachments: LUCENE-1997.patch, LUCENE-1997.patch,
>>> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch,
>>> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch
>>>
>>>
>>> Spinoff from recent "lucene 2.9 sorting algorithm" thread on java-dev,
>>> where a simpler (non-segment-based) comparator API is proposed that
>>> gathers results into multiple PQs (one per segment) and then merges
>>> them in the end.
>>> I started from John's multi-PQ code and worked it into
>>> contrib/benchmark so that we could run perf tests. Then I generified
>>> the Python script I use for running search benchmarks (in
>>> contrib/benchmark/sortBench.py).
>>> The script first creates indexes with 1M docs (based on
>>> SortableSingleDocSource, and based on wikipedia, if available). Then
>>> it runs various combinations:
>>> * Index with 20 balanced segments vs index with the "normal" log
>>> segment size
>>> * Queries with different numbers of hits (only for wikipedia index)
>>> * Different top N
>>> * Different sorts (by title, for wikipedia, and by random string,
>>> random int, and country for the random index)
>>> For each test, 7 search rounds are run and the best QPS is kept. The
>>> script runs singlePQ then multiPQ, and records the resulting best QPS
>>> for each and produces table (in Jira format) as output.
>>>
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
>> For additional commands, e-mail: java-dev-help[at]lucene.apache.org
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-dev-help[at]lucene.apache.org
>
>

markrmiller at gmail

Nov 3, 2009, 12:56 PM

Post #105 of 110 (157 views)
Permalink

Re: [jira] Commented: (LUCENE-1997) Explore performance of multi-PQ vs single-PQ sorting API [In reply to]

The memory issue is just one example of something that's somewhat
worse - I don't see it as a deciding faster. If things were clarified
to be decidedly faster with multi queue, and not 50% worse at 1000
hits, I'd be for the change, more memory or not.

- Mark

http://www.lucidimagination.com (mobile)

On Nov 3, 2009, at 12:42 PM, Jake Mannix <jake.mannix[at]gmail.com> wrote:

> Mark, I'm not stuck on single examples, I'm thinking about all of
> lucene land: what tiny fraction of people need the combined
> intersection of
>
> a) many many segments
>
> AND
>
> b) deep paging
>
> AND
>
> c) high QPS
>
> AND
>
> e) can't handle another 40MB of RAM usage.
>
> Only people in the intersection of all of those bitsets would
> possibly have a problem with the memory requirements of multiPQ.
>
> On Tue, Nov 3, 2009 at 12:32 PM, Mark Miller <markrmiller[at]gmail.com>
> wrote:
> Your obviously too stuck on single examples. We have to consider
> everyone in lucene land.
>
> I'm against 2 Apis. A custom search is advanced - it's not worth the
> baggage of maintaining two APIs or be limited by the APIs and back
> compat when moving forward.
>
> If the advantage of the second API is just going to be it's simpler,
> I'm not for it currently.
>
> - Mark
>
> http://www.lucidimagination.com (mobile)
>
> On Nov 3, 2009, at 10:51 AM, "Jake Mannix (JIRA)" <jira[at]apache.org>
> wrote:
>
>
> [ https://issues.apache.org/jira/browse/LUCENE-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773114#action_12773114
> ]
>
> Jake Mannix commented on LUCENE-1997:
> -------------------------------------
>
> bq. Since each approach has distinct advantages, why not offer both
> ("simple" and "expert") comparator extensions APIs?
>
> +1 from me on this one, as long as the simpler one is around. I'll
> bet we'll find that we regret keeping the "expert" one by 3.2 or so
> though, but I'll take any compromise which gets the simpler API in
> there.
>
> bq. Don't forget that this is multiplied by however many queries are
> currently in flight.
>
> Sure, so if you're running with 100 queries per second on a single
> shard (pretty fast!), with 100 segments, and you want to do sorting
> by value on the top 1000 values (how far down the long tail of
> extreme cases are we at now? Do librarians hit their search servers
> with 100 QPS and have indices poorly built with hundreds of segments
> and can't take downtime to *ever* optimize?), we're now talking
> about 40MB.
>
> *Forty megabytes*. On a beefy machine which is supposed to be
> handling 100QPS across an index big enough to need 100 segments.
> How much heap would such a machine already be allocating? 4GB? 6?
> More?
>
> We're talking about less than 1% of the heap is being used by the
> multiPQ approach in comparison to singlePQ.
>
> Explore performance of multi-PQ vs single-PQ sorting API
> --------------------------------------------------------
>
> Key: LUCENE-1997
> URL: https://issues.apache.org/jira/browse/LUCENE-1997
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.9
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Attachments: LUCENE-1997.patch, LUCENE-1997.patch,
> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch,
> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch,
> LUCENE-1997.patch
>
>
> Spinoff from recent "lucene 2.9 sorting algorithm" thread on java-dev,
> where a simpler (non-segment-based) comparator API is proposed that
> gathers results into multiple PQs (one per segment) and then merges
> them in the end.
> I started from John's multi-PQ code and worked it into
> contrib/benchmark so that we could run perf tests. Then I generified
> the Python script I use for running search benchmarks (in
> contrib/benchmark/sortBench.py).
> The script first creates indexes with 1M docs (based on
> SortableSingleDocSource, and based on wikipedia, if available). Then
> it runs various combinations:
> * Index with 20 balanced segments vs index with the "normal" log
> segment size
> * Queries with different numbers of hits (only for wikipedia index)
> * Different top N
> * Different sorts (by title, for wikipedia, and by random string,
> random int, and country for the random index)
> For each test, 7 search rounds are run and the best QPS is kept. The
> script runs singlePQ then multiPQ, and records the resulting best QPS
> for each and produces table (in Jira format) as output.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-dev-help[at]lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-dev-help[at]lucene.apache.org
>
>

markrmiller at gmail

Nov 3, 2009, 12:58 PM

Post #106 of 110 (157 views)
Permalink

Re: [jira] Commented: (LUCENE-1997) Explore performance of multi-PQ vs single-PQ sorting API [In reply to]

I use a low merge factor now too - and recomend it to others. But a
lot of users like to use crazy high merge factors.

I'm not arguing the absolute worst case is common. It almost never is.

- Mark

http://www.lucidimagination.com (mobile)

On Nov 3, 2009, at 12:52 PM, Jake Mannix <jake.mannix[at]gmail.com> wrote:

> There are really not that many hoops you need to jump through to be
> able to periodically optimize down to 10 segments or so. I've used
> lucene at plenty of other places before LinkedIn, and rarely (since
> 2.3's indexing speed blew through the roof) have I had to worry
> about setting the merge factor too high, and even when I do, you
> simply index into another directory while you're optimizing the
> original (which is now kept read-only while keeping an in-memory
> delete set). It's not that hard, and it does wonders for your
> performance.
>
> Sure, plenty of lucene installations can't optimize, and while many
> of those could do with some much-needed refactoring to allow them
> the possibility of doing that (otherwise, you get what happened at
> my old company before I worked there - there was never any optimize,
> high merge factor, and commit after every document [ouch!], and
> eventually query latency went through the roof and the system just
> fell over), I understand that not everyone is going to do that.
>
> But even in these installations, I'm still saying that you've
> narrowed the field down to a very tiny number if you add up all the
> requirements for multiPQ to be painful for them (seriously: when is
> 40MB going to hurt a system that's designed to handle 100QPS per
> box? Or when does 4MB hurt one designed to handle 10QPS?)
>
> -jake
>
> On Tue, Nov 3, 2009 at 12:40 PM, Mark Miller <markrmiller[at]gmail.com>
> wrote:
> Not *ever* being able to optimize is a common case, without jumping
> a lot of hoops. There are many systems that need to be on nearly
> 24/7 - an optimize on a large index can take many hours - usually an
> unknown number. Linkedin and it's use cases are not the only
> consumers of lucene.
>
>
> - Mark
>
> http://www.lucidimagination.com (mobile)
>
> On Nov 3, 2009, at 10:51 AM, "Jake Mannix (JIRA)" <jira[at]apache.org>
> wrote:
>
>
> [ https://issues.apache.org/jira/browse/LUCENE-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773114#action_12773114
> ]
>
> Jake Mannix commented on LUCENE-1997:
> -------------------------------------
>
> bq. Since each approach has distinct advantages, why not offer both
> ("simple" and "expert") comparator extensions APIs?
>
> +1 from me on this one, as long as the simpler one is around. I'll
> bet we'll find that we regret keeping the "expert" one by 3.2 or so
> though, but I'll take any compromise which gets the simpler API in
> there.
>
> bq. Don't forget that this is multiplied by however many queries are
> currently in flight.
>
> Sure, so if you're running with 100 queries per second on a single
> shard (pretty fast!), with 100 segments, and you want to do sorting
> by value on the top 1000 values (how far down the long tail of
> extreme cases are we at now? Do librarians hit their search servers
> with 100 QPS and have indices poorly built with hundreds of segments
> and can't take downtime to *ever* optimize?), we're now talking
> about 40MB.
>
> *Forty megabytes*. On a beefy machine which is supposed to be
> handling 100QPS across an index big enough to need 100 segments.
> How much heap would such a machine already be allocating? 4GB? 6?
> More?
>
> We're talking about less than 1% of the heap is being used by the
> multiPQ approach in comparison to singlePQ.
>
> Explore performance of multi-PQ vs single-PQ sorting API
> --------------------------------------------------------
>
> Key: LUCENE-1997
> URL: https://issues.apache.org/jira/browse/LUCENE-1997
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.9
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Attachments: LUCENE-1997.patch, LUCENE-1997.patch,
> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch,
> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch,
> LUCENE-1997.patch
>
>
> Spinoff from recent "lucene 2.9 sorting algorithm" thread on java-dev,
> where a simpler (non-segment-based) comparator API is proposed that
> gathers results into multiple PQs (one per segment) and then merges
> them in the end.
> I started from John's multi-PQ code and worked it into
> contrib/benchmark so that we could run perf tests. Then I generified
> the Python script I use for running search benchmarks (in
> contrib/benchmark/sortBench.py).
> The script first creates indexes with 1M docs (based on
> SortableSingleDocSource, and based on wikipedia, if available). Then
> it runs various combinations:
> * Index with 20 balanced segments vs index with the "normal" log
> segment size
> * Queries with different numbers of hits (only for wikipedia index)
> * Different top N
> * Different sorts (by title, for wikipedia, and by random string,
> random int, and country for the random index)
> For each test, 7 search rounds are run and the best QPS is kept. The
> script runs singlePQ then multiPQ, and records the resulting best QPS
> for each and produces table (in Jira format) as output.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-dev-help[at]lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-dev-help[at]lucene.apache.org
>
>

jake.mannix at gmail

Nov 3, 2009, 1:09 PM

Post #107 of 110 (157 views)
Permalink

Re: [jira] Commented: (LUCENE-1997) Explore performance of multi-PQ vs single-PQ sorting API [In reply to]

Um, according to Mike's latest numbers, multiPQ is actually *faster* at 1000
hits sometimes. In fact, all of the most recent tests have shown no clear
winner either way in terms of QPS. Sometimes (look at Yonik's linux
numbers), multiPQ is nearly across the board faster.

Either way, I think it's been made abundantly clear that any advantages
singlePQ has in terms of QPS performance degrade to near zero (if not
reverse to be negative) as the number of hits increases higher and higher.
I could show you what it looks like on a 10 or 20 million document index if
you'd like to see that as well, but I think that's pretty clear from the
difference from 1M to 5M.

-jake

On Tue, Nov 3, 2009 at 12:56 PM, Mark Miller <markrmiller[at]gmail.com> wrote:

> The memory issue is just one example of something that's somewhat worse - I
> don't see it as a deciding faster. If things were clarified to be decidedly
> faster with multi queue, and not 50% worse at 1000 hits, I'd be for the
> change, more memory or not.
>
>
> - Mark
>
> http://www.lucidimagination.com (mobile)
>
> On Nov 3, 2009, at 12:42 PM, Jake Mannix <jake.mannix[at]gmail.com> wrote:
>
> Mark, I'm not stuck on single examples, I'm thinking about all of lucene
> land: what tiny fraction of people need the combined intersection of
>
> a) many many segments
>
> AND
>
> b) deep paging
>
> AND
>
> c) high QPS
>
> AND
>
> e) can't handle another 40MB of RAM usage.
>
> Only people in the intersection of all of those bitsets would possibly have
> a problem with the memory requirements of multiPQ.
>
> On Tue, Nov 3, 2009 at 12:32 PM, Mark Miller < <markrmiller[at]gmail.com>
> markrmiller[at]gmail.com> wrote:
>
>> Your obviously too stuck on single examples. We have to consider everyone
>> in lucene land.
>>
>> I'm against 2 Apis. A custom search is advanced - it's not worth the
>> baggage of maintaining two APIs or be limited by the APIs and back compat
>> when moving forward.
>>
>> If the advantage of the second API is just going to be it's simpler, I'm
>> not for it currently.
>>
>> - Mark
>>
>> <http://www.lucidimagination.com>http://www.lucidimagination.com(mobile)
>>
>> On Nov 3, 2009, at 10:51 AM, "Jake Mannix (JIRA)" < <jira[at]apache.org>
>> jira[at]apache.org> wrote:
>>
>>
>>> [
>>> <https://issues.apache.org/jira/browse/LUCENE-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773114#action_12773114>
>>> https://issues.apache.org/jira/browse/LUCENE-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773114#action_12773114
>>> ]
>>>
>>> Jake Mannix commented on LUCENE-1997:
>>> -------------------------------------
>>>
>>> bq. Since each approach has distinct advantages, why not offer both
>>> ("simple" and "expert") comparator extensions APIs?
>>>
>>> +1 from me on this one, as long as the simpler one is around. I'll bet
>>> we'll find that we regret keeping the "expert" one by 3.2 or so though, but
>>> I'll take any compromise which gets the simpler API in there.
>>>
>>> bq. Don't forget that this is multiplied by however many queries are
>>> currently in flight.
>>>
>>> Sure, so if you're running with 100 queries per second on a single shard
>>> (pretty fast!), with 100 segments, and you want to do sorting by value on
>>> the top 1000 values (how far down the long tail of extreme cases are we at
>>> now? Do librarians hit their search servers with 100 QPS and have indices
>>> poorly built with hundreds of segments and can't take downtime to *ever*
>>> optimize?), we're now talking about 40MB.
>>>
>>> *Forty megabytes*. On a beefy machine which is supposed to be handling
>>> 100QPS across an index big enough to need 100 segments. How much heap would
>>> such a machine already be allocating? 4GB? 6? More?
>>>
>>> We're talking about less than 1% of the heap is being used by the multiPQ
>>> approach in comparison to singlePQ.
>>>
>>> Explore performance of multi-PQ vs single-PQ sorting API
>>>> --------------------------------------------------------
>>>>
>>>> Key: LUCENE-1997
>>>> URL: <https://issues.apache.org/jira/browse/LUCENE-1997>
>>>> https://issues.apache.org/jira/browse/LUCENE-1997
>>>> Project: Lucene - Java
>>>> Issue Type: Improvement
>>>> Components: Search
>>>> Affects Versions: 2.9
>>>> Reporter: Michael McCandless
>>>> Assignee: Michael McCandless
>>>> Attachments: LUCENE-1997.patch, LUCENE-1997.patch,
>>>> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch,
>>>> LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch
>>>>
>>>>
>>>> Spinoff from recent "lucene 2.9 sorting algorithm" thread on java-dev,
>>>> where a simpler (non-segment-based) comparator API is proposed that
>>>> gathers results into multiple PQs (one per segment) and then merges
>>>> them in the end.
>>>> I started from John's multi-PQ code and worked it into
>>>> contrib/benchmark so that we could run perf tests. Then I generified
>>>> the Python script I use for running search benchmarks (in
>>>> contrib/benchmark/sortBench.py).
>>>> The script first creates indexes with 1M docs (based on
>>>> SortableSingleDocSource, and based on wikipedia, if available). Then
>>>> it runs various combinations:
>>>> * Index with 20 balanced segments vs index with the "normal" log
>>>> segment size
>>>> * Queries with different numbers of hits (only for wikipedia index)
>>>> * Different top N
>>>> * Different sorts (by title, for wikipedia, and by random string,
>>>> random int, and country for the random index)
>>>> For each test, 7 search rounds are run and the best QPS is kept. The
>>>> script runs singlePQ then multiPQ, and records the resulting best QPS
>>>> for each and produces table (in Jira format) as output.
>>>>
>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> -
>>> You can reply to this email to add a comment to the issue online.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: <java-dev-unsubscribe[at]lucene.apache.org>
>>> java-dev-unsubscribe[at]lucene.apache.org
>>> For additional commands, e-mail: <java-dev-help[at]lucene.apache.org>
>>> java-dev-help[at]lucene.apache.org
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: <java-dev-unsubscribe[at]lucene.apache.org>
>> java-dev-unsubscribe[at]lucene.apache.org
>> For additional commands, e-mail: <java-dev-help[at]lucene.apache.org>
>> java-dev-help[at]lucene.apache.org
>>
>>
>

jira at apache

Nov 3, 2009, 4:00 PM

Post #108 of 110 (153 views)
Permalink

[jira] Commented: (LUCENE-1997) Explore performance of multi-PQ vs single-PQ sorting API [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773298#action_12773298 ]

Mark Miller commented on LUCENE-1997:
-------------------------------------

If this is about ease of use, its pretty easy to return Comparable to the fieldcache, add a Comparable fieldcomparator, and let users that need it (havn't seen the clamoring yet though) just implement getComparable(String). Its not that hard to support that with a single queue either.

Its still my opinion that users that need this are advanced enough to look at the provided impls and figure it out pretty quick. Its not rocket science, and each method impls is generally a couple simple lines of code.

> Explore performance of multi-PQ vs single-PQ sorting API
> --------------------------------------------------------
>
> Key: LUCENE-1997
> URL: https://issues.apache.org/jira/browse/LUCENE-1997
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.9
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Attachments: LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch, LUCENE-1997.patch
>
>
> Spinoff from recent "lucene 2.9 sorting algorithm" thread on java-dev,
> where a simpler (non-segment-based) comparator API is proposed that
> gathers results into multiple PQs (one per segment) and then merges
> them in the end.
> I started from John's multi-PQ code and worked it into
> contrib/benchmark so that we could run perf tests. Then I generified
> the Python script I use for running search benchmarks (in
> contrib/benchmark/sortBench.py).
> The script first creates indexes with 1M docs (based on
> SortableSingleDocSource, and based on wikipedia, if available). Then
> it runs various combinations:
> * Index with 20 balanced segments vs index with the "normal" log
> segment size
> * Queries with different numbers of hits (only for wikipedia index)
> * Different top N
> * Different sorts (by title, for wikipedia, and by random string,
> random int, and country for the random index)
> For each test, 7 search rounds are run and the best QPS is kept. The
> script runs singlePQ then multiPQ, and records the resulting best QPS
> for each and produces table (in Jira format) as output.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

markrmiller at gmail

Nov 3, 2009, 4:08 PM

Post #109 of 110 (153 views)
Permalink

Re: [jira] Commented: (LUCENE-1997) Explore performance of multi-PQ vs single-PQ sorting API [In reply to]

Jake Mannix wrote:
> Um, according to Mike's latest numbers, multiPQ is actually *faster*
> at 1000 hits sometimes. In fact, all of the most recent tests have
> shown no clear winner either way in terms of QPS. Sometimes (look at
> Yonik's linux numbers), multiPQ is nearly across the board faster.
The numbers are changing and changing. And don't look too appealing on
Yonik's jdk 7 runs. But who knows. I'm not taking any of them as gospel yet.

>
> Either way, I think it's been made abundantly clear that any
> advantages singlePQ has in terms of QPS performance degrade to near
> zero (if not reverse to be negative) as the number of hits increases
> higher and higher. I could show you what it looks like on a 10 or 20
> million document index if you'd like to see that as well, but I think
> that's pretty clear from the difference from 1M to 5M.
>
> -jake
>
> On Tue, Nov 3, 2009 at 12:56 PM, Mark Miller <markrmiller[at]gmail.com
> <mailto:markrmiller[at]gmail.com>> wrote:
>
> The memory issue is just one example of something that's somewhat
> worse - I don't see it as a deciding faster. If things were
> clarified to be decidedly faster with multi queue, and not 50%
> worse at 1000 hits, I'd be for the change, more memory or not.
>
>
> - Mark
>
> http://www.lucidimagination.com (mobile)
>
> On Nov 3, 2009, at 12:42 PM, Jake Mannix <jake.mannix[at]gmail.com
> <mailto:jake.mannix[at]gmail.com>> wrote:
>
>> Mark, I'm not stuck on single examples, I'm thinking about all of
>> lucene land: what tiny fraction of people need the combined
>> intersection of
>>
>> a) many many segments
>>
>> AND
>>
>> b) deep paging
>>
>> AND
>>
>> c) high QPS
>>
>> AND
>>
>> e) can't handle another 40MB of RAM usage.
>>
>> Only people in the intersection of all of those bitsets would
>> possibly have a problem with the memory requirements of multiPQ.
>>
>> On Tue, Nov 3, 2009 at 12:32 PM, Mark Miller
>> <markrmiller[at]gmail.com <mailto:markrmiller[at]gmail.com>> wrote:
>>
>> Your obviously too stuck on single examples. We have to
>> consider everyone in lucene land.
>>
>> I'm against 2 Apis. A custom search is advanced - it's not
>> worth the baggage of maintaining two APIs or be limited by
>> the APIs and back compat when moving forward.
>>
>> If the advantage of the second API is just going to be it's
>> simpler, I'm not for it currently.
>>
>> - Mark
>>
>> http://www.lucidimagination.com (mobile)
>>
>> On Nov 3, 2009, at 10:51 AM, "Jake Mannix (JIRA)"
>> <jira[at]apache.org <mailto:jira[at]apache.org>> wrote:
>>
>>
>> [
>> https://issues.apache.org/jira/browse/LUCENE-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773114#action_12773114
>> <https://issues.apache.org/jira/browse/LUCENE-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773114#action_12773114> ]
>>
>> Jake Mannix commented on LUCENE-1997:
>> -------------------------------------
>>
>> bq. Since each approach has distinct advantages, why not
>> offer both ("simple" and "expert") comparator extensions
>> APIs?
>>
>> +1 from me on this one, as long as the simpler one is
>> around. I'll bet we'll find that we regret keeping the
>> "expert" one by 3.2 or so though, but I'll take any
>> compromise which gets the simpler API in there.
>>
>> bq. Don't forget that this is multiplied by however many
>> queries are currently in flight.
>>
>> Sure, so if you're running with 100 queries per second on
>> a single shard (pretty fast!), with 100 segments, and you
>> want to do sorting by value on the top 1000 values (how
>> far down the long tail of extreme cases are we at now?
>> Do librarians hit their search servers with 100 QPS and
>> have indices poorly built with hundreds of segments and
>> can't take downtime to *ever* optimize?), we're now
>> talking about 40MB.
>>
>> *Forty megabytes*. On a beefy machine which is supposed
>> to be handling 100QPS across an index big enough to need
>> 100 segments. How much heap would such a machine already
>> be allocating? 4GB? 6? More?
>>
>> We're talking about less than 1% of the heap is being
>> used by the multiPQ approach in comparison to singlePQ.
>>
>> Explore performance of multi-PQ vs single-PQ sorting API
>> --------------------------------------------------------
>>
>> Key: LUCENE-1997
>> URL:
>> https://issues.apache.org/jira/browse/LUCENE-1997
>> Project: Lucene - Java
>> Issue Type: Improvement
>> Components: Search
>> Affects Versions: 2.9
>> Reporter: Michael McCandless
>> Assignee: Michael McCandless
>> Attachments: LUCENE-1997.patch,
>> LUCENE-1997.patch, LUCENE-1997.patch,
>> LUCENE-1997.patch, LUCENE-1997.patch,
>> LUCENE-1997.patch, LUCENE-1997.patch,
>> LUCENE-1997.patch, LUCENE-1997.patch
>>
>>
>> Spinoff from recent "lucene 2.9 sorting algorithm"
>> thread on java-dev,
>> where a simpler (non-segment-based) comparator API is
>> proposed that
>> gathers results into multiple PQs (one per segment)
>> and then merges
>> them in the end.
>> I started from John's multi-PQ code and worked it into
>> contrib/benchmark so that we could run perf tests.
>> Then I generified
>> the Python script I use for running search benchmarks (in
>> contrib/benchmark/sortBench.py).
>> The script first creates indexes with 1M docs (based on
>> SortableSingleDocSource, and based on wikipedia, if
>> available). Then
>> it runs various combinations:
>> * Index with 20 balanced segments vs index with the
>> "normal" log
>> segment size
>> * Queries with different numbers of hits (only for
>> wikipedia index)
>> * Different top N
>> * Different sorts (by title, for wikipedia, and by
>> random string,
>> random int, and country for the random index)
>> For each test, 7 search rounds are run and the best
>> QPS is kept. The
>> script runs singlePQ then multiPQ, and records the
>> resulting best QPS
>> for each and produces table (in Jira format) as output.
>>
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue
>> online.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> java-dev-unsubscribe[at]lucene.apache.org
>> <mailto:java-dev-unsubscribe[at]lucene.apache.org>
>> For additional commands, e-mail:
>> java-dev-help[at]lucene.apache.org
>> <mailto:java-dev-help[at]lucene.apache.org>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> java-dev-unsubscribe[at]lucene.apache.org
>> <mailto:java-dev-unsubscribe[at]lucene.apache.org>
>> For additional commands, e-mail:
>> java-dev-help[at]lucene.apache.org
>> <mailto:java-dev-help[at]lucene.apache.org>
>>
>>
>

--
- Mark

http://www.lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

jason.rutherglen at gmail

Nov 4, 2009, 10:02 AM

Post #110 of 110 (135 views)
Permalink

Re: [jira] Commented: (LUCENE-1997) Explore performance of multi-PQ vs single-PQ sorting API [In reply to]

If I'm understanding the multi-PQ approach correctly, the
multi-PQ could enable Solr for example to cache results per
segment, whereas the single-PQ will not.

This raises the question of how useful results caching is to
Solr going forward? Will the lack of results caching in the NRT
use case be a drawback or a non-issue?

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-dev-help[at]lucene.apache.org

1 2 3 4 5

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com