Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Lucene: General

Performance issues with high query volume

 

 

Lucene general RSS feed   Index | Next | Previous | View Threaded


rstaudinger at gmail

Dec 12, 2011, 3:00 PM

Post #1 of 2 (419 views)
Permalink
Performance issues with high query volume

Hi,

I inherited a fairly large SOLR installation that has some performance
issues in terms of query times. Since I am newbie to SOLR, I am looking for
pointers where to start looking for the problem. Here is the basic setup of
the system:

- sharded across 4 servers (24 CPU, 48GB RAM)
- the index size on each shard is 11GB
- each shard contains about 4.5 million documents

The system receives a pretty constant load of search requests, at a rate of
about 110 requests/second. I can control the rate of request sent to SOLR,
and this is about the best rate I can get, which CPU maxed out on all 4
servers. Average search time is around 4000ms on the master and anywhere
from 700ms to 1500ms on the slaves. Search times drop proportionally with
the number of requests. The best search time average I can get is around
500ms.

I suspect that this configuration is not appropriate, since the sharding is
geared more toward large datasets, not so much query volume, and a
master-slave configuration might be more appropriate. Is an index size of
11GB appropriate, or should it be larger or smaller in this situation?

We are also on SORL 1.4.1, which might be part of the issue as well. I am
planning to upgrade to the latest version in a few months. Are there
performance improvements from that alone even with the current
configuration?

Any pointers to either narrow down the problem or suggest alternate
configurations would be greatly appreciated.

Thanks!

--
View this message in context: http://lucene.472066.n3.nabble.com/Performance-issues-with-high-query-volume-tp3580962p3580962.html
Sent from the Lucene - General mailing list archive at Nabble.com.


otis_gospodnetic at yahoo

Dec 16, 2011, 9:17 AM

Post #2 of 2 (390 views)
Permalink
Re: Performance issues with high query volume [In reply to]

Hi,

It's hard to tell.  For example, the problem could "simply" be with the JVM and how it is tuned or not tuned, not at all a Solr problem.  Or it could be a Solr problem.

Try using SPM (link below, it's free) - this will help you figure out what is going on and see what effect any changes you make to Solr have.

Otis
----
Performance Monitoring SaaS for Solr - http://sematext.com/spm/solr-performance-monitoring/index.html



----- Original Message -----
> From: rstaudinger <rstaudinger [at] gmail>
> To: general [at] lucene
> Cc:
> Sent: Monday, December 12, 2011 6:00 PM
> Subject: Performance issues with high query volume
>
> Hi,
>
> I inherited a fairly large SOLR installation that has some performance
> issues in terms of query times.  Since I am newbie to SOLR, I am looking for
> pointers where to start looking for the problem.  Here is the basic setup of
> the system:
>
> - sharded across 4 servers (24 CPU, 48GB RAM)
> - the index size on each shard is 11GB
> - each shard contains about 4.5 million documents
>
> The system receives a pretty constant load of search requests, at a rate of
> about 110 requests/second.  I can control the rate of request sent to SOLR,
> and this is about the best rate I can get, which CPU maxed out on all 4
> servers.  Average search time is around 4000ms on the master and anywhere
> from 700ms to 1500ms on the slaves.  Search times drop proportionally with
> the number of requests.  The best search time average I can get is around
> 500ms.
>
> I suspect that this configuration is not appropriate, since the sharding is
> geared more toward large datasets, not so much query volume, and a
> master-slave configuration might be more appropriate.  Is an index size of
> 11GB appropriate, or should it be larger or smaller in this situation? 
>  
> We are also on SORL 1.4.1, which might be part of the issue as well.  I am
> planning to upgrade to the latest version in a few months.  Are there
> performance improvements from that alone even with the current
> configuration?
>
> Any pointers to either narrow down the problem or suggest alternate
> configurations would be greatly appreciated.
>
> Thanks!
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Performance-issues-with-high-query-volume-tp3580962p3580962.html
> Sent from the Lucene - General mailing list archive at Nabble.com.
>

Lucene general RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.