Mailing List Archive: Scaling out/up or a mix

Mailing List Archive: Lucene: Java-User
Scaling out/up or a mix

Index | Next | Previous | View Flat

marcus.herou at tailsweep

Jun 26, 2009, 3:00 PM

Views: 848
Permalink

Scaling out/up or a mix

Hi.

I currently have an index which is 16GB per machine (8 machines = 128GB)
(data is stored externally, not in index) and is growing like crazy (we are
indexing blogs which is crazy by nature) and have only allocated 2GB per
machine to the Lucene app since we are running some other stuff there in
parallell.

Each doc should be roughly the size of a blog post, no more than 20k.

We currently have about 90M documents and it is increasing rapidly so
getting into the G+ document range is not going to be too far away.

Now due to search performance I think I need to move these instances to
dedicated index/search machines (or index on some machines and search on
others). Anyway I would like to get some feedback about two things:

1. What is the most important hardware aspect when it comes to add document
to the index and optimize it.
1.1 Is it disk I|O write throghput ? (sequential or random-io ?)
1.2 Is it RAM ?
1.3 Is is CPU ?

My guess would be disk-io, right, wrong ?

2. What is the most important hardware aspect when it comes to searching
documents in my setup ? (result-set is limited to return only the top 10
matches with page handling)
2.1 Is it disk read throughput ? (sequential or random-io ?)
2.2 Is it RAM ?
2.3 Is is CPU ?

I have no clue since the data might not fit into memory. What is then the
most important factor ? read-performance while scanning the index ? CPU
while comparing fields and collecting results ?

What I'm trying to find out is what I can do to get most bang for the buck
with a limited (aren't we all limited?) budget.

Kindly

//Marcus

--
Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou[at]tailsweep.com
http://www.tailsweep.com/

Subject	User	Time
Scaling out/up or a mix	marcus.herou at tailsweep	Jun 26, 2009, 3:00 PM
Re: Scaling out/up or a mix	marcus.herou at tailsweep	Jun 28, 2009, 2:13 PM
Re: Scaling out/up or a mix	ebowman at boboco	Jun 28, 2009, 11:53 PM
Re: Scaling out/up or a mix	marcus.herou at tailsweep	Jun 29, 2009, 12:47 AM
Re: Scaling out/up or a mix	eksdev at yahoo	Jun 29, 2009, 1:06 AM
Re: Scaling out/up or a mix	marcus.herou at tailsweep	Jun 29, 2009, 7:40 AM
Re: Scaling out/up or a mix	te at statsbiblioteket	Jun 29, 2009, 2:41 AM
Re: Scaling out/up or a mix	te at statsbiblioteket	Jun 30, 2009, 1:49 AM
RE: Scaling out/up or a mix	uschindler at pangaea	Jun 30, 2009, 2:29 AM
Re: Scaling out/up or a mix	marcus.herou at tailsweep	Jun 30, 2009, 2:07 PM
RE: Scaling out/up or a mix	uwe at thetaphi	Jun 30, 2009, 2:29 AM
Re: Scaling out/up or a mix	marcus.herou at tailsweep	Jun 30, 2009, 1:59 PM
RE: Scaling out/up or a mix	te at statsbiblioteket	Jun 30, 2009, 2:51 AM
Re: Scaling out/up or a mix	goodell at gmail	Jun 30, 2009, 9:07 AM
Re: Scaling out/up or a mix	te at statsbiblioteket	Jul 1, 2009, 4:31 AM
Re: Scaling out/up or a mix	marcus.herou at tailsweep	Jul 1, 2009, 7:31 AM
Re: Scaling out/up or a mix	otis_gospodnetic at yahoo	Jul 2, 2009, 8:33 PM

Index | Next | Previous | View Flat

Interested in having your list archived? Contact lists@gossamer-threads.com