Mailing List Archive: Lucene: Java-User

Doesn it make sense cache IndexReader?

Index | Next | Previous | View Threaded

mohsens at gmail

Jul 16, 2008, 11:29 PM

Post #1 of 4 (280 views)
Permalink

Doesn it make sense cache IndexReader?

Hi,
I have a set of indices in different languages (very smal indices: on
average each index directory has 10,000 documents, which has an overall size
of less than 2mb). I want to know if this is a good idea to cache
IndexReader (once opened) somewhere and further reuse it? My application is
single-threaded, but I have memory concerns, since the indexing and search
is totally done on client machines.

Does it make sense to have all IndexReaders cached (for example if opening
an index reader takes some time). Does Lucene load some part of index in
memory as soon as a call to IndexReader.open() is done?

Mohsen.
--
View this message in context: http://www.nabble.com/Doesn-it-make-sense-cache-IndexReader--tp18502361p18502361.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

karsten-lucene at fiz-technik

Jul 17, 2008, 2:35 AM

Post #2 of 4 (265 views)
Permalink

Re: Doesn it make sense cache IndexReader? [In reply to]

hi Mohsen,

IndexReader is thread-save so you don't need a cache.

Opening and closing an IndexReader takes some time. But I don't now if this is measurable.
So if you know that you don't want to change a index, let the IndexReader open.
(somebody disagree with me?)

If you want to use your main memory you can switch from FSDirectory to RamDirectory.
This will boost your performance for sure.

Best regards
Karsten

> Does it make sense to have all IndexReaders cached (for example if opening
> an index reader takes some time). Does Lucene load some part of index in
> memory as soon as a call to IndexReader.open() is done?

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

mohsens at gmail

Jul 17, 2008, 8:49 AM

Post #3 of 4 (260 views)
Permalink

Re: Doesn it make sense cache IndexReader? [In reply to]

Kastern, thanks for the reply.

> IndexReader is thread-save so you don't need a cache.

If it wasn't thread-safe I may need a cache? I think the cache cannot be
used on a non-thread-safe environment since it exposes same instances to
different threads.

> Opening and closing an IndexReader takes some time. But I don't
> now if this is measurable.
> So if you know that you don't want to change a index, let the IndexReader
> open.
> (somebody disagree with me?)

My indices are readonly. Once created never change. If the memory footprint
for an open IndexReader is very low, I can keep all the gradually opened
index readers open.

> If you want to use your main memory you can switch from FSDirectory to
> RamDirectory.
> This will boost your performance for sure.

Thanks for the suggestion. I will use it for the most commonly used index.

Mohsen.
--
View this message in context: http://www.nabble.com/Doesn-it-make-sense-cache-IndexReader--tp18502361p18511667.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

markrmiller at gmail

Jul 20, 2008, 7:05 PM

Post #4 of 4 (220 views)
Permalink

Re: Doesn it make sense cache IndexReader? [In reply to]

With very small indexes and no sort fields (eg. you just use relevance)
loading an IndexReader does not take very long. I think it does always
make sense to cache it and reuse it though - unless the index has
changed, there is no reason to pay the price of opening a new
IndexReader. As your index grows, and especially if you sort on fields,
the price of 'warming' a new IndexReader can get quite high.

However, if memory is an issue, you might not cache just so that its not
guaranteed that all of your IndexReaders for each index will be in
memory at the same time - this is not a great solution to your memory
problem though - if all indecies are ever searched at the same time,
you will need to be able to accommodate that many IndexReaders in RAM
anyway.

- Mark

Mohsen Saboorian wrote:
> Hi,
> I have a set of indices in different languages (very smal indices: on
> average each index directory has 10,000 documents, which has an overall size
> of less than 2mb). I want to know if this is a good idea to cache
> IndexReader (once opened) somewhere and further reuse it? My application is
> single-threaded, but I have memory concerns, since the indexing and search
> is totally done on client machines.
>
> Does it make sense to have all IndexReaders cached (for example if opening
> an index reader takes some time). Does Lucene load some part of index in
> memory as soon as a call to IndexReader.open() is done?
>
> Mohsen.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com