Mailing List Archive: Lucene: Java-User

Max number of open IndexWriters

Index | Next | Previous | View Threaded

hrishikesh_agashe at persistent

Nov 14, 2009, 8:35 AM

Post #1 of 6 (691 views)
Permalink

Max number of open IndexWriters

Is there any limit on how many IndexWriter can I keep open at same time? What does it depends on (RAM?) Can I keep 100 or 200 IndexWriters open in say HashMap and use them as I process documents?

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

erickerickson at gmail

Nov 14, 2009, 3:58 PM

Post #2 of 6 (639 views)
Permalink

Re: Max number of open IndexWriters [In reply to]

Hmmm, why would you want to do this? What problem are
you trying to address with so many readers?

I ask because I'm having a hard time thinking of a reasonable
use-case.

Best
Erick

On Sat, Nov 14, 2009 at 11:35 AM, Hrishikesh Agashe <
hrishikesh_agashe [at] persistent> wrote:

> Is there any limit on how many IndexWriter can I keep open at same time?
> What does it depends on (RAM?) Can I keep 100 or 200 IndexWriters open in
> say HashMap and use them as I process documents?
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>

simon.willnauer at googlemail

Nov 14, 2009, 4:03 PM

Post #3 of 6 (647 views)
Permalink

Re: Max number of open IndexWriters [In reply to]

In general you should only use one single writer and use it with
multiple threads. IndexWriter is optimized for concurrency and fully
thread-safe. Is that what you trying to achieve?

otherwise I would 100% agree with eric not knowing of a use-case.

simon

On Sun, Nov 15, 2009 at 12:58 AM, Erick Erickson
<erickerickson [at] gmail> wrote:
> Hmmm, why would you want to do this? What problem are
> you trying to address with so many readers?
>
> I ask because I'm having a hard time thinking of a reasonable
> use-case.
>
> Best
> Erick
>
> On Sat, Nov 14, 2009 at 11:35 AM, Hrishikesh Agashe <
> hrishikesh_agashe [at] persistent> wrote:
>
>> Is there any limit on how many IndexWriter can I keep open at same time?
>> What does it depends on (RAM?) Can I keep 100 or 200 IndexWriters open in
>> say HashMap and use them as I process documents?
>>
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is
>> the property of Persistent Systems Ltd. It is intended only for the use of
>> the individual or entity to which it is addressed. If you are not the
>> intended recipient, you are not authorized to read, retain, copy, print,
>> distribute or use this message. If you have received this communication in
>> error, please notify the sender and delete all copies of this message.
>> Persistent Systems Ltd. does not accept any liability for virus infected
>> mails.
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

hrishikesh_agashe at persistent

Nov 15, 2009, 8:14 PM

Post #4 of 6 (621 views)
Permalink

RE: Max number of open IndexWriters [In reply to]

Simon and Eric, thanks for the reply.

I want to create multiple indexes depending on the data present. Like if month of record is Nov, I want to add it in index for November. If it's October, add it in index for October. I don't want to open and close indexes so many times so just maintaining IndexWriters for months in HashMap.
Is there better way of doing this?

--Hrishi

-----Original Message-----
From: Simon Willnauer [mailto:simon.willnauer [at] googlemail]
Sent: Sunday, November 15, 2009 5:34 AM
To: java-user [at] lucene
Subject: Re: Max number of open IndexWriters

In general you should only use one single writer and use it with
multiple threads. IndexWriter is optimized for concurrency and fully
thread-safe. Is that what you trying to achieve?

otherwise I would 100% agree with eric not knowing of a use-case.

simon

On Sun, Nov 15, 2009 at 12:58 AM, Erick Erickson
<erickerickson [at] gmail> wrote:
> Hmmm, why would you want to do this? What problem are
> you trying to address with so many readers?
>
> I ask because I'm having a hard time thinking of a reasonable
> use-case.
>
> Best
> Erick
>
> On Sat, Nov 14, 2009 at 11:35 AM, Hrishikesh Agashe <
> hrishikesh_agashe [at] persistent> wrote:
>
>> Is there any limit on how many IndexWriter can I keep open at same time?
>> What does it depends on (RAM?) Can I keep 100 or 200 IndexWriters open in
>> say HashMap and use them as I process documents?
>>
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is
>> the property of Persistent Systems Ltd. It is intended only for the use of
>> the individual or entity to which it is addressed. If you are not the
>> intended recipient, you are not authorized to read, retain, copy, print,
>> distribute or use this message. If you have received this communication in
>> error, please notify the sender and delete all copies of this message.
>> Persistent Systems Ltd. does not accept any liability for virus infected
>> mails.
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

emailgane at yahoo

Nov 15, 2009, 9:32 PM

Post #5 of 6 (626 views)
Permalink

Re: Max number of open IndexWriters [In reply to]

You could keep multiple writers opened and it will do no harm. I am doing this.

Its good to reopen (close and open) the writer at certain interval. This will release the memory it holds. Whenever i create a new database, I reopen writers belong to all database.

Regards
Ganesh

----- Original Message -----
From: "Hrishikesh Agashe" <hrishikesh_agashe [at] persistent>
To: <java-user [at] lucene>; <simon.willnauer [at] gmail>
Sent: Monday, November 16, 2009 9:44 AM
Subject: RE: Max number of open IndexWriters

> Simon and Eric, thanks for the reply.
>
> I want to create multiple indexes depending on the data present. Like if month of record is Nov, I want to add it in index for November. If it's October, add it in index for October. I don't want to open and close indexes so many times so just maintaining IndexWriters for months in HashMap.
> Is there better way of doing this?
>
> --Hrishi
>
> -----Original Message-----
> From: Simon Willnauer [mailto:simon.willnauer [at] googlemail]
> Sent: Sunday, November 15, 2009 5:34 AM
> To: java-user [at] lucene
> Subject: Re: Max number of open IndexWriters
>
> In general you should only use one single writer and use it with
> multiple threads. IndexWriter is optimized for concurrency and fully
> thread-safe. Is that what you trying to achieve?
>
> otherwise I would 100% agree with eric not knowing of a use-case.
>
> simon
>
> On Sun, Nov 15, 2009 at 12:58 AM, Erick Erickson
> <erickerickson [at] gmail> wrote:
>> Hmmm, why would you want to do this? What problem are
>> you trying to address with so many readers?
>>
>> I ask because I'm having a hard time thinking of a reasonable
>> use-case.
>>
>> Best
>> Erick
>>
>> On Sat, Nov 14, 2009 at 11:35 AM, Hrishikesh Agashe <
>> hrishikesh_agashe [at] persistent> wrote:
>>
>>> Is there any limit on how many IndexWriter can I keep open at same time?
>>> What does it depends on (RAM?) Can I keep 100 or 200 IndexWriters open in
>>> say HashMap and use them as I process documents?
>>>
>>>
>>> DISCLAIMER
>>> ==========
>>> This e-mail may contain privileged and confidential information which is
>>> the property of Persistent Systems Ltd. It is intended only for the use of
>>> the individual or entity to which it is addressed. If you are not the
>>> intended recipient, you are not authorized to read, retain, copy, print,
>>> distribute or use this message. If you have received this communication in
>>> error, please notify the sender and delete all copies of this message.
>>> Persistent Systems Ltd. does not accept any liability for virus infected
>>> mails.
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
>
Send instant messages to your online friends http://in.messenger.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

erickerickson at gmail

Nov 16, 2009, 5:10 AM

Post #6 of 6 (624 views)
Permalink

Re: Max number of open IndexWriters [In reply to]

Ah, I understand now. No, Lucene imposes no limits that I know of. But see
Ganesh's comments. How many you want to keep open is going to depend
upon your traffic. It's the classic tradeoff between efficiency and simple
code.

Best
Erick

On Sun, Nov 15, 2009 at 11:14 PM, Hrishikesh Agashe <
hrishikesh_agashe [at] persistent> wrote:

> Simon and Eric, thanks for the reply.
>
> I want to create multiple indexes depending on the data present. Like if
> month of record is Nov, I want to add it in index for November. If it's
> October, add it in index for October. I don't want to open and close indexes
> so many times so just maintaining IndexWriters for months in HashMap.
> Is there better way of doing this?
>
> --Hrishi
>
> -----Original Message-----
> From: Simon Willnauer [mailto:simon.willnauer [at] googlemail]
> Sent: Sunday, November 15, 2009 5:34 AM
> To: java-user [at] lucene
> Subject: Re: Max number of open IndexWriters
>
> In general you should only use one single writer and use it with
> multiple threads. IndexWriter is optimized for concurrency and fully
> thread-safe. Is that what you trying to achieve?
>
> otherwise I would 100% agree with eric not knowing of a use-case.
>
> simon
>
> On Sun, Nov 15, 2009 at 12:58 AM, Erick Erickson
> <erickerickson [at] gmail> wrote:
> > Hmmm, why would you want to do this? What problem are
> > you trying to address with so many readers?
> >
> > I ask because I'm having a hard time thinking of a reasonable
> > use-case.
> >
> > Best
> > Erick
> >
> > On Sat, Nov 14, 2009 at 11:35 AM, Hrishikesh Agashe <
> > hrishikesh_agashe [at] persistent> wrote:
> >
> >> Is there any limit on how many IndexWriter can I keep open at same time?
> >> What does it depends on (RAM?) Can I keep 100 or 200 IndexWriters open
> in
> >> say HashMap and use them as I process documents?
> >>
> >>
> >> DISCLAIMER
> >> ==========
> >> This e-mail may contain privileged and confidential information which is
> >> the property of Persistent Systems Ltd. It is intended only for the use
> of
> >> the individual or entity to which it is addressed. If you are not the
> >> intended recipient, you are not authorized to read, retain, copy, print,
> >> distribute or use this message. If you have received this communication
> in
> >> error, please notify the sender and delete all copies of this message.
> >> Persistent Systems Ltd. does not accept any liability for virus infected
> >> mails.
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads