Mailing List Archive: Lucene: Java-User

Transaction support in Lucene

Index | Next | Previous | View Threaded

rajesh_parab_1 at yahoo

Nov 14, 2006, 9:24 AM

Post #1 of 5 (1133 views)
Permalink

Transaction support in Lucene

Hi,

Does anyone know if there is any plan in adding transaction support in Lucene?

Regards,
Rajesh

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Nov 14, 2006, 10:09 AM

Post #2 of 5 (1081 views)
Permalink

Re: Transaction support in Lucene [In reply to]

Rajesh parab wrote:

> Does anyone know if there is any plan in adding transaction support in Lucene?

I don't know of specific plans.

This has been discussed before on user & dev lists. I know the
Compass project builds transactional support on top of Lucene.

Are you asking for transaction support shared with something external
(eg a database)? Meaning updates to the DB and to Lucene either
atomically succeed or fail, together?

Or, are you asking for transactional behaviour of updates just to
Lucene, eg, you want to do a bunch of adds & deletes but have them not
be visible (committed) until the end of your transaction and roll back
on any failure?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

rajesh_parab_1 at yahoo

Nov 14, 2006, 10:19 AM

Post #3 of 5 (1082 views)
Permalink

Re: Transaction support in Lucene [In reply to]

Hi Mike,

Thanks for the feedback.

I am talking about transaction support in Lucene only. If there is a failure during insert/update/delete of document inside the index, there is no way to roll back the operation and this will keep the index in in-consistent state.

I read about Compass providing transaction support on top of Lucene APIs. But, I am not sure, if it is a good idea to use Compass instead of directly using Lucene APIs. There will always be a dependency on Compass to support the latest version/additions to Lucene.

Regards,
Rajesh

----- Original Message ----
From: Michael McCandless <lucene [at] mikemccandless>
To: java-user [at] lucene
Sent: Tuesday, November 14, 2006 10:09:49 AM
Subject: Re: Transaction support in Lucene

Rajesh parab wrote:

> Does anyone know if there is any plan in adding transaction support in Lucene?

I don't know of specific plans.

This has been discussed before on user & dev lists. I know the
Compass project builds transactional support on top of Lucene.

Are you asking for transaction support shared with something external
(eg a database)? Meaning updates to the DB and to Lucene either
atomically succeed or fail, together?

Or, are you asking for transactional behaviour of updates just to
Lucene, eg, you want to do a bunch of adds & deletes but have them not
be visible (committed) until the end of your transaction and roll back
on any failure?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

greg at colvin

Nov 14, 2006, 10:28 AM

Post #4 of 5 (1072 views)
Permalink

Re: Transaction support in Lucene [In reply to]

Last I looked at this, I thought that mapping transactions
onto Lucene segments was the way to go.

On Nov 14, 2006, at 11:19 AM, Rajesh parab wrote:
> Hi Mike,
>
> Thanks for the feedback.
>
> I am talking about transaction support in Lucene only. If there is
> a failure during insert/update/delete of document inside the index,
> there is no way to roll back the operation and this will keep the
> index in in-consistent state.
>
> I read about Compass providing transaction support on top of Lucene
> APIs. But, I am not sure, if it is a good idea to use Compass
> instead of directly using Lucene APIs. There will always be a
> dependency on Compass to support the latest version/additions to
> Lucene.
>
> Regards,
> Rajesh
>
> ----- Original Message ----
> From: Michael McCandless <lucene [at] mikemccandless>
> To: java-user [at] lucene
> Sent: Tuesday, November 14, 2006 10:09:49 AM
> Subject: Re: Transaction support in Lucene
>
> Rajesh parab wrote:
>
>> Does anyone know if there is any plan in adding transaction
>> support in Lucene?
>
> I don't know of specific plans.
>
> This has been discussed before on user & dev lists. I know the
> Compass project builds transactional support on top of Lucene.
>
> Are you asking for transaction support shared with something external
> (eg a database)? Meaning updates to the DB and to Lucene either
> atomically succeed or fail, together?
>
> Or, are you asking for transactional behaviour of updates just to
> Lucene, eg, you want to do a bunch of adds & deletes but have them not
> be visible (committed) until the end of your transaction and roll back
> on any failure?
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

lucene at mikemccandless

Nov 14, 2006, 10:57 AM

Post #5 of 5 (1076 views)
Permalink

Re: Transaction support in Lucene [In reply to]

Rajesh parab wrote:

> I am talking about transaction support in Lucene only. If there is a failure during insert/update/delete of document inside the index, there is no way to roll back the operation and this will keep the index in in-consistent state.

OK, I see. Then you should also look at Solr, which puts "Lucene
only" transactional semantics on top, in that you can do a bunch of
add/update/deletes to your index but only on calling "commit" are
these changes copied to a snapshot (using hard links in the
filesystem) and distributed to searchers.

Also, the lockless commits patch (not committed yet but I think
close):

http://issues.apache.org/jira/browse/LUCENE-701

gets us closer to enabling transactions. With that patch, every
commit from Lucene writes to a segments_N file (N increments on every
commit). We keep only the last segments_N file.

But, with a small-ish change beyond that patch, which has been briefly
touched on the dev list & Jira, we could have Lucene keep more than
just the last segments_N file (and all segment files referenced by
them). This would enable you to open a Lucene index explicitly at
prior commit point. Then we could enable transactions over that by
recording the segments_N at which you began your transaction, not
deleting it, and then rolling back to it if any error had occurred.

This same approach could be used to make Lucene's "point in time"
searching explicit, and, work correctly on NFS.

I will open a Jira issue to start tracking these ideas ... but I think
it's a ways off before this would be available in a release.

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com