Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Lucene: Java-Dev

patch: adding IndexReader.isCurrent()

 

 

Lucene java-dev RSS feed   Index | Next | Previous | View Threaded


lucenelist at danielnaber

Jun 3, 2005, 1:18 PM

Post #1 of 7 (102 views)
Permalink
patch: adding IndexReader.isCurrent()

Hi,

a patch similar to this was proposed a year ago but then got lost. I will
apply it unless someone objects. It fixes the problem that you cannot
safely use IndexReader.getCurrentVersion() to detect changes, as the
version number is reset to 0 if the index is re-created.

Also see
http://www.mail-archive.com/lucene-dev%40jakarta.apache.org/msg06143.html

Regards
Daniel

--
http://www.danielnaber.de
Attachments: version.diff (3.38 KB)


otis_gospodnetic at yahoo

Jun 3, 2005, 2:18 PM

Post #2 of 7 (103 views)
Permalink
Re: patch: adding IndexReader.isCurrent() [In reply to]

Looks good to me. You may also want to add that instance method to
IndexReader that somebody asked for the other day...

Here:

"Hello,

Am I missing an obvious way to return version of already opened index?

Static method IndexReader.getCurrentVersion() uses SegmentInfos to
read version number from "segments" files. This returns current
version of index in Directory but I need a reliable way to obtain
version of IndexReader.

This can be solved using some locking in higher level API. However,
all what's actually needed is a simple method addition to IndexReader:

/**
* Version number when this IndexReader was opened.
*/
public long getVersion() {
return segmentInfos.getVersion();
}

that just passes version from segmentInfos instance used by
IndexReader. Is there a commiter willing to make such change please?

BTW, SigmentInfos.readCurrentVersion() reads ''segments" without
locking (COMMIT_LOCK). Is there a good reason or just an omission?

Best regards,
Radomir
"



Otis


--- Daniel Naber <lucenelist [at] danielnaber> wrote:

> Hi,
>
> a patch similar to this was proposed a year ago but then got lost. I
> will
> apply it unless someone objects. It fixes the problem that you cannot
>
> safely use IndexReader.getCurrentVersion() to detect changes, as the
> version number is reset to 0 if the index is re-created.
>
> Also see
>
http://www.mail-archive.com/lucene-dev%40jakarta.apache.org/msg06143.html
>
> Regards
> Daniel
>
> --
> http://www.danielnaber.de
> > Index:
>
/home/dnaber/workspace/LuceneSVN/src/java/org/apache/lucene/index/IndexReader.java
> ===================================================================
> ---
>
/home/dnaber/workspace/LuceneSVN/src/java/org/apache/lucene/index/IndexReader.java
> (Revision 178894)
> +++
>
/home/dnaber/workspace/LuceneSVN/src/java/org/apache/lucene/index/IndexReader.java
> (Arbeitskopie)
> @@ -224,6 +224,7 @@
> * @param directory where the index resides.
> * @return version number.
> * @throws IOException if segments file cannot be read
> + * @deprecated use {@link #isCurrent()} instead
> */
> public static long getCurrentVersion(String directory) throws
> IOException {
> return getCurrentVersion(new File(directory));
> @@ -236,6 +237,7 @@
> * @param directory where the index resides.
> * @return version number.
> * @throws IOException if segments file cannot be read
> + * @deprecated use {@link #isCurrent()} instead
> */
> public static long getCurrentVersion(File directory) throws
> IOException {
> Directory dir = FSDirectory.getDirectory(directory, false);
> @@ -251,12 +253,27 @@
> * @param directory where the index resides.
> * @return version number.
> * @throws IOException if segments file cannot be read.
> + * @deprecated use {@link #isCurrent()} instead
> */
> public static long getCurrentVersion(Directory directory) throws
> IOException {
> return SegmentInfos.readCurrentVersion(directory);
> }
>
> /**
> + * Check whether this IndexReader still works on a current version
> of the index.
> + * If this is not the case you will need to re-open the
> IndexReader to
> + * make sure you see the latest changes made to the index.
> + *
> + * @throws IOException
> + */
> + public boolean isCurrent() throws IOException {
> + if (SegmentInfos.readCurrentVersion(directory) >
> segmentInfos.getVersion()) {
> + return false;
> + }
> + return true;
> + }
> +
> + /**
> * Return an array of term frequency vectors for the specified
> document.
> * The array contains a vector for each vectorized field in the
> document.
> * Each vector contains terms and frequencies for all terms in a
> given vectorized field.
> Index:
>
/home/dnaber/workspace/LuceneSVN/src/java/org/apache/lucene/index/SegmentInfos.java
> ===================================================================
> ---
>
/home/dnaber/workspace/LuceneSVN/src/java/org/apache/lucene/index/SegmentInfos.java
> (Revision 171375)
> +++
>
/home/dnaber/workspace/LuceneSVN/src/java/org/apache/lucene/index/SegmentInfos.java
> (Arbeitskopie)
> @@ -29,7 +29,11 @@
> public static final int FORMAT = -1;
>
> public int counter = 0; // used to name new segments
> - private long version = 0; //counts how often the index has been
> changed by adding or deleting docs
> + /**
> + * counts how often the index has been changed by adding or
> deleting docs.
> + * starting with the current time in milliseconds forces to create
> unique version numbers.
> + */
> + private long version = System.currentTimeMillis();
>
> public final SegmentInfo info(int i) {
> return (SegmentInfo) elementAt(i);
> @@ -59,7 +63,7 @@
>
> if(format >= 0){ // in old format the version number may be
> at the end of the file
> if (input.getFilePointer() >= input.length())
> - version = 0; // old file format without version number
> + version = System.currentTimeMillis(); // old file format
> without version number
> else
> version = input.readLong(); // read version
> }
>
> >
---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
> For additional commands, e-mail: java-dev-help [at] lucene


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene


lucenelist at danielnaber

Jun 3, 2005, 2:31 PM

Post #3 of 7 (104 views)
Permalink
Re: patch: adding IndexReader.isCurrent() [In reply to]

On Friday 03 June 2005 23:18, Otis Gospodnetic wrote:

> Looks good to me.  You may also want to add that instance method to
> IndexReader that somebody asked for the other day...

Well, the "version" isn't a version number anymore, at least not one that
starts with 0 or 1. That's why those methods are deprecated. I don't know
what exactly the proposed getVersion() could be useful for besides solving
the problem that isCurrent() now solves?

Regards
Daniel

--
http://www.danielnaber.de

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene


otis_gospodnetic at yahoo

Jun 3, 2005, 2:51 PM

Post #4 of 7 (104 views)
Permalink
Re: patch: adding IndexReader.isCurrent() [In reply to]

Isn't version a time stamp now? Somebody may still want to have that
and keep track of that "long" outside Lucene. I think that's what that
person wanted.

Otis

--- Daniel Naber <lucenelist [at] danielnaber> wrote:

> On Friday 03 June 2005 23:18, Otis Gospodnetic wrote:
>
> > Looks good to me. You may also want to add that instance method to
> > IndexReader that somebody asked for the other day...
>
> Well, the "version" isn't a version number anymore, at least not one
> that
> starts with 0 or 1. That's why those methods are deprecated. I don't
> know
> what exactly the proposed getVersion() could be useful for besides
> solving
> the problem that isCurrent() now solves?
>
> Regards
> Daniel


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene


lucenelist at danielnaber

Jun 3, 2005, 3:05 PM

Post #5 of 7 (104 views)
Permalink
Re: patch: adding IndexReader.isCurrent() [In reply to]

On Friday 03 June 2005 23:51, Otis Gospodnetic wrote:

> Isn't version a time stamp now?

It starts as a time stamp, but then +1 is added for each change. So that
number has no useful meaning I think. It's only useful for comparison. Or
am I missing something?

Regards
Daniel

--
http://www.danielnaber.de

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene


otis_gospodnetic at yahoo

Jun 3, 2005, 3:29 PM

Post #6 of 7 (100 views)
Permalink
Re: patch: adding IndexReader.isCurrent() [In reply to]

Well, Radomir, the person who asked about that method should be on
java-dev@, so I'll assume he'll speak up if he needs that method. I
think isCurrent() is good to go.

Otis


--- Daniel Naber <lucenelist [at] danielnaber> wrote:

> On Friday 03 June 2005 23:51, Otis Gospodnetic wrote:
>
> > Isn't version a time stamp now?
>
> It starts as a time stamp, but then +1 is added for each change. So
> that
> number has no useful meaning I think. It's only useful for
> comparison. Or
> am I missing something?
>
> Regards
> Daniel
>
> --
> http://www.danielnaber.de
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
> For additional commands, e-mail: java-dev-help [at] lucene
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene


radomirml at gmail

Jun 6, 2005, 12:38 AM

Post #7 of 7 (101 views)
Permalink
Re: patch: adding IndexReader.isCurrent() [In reply to]

Method isCurrrent() is a useful addition and I'm glad fix with
timestamp use for initial version is finally commited. However,
although version is not meaningful, I believe it should stay in API...

The software I'm working on is using pair "version, document_id" as a
fast reference to lucene document - if version is not current we have
to make an additional query first in order to get document id. Without
version information method isCurrent() is useless in this context as
we don't know which version of search results client has.

Here's one simple example... Let's say we have web application that's
paging search results (20 or so per page) and we re-run query for each
"next results page" request. Without version information, how can we
eventually warn user that index changed between its two "next results
page" requests?

So I think getCurentVersion() should stay and (as I suggested earler)
would like to see getVersion() added to IndexReader.

In addition, shouldn't SigmentInfos.readCurrentVersion() rely on
commit lock when reading segments file? Could someone with deeper
understanding of lucene internals check this?

Radomir


On 6/4/05, Otis Gospodnetic <otis_gospodnetic [at] yahoo> wrote:
> Well, Radomir, the person who asked about that method should be on
> java-dev@, so I'll assume he'll speak up if he needs that method. I
> think isCurrent() is good to go.
>
> Otis
>
>
> --- Daniel Naber <lucenelist [at] danielnaber> wrote:
>
> > On Friday 03 June 2005 23:51, Otis Gospodnetic wrote:
> >
> > > Isn't version a time stamp now?
> >
> > It starts as a time stamp, but then +1 is added for each change. So
> > that
> > number has no useful meaning I think. It's only useful for
> > comparison. Or
> > am I missing something?
> >
> > Regards
> > Daniel
> >
> > --
> > http://www.danielnaber.de
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
> > For additional commands, e-mail: java-dev-help [at] lucene
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
> For additional commands, e-mail: java-dev-help [at] lucene
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

Lucene java-dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.