Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Lucene: Java-Dev

Bug in DocInvertedPerField?

 

 

Lucene java-dev RSS feed   Index | Next | Previous | View Threaded


serera at gmail

Jul 2, 2009, 7:25 AM

Post #1 of 5 (352 views)
Permalink
Bug in DocInvertedPerField?

I hit NPE in DocInvertedPerField in the following scenario:
1) Create IndexWriter w/ a null Analyzer.
2) Add two fields to a Document, w/ a TokenStream (or one as not analyzed
and one as TokenStream, or two not analyzed).
3) Add the document to the IndexWriter.
The NPE comes from processFields (line 79), when it tries to read the
position increment gap.

I think I'm using it right, but want to verify. If I always add fields as
TokenStream, or untokenized, I don't really need the Analyzer on
IndexWriter. Also, in case of a TokenStream field, DocInverterPerField just
uses the TokenStream and not the Analyzer. So maybe it should read the
posIncrGap only if analyzer is not null?

Is this a bug or a misuse of IndexWriter by me?

Shai


lucene at mikemccandless

Jul 6, 2009, 4:05 AM

Post #2 of 5 (292 views)
Permalink
Re: Bug in DocInvertedPerField? [In reply to]

Were the two fields that you added to the doc the same field name?

In which case, the pos incr gap is in fact needed, even if the fields
are pre-analyzed (have TokenStream values)?

Mike

On Thu, Jul 2, 2009 at 10:25 AM, Shai Erera<serera [at] gmail> wrote:
> I hit NPE in DocInvertedPerField in the following scenario:
> 1) Create IndexWriter w/ a null Analyzer.
> 2) Add two fields to a Document, w/ a TokenStream (or one as not analyzed
> and one as TokenStream, or two not analyzed).
> 3) Add the document to the IndexWriter.
> The NPE comes from processFields (line 79), when it tries to read the
> position increment gap.
>
> I think I'm using it right, but want to verify. If I always add fields as
> TokenStream, or untokenized, I don't really need the Analyzer on
> IndexWriter. Also, in case of a TokenStream field, DocInverterPerField just
> uses the TokenStream and not the Analyzer. So maybe it should read the
> posIncrGap only if analyzer is not null?
>
> Is this a bug or a misuse of IndexWriter by me?
>
> Shai
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene


serera at gmail

Jul 6, 2009, 4:12 AM

Post #3 of 5 (293 views)
Permalink
Re: Bug in DocInvertedPerField? [In reply to]

Yes they have the same field name.
Can we use the default posIncr? If I want to create an IndexWriter w/o an
Analyzer, why should I be forced to do new IndexWriter(new SimpleAnalyzer()
/* for example */ ...), when the analyzer will never be used?

It is an edge case though which I can easily reproduce, but not sure how
important it is to fix it (the fix should be simple though).

Shai

On Mon, Jul 6, 2009 at 2:05 PM, Michael McCandless <
lucene [at] mikemccandless> wrote:

> Were the two fields that you added to the doc the same field name?
>
> In which case, the pos incr gap is in fact needed, even if the fields
> are pre-analyzed (have TokenStream values)?
>
> Mike
>
> On Thu, Jul 2, 2009 at 10:25 AM, Shai Erera<serera [at] gmail> wrote:
> > I hit NPE in DocInvertedPerField in the following scenario:
> > 1) Create IndexWriter w/ a null Analyzer.
> > 2) Add two fields to a Document, w/ a TokenStream (or one as not analyzed
> > and one as TokenStream, or two not analyzed).
> > 3) Add the document to the IndexWriter.
> > The NPE comes from processFields (line 79), when it tries to read the
> > position increment gap.
> >
> > I think I'm using it right, but want to verify. If I always add fields as
> > TokenStream, or untokenized, I don't really need the Analyzer on
> > IndexWriter. Also, in case of a TokenStream field, DocInverterPerField
> just
> > uses the TokenStream and not the Analyzer. So maybe it should read the
> > posIncrGap only if analyzer is not null?
> >
> > Is this a bug or a misuse of IndexWriter by me?
> >
> > Shai
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
> For additional commands, e-mail: java-dev-help [at] lucene
>
>


yonik at lucidimagination

Jul 6, 2009, 7:22 AM

Post #4 of 5 (291 views)
Permalink
Re: Bug in DocInvertedPerField? [In reply to]

On Mon, Jul 6, 2009 at 7:12 AM, Shai Erera<serera [at] gmail> wrote:
> If I want to create an IndexWriter w/o an
> Analyzer, why should I be forced to do new IndexWriter(new SimpleAnalyzer()

Passing an Analyzer really doesn't seem like a hardship... it's the
current interface that defines analysis, and it would complicate both
code and semantics to start creating defaults when the analyzer==null.

-Yonik
http://www.lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene


serera at gmail

Jul 6, 2009, 9:06 AM

Post #5 of 5 (291 views)
Permalink
Re: Bug in DocInvertedPerField? [In reply to]

Ok. BTW, maybe we want to ensure then that the Analyzer passed to
IndexWriter is not null, since it looks to be a required argument, unless I
always addDocument w/ an Analyzer.

Thanks for the replies guys.

Shai

On Mon, Jul 6, 2009 at 5:22 PM, Yonik Seeley <yonik [at] lucidimagination>wrote:

> On Mon, Jul 6, 2009 at 7:12 AM, Shai Erera<serera [at] gmail> wrote:
> > If I want to create an IndexWriter w/o an
> > Analyzer, why should I be forced to do new IndexWriter(new
> SimpleAnalyzer()
>
> Passing an Analyzer really doesn't seem like a hardship... it's the
> current interface that defines analysis, and it would complicate both
> code and semantics to start creating defaults when the analyzer==null.
>
> -Yonik
> http://www.lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
> For additional commands, e-mail: java-dev-help [at] lucene
>
>

Lucene java-dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.