Mailing List Archive: Lucene: Java-User

Scoring for all the documents in the index relative to a query

Index | Next | Previous | View Threaded

haiduc_sonia at yahoo

Nov 19, 2007, 8:05 AM

Post #1 of 14 (514 views)
Permalink

Scoring for all the documents in the index relative to a query

Hi everyone,

I am trying to obtain the score for each document in the index relative to a given query. For example, if I have the query "search file", I am trying to get the list of all documents in the index and their scores relative to the given query. I tried first using Hits, which gave me the normalized score. I thought that I don't see the whole list of documents and their scores because of the normalization, so I tried using HitsCollector. But even after using HitsCollector, I get the same number of matching documents, so the normalization didn't exclude documents because of negative scoring. Does Lucene actually compute the score for all the documents in the index or just for matching documents? I really need to have the scores for all the documents in the index relative to the query (even if negative), not just the ones that contain the query terms(this is what Lucene considers "matching documents", right?). Is this possible using Lucene?

I really appreciate your time and effort!
Thanks,
Sonia

____________________________________________________________________________________
Get easy, one-click access to your favorites.
Make Yahoo! your homepage.
http://www.yahoo.com/r/hs

erickerickson at gmail

Nov 19, 2007, 8:55 AM

Post #2 of 14 (493 views)
Permalink

Re: Scoring for all the documents in the index relative to a query [In reply to]

Could you explain a bit more what problem you're trying to solve?
The reason I ask is that your question doesn't make sense to me,
since I have no idea what you expect by the term "negative score".

My simplistic view has been that all the docs returned via Hits
or HitCollector have scores > 0, and all the rest have scores of 0,
and this view is supported by the explanation of
HitCollector.collect

" Called once for every non-zero scoring document, with the
document number and its score."

You might also get value from this page:
http://lucene.apache.org/java/docs/scoring.html#Scoring

Best
Erick

On Nov 19, 2007 11:05 AM, HAIDUC SONIA <haiduc_sonia[at]yahoo.com> wrote:

> Hi everyone,
>
> I am trying to obtain the score for each document in the index relative to
> a given query. For example, if I have the query "search file", I am trying
> to get the list of all documents in the index and their scores relative to
> the given query. I tried first using Hits, which gave me the normalized
> score. I thought that I don't see the whole list of documents and their
> scores because of the normalization, so I tried using HitsCollector. But
> even after using HitsCollector, I get the same number of matching documents,
> so the normalization didn't exclude documents because of negative scoring.
> Does Lucene actually compute the score for all the documents in the index or
> just for matching documents? I really need to have the scores for all the
> documents in the index relative to the query (even if negative), not just
> the ones that contain the query terms(this is what Lucene considers
> "matching documents", right?). Is this possible using Lucene?
>
> I really appreciate your time and effort!
> Thanks,
> Sonia
>
>
>
>
>
> ____________________________________________________________________________________
> Get easy, one-click access to your favorites.
> Make Yahoo! your homepage.
> http://www.yahoo.com/r/hs
>

haiduc_sonia at yahoo

Nov 19, 2007, 10:25 AM

Post #3 of 14 (492 views)
Permalink

Re: Scoring for all the documents in the index relative to a query [In reply to]

I am trying to order all the documents in the index according to their similarity to a given query. I am interested in having a complete list of *all* the documents in the index with their score. From what I understood by reading some documentation, Lucene internally assigns scores to all the documents in the index according to their similarity to the query, but when returning the hits, all the scores that are less than 0 are rounded to 0 and only the documents with the score > 0 are returned as hits. But what I would like to get is the list before this intermediate processing, so the list of all the documents with their raw score. I am trying to compare Lucene with LSI and for the comparison I want to do, I need the entire list of documents. Is there a way that I can get that with Lucene?
I hope I explained it clearly this time. If you need more details let me know.

Thank you,
Sonia

----- Original Message ----
From: Erick Erickson <erickerickson[at]gmail.com>
To: java-user[at]lucene.apache.org
Sent: Monday, November 19, 2007 11:55:00 AM
Subject: Re: Scoring for all the documents in the index relative to a query

Could you explain a bit more what problem you're trying to solve?
The reason I ask is that your question doesn't make sense to me,
since I have no idea what you expect by the term "negative score".

My simplistic view has been that all the docs returned via Hits
or HitCollector have scores > 0, and all the rest have scores of 0,
and this view is supported by the explanation of
HitCollector.collect

" Called once for every non-zero scoring document, with the
document number and its score."

You might also get value from this page:
http://lucene.apache.org/java/docs/scoring.html#Scoring

Best
Erick

On Nov 19, 2007 11:05 AM, HAIDUC SONIA <haiduc_sonia[at]yahoo.com> wrote:

> Hi everyone,
>
> I am trying to obtain the score for each document in the index
relative to
> a given query. For example, if I have the query "search file", I am
trying
> to get the list of all documents in the index and their scores
relative to
> the given query. I tried first using Hits, which gave me the
normalized
> score. I thought that I don't see the whole list of documents and
their
> scores because of the normalization, so I tried using HitsCollector.
But
> even after using HitsCollector, I get the same number of matching
documents,
> so the normalization didn't exclude documents because of negative
scoring.
> Does Lucene actually compute the score for all the documents in the
index or
> just for matching documents? I really need to have the scores for all
the
> documents in the index relative to the query (even if negative), not
just
> the ones that contain the query terms(this is what Lucene considers
> "matching documents", right?). Is this possible using Lucene?
>
> I really appreciate your time and effort!
> Thanks,
> Sonia
>
>
>
>
>
>
____________________________________________________________________________________
> Get easy, one-click access to your favorites.
> Make Yahoo! your homepage.
> http://www.yahoo.com/r/hs
>

____________________________________________________________________________________
Get easy, one-click access to your favorites.
Make Yahoo! your homepage.
http://www.yahoo.com/r/hs

haiduc_sonia at yahoo

Nov 19, 2007, 10:25 AM

Post #4 of 14 (494 views)
Permalink

Re: Scoring for all the documents in the index relative to a query [In reply to]

I am trying to order all the documents in the index according to their similarity to a given query. I am interested in having a complete list of *all* the documents in the index with their score. From what I understood by reading some documentation, Lucene internally assigns scores to all the documents in the index according to their similarity to the query, but when returning the hits, all the scores that are less than 0 are rounded to 0 and only the documents with the score > 0 are returned as hits. But what I would like to get is the list before this intermediate processing, so the list of all the documents with their raw score. I am trying to compare Lucene with LSI and for the comparison I want to do, I need the entire list of documents. Is there a way that I can get that with Lucene?
I hope I explained it clearly this time. If you need more details let me know.

Thank you,
Sonia

----- Original Message ----
From: Erick Erickson <erickerickson[at]gmail.com>
To: java-user[at]lucene.apache.org
Sent: Monday, November 19, 2007 11:55:00 AM
Subject: Re: Scoring for all the documents in the index relative to a query

Could you explain a bit more what problem you're trying to solve?
The reason I ask is that your question doesn't make sense to me,
since I have no idea what you expect by the term "negative score".

My simplistic view has been that all the docs returned via Hits
or HitCollector have scores > 0, and all the rest have scores of 0,
and this view is supported by the explanation of
HitCollector.collect

" Called once for every non-zero scoring document, with the
document number and its score."

You might also get value from this page:
http://lucene.apache.org/java/docs/scoring.html#Scoring

Best
Erick

On Nov 19, 2007 11:05 AM, HAIDUC SONIA <haiduc_sonia[at]yahoo.com> wrote:

> Hi everyone,
>
> I am trying to obtain the score for each document in the index
relative to
> a given query. For example, if I have the query "search file", I am
trying
> to get the list of all documents in the index and their scores
relative to
> the given query. I tried first using Hits, which gave me the
normalized
> score. I thought that I don't see the whole list of documents and
their
> scores because of the normalization, so I tried using HitsCollector.
But
> even after using HitsCollector, I get the same number of matching
documents,
> so the normalization didn't exclude documents because of negative
scoring.
> Does Lucene actually compute the score for all the documents in the
index or
> just for matching documents? I really need to have the scores for all
the
> documents in the index relative to the query (even if negative), not
just
> the ones that contain the query terms(this is what Lucene considers
> "matching documents", right?). Is this possible using Lucene?
>
> I really appreciate your time and effort!
> Thanks,
> Sonia
>
>
>
>
>
>
____________________________________________________________________________________
> Get easy, one-click access to your favorites.
> Make Yahoo! your homepage.
> http://www.yahoo.com/r/hs
>

____________________________________________________________________________________
Never miss a thing. Make Yahoo your home page.
http://www.yahoo.com/r/hs

buschmic at gmail

Nov 19, 2007, 10:32 AM

Post #5 of 14 (493 views)
Permalink

Re: Scoring for all the documents in the index relative to a query [In reply to]

Hi Sonia,

I agree with Erick here. Negative scores don't make sense and Lucene
never computes scores for documents that don't match a query.

E. g. if your query is: "term1 OR term2", then every document that
contains term1 or term2 or both will have a score greater than 0. But if
two docs don't contain either of the two terms, why should one of them
have a lower score than the other one? Both would have 0.

-Michael

HAIDUC SONIA wrote:
> I am trying to order all the documents in the index according to their similarity to a given query. I am interested in having a complete list of *all* the documents in the index with their score. From what I understood by reading some documentation, Lucene internally assigns scores to all the documents in the index according to their similarity to the query, but when returning the hits, all the scores that are less than 0 are rounded to 0 and only the documents with the score > 0 are returned as hits. But what I would like to get is the list before this intermediate processing, so the list of all the documents with their raw score. I am trying to compare Lucene with LSI and for the comparison I want to do, I need the entire list of documents. Is there a way that I can get that with Lucene?
> I hope I explained it clearly this time. If you need more details let me know.
>
> Thank you,
> Sonia
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

gsingers at apache

Nov 19, 2007, 10:38 AM

Post #6 of 14 (493 views)
Permalink

Re: Scoring for all the documents in the index relative to a query [In reply to]

Lucene only scores those documents that have at least one match term,
it doesn't implement a pure vector space model whereby all documents
are scored (it uses a combination of the Boolean Model and VSM).
Thus, I am not sure you can do a pure comparison. I suppose you could
simulating the relevance by using TermVectors and looping over all
documents, but I think one could argue this isn't exactly what Lucene
does, so it isn't comparable.

http://lucene.apache.org/java/docs/scoring.html might help in
understanding this stuff.

HTH,
Grant

On Nov 19, 2007, at 1:25 PM, HAIDUC SONIA wrote:

> I am trying to order all the documents in the index according to
> their similarity to a given query. I am interested in having a
> complete list of *all* the documents in the index with their score.
> From what I understood by reading some documentation, Lucene
> internally assigns scores to all the documents in the index
> according to their similarity to the query, but when returning the
> hits, all the scores that are less than 0 are rounded to 0 and only
> the documents with the score > 0 are returned as hits. But what I
> would like to get is the list before this intermediate processing,
> so the list of all the documents with their raw score. I am trying
> to compare Lucene with LSI and for the comparison I want to do, I
> need the entire list of documents. Is there a way that I can get
> that with Lucene?
> I hope I explained it clearly this time. If you need more details
> let me know.
>
> Thank you,
> Sonia
>
> ----- Original Message ----
> From: Erick Erickson <erickerickson[at]gmail.com>
> To: java-user[at]lucene.apache.org
> Sent: Monday, November 19, 2007 11:55:00 AM
> Subject: Re: Scoring for all the documents in the index relative to
> a query
>
>
> Could you explain a bit more what problem you're trying to solve?
> The reason I ask is that your question doesn't make sense to me,
> since I have no idea what you expect by the term "negative score".
>
> My simplistic view has been that all the docs returned via Hits
> or HitCollector have scores > 0, and all the rest have scores of 0,
> and this view is supported by the explanation of
> HitCollector.collect
>
> " Called once for every non-zero scoring document, with the
> document number and its score."
>
> You might also get value from this page:
> http://lucene.apache.org/java/docs/scoring.html#Scoring
>
> Best
> Erick
>
> On Nov 19, 2007 11:05 AM, HAIDUC SONIA <haiduc_sonia[at]yahoo.com> wrote:
>
>> Hi everyone,
>>
>> I am trying to obtain the score for each document in the index
> relative to
>> a given query. For example, if I have the query "search file", I am
> trying
>> to get the list of all documents in the index and their scores
> relative to
>> the given query. I tried first using Hits, which gave me the
> normalized
>> score. I thought that I don't see the whole list of documents and
> their
>> scores because of the normalization, so I tried using HitsCollector.
> But
>> even after using HitsCollector, I get the same number of matching
> documents,
>> so the normalization didn't exclude documents because of negative
> scoring.
>> Does Lucene actually compute the score for all the documents in the
> index or
>> just for matching documents? I really need to have the scores for all
> the
>> documents in the index relative to the query (even if negative), not
> just
>> the ones that contain the query terms(this is what Lucene considers
>> "matching documents", right?). Is this possible using Lucene?
>>
>> I really appreciate your time and effort!
>> Thanks,
>> Sonia
>>
>>
>>
>>
>>
>>
>
> ____________________________________________________________________________________
>> Get easy, one-click access to your favorites.
>> Make Yahoo! your homepage.
>> http://www.yahoo.com/r/hs
>>
>
>
>
>
>
>
>
> ____________________________________________________________________________________
> Get easy, one-click access to your favorites.
> Make Yahoo! your homepage.
> http://www.yahoo.com/r/hs

--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

gresh at us

Nov 19, 2007, 10:45 AM

Post #7 of 14 (493 views)
Permalink

Re: Scoring for all the documents in the index relative to a query [In reply to]

I could be mistaken, but I think the earlier answer was right; a document
with no terms matching has a score of 0, so you can assume that all
documents NOT returned by the query have a score of 0. If you look at the
scoring formula on this page, it is hard to see how you can get a negative
score.

http://lucene.zones.apache.org:8080/hudson/job/Lucene-Nightly/javadoc/org/apache/lucene/search/Similarity.html

HAIDUC SONIA <haiduc_sonia[at]yahoo.com> wrote on 11/19/2007 01:25:13 PM:

> I am trying to order all the documents in the index according to
> their similarity to a given query. I am interested in having a
> complete list of *all* the documents in the index with their score.
> From what I understood by reading some documentation, Lucene
> internally assigns scores to all the documents in the index
> according to their similarity to the query, but when returning the
> hits, all the scores that are less than 0 are rounded to 0 and only
> the documents with the score > 0 are returned as hits. But what I
> would like to get is the list before this intermediate processing,
> so the list of all the documents with their raw score. I am trying
> to compare Lucene with LSI and for the comparison I want to do, I
> need the entire list of documents. Is there a way that I can get
> that with Lucene?
> I hope I explained it clearly this time. If you need more details let me
know.
>
> Thank you,
> Sonia
>
> ----- Original Message ----
> From: Erick Erickson <erickerickson[at]gmail.com>
> To: java-user[at]lucene.apache.org
> Sent: Monday, November 19, 2007 11:55:00 AM
> Subject: Re: Scoring for all the documents in the index relative to a
query
>
>
> Could you explain a bit more what problem you're trying to solve?
> The reason I ask is that your question doesn't make sense to me,
> since I have no idea what you expect by the term "negative score".
>
> My simplistic view has been that all the docs returned via Hits
> or HitCollector have scores > 0, and all the rest have scores of 0,
> and this view is supported by the explanation of
> HitCollector.collect
>
> " Called once for every non-zero scoring document, with the
> document number and its score."
>
> You might also get value from this page:
> http://lucene.apache.org/java/docs/scoring.html#Scoring
>
> Best
> Erick
>
> On Nov 19, 2007 11:05 AM, HAIDUC SONIA <haiduc_sonia[at]yahoo.com> wrote:
>
> > Hi everyone,
> >
> > I am trying to obtain the score for each document in the index
> relative to
> > a given query. For example, if I have the query "search file", I am
> trying
> > to get the list of all documents in the index and their scores
> relative to
> > the given query. I tried first using Hits, which gave me the
> normalized
> > score. I thought that I don't see the whole list of documents and
> their
> > scores because of the normalization, so I tried using HitsCollector.
> But
> > even after using HitsCollector, I get the same number of matching
> documents,
> > so the normalization didn't exclude documents because of negative
> scoring.
> > Does Lucene actually compute the score for all the documents in the
> index or
> > just for matching documents? I really need to have the scores for all
> the
> > documents in the index relative to the query (even if negative), not
> just
> > the ones that contain the query terms(this is what Lucene considers
> > "matching documents", right?). Is this possible using Lucene?
> >
> > I really appreciate your time and effort!
> > Thanks,
> > Sonia
> >
> >
> >
> >
> >
> >
>
>
____________________________________________________________________________________
> > Get easy, one-click access to your favorites.
> > Make Yahoo! your homepage.
> > http://www.yahoo.com/r/hs
> >
>
>
>
>
>
>
>
>
____________________________________________________________________________________
> Never miss a thing. Make Yahoo your home page.
> http://www.yahoo.com/r/hs

haiduc_sonia at yahoo

Nov 19, 2007, 10:54 AM

Post #8 of 14 (492 views)
Permalink

Re: Scoring for all the documents in the index relative to a query [In reply to]

Thank you guys for your prompt answers. I'm a beginner with Lucene and I still had some unclarities regarding its scoring function. Your answers really cleared things up for me. I guess a direct comparison with LSI is not possible after all, only the comparison between LSI and the pure VSM.

Thanks again,
Sonia

____________________________________________________________________________________
Be a better pen pal.
Text or chat with friends inside Yahoo! Mail. See how. http://overview.mail.yahoo.com/

hossman_lucene at fucit

Nov 19, 2007, 2:03 PM

Post #9 of 14 (492 views)
Permalink

Re: Scoring for all the documents in the index relative to a query [In reply to]

: My simplistic view has been that all the docs returned via Hits
: or HitCollector have scores > 0, and all the rest have scores of 0,
: and this view is supported by the explanation of
: HitCollector.collect
:
: " Called once for every non-zero scoring document, with the
: document number and its score."

(Hmmm... i thought those javadocs had been fixed a while back? oh well,
i just did: r596462)

Strictly speaking there's nothing to stop a document from having a zero or
negative *raw* score for a query (ie: a ValueSourceQuery for example) it's
just not common.

the HitCollector.collect method is called for every *matching* document,
with it's raw score.

(I'm not actually sure how the Hits class treats negative values ... odds
are no one using the Hits class is going to be dealing with a query that
can product negative scores though)

-Hoss

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

yonik at apache

Nov 19, 2007, 2:35 PM

Post #10 of 14 (493 views)
Permalink

Re: Scoring for all the documents in the index relative to a query [In reply to]

On Nov 19, 2007 5:03 PM, Chris Hostetter <hossman_lucene[at]fucit.org> wrote:
> (I'm not actually sure how the Hits class treats negative values

All Lucene search methods except ones that take a HitCollector filter
out final scores <= 0

Solr does allow scores <=0 through since it had different collection
methods to avoid score normalization (back when Lucene still did it
for TopDocs).

-Yonik

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

paul.elschot at xs4all

Nov 19, 2007, 3:09 PM

Post #11 of 14 (491 views)
Permalink

Re: Scoring for all the documents in the index relative to a query [In reply to]

Gentlefolk,

Well, the javadocs as patched at LUCENE-584 try to change all
the cases of zero scoring to 'non matching'.

I'm happily bracing for a minor conflict with that patch. In case
someone wants to take another look at the javadocs as
patched there, don't let me stop you...

Regards,
Paul Elschot

On Monday 19 November 2007 23:35:07 Yonik Seeley wrote:
> On Nov 19, 2007 5:03 PM, Chris Hostetter <hossman_lucene[at]fucit.org> wrote:
> > (I'm not actually sure how the Hits class treats negative values
>
> All Lucene search methods except ones that take a HitCollector filter
> out final scores <= 0
>
> Solr does allow scores <=0 through since it had different collection
> methods to avoid score normalization (back when Lucene still did it
> for TopDocs).
>
> -Yonik
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-user-help[at]lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

DORONC at il

Nov 20, 2007, 4:23 AM

Post #12 of 14 (476 views)
Permalink

Re: Scoring for all the documents in the index relative to a query [In reply to]

You can also rely on that by default documents are
collected in-docid-order. You can therefore use your own
hit collector that when collecting doc with id n2,
assuming the previous doc collected had id n1,
would (know to) assign score 0 to all docs
with: n1 < id < n2.

In other words, you can know earlier that a certain
doc is a non-match, not having to wait until all
matching docs were collected. (well, at least I had
the impression that something like this might help you.)

Doron

HAIDUC SONIA wrote:
> I am trying to order all the documents in the index according
> to their similarity to a given query. I am interested in having
> a complete list of *all* the documents in the index with their
> score. From what I understood by reading some documentation,
> Lucene internally assigns scores to all the documents in the
> index according to their similarity to the query, but when
> returning the hits, all the scores that are less than 0 are
> rounded to 0 and only the documents with the score > 0 are
> returned as hits. But what I would like to get is the list
> before this intermediate processing, so the list of all the
> documents with their raw score. I am trying to compare Lucene
> with LSI and for the comparison I want to do, I need the entire
> list of documents. Is there a way that I can get that with Lucene?
> I hope I explained it clearly this time. If you need more
> details let me know.
>
> Thank you,
> Sonia

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

hossman_lucene at fucit

Nov 20, 2007, 9:33 AM

Post #13 of 14 (475 views)
Permalink

Re: Scoring for all the documents in the index relative to a query [In reply to]

: Well, the javadocs as patched at LUCENE-584 try to change all
: the cases of zero scoring to 'non matching'.

I'm very out of the loop on LUCENE-584, but i think supporting scores <= 0
is an important use case that shouldn't go away ... it makes
CustomScoringQueries a lot morepowerful, and easier to understand.

if nothing else, we want to ensure that
if...
search("+foo^0 bar baz yak") => returns X results
then...
search("foo^0") => returns the same X results even if all scores are 0

-Hoss

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

paul.elschot at xs4all

Nov 20, 2007, 3:09 PM

Post #14 of 14 (473 views)
Permalink

Re: Scoring for all the documents in the index relative to a query [In reply to]

On Tuesday 20 November 2007 18:33:26 Chris Hostetter wrote:
> : Well, the javadocs as patched at LUCENE-584 try to change all
> : the cases of zero scoring to 'non matching'.
>
> I'm very out of the loop on LUCENE-584, but i think supporting scores <= 0
> is an important use case that shouldn't go away ... it makes
> CustomScoringQueries a lot morepowerful, and easier to understand.
>
> if nothing else, we want to ensure that
> if...
> search("+foo^0 bar baz yak") => returns X results
> then...
> search("foo^0") => returns the same X results even if all scores are 0

In the patch, Matcher is a superclass of Scorer and it does not have the
score() method, so 'matching' is independent of the any score value.
The matching docs are determined by the next(), skipTo() and doc() methods
only.

Regards,
Paul Elschot

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com