Mailing List Archive: Lucene: Java-User

Highligheter fails using JapaneseAnalyzer

Index | Next | Previous | View Threaded

sake-ginjyo at nifty

Jun 30, 2009, 7:16 AM

Post #1 of 10 (661 views)
Permalink

Highligheter fails using JapaneseAnalyzer

hello.

i've tried to highlight string using Highligheter(2.4.1) and
JapaneseAnalyzer
but the following code extract show the problem

String F = "f";
String CONTENTS = "AAA :BBB CCC";
JapaneseAnalyzer analyzer = new JapaneseAnalyzer();
QueryParser qp = new QueryParser( F, analyzer );
Query query = qp.parse( "BBB" );
Highlighter h = new Highlighter( new QueryScorer( query, F ) );

System.out.println( h.getBestFragment( analyzer, F, CONTENTS ) );

The sytsem outputs
AAA :BBB CCC

When you change CONTENTS to "AAA _BBB CCC"
the system outputs

AAA _BBB CCC

Are there any problems?
Thanks in advance

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

mhall at informatics

Jun 30, 2009, 7:41 AM

Post #2 of 10 (629 views)
Permalink

Re: Highligheter fails using JapaneseAnalyzer [In reply to]

Does the same thing happen when you use SimpleAnalyzer, or StandardAnalyzer?

I have a sneaking suspicion that the : in your contents string is what's
causing your issue here, as : is a reserved character that denotes a
field specification. But I could be wrong.

Try swapping analyzers, if you no longer have the same issue with
Simple, try Standard. Assuming the same problem shows up there, I think
you might need to do something about the :.

Matt

k.sayama wrote:
> hello.
>
> i've tried to highlight string using Highligheter(2.4.1) and
> JapaneseAnalyzer
> but the following code extract show the problem
>
> String F = "f";
> String CONTENTS = "AAA :BBB CCC";
> JapaneseAnalyzer analyzer = new JapaneseAnalyzer();
> QueryParser qp = new QueryParser( F, analyzer );
> Query query = qp.parse( "BBB" );
> Highlighter h = new Highlighter( new QueryScorer( query, F ) );
>
> System.out.println( h.getBestFragment( analyzer, F, CONTENTS ) );
>
> The sytsem outputs
> AAA :BBB CCC
>
> When you change CONTENTS to "AAA _BBB CCC"
> the system outputs
>
> AAA _BBB CCC
>
> Are there any problems?
> Thanks in advance
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-user-help[at]lucene.apache.org
>

--
Matthew Hall
Software Engineer
Mouse Genome Informatics
mhall[at]informatics.jax.org
(207) 288-6012

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

sake-ginjyo at nifty

Jul 1, 2009, 7:22 AM

Post #3 of 10 (607 views)
Permalink

Re: Highligheter fails using JapaneseAnalyzer [In reply to]

hi

I verified it by using SimpleAnalyzer, StandardAnalyzer, and CJKAnalyzer.
but, The problem did not happen.

I think the problem of JapaneseAnalyzer.
Can this problem be solved?

> Does the same thing happen when you use SimpleAnalyzer, or
> StandardAnalyzer?
>
> I have a sneaking suspicion that the : in your contents string is what's
> causing your issue here, as : is a reserved character that denotes a
> field specification. But I could be wrong.
>
> Try swapping analyzers, if you no longer have the same issue with
> Simple, try Standard. Assuming the same problem shows up there, I think
> you might need to do something about the :.
>
> Matt
>
> k.sayama wrote:
>> hello.
>>
>> i've tried to highlight string using Highligheter(2.4.1) and
>> JapaneseAnalyzer
>> but the following code extract show the problem
>>
>> String F = "f";
>> String CONTENTS = "AAA :BBB CCC";
>> JapaneseAnalyzer analyzer = new JapaneseAnalyzer();
>> QueryParser qp = new QueryParser( F, analyzer );
>> Query query = qp.parse( "BBB" );
>> Highlighter h = new Highlighter( new QueryScorer( query, F ) );
>>
>> System.out.println( h.getBestFragment( analyzer, F, CONTENTS ) );
>>
>> The sytsem outputs
>> AAA :BBB CCC
>>
>> When you change CONTENTS to "AAA _BBB CCC"
>> the system outputs
>>
>> AAA _BBB CCC
>>
>> Are there any problems?
>> Thanks in advance
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
>> For additional commands, e-mail: java-user-help[at]lucene.apache.org
>>
>
>
> --
> Matthew Hall
> Software Engineer
> Mouse Genome Informatics
> mhall[at]informatics.jax.org
> (207) 288-6012
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-user-help[at]lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

markharw00d at yahoo

Jul 1, 2009, 7:31 AM

Post #4 of 10 (607 views)
Permalink

Re: Highligheter fails using JapaneseAnalyzer [In reply to]

Can you verify the Token byte offsets produced by this particular analyzer are correct?

----- Original Message ----
From: k.sayama <sake-ginjyo[at]nifty.com>
To: java-user[at]lucene.apache.org
Sent: Wednesday, 1 July, 2009 15:22:37
Subject: Re: Highligheter fails using JapaneseAnalyzer

hi

I verified it by using SimpleAnalyzer, StandardAnalyzer, and CJKAnalyzer.
but, The problem did not happen.

I think the problem of JapaneseAnalyzer.
Can this problem be solved?

> Does the same thing happen when you use SimpleAnalyzer, or
> StandardAnalyzer?
>
> I have a sneaking suspicion that the : in your contents string is what's
> causing your issue here, as : is a reserved character that denotes a
> field specification. But I could be wrong.
>
> Try swapping analyzers, if you no longer have the same issue with
> Simple, try Standard. Assuming the same problem shows up there, I think
> you might need to do something about the :.
>
> Matt
>
> k.sayama wrote:
>> hello.
>>
>> i've tried to highlight string using Highligheter(2.4.1) and
>> JapaneseAnalyzer
>> but the following code extract show the problem
>>
>> String F = "f";
>> String CONTENTS = "AAA :BBB CCC";
>> JapaneseAnalyzer analyzer = new JapaneseAnalyzer();
>> QueryParser qp = new QueryParser( F, analyzer );
>> Query query = qp.parse( "BBB" );
>> Highlighter h = new Highlighter( new QueryScorer( query, F ) );
>>
>> System.out.println( h.getBestFragment( analyzer, F, CONTENTS ) );
>>
>> The sytsem outputs
>> AAA :BBB CCC
>>
>> When you change CONTENTS to "AAA _BBB CCC"
>> the system outputs
>>
>> AAA _BBB CCC
>>
>> Are there any problems?
>> Thanks in advance
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
>> For additional commands, e-mail: java-user-help[at]lucene.apache.org
>>
>
>
> --
> Matthew Hall
> Software Engineer
> Mouse Genome Informatics
> mhall[at]informatics.jax.org
> (207) 288-6012
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-user-help[at]lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

sake-ginjyo at nifty

Jul 1, 2009, 8:13 AM

Post #5 of 10 (606 views)
Permalink

Re: Highligheter fails using JapaneseAnalyzer [In reply to]

Sorry
I can not verify the Token byte offsets produced by JapaneseAnalyzer
How should I verify it?

----- Original Message -----
From: "mark harwood" <markharw00d[at]yahoo.co.uk>
To: <java-user[at]lucene.apache.org>
Sent: Wednesday, July 01, 2009 11:31 PM
Subject: Re: Highligheter fails using JapaneseAnalyzer

Can you verify the Token byte offsets produced by this particular analyzer
are correct?

----- Original Message ----
From: k.sayama <sake-ginjyo[at]nifty.com>
To: java-user[at]lucene.apache.org
Sent: Wednesday, 1 July, 2009 15:22:37
Subject: Re: Highligheter fails using JapaneseAnalyzer

hi

I verified it by using SimpleAnalyzer, StandardAnalyzer, and CJKAnalyzer.
but, The problem did not happen.

I think the problem of JapaneseAnalyzer.
Can this problem be solved?

> Does the same thing happen when you use SimpleAnalyzer, or
> StandardAnalyzer?
>
> I have a sneaking suspicion that the : in your contents string is what's
> causing your issue here, as : is a reserved character that denotes a
> field specification. But I could be wrong.
>
> Try swapping analyzers, if you no longer have the same issue with
> Simple, try Standard. Assuming the same problem shows up there, I think
> you might need to do something about the :.
>
> Matt
>
> k.sayama wrote:
>> hello.
>>
>> i've tried to highlight string using Highligheter(2.4.1) and
>> JapaneseAnalyzer
>> but the following code extract show the problem
>>
>> String F = "f";
>> String CONTENTS = "AAA :BBB CCC";
>> JapaneseAnalyzer analyzer = new JapaneseAnalyzer();
>> QueryParser qp = new QueryParser( F, analyzer );
>> Query query = qp.parse( "BBB" );
>> Highlighter h = new Highlighter( new QueryScorer( query, F ) );
>>
>> System.out.println( h.getBestFragment( analyzer, F, CONTENTS ) );
>>
>> The sytsem outputs
>> AAA :BBB CCC
>>
>> When you change CONTENTS to "AAA _BBB CCC"
>> the system outputs
>>
>> AAA _BBB CCC
>>
>> Are there any problems?
>> Thanks in advance
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
>> For additional commands, e-mail: java-user-help[at]lucene.apache.org
>>
>
>
> --
> Matthew Hall
> Software Engineer
> Mouse Genome Informatics
> mhall[at]informatics.jax.org
> (207) 288-6012
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-user-help[at]lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

markharw00d at yahoo

Jul 1, 2009, 8:55 AM

Post #6 of 10 (607 views)
Permalink

Re: Highligheter fails using JapaneseAnalyzer [In reply to]

>>How should I verify it?

Make sure the Token.startOffset and endOffset properties of Tokens produced by your TokenStream correctly define the location of Token.termBuffer in the original text.

----- Original Message ----
From: k.sayama <sake-ginjyo[at]nifty.com>
To: java-user[at]lucene.apache.org
Sent: Wednesday, 1 July, 2009 16:13:17
Subject: Re: Highligheter fails using JapaneseAnalyzer

Sorry
I can not verify the Token byte offsets produced by JapaneseAnalyzer
How should I verify it?

----- Original Message -----
From: "mark harwood" <markharw00d[at]yahoo.co.uk>
To: <java-user[at]lucene.apache.org>
Sent: Wednesday, July 01, 2009 11:31 PM
Subject: Re: Highligheter fails using JapaneseAnalyzer

Can you verify the Token byte offsets produced by this particular analyzer
are correct?

----- Original Message ----
From: k.sayama <sake-ginjyo[at]nifty.com>
To: java-user[at]lucene.apache.org
Sent: Wednesday, 1 July, 2009 15:22:37
Subject: Re: Highligheter fails using JapaneseAnalyzer

hi

I verified it by using SimpleAnalyzer, StandardAnalyzer, and CJKAnalyzer.
but, The problem did not happen.

I think the problem of JapaneseAnalyzer.
Can this problem be solved?

> Does the same thing happen when you use SimpleAnalyzer, or
> StandardAnalyzer?
>
> I have a sneaking suspicion that the : in your contents string is what's
> causing your issue here, as : is a reserved character that denotes a
> field specification. But I could be wrong.
>
> Try swapping analyzers, if you no longer have the same issue with
> Simple, try Standard. Assuming the same problem shows up there, I think
> you might need to do something about the :.
>
> Matt
>
> k.sayama wrote:
>> hello.
>>
>> i've tried to highlight string using Highligheter(2.4.1) and
>> JapaneseAnalyzer
>> but the following code extract show the problem
>>
>> String F = "f";
>> String CONTENTS = "AAA :BBB CCC";
>> JapaneseAnalyzer analyzer = new JapaneseAnalyzer();
>> QueryParser qp = new QueryParser( F, analyzer );
>> Query query = qp.parse( "BBB" );
>> Highlighter h = new Highlighter( new QueryScorer( query, F ) );
>>
>> System.out.println( h.getBestFragment( analyzer, F, CONTENTS ) );
>>
>> The sytsem outputs
>> AAA :BBB CCC
>>
>> When you change CONTENTS to "AAA _BBB CCC"
>> the system outputs
>>
>> AAA _BBB CCC
>>
>> Are there any problems?
>> Thanks in advance
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
>> For additional commands, e-mail: java-user-help[at]lucene.apache.org
>>
>
>
> --
> Matthew Hall
> Software Engineer
> Mouse Genome Informatics
> mhall[at]informatics.jax.org
> (207) 288-6012
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-user-help[at]lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

sake-ginjyo at nifty

Jul 1, 2009, 9:39 AM

Post #7 of 10 (606 views)
Permalink

Re: Highligheter fails using JapaneseAnalyzer [In reply to]

I could verify Token byte offsets

The sytsem outputs
aaa:0:3
bbb:0:3
ccc:4:7

offset is initialized

Is this problem Analyzer?　Or, is it Tokenizer?

----- Original Message -----
From: "mark harwood" <markharw00d[at]yahoo.co.uk>
To: <java-user[at]lucene.apache.org>
Sent: Thursday, July 02, 2009 12:55 AM
Subject: Re: Highligheter fails using JapaneseAnalyzer

>>How should I verify it?

Make sure the Token.startOffset and endOffset properties of Tokens produced
by your TokenStream correctly define the location of Token.termBuffer in the
original text.

----- Original Message ----
From: k.sayama <sake-ginjyo[at]nifty.com>
To: java-user[at]lucene.apache.org
Sent: Wednesday, 1 July, 2009 16:13:17
Subject: Re: Highligheter fails using JapaneseAnalyzer

Sorry
I can not verify the Token byte offsets produced by JapaneseAnalyzer
How should I verify it?

----- Original Message -----
From: "mark harwood" <markharw00d[at]yahoo.co.uk>
To: <java-user[at]lucene.apache.org>
Sent: Wednesday, July 01, 2009 11:31 PM
Subject: Re: Highligheter fails using JapaneseAnalyzer

Can you verify the Token byte offsets produced by this particular analyzer
are correct?

----- Original Message ----
From: k.sayama <sake-ginjyo[at]nifty.com>
To: java-user[at]lucene.apache.org
Sent: Wednesday, 1 July, 2009 15:22:37
Subject: Re: Highligheter fails using JapaneseAnalyzer

hi

I verified it by using SimpleAnalyzer, StandardAnalyzer, and CJKAnalyzer.
but, The problem did not happen.

I think the problem of JapaneseAnalyzer.
Can this problem be solved?

> Does the same thing happen when you use SimpleAnalyzer, or
> StandardAnalyzer?
>
> I have a sneaking suspicion that the : in your contents string is what's
> causing your issue here, as : is a reserved character that denotes a
> field specification. But I could be wrong.
>
> Try swapping analyzers, if you no longer have the same issue with
> Simple, try Standard. Assuming the same problem shows up there, I think
> you might need to do something about the :.
>
> Matt
>
> k.sayama wrote:
>> hello.
>>
>> i've tried to highlight string using Highligheter(2.4.1) and
>> JapaneseAnalyzer
>> but the following code extract show the problem
>>
>> String F = "f";
>> String CONTENTS = "AAA :BBB CCC";
>> JapaneseAnalyzer analyzer = new JapaneseAnalyzer();
>> QueryParser qp = new QueryParser( F, analyzer );
>> Query query = qp.parse( "BBB" );
>> Highlighter h = new Highlighter( new QueryScorer( query, F ) );
>>
>> System.out.println( h.getBestFragment( analyzer, F, CONTENTS ) );
>>
>> The sytsem outputs
>> AAA :BBB CCC
>>
>> When you change CONTENTS to "AAA _BBB CCC"
>> the system outputs
>>
>> AAA _BBB CCC
>>
>> Are there any problems?
>> Thanks in advance
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
>> For additional commands, e-mail: java-user-help[at]lucene.apache.org
>>
>
>
> --
> Matthew Hall
> Software Engineer
> Mouse Genome Informatics
> mhall[at]informatics.jax.org
> (207) 288-6012
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-user-help[at]lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

markharw00d at yahoo

Jul 1, 2009, 2:25 PM

Post #8 of 10 (593 views)
Permalink

Re: Highligheter fails using JapaneseAnalyzer [In reply to]

On 1 Jul 2009, at 17:39, k.sayama wrote:

> I could verify Token byte offsets
>
> The sytsem outputs
> aaa:0:3
> bbb:0:3
> ccc:4:7
>

That explains the highlighter behaviour. Clearly BBB is not at
position 0-3 in the String you supplied

>>> String CONTENTS = "AAA :BBB CCC";

Looks like the Tokenizer needs fixing. Is this yours or a standard
Lucene class? If the latter, raising a JIRA bug with a Junit test
would be the best way to get things moving.

Cheers
Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

mhall at informatics

Jul 2, 2009, 5:59 AM

Post #9 of 10 (580 views)
Permalink

Re: Highligheter fails using JapaneseAnalyzer [In reply to]

Out of curiosity, when you try your other test string "aaa _bbb ccc"
what do the token byte offsets show?

Matt

Mark Harwood wrote:
>
> On 1 Jul 2009, at 17:39, k.sayama wrote:
>
>> I could verify Token byte offsets
>>
>> The sytsem outputs
>> aaa:0:3
>> bbb:0:3
>> ccc:4:7
>>
>
> That explains the highlighter behaviour. Clearly BBB is not at
> position 0-3 in the String you supplied
>
>>>> String CONTENTS = "AAA :BBB CCC";
>
> Looks like the Tokenizer needs fixing. Is this yours or a standard
> Lucene class? If the latter, raising a JIRA bug with a Junit test
> would be the best way to get things moving.
>
>
> Cheers
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-user-help[at]lucene.apache.org
>

--
Matthew Hall
Software Engineer
Mouse Genome Informatics
mhall[at]informatics.jax.org
(207) 288-6012

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

sake-ginjyo at nifty

Jul 2, 2009, 3:28 PM

Post #10 of 10 (556 views)
Permalink

Re: Highligheter fails using JapaneseAnalyzer [In reply to]

Hi

Tokenizer is not standard Lucene class.
but to acquire startOffset and endOffset correctly, I edited Tokenizer.
It is operating correctly now.

I want to verify more patterns.

thanks

----- Original Message -----
From: "Mark Harwood" <markharw00d[at]yahoo.co.uk>
To: <java-user[at]lucene.apache.org>
Sent: Thursday, July 02, 2009 6:25 AM
Subject: Re: Highligheter fails using JapaneseAnalyzer

>
> On 1 Jul 2009, at 17:39, k.sayama wrote:
>
>> I could verify Token byte offsets
>>
>> The sytsem outputs
>> aaa:0:3
>> bbb:0:3
>> ccc:4:7
>>
>
> That explains the highlighter behaviour. Clearly BBB is not at
> position 0-3 in the String you supplied
>
>>>> String CONTENTS = "AAA :BBB CCC";
>
> Looks like the Tokenizer needs fixing. Is this yours or a standard
> Lucene class? If the latter, raising a JIRA bug with a Junit test
> would be the best way to get things moving.
>
>
> Cheers
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
> For additional commands, e-mail: java-user-help[at]lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe[at]lucene.apache.org
For additional commands, e-mail: java-user-help[at]lucene.apache.org

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com