Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Lucene: Java-Dev

[jira] Issue Comment Edited: (LUCENE-2090) convert automaton to char[] based processing and TermRef / TermsEnum api

 

 

Lucene java-dev RSS feed   Index | Next | Previous | View Threaded


jira at apache

Nov 25, 2009, 5:06 AM

Post #1 of 3 (158 views)
Permalink
[jira] Issue Comment Edited: (LUCENE-2090) convert automaton to char[] based processing and TermRef / TermsEnum api

[ https://issues.apache.org/jira/browse/LUCENE-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782409#action_12782409 ]

Michael McCandless edited comment on LUCENE-2090 at 11/25/09 1:06 PM:
----------------------------------------------------------------------

BTW, we've discussed someday having a codec whose terms dict (or maybe just terms index) is represented as an FST, at which point AutomatonTermsEnum would be an intersection + walk of two FSTs. Because suffix's are also shared in the FST, you could more easily (more efficiently) handle \*XXX cases as well (it'd just be symmetic with the XXX\* cases).

was (Author: mikemccand):
BTW, we've discussed someday having a codec whose terms dict (or maybe just terms index) is represented as an FST, at which point AutomatonTermsEnum would be an intersection + walk of two FSTs. Because suffix's are also shared in the FST, you could more easily (more efficiently) handle *XXX cases as well (it'd just be symmetic with the XXX* cases).

> convert automaton to char[] based processing and TermRef / TermsEnum api
> ------------------------------------------------------------------------
>
> Key: LUCENE-2090
> URL: https://issues.apache.org/jira/browse/LUCENE-2090
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2090_TermRef_flex.patch, LUCENE-2090_TermRef_flex2.patch
>
>
> The automaton processing is currently done with String, mostly because TermEnum is based on String.
> it is easy to change the processing to work with char[], since behind the scenes this is used anyway.
> in general I think we should make sure char[] based processing is exposed in the automaton pkg anyway, for things like pattern-based tokenizers and such.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene


jira at apache

Dec 7, 2009, 2:36 PM

Post #2 of 3 (105 views)
Permalink
[jira] Issue Comment Edited: (LUCENE-2090) convert automaton to char[] based processing and TermRef / TermsEnum api [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787175#action_12787175 ]

Robert Muir edited comment on LUCENE-2090 at 12/7/09 10:34 PM:
---------------------------------------------------------------

Mike, I converted this to char[] api (see LUCENE-1606 for the patch).

In order for this to work, I needed to expose UnicodeUtil.nextUTF16ValidString(UTF16Result).
The code is not duplicated, the String based method is just a wrapper for this, take a look if you get a chance.

the other thing I forgot, I think TermRef.copy(UTF16Result) would be handy... is there anywhere you could use this too?

was (Author: rcmuir):
Mike, I converted this to char[] api (see LUCENE-1606 for the patch).

In order for this to work, I needed to expose UnicodeUtil.nextUTF16ValidString(UTF16Result).
The code is not duplicated, the String based method is just a wrapper for this, take a look if you get a chance.


> convert automaton to char[] based processing and TermRef / TermsEnum api
> ------------------------------------------------------------------------
>
> Key: LUCENE-2090
> URL: https://issues.apache.org/jira/browse/LUCENE-2090
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2090_TermRef_flex.patch, LUCENE-2090_TermRef_flex2.patch, LUCENE-2090_TermRef_flex3.patch
>
>
> The automaton processing is currently done with String, mostly because TermEnum is based on String.
> it is easy to change the processing to work with char[], since behind the scenes this is used anyway.
> in general I think we should make sure char[] based processing is exposed in the automaton pkg anyway, for things like pattern-based tokenizers and such.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene


jira at apache

Dec 7, 2009, 2:36 PM

Post #3 of 3 (104 views)
Permalink
[jira] Issue Comment Edited: (LUCENE-2090) convert automaton to char[] based processing and TermRef / TermsEnum api [In reply to]

[ https://issues.apache.org/jira/browse/LUCENE-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787175#action_12787175 ]

Robert Muir edited comment on LUCENE-2090 at 12/7/09 10:35 PM:
---------------------------------------------------------------

Mike, I converted this to char[] api (see LUCENE-1606 for the patch).

In order for this to work, I needed to expose UnicodeUtil.nextUTF16ValidString(UTF16Result).
The code is not duplicated, the String based method is just a wrapper for this, take a look if you get a chance.

the other thing I forgot, I think TermRef.copy(UTF8Result) would be handy... is there anywhere you could use this too?

edit: i meant utf-8 result, sorry

was (Author: rcmuir):
Mike, I converted this to char[] api (see LUCENE-1606 for the patch).

In order for this to work, I needed to expose UnicodeUtil.nextUTF16ValidString(UTF16Result).
The code is not duplicated, the String based method is just a wrapper for this, take a look if you get a chance.

the other thing I forgot, I think TermRef.copy(UTF16Result) would be handy... is there anywhere you could use this too?

> convert automaton to char[] based processing and TermRef / TermsEnum api
> ------------------------------------------------------------------------
>
> Key: LUCENE-2090
> URL: https://issues.apache.org/jira/browse/LUCENE-2090
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Robert Muir
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2090_TermRef_flex.patch, LUCENE-2090_TermRef_flex2.patch, LUCENE-2090_TermRef_flex3.patch
>
>
> The automaton processing is currently done with String, mostly because TermEnum is based on String.
> it is easy to change the processing to work with char[], since behind the scenes this is used anyway.
> in general I think we should make sure char[] based processing is exposed in the automaton pkg anyway, for things like pattern-based tokenizers and such.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene

Lucene java-dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.