jira at apache
Jul 14, 2008, 2:28 PM
Post #1 of 1
(555 views)
Permalink
|
|
[jira] Updated: (LUCENE-1333) Token implementation needs improvements
|
|
[ https://issues.apache.org/jira/browse/LUCENE-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
DM Smith updated LUCENE-1333:
-----------------------------
Attachment: LUCENE-1333a.txt
The following has been addressed in this patch:
1. JavaDoc is improved (as always, there is still room for improvement. For example, it says the field type is interned, but it is not.)
2. Deprecated the Token constructors taking a String.
3. Changed the allocation policy to be less aggressive.
5. Optimized the growing of the internal termBuffer immediately followed by storing a new term. In doing this added, setTermBuffer(String) and setTermBuffer(String, int, int). Setting from a string is roughly the same cost as setting from a char[].
6. TokenStream has next() has been deprecated. The javadoc has been updated to recommend next(Token) over next().
7. Rather than modifying Term to take a Token, public String term() has been added. With termText() still deprecated, this gives upgraders a clean choice to use term() or termBuffer(), with the knowledge of the performance differences.
I also updated TestToken to test all the changes.
Left to do: (I'd like to get a nod of whether I need to make further changes to the supplied patch before doing #4)
4. Changing of the remainder of core and contrib to remove calls to deprecated Token and TokenStream methods, i.e. to use the reuse API.
> Token implementation needs improvements
> ---------------------------------------
>
> Key: LUCENE-1333
> URL: https://issues.apache.org/jira/browse/LUCENE-1333
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 2.3.1
> Environment: All
> Reporter: DM Smith
> Priority: Minor
> Fix For: 2.4
>
> Attachments: LUCENE-1333a.txt
>
>
> This was discussed in the thread (not sure which place is best to reference so here are two):
> http://mail-archives.apache.org/mod_mbox/lucene-java-dev/200805.mbox/%3C21F67CC2-EBB4-48A0-894E-FBA4AECC0D50 [at] gmail%3E
> or to see it all at once:
> http://www.gossamer-threads.com/lists/lucene/java-dev/62851
> Issues:
> 1. JavaDoc is insufficient, leading one to read the code to figure out how to use the class.
> 2. Deprecations are incomplete. The constructors that take String as an argument and the methods that take and/or return String should *all* be deprecated.
> 3. The allocation policy is too aggressive. With large tokens the resulting buffer can be over-allocated. A less aggressive algorithm would be better. In the thread, the Python example is good as it is computationally simple.
> 4. The parts of the code that currently use Token's deprecated methods can be upgraded now rather than waiting for 3.0. As it stands, filter chains that alternate between char[] and String are sub-optimal. Currently, it is used in core by Query classes. The rest are in contrib, mostly in analyzers.
> 5. Some internal optimizations can be done with regard to char[] allocation.
> 6. TokenStream has next() and next(Token), next() should be deprecated, so that reuse is maximized and descendant classes should be rewritten to over-ride next(Token)
> 7. Tokens are often stored as a String in a Term. It would be good to add constructors that took a Token. This would simplify the use of the two together.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe [at] lucene
For additional commands, e-mail: java-dev-help [at] lucene
|
