Mailing List Archive by Gossamer Threads http://www.gossamer-threads.com/lists/lucene/java-user/ en-us (c) Gossamer Threads Inc. All rights reserved. 12 Oct 2008 21:12:37 -0800 120 75 23 http://www.gossamer-threads.com/lists/lucene/java-user/ http://www.gossamer-threads.com/images/lists/rss_logo.jpg Someone just asked this question a week ago (unforunatley they asked it on the wrong list)... http://www.nabble.com/Can-I-filter-the-results-returne 12 Oct 2008 18:59:27 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66248 all folders which match "A AND Y", do you search for file name? If yes, A or Y in "A AND Y" is a Strring too, so you can do it by: construct a Lucene 12 Oct 2008 17:27:54 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66247 Hello, How can I get a list of all the terms for a particular field? Is the right approach to extend FilteredTermEnum? Thanks!! 12 Oct 2008 15:13:01 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66246 Hello, I noticed that indexSearcher.explain() method is not supposed to be run for a large collection of documents, so I am looking for an alternativ 12 Oct 2008 14:50:27 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66245 Hi, I want to search for sets of documents. For instance I index some folders with documents in it and now I do not want to find certain documents bu 12 Oct 2008 11:11:57 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66240 Have a look at the o.a.lucene.search.function package and the ValueSourceQuery. You will probably be able to factor in those pieces during scorin 12 Oct 2008 08:40:02 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66235 How large of a subset are you talking? You might look at the FitleredTermEnum class, but you will probably have to do some work on it to extend it 12 Oct 2008 08:38:51 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66234 Lucene should work quite well for this, you'll just need some infrastructure around it to get the file and extract the contents (see Lucene's Tika 12 Oct 2008 07:53:51 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66233 Congratulations! A truly stellar achievement. Can't wait to dive in! On Sat, 2008-10-11 at 11:50 -0400, Michael McCandless wrote: > Release 2.4.0 11 Oct 2008 19:28:05 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66228 Release 2.4.0 of Lucene is now available! With 2.4.0 we have relaxed the backwards compatibility policy of the Fieldable interface: we now allow chan 11 Oct 2008 08:50:45 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66225 Hello All, At my university we have over 20,000 small file ranging from 20k to 500k per directory and we would like to index them. I was wondering if 11 Oct 2008 07:22:28 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66223 Hello everyone. I've been fiddeling with the idea of retrieving the top terms from a subset of the index (i.e. top terms from the documents retrie 11 Oct 2008 03:36:04 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66221 Glen, Thank you for the details there. Its really great what you've done and I will study it some more! I too though about using multiple writers i 11 Oct 2008 02:21:08 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66220 IndexWriter is thread-safe and has been for a while (http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg00157.html) so you don't have to wor 10 Oct 2008 19:17:45 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66219 Hi gang, Wondering how folks have address scaled up indexing. I saw old threads about using clustered webapp with JNDI singleton index writer due to 10 Oct 2008 14:05:16 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66218 Actually looking at this a little deeper maybe Lucene could/should automatically be doing this "short" optimisation here? Given a comparitively small 10 Oct 2008 09:22:13 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66214 I have had a similar problem. What I do is load all the date field values at index startup, convert dates (timestamps) to a Julian date (# of seconds 10 Oct 2008 08:57:19 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66213 That's a really good idea Mark! :) Thanks! Will try to see if can make a quick change with your suggestion. (Too bad quick isn't really a word in my 10 Oct 2008 08:45:02 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66212 Update: The statement "...cost is field size (10 bytes ?) times number of documents" is wrong. What you actually have is the cost of the unique strin 10 Oct 2008 08:07:31 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66209 Yes, I understand that, and I did mean the number of documents, but I read in the javadoc that: "For String fields, the cache is larger: in additio 10 Oct 2008 07:58:14 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66208 I think you have your memory cost calculation wrong. The cost is field size (10 bytes ?) times number of documents NOT number of unique terms. The cac 10 Oct 2008 07:43:35 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66207 Unfortunately no, since the documents that are added may come form a new "source" containing old documents aswell..:/ I tried deploying our webappli 10 Oct 2008 07:25:29 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66206 Assuming content is added in chronological order and with no updates to existing docs couldn't you rely on internal Lucene document id to give a chron 10 Oct 2008 06:18:46 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66205 I'll follow up on my own question... Let's say that we have 4 years of data, meaning that there will be roughly 4 * 365 = 1460 unique terms for our 10 Oct 2008 05:51:50 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66202 True, I guess I was thinking of things from a search-only perspective when I claimed they were identical... But you're absolutely right in that you ca 10 Oct 2008 05:47:34 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66201 The release bits are indeed propagating through all mirrors, but I'm going to wait until tomorrow to do the announcement, to make sure all mirrors 10 Oct 2008 05:31:25 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66200 Hi there, I've just noticed that there is already a 2.4 release available on ibiblio (http://mirrors.ibiblio.org/pub/mirrors/maven2/org/apache/luce 10 Oct 2008 05:27:40 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66199 Hello, I've read a lot of threads now on memory consumption and sorting, and I think I have a pretty good understanding of how things work, but I 10 Oct 2008 05:09:36 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66198 Ah, sorry. Just saw the bit about the free text query too. A FieldCache is the answer here I suspect in order to quickly retrieve the date values for 10 Oct 2008 03:04:25 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66196 Assuming your date data is held as YYYYMMDD and you want daily totals.... Term startTerm=new Term("date","20080101"); TermEnum termEn 10 Oct 2008 02:40:32 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66195 Hi. Anyone have an idea of how I would create a query which finds the data backing a trend graph where date is X and num(docs) is on Y axis ? This i 10 Oct 2008 02:12:35 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66194 From what I can understand, you want to insert the word "history" and then get proposed "related" terms in combination with your input query. In ess 10 Oct 2008 00:33:58 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66193 Hi Ganesh, Your situation seems pretty straight. I did not really split my database (storage), just that while indexing, I indexed the data into 'n' 09 Oct 2008 21:49:16 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66192 Kalani, They are put into the document as separate fields. That's why there are getFields() and deleteFields() methods on Document. John G. -----Or 09 Oct 2008 19:29:36 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66191 Albert Juhe: > > Hi, > > I want to make a wizard that can help to find n-grams terms. > For example: > If i want to search History, after write it t 09 Oct 2008 09:49:26 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66186 Hi, I want to make a wizard that can help to find n-grams terms. For example: If i want to search History, after write it the system propose you the 09 Oct 2008 07:33:06 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66175 We have a custom "tagger" application which identifies certain entities (such as companies, etc.) and applies a "relevance" value to each entity based 08 Oct 2008 08:48:12 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66162 Hi everybody, I am using Lucene for searching items in a online shop. E.g. I search the shop for "shirt" I get a resultset from lucene. Now I want to 08 Oct 2008 08:15:21 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66161 Kalani: You are mistaken to think this is not possible. In fact it is explicitly supported, and I've done it in many situations. These two are entire 08 Oct 2008 05:58:05 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66157 Let's say my index has two fields (Type and Description). Type is either 0 or 1 and Description is a string (up to 250 characters). I'd like to exec 08 Oct 2008 05:44:56 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66156 Hello Anshum, In my case i have to add /modify records to the current index database and there will be only delete in older index DB. I will not ha 08 Oct 2008 05:09:21 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66155 You are doing this kind of a thing? document.add(new Field("id", id1, Field.Store.YES, Field.Index.TOKENIZED)); document.add(new Field("id", id2, Fie 08 Oct 2008 02:01:45 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66152 Thanks very much for your response, and for pointing me in the direction towards Function Queries - you saved me a ton of time! You're right, that see 07 Oct 2008 22:42:57 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66148 Not sure if I fully get it, but bear with me... Inline below. On Oct 6, 2008, at 11:37 PM, Alexander Devine wrote: > Hi Luceners, > > I have a part 07 Oct 2008 19:11:06 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66147 See below (and your other mail) On Tue, Oct 7, 2008 at 1:59 PM, John G <griffij@sosstaffing.com> wrote: > > Thanks Erick, > > Yes PerFieldAnalyzerWr 07 Oct 2008 11:25:00 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66142 Let's see the indexing code. It is perfectly reasonable to add data to a field multiple times, so I suspect you're doing something wrong..... What ev 07 Oct 2008 11:23:51 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66141 Thanks Erick, Yes PerFieldAnalyzerWrapper is my friend :>). Another related question, I'm putting these values into a document in fields with the sa 07 Oct 2008 10:59:44 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66140 Guys, I'm adding multiple fields with the same name to a document as Store.YES, Indexed.TOKENIZED and it seems that only the last field entered is in 07 Oct 2008 10:28:45 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66139 I found it. My reader was returning 0 at the end of the stream instead of -1. Doh.   Thanks again for the suggestions. They did ultimately lead me to 07 Oct 2008 08:50:19 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66136 Hello, I've been exploring usage of the spellcheck feature via solr 1.3. I have it working, but there are some issues I'm seeing that make it less us 07 Oct 2008 07:56:12 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66133 Thanks for the tip. I tried your experiment and, sure enough, it works just fine, so it's not the contents but obviously some other behavior of my cus 07 Oct 2008 07:43:06 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66132 This is going to get really sticky given StandardAnalyzer. Let's say that you have codesearch:B05 1 codesearch:B05 2 codesearch:B05 3 When you index 07 Oct 2008 05:52:24 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66130 There were all of the links I could find: http://findmeajob.wordpress.com/2007/07/31/lucene-singlesearcher-vs-multisearcher/ http://archives.devshed.c 07 Oct 2008 05:25:47 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66129 Hello Anusham, My intention is to shard the index after every 7 days (week). After 30 days, (4th week) the first DB may get deleted. At any point o 07 Oct 2008 02:59:33 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66127 If you capture the exact text produced by the reader, and wrap it in a StringReader and pass that to StandardAnalyzer, do you then see the same ex 07 Oct 2008 02:12:05 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66125 Hi Edwin, Would love to pose the code, but I don't have the Terracotta source code at the moment as I only downloaded the binaries. But Terracotta i 06 Oct 2008 22:35:21 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66123 Hi Paul, now i'm curious to know how the overriden java.util.Vector implements the clone method. Any chance of posting a snippet? ;) Cheers, Edwin 06 Oct 2008 21:34:06 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66121 Seems like I found the culprit to the problem. Because I am using Terracotta to doing JVM clustering, it has overriden many classes of java.util.*. 06 Oct 2008 20:51:38 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66120 My previous question may be moot but as is it is still a problem. Here's a little more info on my problem. The same named fields contain two pieces of 06 Oct 2008 20:38:38 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66119 Hi Luceners, I have a particular sorting problem and I wanted some advice on what the best implementation approach would be. We currently use Lucene 06 Oct 2008 20:37:18 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66118 Guys, I have documents with multiple stored, tokenized fields of the same name but different values in them such as: "codesearch", "B01" "cod 06 Oct 2008 19:39:28 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66115 I upgraded to the latest, 3.3.2 and had the same problem, even though it was clearly a different lexer reading the text.   I did find some problems w 06 Oct 2008 15:56:08 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66110 Hoss -- Thanks for reply -- >Paul: your subject linke seems to indicate that your email is a followup >to previous questions, however i don't see 06 Oct 2008 15:09:59 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66109 Hello and thanks for the response. Yes the title should have been values. I'll take at look at the link you provided. Thanks again! Erick Erickson 06 Oct 2008 14:22:30 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66108 You have to escape it in a query. And did you mean to title your post as field *names*? because I can't imagine how that would work. Field *values*, s 06 Oct 2008 13:29:36 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66101 It appears if a search field value entered contains a ':' causes some issues since if I have a field named a.b.c and the user enters a value of foo t 06 Oct 2008 12:54:12 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66100 No particular reason. It is just what I had loaded last and hadn't upgraded. It sounds like there might be good reason to do that now.   Thanks for th 06 Oct 2008 12:20:51 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66099 Hi Edwin, I don't know specifically what's causing the exception you're seeing, but note that in Lucene 2.3.0+, the JavaCC-generated version of Stand 06 Oct 2008 12:18:20 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66094 Hi Ganesh, About the memory consumption while sorting, it would end up using similar amounts, perhaps even more.. like in the case of regular parallel 06 Oct 2008 11:48:15 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66093 Oh, and in case it matters, I'm using Lucene 2.2.0. Ed ----- Original Message ---- I am stumped and have not seen any other reference to this pr 06 Oct 2008 11:32:31 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66091 : Subject: bunch of newbie queries, PS Paul: your subject linke seems to indicate that your email is a followup to previous questions, however i don 06 Oct 2008 11:24:53 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66090 I am stumped and have not seen any other reference to this problem. I am getting the following exception on everything I try to index. Does anyone kno 06 Oct 2008 11:23:58 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66092 Hi Edwin, Yes, I am running the TestVector against the same class path as my actual application. My class path contains the Lucene JAR files. This 06 Oct 2008 09:26:23 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66084 Hi Paul, When you say that your dummy TestVector class works, are you running it using the same classpaths that you used to run your actual applicati 06 Oct 2008 09:04:23 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66083 Hi, Probably off-topic, but just like to plug a bit on my blog post here: http://tinyurl.com/4vytcc :p (incidentally, Java GC is one of my favourite 06 Oct 2008 08:58:59 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66082 Oh yes, I forget to mention that MaxPermSize is very useful, and may be another key that solves my problem. I haven't tried UseConcMarkSweepGC and the 06 Oct 2008 08:11:25 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66078 OK it sounds like you need to increase the RAM your JVM is allowed to use, or, make your documents smaller. Mike Aditi Goyal wrote: > Thanks for 06 Oct 2008 08:06:04 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66077 I played around with GC quite a bit in our app and found the following java settings to help a lot (Used with jboss, but should be good for any jvm). 06 Oct 2008 05:48:20 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66076 Thanks for showing interest Mike. The OOME comes in the middle of setting a value of one of the field in the doc. That field has a fairly large value. 06 Oct 2008 05:17:59 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66075 Hello Mathias, IMHO sounds like you are planning to re-invent the wheel while all things you want (AFAICS) are already largely available as open sou 06 Oct 2008 01:35:08 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66074 I have tried to recompile Lucene 2.3.2 under my environment (SUN JDK 1.6) and it still doesn't work. I am beginning to think that maybe there is some 05 Oct 2008 23:16:38 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66071 Hello Anshum, My index is growing 1 million documents per day. Initially i planned to have a single database but the sorting of one or more fields c 05 Oct 2008 21:36:47 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66070 Thank you all for your input. However, this is what I have discovered so far: - I tried running the test that Edwin suggested and it failed for m 05 Oct 2008 19:45:19 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66068 I have confirmed that the OutOfMemoryError is not Lucene's problem. It's just because JVM failed to perform GC when necessary, and I don't know why. T 05 Oct 2008 04:55:25 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66059 i think, very likely, you have another copy of java.util.Vector loaded, and this one tries to be too clever with its implementation of clone (instanti 03 Oct 2008 23:21:40 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66053 Hi Paul, The clone() in SegmentInfos is correct. The best practice of clone is to delegate the clone to the super class (if you look at the source co 03 Oct 2008 23:11:32 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66052 : SegmentInfos sis = (SegmentInfos) super.clone(); : We see that it is trying to cast a Vector into SegmentInfos which explains : the ClassCas 03 Oct 2008 23:08:31 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66051 I am using Sun's JRE 1.6.0_02 on Windows XP Actually...are you sure it would work? java.util.Vector has the following clone() method: public Object 03 Oct 2008 15:38:21 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66048 That's Sun's JRE? That should be fine, unless there's something seriously wrong with it's java.util.Vector implementation. But, this is an excepti 03 Oct 2008 15:16:00 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66047 I am using Java 1.6.0_02. Is this a problem? On Fri, Oct 3, 2008 at 5:35 PM, Michael McCandless < lucene@mikemccandless.com> wrote: > > Which Java 03 Oct 2008 15:07:12 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66046 Which Java environment are you running? super.clone() from SegmentInfos should produce a new SegmentInfos object. It seems like in your case it's 03 Oct 2008 14:35:36 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66045 Hi Mike, I am actually using the Compass Search Engine which in turn makes use of Lucene. They are doing the following in their code: Index 03 Oct 2008 14:24:19 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66044 Can you describe what led up to this exception? Ie, what calls you made to Lucene before this. Mike Paul Chan wrote: > I think I know what the p 03 Oct 2008 13:54:51 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66043 I think I know what the problem is looking at the code: In SegmentInfos.java (line 321): class SegmentInfos extends Vector { public Object clone 03 Oct 2008 13:41:23 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66042 Hi, I am using lucene 2.3.2 and I encounter the following exception when I try to insert a object into the index. Caused by: java.lang.ClassCastExce 03 Oct 2008 12:40:10 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66041 David, this is not really a Lucene issue. Here is some Perl code that you could either use or rewrite in Java if you need it in Java: http://search.c 03 Oct 2008 09:49:57 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66029 Hi Ganesh, I have experimented with sharded indexes and they seem to benefit me(atleast in my case). I would like to know a few things before I answe 03 Oct 2008 09:18:57 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66028 Note that large stored fields do not use up any RAM in IndexWriter's RAM buffer because these stored fields are immediately written to the directo 03 Oct 2008 04:27:35 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66021 Single document of 16 MB seems to be big. I think you are trying to store the entire document content. If it is so drop the stored field and store it 03 Oct 2008 03:11:06 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66019 First off, IndexWriter's RAM buffer size is "approximate": after each doc is added, we check if the RAM consumed is greater than our bugdet, and i 03 Oct 2008 03:05:50 -0800 http://www.gossamer-threads.com/lists/lucene/java-user/66018