Mailing List Archive: Lucene: Java-User

Build RAMDirectory on FSDirectory, and then synchronzing the two

Index | Next | Previous | View Threaded

zhoucheng2008 at gmail

Jan 8, 2012, 8:04 PM

Post #1 of 8 (177 views)
Permalink

Build RAMDirectory on FSDirectory, and then synchronzing the two

Hi,

I new a RAMDirectory based upon a FSDirectory. After a few modifications, I
would like to synchronize the two.

Some on the mailing list provided a solution that uses addIndex() function.

However, the FSDirectory simply combines with the RAMDirectory, and the
size doubled.

How can I do a real synchronization?

Thanks

1393975679 at qq

Jan 8, 2012, 8:29 PM

Post #2 of 8 (168 views)
Permalink

Re:Build RAMDirectory on FSDirectory, and then synchronzing the two [In reply to]

I'd better provide a snapshot of my code for people to understand my issues:

File file=new File("c:/index_files");
FSDirectory fsDir=new FSDirectory(file);
RAMDirectory ramDir=new RAMDirectory(fsDir, new IndexWriterConfig(Version.LUCENE_35, new StandardAnalyzer());

IndexWriter iw = new IndexWriter(ramDir, iwc);

......DO something here with iw (associated with ramDir).....

Now I am trying to synchronize ramDir with fsDir:

//close iw prior to synchronization
iw.close();

// synchronize RAM with FS
IndexWriter writer = new IndexWriter(fsDir, new IndexWriterConfig(Version.LUCENE_35, ik));
writer.addIndexes(ramDir);
writer.close();
ramDir.close();

Now I end up with duplicate copies of index files in c:/index_files

Is there something that I miss here?

------------------ Original ------------------
From: "zhoucheng2008"<zhoucheng2008 [at] gmail>;
Date: Mon, Jan 9, 2012 12:04 PM
To: "java-user"<java-user [at] lucene>;

Subject: Build RAMDirectory on FSDirectory, and then synchronzing the two

Hi,

I new a RAMDirectory based upon a FSDirectory. After a few modifications, I would like to synchronize the two.

Some on the mailing list provided a solution that uses addIndex() function.

However, the FSDirectory simply combines with the RAMDirectory, and the size doubled.

How can I do a real synchronization?

Thanks

ian.lea at gmail

Jan 9, 2012, 2:23 AM

Post #3 of 8 (168 views)
Permalink

Re: Build RAMDirectory on FSDirectory, and then synchronzing the two [In reply to]

If you load an existing disk index into a RAMDirectory, make some
changes in RAM and call addIndexes to add the contents of the
RAMDirectory to the original disk index, you are likely to end up with
duplicate data on disk. Depending of course on what you've done to
the RAM index.

Sounds you want to call addIndexes using a writer on a new, empty,
index or overwrite the original. IndexWriterConfig.OpenMode CREATE.

--
Ian.

On Mon, Jan 9, 2012 at 4:29 AM, dyzc <1393975679 [at] qq> wrote:
> I'd better provide a snapshot of my code for people to understand my issues:
>
>
> File file=new File("c:/index_files");
> FSDirectory fsDir=new FSDirectory(file);
> RAMDirectory ramDir=new RAMDirectory(fsDir, new IndexWriterConfig(Version.LUCENE_35, new StandardAnalyzer());
>
>
> IndexWriter iw = new IndexWriter(ramDir, iwc);
>
>
> ......DO something here with iw (associated with ramDir).....
>
>
> Now I am trying to synchronize ramDir with fsDir:
>
>
> //close iw prior to synchronization
> iw.close();
>
>
> // synchronize RAM with FS
> IndexWriter writer = new IndexWriter(fsDir, new IndexWriterConfig(Version.LUCENE_35, ik));
> writer.addIndexes(ramDir);
> writer.close();
> ramDir.close();
>
>
>
> Now I end up with duplicate copies of index files in c:/index_files
>
>
> Is there something that I miss here?
>
>
> ------------------ Original ------------------
> From: "zhoucheng2008"<zhoucheng2008 [at] gmail>;
> Date: Mon, Jan 9, 2012 12:04 PM
> To: "java-user"<java-user [at] lucene>;
>
> Subject: Build RAMDirectory on FSDirectory, and then synchronzing the two
>
>
> Hi,
>
> I new a RAMDirectory based upon a FSDirectory. After a few modifications, I would like to synchronize the two.
>
>
> Some on the mailing list provided a solution that uses addIndex() function.
>
>
> However, the FSDirectory simply combines with the RAMDirectory, and the size doubled.
>
>
> How can I do a real synchronization?
>
>
> Thanks

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

zhoucheng2008 at gmail

Jan 10, 2012, 5:18 PM

Post #4 of 8 (162 views)
Permalink

Re: Build RAMDirectory on FSDirectory, and then synchronzing the two [In reply to]

I tried IndexWriterConfig.OpenMode CREATE, and the size is doubled.

The only way that is effective is the writer's deleteAll() methods.

On Mon, Jan 9, 2012 at 5:23 AM, Ian Lea <ian.lea [at] gmail> wrote:

> If you load an existing disk index into a RAMDirectory, make some
> changes in RAM and call addIndexes to add the contents of the
> RAMDirectory to the original disk index, you are likely to end up with
> duplicate data on disk. Depending of course on what you've done to
> the RAM index.
>
> Sounds you want to call addIndexes using a writer on a new, empty,
> index or overwrite the original. IndexWriterConfig.OpenMode CREATE.
>
>
> --
> Ian.
>
>
> On Mon, Jan 9, 2012 at 4:29 AM, dyzc <1393975679 [at] qq> wrote:
> > I'd better provide a snapshot of my code for people to understand my
> issues:
> >
> >
> > File file=new File("c:/index_files");
> > FSDirectory fsDir=new FSDirectory(file);
> > RAMDirectory ramDir=new RAMDirectory(fsDir, new
> IndexWriterConfig(Version.LUCENE_35, new StandardAnalyzer());
> >
> >
> > IndexWriter iw = new IndexWriter(ramDir, iwc);
> >
> >
> > ......DO something here with iw (associated with ramDir).....
> >
> >
> > Now I am trying to synchronize ramDir with fsDir:
> >
> >
> > //close iw prior to synchronization
> > iw.close();
> >
> >
> > // synchronize RAM with FS
> > IndexWriter writer = new IndexWriter(fsDir, new
> IndexWriterConfig(Version.LUCENE_35, ik));
> > writer.addIndexes(ramDir);
> > writer.close();
> > ramDir.close();
> >
> >
> >
> > Now I end up with duplicate copies of index files in c:/index_files
> >
> >
> > Is there something that I miss here?
> >
> >
> > ------------------ Original ------------------
> > From: "zhoucheng2008"<zhoucheng2008 [at] gmail>;
> > Date: Mon, Jan 9, 2012 12:04 PM
> > To: "java-user"<java-user [at] lucene>;
> >
> > Subject: Build RAMDirectory on FSDirectory, and then synchronzing the
> two
> >
> >
> > Hi,
> >
> > I new a RAMDirectory based upon a FSDirectory. After a few
> modifications, I would like to synchronize the two.
> >
> >
> > Some on the mailing list provided a solution that uses addIndex()
> function.
> >
> >
> > However, the FSDirectory simply combines with the RAMDirectory, and the
> size doubled.
> >
> >
> > How can I do a real synchronization?
> >
> >
> > Thanks
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene
>
>

ian.lea at gmail

Jan 11, 2012, 1:20 AM

Post #5 of 8 (163 views)
Permalink

Re: Build RAMDirectory on FSDirectory, and then synchronzing the two [In reply to]

> I tried IndexWriterConfig.OpenMode CREATE, and the size is doubled.

Prove it.

--
Ian.

> The only way that is effective is the writer's deleteAll() methods.
>
> On Mon, Jan 9, 2012 at 5:23 AM, Ian Lea <ian.lea [at] gmail> wrote:
>
>> If you load an existing disk index into a RAMDirectory, make some
>> changes in RAM and call addIndexes to add the contents of the
>> RAMDirectory to the original disk index, you are likely to end up with
>> duplicate data on disk. Depending of course on what you've done to
>> the RAM index.
>>
>> Sounds you want to call addIndexes using a writer on a new, empty,
>> index or overwrite the original. IndexWriterConfig.OpenMode CREATE.
>>
>>
>> --
>> Ian.
>>
>>
>> On Mon, Jan 9, 2012 at 4:29 AM, dyzc <1393975679 [at] qq> wrote:
>> > I'd better provide a snapshot of my code for people to understand my
>> issues:
>> >
>> >
>> > File file=new File("c:/index_files");
>> > FSDirectory fsDir=new FSDirectory(file);
>> > RAMDirectory ramDir=new RAMDirectory(fsDir, new
>> IndexWriterConfig(Version.LUCENE_35, new StandardAnalyzer());
>> >
>> >
>> > IndexWriter iw = new IndexWriter(ramDir, iwc);
>> >
>> >
>> > ......DO something here with iw (associated with ramDir).....
>> >
>> >
>> > Now I am trying to synchronize ramDir with fsDir:
>> >
>> >
>> > //close iw prior to synchronization
>> > iw.close();
>> >
>> >
>> > // synchronize RAM with FS
>> > IndexWriter writer = new IndexWriter(fsDir, new
>> IndexWriterConfig(Version.LUCENE_35, ik));
>> > writer.addIndexes(ramDir);
>> > writer.close();
>> > ramDir.close();
>> >
>> >
>> >
>> > Now I end up with duplicate copies of index files in c:/index_files
>> >
>> >
>> > Is there something that I miss here?
>> >
>> >
>> > ------------------ Original ------------------
>> > From: "zhoucheng2008"<zhoucheng2008 [at] gmail>;
>> > Date: Mon, Jan 9, 2012 12:04 PM
>> > To: "java-user"<java-user [at] lucene>;
>> >
>> > Subject: Build RAMDirectory on FSDirectory, and then synchronzing the
>> two
>> >
>> >
>> > Hi,
>> >
>> > I new a RAMDirectory based upon a FSDirectory. After a few
>> modifications, I would like to synchronize the two.
>> >
>> >
>> > Some on the mailing list provided a solution that uses addIndex()
>> function.
>> >
>> >
>> > However, the FSDirectory simply combines with the RAMDirectory, and the
>> size doubled.
>> >
>> >
>> > How can I do a real synchronization?
>> >
>> >
>> > Thanks
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

1393975679 at qq

Jan 11, 2012, 8:01 PM

Post #6 of 8 (162 views)
Permalink

Re: Build RAMDirectory on FSDirectory, and then synchronzing the two [In reply to]

That lies in that my apps add indexes to those in RAM rather than update them. So the size doubled. Seem not related to the OpenMode.CREATE option.

------------------ Original ------------------
From: "Ian Lea"<ian.lea [at] gmail>;
Date: Wed, Jan 11, 2012 05:20 PM
To: "java-user"<java-user [at] lucene>;

Subject: Re: Build RAMDirectory on FSDirectory, and then synchronzing the two

> I tried IndexWriterConfig.OpenMode CREATE, and the size is doubled.

Prove it.

--
Ian.

> The only way that is effective is the writer's deleteAll() methods.
>
> On Mon, Jan 9, 2012 at 5:23 AM, Ian Lea <ian.lea [at] gmail> wrote:
>
>> If you load an existing disk index into a RAMDirectory, make some
>> changes in RAM and call addIndexes to add the contents of the
>> RAMDirectory to the original disk index, you are likely to end up with
>> duplicate data on disk. Depending of course on what you've done to
>> the RAM index.
>>
>> Sounds you want to call addIndexes using a writer on a new, empty,
>> index or overwrite the original. IndexWriterConfig.OpenMode CREATE.
>>
>>
>> --
>> Ian.
>>
>>
>> On Mon, Jan 9, 2012 at 4:29 AM, dyzc <1393975679 [at] qq> wrote:
>> > I'd better provide a snapshot of my code for people to understand my
>> issues:
>> >
>> >
>> > File file=new File("c:/index_files");
>> > FSDirectory fsDir=new FSDirectory(file);
>> > RAMDirectory ramDir=new RAMDirectory(fsDir, new
>> IndexWriterConfig(Version.LUCENE_35, new StandardAnalyzer());
>> >
>> >
>> > IndexWriter iw = new IndexWriter(ramDir, iwc);
>> >
>> >
>> > ......DO something here with iw (associated with ramDir).....
>> >
>> >
>> > Now I am trying to synchronize ramDir with fsDir:
>> >
>> >
>> > //close iw prior to synchronization
>> > iw.close();
>> >
>> >
>> > // synchronize RAM with FS
>> > IndexWriter writer = new IndexWriter(fsDir, new
>> IndexWriterConfig(Version.LUCENE_35, ik));
>> > writer.addIndexes(ramDir);
>> > writer.close();
>> > ramDir.close();
>> >
>> >
>> >
>> > Now I end up with duplicate copies of index files in c:/index_files
>> >
>> >
>> > Is there something that I miss here?
>> >
>> >
>> > ------------------ Original ------------------
>> > From: "zhoucheng2008"<zhoucheng2008 [at] gmail>;
>> > Date: Mon, Jan 9, 2012 12:04 PM
>> > To: "java-user"<java-user [at] lucene>;
>> >
>> > Subject: Build RAMDirectory on FSDirectory, and then synchronzing the
>> two
>> >
>> >
>> > Hi,
>> >
>> > I new a RAMDirectory based upon a FSDirectory. After a few
>> modifications, I would like to synchronize the two.
>> >
>> >
>> > Some on the mailing list provided a solution that uses addIndex()
>> function.
>> >
>> >
>> > However, the FSDirectory simply combines with the RAMDirectory, and the
>> size doubled.
>> >
>> >
>> > How can I do a real synchronization?
>> >
>> >
>> > Thanks
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
>> For additional commands, e-mail: java-user-help [at] lucene
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
For additional commands, e-mail: java-user-help [at] lucene

sanne.grinovero at gmail

Jan 12, 2012, 3:36 PM

Post #7 of 8 (157 views)
Permalink

Re: Build RAMDirectory on FSDirectory, and then synchronzing the two [In reply to]

Maybe you could explain why you are doing this? Someone could suggest
alternative approaches.

Regards,
Sanne
On Jan 12, 2012 4:02 AM, "dyzc" <1393975679 [at] qq> wrote:

> That lies in that my apps add indexes to those in RAM rather than update
> them. So the size doubled. Seem not related to the OpenMode.CREATE option.
>
>
> ------------------ Original ------------------
> From: "Ian Lea"<ian.lea [at] gmail>;
> Date: Wed, Jan 11, 2012 05:20 PM
> To: "java-user"<java-user [at] lucene>;
>
> Subject: Re: Build RAMDirectory on FSDirectory, and then synchronzing the
> two
>
>
> > I tried IndexWriterConfig.OpenMode CREATE, and the size is doubled.
>
> Prove it.
>
>
> --
> Ian.
>
> > The only way that is effective is the writer's deleteAll() methods.
> >
> > On Mon, Jan 9, 2012 at 5:23 AM, Ian Lea <ian.lea [at] gmail> wrote:
> >
> >> If you load an existing disk index into a RAMDirectory, make some
> >> changes in RAM and call addIndexes to add the contents of the
> >> RAMDirectory to the original disk index, you are likely to end up with
> >> duplicate data on disk. Depending of course on what you've done to
> >> the RAM index.
> >>
> >> Sounds you want to call addIndexes using a writer on a new, empty,
> >> index or overwrite the original. IndexWriterConfig.OpenMode CREATE.
> >>
> >>
> >> --
> >> Ian.
> >>
> >>
> >> On Mon, Jan 9, 2012 at 4:29 AM, dyzc <1393975679 [at] qq> wrote:
> >> > I'd better provide a snapshot of my code for people to understand my
> >> issues:
> >> >
> >> >
> >> > File file=new File("c:/index_files");
> >> > FSDirectory fsDir=new FSDirectory(file);
> >> > RAMDirectory ramDir=new RAMDirectory(fsDir, new
> >> IndexWriterConfig(Version.LUCENE_35, new StandardAnalyzer());
> >> >
> >> >
> >> > IndexWriter iw = new IndexWriter(ramDir, iwc);
> >> >
> >> >
> >> > ......DO something here with iw (associated with ramDir).....
> >> >
> >> >
> >> > Now I am trying to synchronize ramDir with fsDir:
> >> >
> >> >
> >> > //close iw prior to synchronization
> >> > iw.close();
> >> >
> >> >
> >> > // synchronize RAM with FS
> >> > IndexWriter writer = new IndexWriter(fsDir, new
> >> IndexWriterConfig(Version.LUCENE_35, ik));
> >> > writer.addIndexes(ramDir);
> >> > writer.close();
> >> > ramDir.close();
> >> >
> >> >
> >> >
> >> > Now I end up with duplicate copies of index files in c:/index_files
> >> >
> >> >
> >> > Is there something that I miss here?
> >> >
> >> >
> >> > ------------------ Original ------------------
> >> > From: "zhoucheng2008"<zhoucheng2008 [at] gmail>;
> >> > Date: Mon, Jan 9, 2012 12:04 PM
> >> > To: "java-user"<java-user [at] lucene>;
> >> >
> >> > Subject: Build RAMDirectory on FSDirectory, and then synchronzing the
> >> two
> >> >
> >> >
> >> > Hi,
> >> >
> >> > I new a RAMDirectory based upon a FSDirectory. After a few
> >> modifications, I would like to synchronize the two.
> >> >
> >> >
> >> > Some on the mailing list provided a solution that uses addIndex()
> >> function.
> >> >
> >> >
> >> > However, the FSDirectory simply combines with the RAMDirectory, and
> the
> >> size doubled.
> >> >
> >> >
> >> > How can I do a real synchronization?
> >> >
> >> >
> >> > Thanks
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> >> For additional commands, e-mail: java-user-help [at] lucene
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> For additional commands, e-mail: java-user-help [at] lucene

zhoucheng2008 at gmail

Jan 12, 2012, 4:27 PM

Post #8 of 8 (155 views)
Permalink

Re: Build RAMDirectory on FSDirectory, and then synchronzing the two [In reply to]

The reason is I have indexes on hard drive but want to load them into ram
for faster searching, adding, deleting, etc.

Using RAMDirectory can help achieve this goal.

On Thu, Jan 12, 2012 at 6:36 PM, Sanne Grinovero
<sanne.grinovero [at] gmail>wrote:

> Maybe you could explain why you are doing this? Someone could suggest
> alternative approaches.
>
> Regards,
> Sanne
> On Jan 12, 2012 4:02 AM, "dyzc" <1393975679 [at] qq> wrote:
>
> > That lies in that my apps add indexes to those in RAM rather than update
> > them. So the size doubled. Seem not related to the OpenMode.CREATE
> option.
> >
> >
> > ------------------ Original ------------------
> > From: "Ian Lea"<ian.lea [at] gmail>;
> > Date: Wed, Jan 11, 2012 05:20 PM
> > To: "java-user"<java-user [at] lucene>;
> >
> > Subject: Re: Build RAMDirectory on FSDirectory, and then synchronzing
> the
> > two
> >
> >
> > > I tried IndexWriterConfig.OpenMode CREATE, and the size is doubled.
> >
> > Prove it.
> >
> >
> > --
> > Ian.
> >
> > > The only way that is effective is the writer's deleteAll() methods.
> > >
> > > On Mon, Jan 9, 2012 at 5:23 AM, Ian Lea <ian.lea [at] gmail> wrote:
> > >
> > >> If you load an existing disk index into a RAMDirectory, make some
> > >> changes in RAM and call addIndexes to add the contents of the
> > >> RAMDirectory to the original disk index, you are likely to end up with
> > >> duplicate data on disk. Depending of course on what you've done to
> > >> the RAM index.
> > >>
> > >> Sounds you want to call addIndexes using a writer on a new, empty,
> > >> index or overwrite the original. IndexWriterConfig.OpenMode CREATE.
> > >>
> > >>
> > >> --
> > >> Ian.
> > >>
> > >>
> > >> On Mon, Jan 9, 2012 at 4:29 AM, dyzc <1393975679 [at] qq> wrote:
> > >> > I'd better provide a snapshot of my code for people to understand my
> > >> issues:
> > >> >
> > >> >
> > >> > File file=new File("c:/index_files");
> > >> > FSDirectory fsDir=new FSDirectory(file);
> > >> > RAMDirectory ramDir=new RAMDirectory(fsDir, new
> > >> IndexWriterConfig(Version.LUCENE_35, new StandardAnalyzer());
> > >> >
> > >> >
> > >> > IndexWriter iw = new IndexWriter(ramDir, iwc);
> > >> >
> > >> >
> > >> > ......DO something here with iw (associated with ramDir).....
> > >> >
> > >> >
> > >> > Now I am trying to synchronize ramDir with fsDir:
> > >> >
> > >> >
> > >> > //close iw prior to synchronization
> > >> > iw.close();
> > >> >
> > >> >
> > >> > // synchronize RAM with FS
> > >> > IndexWriter writer = new IndexWriter(fsDir, new
> > >> IndexWriterConfig(Version.LUCENE_35, ik));
> > >> > writer.addIndexes(ramDir);
> > >> > writer.close();
> > >> > ramDir.close();
> > >> >
> > >> >
> > >> >
> > >> > Now I end up with duplicate copies of index files in c:/index_files
> > >> >
> > >> >
> > >> > Is there something that I miss here?
> > >> >
> > >> >
> > >> > ------------------ Original ------------------
> > >> > From: "zhoucheng2008"<zhoucheng2008 [at] gmail>;
> > >> > Date: Mon, Jan 9, 2012 12:04 PM
> > >> > To: "java-user"<java-user [at] lucene>;
> > >> >
> > >> > Subject: Build RAMDirectory on FSDirectory, and then synchronzing
> the
> > >> two
> > >> >
> > >> >
> > >> > Hi,
> > >> >
> > >> > I new a RAMDirectory based upon a FSDirectory. After a few
> > >> modifications, I would like to synchronize the two.
> > >> >
> > >> >
> > >> > Some on the mailing list provided a solution that uses addIndex()
> > >> function.
> > >> >
> > >> >
> > >> > However, the FSDirectory simply combines with the RAMDirectory, and
> > the
> > >> size doubled.
> > >> >
> > >> >
> > >> > How can I do a real synchronization?
> > >> >
> > >> >
> > >> > Thanks
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > >> For additional commands, e-mail: java-user-help [at] lucene
> > >>
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe [at] lucene
> > For additional commands, e-mail: java-user-help [at] lucene
>

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads