Mailing List Archive: kinosearch: discuss

optimize broken?

Index | Next | Previous | View Threaded

mark at thinkfoo

Aug 14, 2008, 10:48 AM

Post #1 of 2 (3286 views)
Permalink

optimize broken?

Hi All,

I'm doing some parallel indexing by firing off batches using gearman,
then once a task completes using ->add_invindex() to merge the batch to
a single index. After all batches are merged I optimize the index. This
works pretty reliably with around 27m records.

However, in the latest trunk of KinoSearch, ->finish(optimize => 1) no
longer seems to work. It returns almost immediately and none of the
segments are collapsed. The index is searchable but it's pretty slow.

The test suite runs successfully. Any ideas?

Cheers
~mark

p.s. I was pleased to see indexing speed significantly improved in the
latest trunk. It's now taking perhaps half the time of the
KinoSearch-0.20_051 cpan release and getting close to speeds I'm used to
seeing with Solr.

_______________________________________________
KinoSearch mailing list
KinoSearch [at] rectangular
http://www.rectangular.com/mailman/listinfo/kinosearch

marvin at rectangular

Aug 16, 2008, 7:42 PM

Post #2 of 2 (3034 views)
Permalink

Re: optimize broken? [In reply to]

On Aug 14, 2008, at 10:48 AM, Mark Holland wrote:

> However, in the latest trunk of KinoSearch, ->finish(optimize => 1)
> no longer seems to work. It returns almost immediately and none of
> the segments are collapsed. The index is searchable but it's pretty
> slow.
>
> The test suite runs successfully. Any ideas?

Sorry, but I've been unable to duplicate the problem so far. I've
now added a test verifying that optimization is working at least under
normal circumstances (commits 3730-3731, affecting t/213-
segment_merging.t). I'll keep looking, but if you can give me any
other hints, I'd appreciate it.

> p.s. I was pleased to see indexing speed significantly improved in
> the latest trunk. It's now taking perhaps half the time of the
> KinoSearch-0.20_051 cpan release and getting close to speeds I'm
> used to seeing with Solr.

Groovy. A lot has happened since 0.20_051.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

_______________________________________________
KinoSearch mailing list
KinoSearch [at] rectangular
http://www.rectangular.com/mailman/listinfo/kinosearch

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads