Mailing List Archive: kinosearch: discuss

Index optimize failure

Index | Next | Previous | View Threaded

henka at cityweb

Sep 18, 2008, 1:49 AM

Post #1 of 8 (6793 views)
Permalink

Index optimize failure

Greetings,

Revision 3875 consistently fails on an optimize of *one* of my
indexes. The script simply opens the index (a previously merged
multi-index), then closes it with optimize => 1.

Other index optimizes run successfully.

The error:
----------
Out of bounds: -2147406182 >= 166543 at
../c_src/KinoSearch/Util/I32Array.c:33 kino_I32Arr_get
at /etc/test/testindexer/optimize_master_index line 97

Line 97 is the expected:
------------------------
$invindexer->finish( optimize => 1 );

I'm busy running it through gdb to obtain more info, and will repost shortly.

Let me know if you require anything else. A test case might be a bit
difficult considering the size of the index (couple of gigs) and the
number of subindexes merged therein, but I'll keep digging to narrow
it down.

Regards
Henry

_______________________________________________
KinoSearch mailing list
KinoSearch [at] rectangular
http://www.rectangular.com/mailman/listinfo/kinosearch

henka at cityweb

Sep 18, 2008, 4:18 AM

Post #2 of 8 (6526 views)
Permalink

Re: Index optimize failure [In reply to]

Quoting Henka <henka [at] cityweb>:
> I'm busy running it through gdb to obtain more info, and will repost shortly.

gdb and perl debug don't provide anything useful (this not being a
nice segv or related genus) - looks like the problem lies in the
caller of I32Arr_get() or higher. Short of setting a breakpoint
(somewhere) and stepping around (probably blindly), I think it best to
await instructions from Our Main Man.

Cheers
Henry

_______________________________________________
KinoSearch mailing list
KinoSearch [at] rectangular
http://www.rectangular.com/mailman/listinfo/kinosearch

henka at cityweb

Sep 18, 2008, 5:05 AM

Post #3 of 8 (6516 views)
Permalink

Re: Index optimize failure [In reply to]

Quoting Henka <henka [at] cityweb>:
> Revision 3875 consistently fails on an optimize of *one* of my
> indexes. The script simply opens the index (a previously merged
> multi-index), then closes it with optimize => 1.

The latest, revision 3879 does the same.

Regards
Henry

_______________________________________________
KinoSearch mailing list
KinoSearch [at] rectangular
http://www.rectangular.com/mailman/listinfo/kinosearch

marvin at rectangular

Sep 18, 2008, 7:45 AM

Post #4 of 8 (6519 views)
Permalink

Re: Index optimize failure [In reply to]

On Sep 18, 2008, at 1:49 AM, Henka wrote:

> Revision 3875 consistently fails on an optimize of *one* of my
> indexes.

Do you recall the last time you updated before r3875? r3875 itself is
definitely not the culprit -- it's a one-line fix for a memory leak in
a testing-only file.

I know you watch the commits list for stuff like this, but did r3737
from a month ago slip by?

------------------------------------------------------------------------
r3737 | creamyg | 2008-08-19 12:53:41 -0700 (Tue, 19 Aug 2008) | 3
lines

Change field numbers to start at 1 instead of 0. This is a
backwards-incompatible index format change.

> The script simply opens the index (a previously merged multi-index),
> then closes it with optimize => 1.
>
> Other index optimizes run successfully.
>
> The error:
> ----------
> Out of bounds: -2147406182 >= 166543 at ../c_src/KinoSearch/Util/
> I32Array.c:33 kino_I32Arr_get
> at /etc/test/testindexer/optimize_master_index line 97
>
>
> Line 97 is the expected:
> ------------------------
> $invindexer->finish( optimize => 1 );

That's an index-out-of-bounds error from I32Array, a "safe" array
class that throws exceptions when ordinary C array access would
trigger a memory error. I32Array is used in a few places around KS,
but not that many where the capacity would be as high as 166543 and
that would be called during InvIndexer_Finish(). My guess that those
are doc numbers and the call in question is coming from PostingPool.c:

/* Skip deletions. */
if (doc_map != NULL) {
const i32_t remapped = I32Arr_Get(doc_map,
raw_posting->doc_num - doc_base);
if ( !remapped )
continue;
raw_posting->doc_num = remapped;
}

It would be helpful to see a C stack trace to confirm the suspicion.
If it's the same number every time, can you put a watch point into
I32Arr_get() looking for it?

> A test case might be a bit difficult considering the size of the
> index (couple of gigs) and the number of subindexes merged therein,
> but I'll keep digging to narrow it down.

Will you be able to recreate the circumstances that led to this bug,
even if we can't condense a test case? Meaning, can you duplicate the
sequence of subindex creation and merging?

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

_______________________________________________
KinoSearch mailing list
KinoSearch [at] rectangular
http://www.rectangular.com/mailman/listinfo/kinosearch

henka at cityweb

Sep 18, 2008, 12:01 PM

Post #5 of 8 (6515 views)
Permalink

Re: Index optimize failure [In reply to]

Quoting "Marvin Humphrey" <marvin [at] rectangular>:
> Do you recall the last time you updated before r3875? r3875 itself
> is definitely not the culprit -- it's a one-line fix for a memory
> leak in a testing-only file.
>
> I know you watch the commits list for stuff like this, but did r3737
> from a month ago slip by?
>
> ------------------------------------------------------------------------
> r3737 | creamyg | 2008-08-19 12:53:41 -0700 (Tue, 19 Aug 2008) | 3 lines
>
> Change field numbers to start at 1 instead of 0. This is a
> backwards-incompatible index format change.

Nah, I remember noting this commit.

I seem to recall doing an update about a week or two ago, followed by
an index run over a few days to build the test indexes.

Might be a good idea to reindex the subs in question and re-merge and
see what happens.

> Will you be able to recreate the circumstances that led to this bug,
> even if we can't condense a test case? Meaning, can you duplicate
> the sequence of subindex creation and merging?

Yes. You'll recall I do a lot of logging... I can recreate the
sequence of events exactly.

In fact, that's what I'll start tomorrow and let you know how it goes.

I'll also see about triggering that breakpoint and doing a bt before
rerunning everything for that particular master index.

Cheers
Henry

_______________________________________________
KinoSearch mailing list
KinoSearch [at] rectangular
http://www.rectangular.com/mailman/listinfo/kinosearch

eric at pieinsky

Sep 18, 2008, 1:31 PM

Post #6 of 8 (6522 views)
Permalink

Re: Index optimize failure [In reply to]

Henka, what does your schema look like? Do you have any vectorized
fields? I think I came across a similar issue (but neglected to
report it, slipped through the too-busy cracks) with a schema that
had no vectorized fields, vectorizing one field made the problem go
away.

Eric Howe

On Sep 18, 2008, at 12:01, Henka wrote:

> Quoting "Marvin Humphrey" <marvin [at] rectangular>:
>> Do you recall the last time you updated before r3875? r3875
>> itself is definitely not the culprit -- it's a one-line fix for a
>> memory leak in a testing-only file.
>>
>> I know you watch the commits list for stuff like this, but did
>> r3737 from a month ago slip by?
>>
>>
>> ---------------------------------------------------------------------
>> ---
>> r3737 | creamyg | 2008-08-19 12:53:41 -0700 (Tue, 19 Aug 2008) |
>> 3 lines
>>
>> Change field numbers to start at 1 instead of 0. This is a
>> backwards-incompatible index format change.
>
> Nah, I remember noting this commit.
>
> I seem to recall doing an update about a week or two ago, followed
> by an index run over a few days to build the test indexes.
>
> Might be a good idea to reindex the subs in question and re-merge
> and see what happens.
>
>> Will you be able to recreate the circumstances that led to this
>> bug, even if we can't condense a test case? Meaning, can you
>> duplicate the sequence of subindex creation and merging?
>
> Yes. You'll recall I do a lot of logging... I can recreate the
> sequence of events exactly.
>
> In fact, that's what I'll start tomorrow and let you know how it goes.
>
> I'll also see about triggering that breakpoint and doing a bt
> before rerunning everything for that particular master index.
>
> Cheers
> Henry
>
>
>
> _______________________________________________
> KinoSearch mailing list
> KinoSearch [at] rectangular
> http://www.rectangular.com/mailman/listinfo/kinosearch

_______________________________________________
KinoSearch mailing list
KinoSearch [at] rectangular
http://www.rectangular.com/mailman/listinfo/kinosearch

henka at cityweb

Sep 19, 2008, 12:17 AM

Post #7 of 8 (6512 views)
Permalink

Re: Index optimize failure [In reply to]

Quoting "Eric Howe" <eric [at] pieinsky>:
> Henka, what does your schema look like? Do you have any vectorized
> fields? I think I came across a similar issue (but neglected to
> report it, slipped through the too-busy cracks) with a schema that
> had no vectorized fields, vectorizing one field made the problem go
> away.

Howzit Eric; no, several vectorized fields in my schema.

Cheers
Henry

_______________________________________________
KinoSearch mailing list
KinoSearch [at] rectangular
http://www.rectangular.com/mailman/listinfo/kinosearch

henka at cityweb

Sep 19, 2008, 3:12 AM

Post #8 of 8 (6517 views)
Permalink

Re: Index optimize failure [In reply to]

Quoting "Marvin Humphrey" <marvin [at] rectangular>:
> It would be helpful to see a C stack trace to confirm the suspicion.
> If it's the same number every time, can you put a watch point into
> I32Arr_get() looking for it?

Here's the backtrace. I had trouble triggering the breakpoint with a
condition, so I just spliced in a function call to break on before
CONFESS("Out of bounds...:

(gdb) bt
#0 bloopy () at ../c_src/KinoSearch/Util/I32Array.c:31
#1 0x0023d2be in kino_I32Arr_get (self=0x9dbd1d8, num=-2147406182) at
../c_src/KinoSearch/Util/I32Array.c:37
#2 0x00262309 in kino_PostPool_refill (self=0x9dd88c8) at
../c_src/h/KinoSearch/Util/I32Array.h:225
#3 0x0023ae7c in refill_cache (self=0x9dbfb48) at
../c_src/h/KinoSearch/Util/SortExRun.h:264
#4 0x0023b260 in kino_SortEx_fetch (self=0x9dbfb48) at
../c_src/KinoSearch/Util/SortExternal.c:120
#5 0x002642d2 in write_terms_and_postings (self=<value optimized
out>, raw_post_source=0x9dbfb48, post_stream=0x9e816a8,
skip_stream=0x9f591a8)
at ../c_src/KinoSearch/Index/PostingsWriter.c:332
#6 0x002645b8 in kino_PostWriter_finish (self=0x9dba8a8) at
../c_src/KinoSearch/Index/PostingsWriter.c:380
#7 0x00266c13 in kino_SegWriter_finish (self=0x9dbb698) at
../c_src/h/KinoSearch/Index/PostingsWriter.h:266
#8 0x00231e4d in kino_InvIndexer_finish (self=0x9718b70, optimize=1)
at ../c_src/h/KinoSearch/Index/SegWriter.h:259
#9 0x001aabe5 in XS_KinoSearch__InvIndexer__finish
(my_perl=0x8a97008, cv=0x8bed7ec) at KinoSearch.xs:24903
#10 0x0050949d in Perl_pp_entersub () from
/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
#11 0x0050290f in Perl_runops_standard () from
/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
#12 0x004a810e in perl_run () from
/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
#13 0x0804921e in main ()

...and here's a bt with vars, just in case:

(gdb) bt full
#0 bloopy () at ../c_src/KinoSearch/Util/I32Array.c:31
No locals.
#1 0x0023d2be in kino_I32Arr_get (self=0x9dbd1d8, num=-2147406182) at
../c_src/KinoSearch/Util/I32Array.c:37
__func__ = "kino_I32Arr_get"
#2 0x00262309 in kino_PostPool_refill (self=0x9dd88c8) at
../c_src/h/KinoSearch/Util/I32Array.h:225
remapped = <value optimized out>
lex_stepper = (kino_LexStepper * const) 0x9dd8950
main_posting = (kino_Posting * const) 0x9dd89b8
lex_instream = (kino_InStream * const) 0x9dd8a40
post_instream = (kino_InStream * const) 0x9dd8aa8
doc_map = (kino_I32Array * const) 0x9dbd1d8
mem_thresh = 1048576
doc_base = 338003
lex_end = 503
num_elems = 6215
term_text = (kino_CharBuf *) 0x9e67ad8
mem_pool = (kino_MemoryPool *) 0x9e98898
__func__ = "kino_PostPool_refill"
#3 0x0023ae7c in refill_cache (self=0x9dbfb48) at
../c_src/h/KinoSearch/Util/SortExRun.h:264
endpost = <value optimized out>
#4 0x0023b260 in kino_SortEx_fetch (self=0x9dbfb48) at
../c_src/KinoSearch/Util/SortExternal.c:120
__func__ = "kino_SortEx_fetch"
#5 0x002642d2 in write_terms_and_postings (self=<value optimized
out>, raw_post_source=0x9dbfb48, post_stream=0x9e816a8,
skip_stream=0x9f591a8)
at ../c_src/KinoSearch/Index/PostingsWriter.c:332
same_text_as_last = 1
tinfo = (kino_TermInfo * const) 0x9e69b18
skip_stepper = (kino_SkipStepper * const) 0x9dbd0a0
lex_writer = (kino_LexWriter * const) 0x9dba8f0
skip_interval = 16
last_term_text = (kino_CharBuf * const) 0x9dd7420
last_doc_num = 426852
last_skip_doc = 426833
last_skip_filepos = <value optimized out>
posting = (kino_RawPosting *) 0xb79d8ff8
fetch = (fetcher_t) 0x23b230 <kino_SortEx_fetch>
__func__ = "write_terms_and_postings"
#6 0x002645b8 in kino_PostWriter_finish (self=0x9dba8a8) at
../c_src/KinoSearch/Index/PostingsWriter.c:380
folder = (kino_Folder *) 0x9716fa8
i = 24
metadata = (kino_Hash *) 0x9dbc800
#7 0x00266c13 in kino_SegWriter_finish (self=0x9dbb698) at
../c_src/h/KinoSearch/Index/PostingsWriter.h:266
No locals.
#8 0x00231e4d in kino_InvIndexer_finish (self=0x9718b70, optimize=1)
at ../c_src/h/KinoSearch/Index/SegWriter.h:259
to_merge = (kino_VArray *) 0x9db78b8
i = 15
merge_happened = 1
__func__ = "kino_InvIndexer_finish"
#9 0x001aabe5 in XS_KinoSearch__InvIndexer__finish
(my_perl=0x8a97008, cv=0x8bed7ec) at KinoSearch.xs:24903
optimize = <value optimized out>
optimize_sv = (SV *) 0x962d920
sp = (SV **) 0x9135084
ax = <value optimized out>
__func__ = "XS_KinoSearch__InvIndexer__finish"
#10 0x0050949d in Perl_pp_entersub () from
/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#11 0x0050290f in Perl_runops_standard () from
/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
No symbol table info available.
#12 0x004a810e in perl_run () from
/usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#13 0x0804921e in main ()
No symbol table info available.

_______________________________________________
KinoSearch mailing list
KinoSearch [at] rectangular
http://www.rectangular.com/mailman/listinfo/kinosearch

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads