Mailing List Archive: kinosearch: discuss

Issue with KinoSearch

Index | Next | Previous | View Threaded

ox.jobs at gmail

Apr 24, 2008, 6:01 PM

Post #1 of 4 (1034 views)
Permalink

Issue with KinoSearch

When trying to create a new invIndexer, I keeping getting this error...

[error] Caught exception in
GMail::Controller::User::Mail::Folder->begin "Error in function
kino_FSFolder_open_outstream at c_src/KinoSearch/Store/FSFolder.c:55: Can't
open 'seg_t.ds': File exists

at /usr/lib/perl5/vendor_perl/5.8.4/i686-linux/KinoSearch/Index/SegWriter.pm
line 42

KinoSearch::Index::SegWriter::new('KinoSearch::Index::SegWriter',
'invindex', 'KinoSearch::InvIndex=SCALAR(0x87b8198)', 'seg_info',
'KinoSearch::Index::SegInfo=SCALAR(0x87c93fc)')
called at /usr/lib/
perl5/vendor_perl/5.8.4/i686-linux/KinoSearch/InvIndexer.pm line 110

KinoSearch::InvIndexer::new('KinoSearch::InvIndexer', 'invindex',
'KinoSearch::InvIndex=SCALAR(0x87b8198)', 'lock_factory',
'KinoSearch::Store::LockFactory=SCALAR(0x87b8174)')
called at /usr/lib/gmail_maildir/GT/Maildir/KinoSearch/Indexer.pm line 202

GT::Maildir::KinoSearch::Indexer::invindexer('GT::Maildir::KinoSearch::Indexer=HASH(0x8a2ec78)')
called at /usr/lib/gmail_maildir/GT/Maildir/KinoSearch/Indexer.pm line 81

I am using the following code to create a new invIndexer:

sub invindexer {
my $self = shift;
my $path = $self->get_maildir->get_config_path . "/kino";
my $hostname = hostname();
die "Can't get unique hostname" unless $hostname;

my $invindex = GT::Maildir::KinoSearch::Schema->open($path);
my $lock_factory = KinoSearch::Store::LockFactory->new(
folder => $invindex->get_folder,
agent_id => $hostname,
);
return KinoSearch::InvIndexer->new(
invindex => $invindex,
lock_factory => $lock_factory
);
}

Anyone have an idea what might be causing this?

______________________________
J. "Olyx" Gibson
Software Programmer
Austin, Texas
Cell: 512-922-9825

marvin at rectangular

Apr 25, 2008, 12:19 AM

Post #2 of 4 (971 views)
Permalink

Re: Issue with KinoSearch [In reply to]

On Apr 25, 2008, at 12:30 AM, Henry wrote:

>> When trying to create a new invIndexer, I keeping getting this
>> error...
>>
>> [error] Caught exception in
>> GMail::Controller::User::Mail::Folder->begin "Error in function
>> kino_FSFolder_open_outstream at c_src/KinoSearch/Store/FSFolder.c:55:
>> Can't open 'seg_t.ds': File exists
>
> I may be off base here, but that looks like an existing index which
> KS is
> protecting. Try using:

You're right, Henry. I'm investigating. More in a bit.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

_______________________________________________
KinoSearch mailing list
KinoSearch [at] rectangular
http://www.rectangular.com/mailman/listinfo/kinosearch

henka at cityweb

Apr 25, 2008, 12:23 AM

Post #3 of 4 (967 views)
Permalink

Re: Issue with KinoSearch [In reply to]

> When trying to create a new invIndexer, I keeping getting this error...
>
> [error] Caught exception in
> GMail::Controller::User::Mail::Folder->begin "Error in function
> kino_FSFolder_open_outstream at c_src/KinoSearch/Store/FSFolder.c:55:
> Can't open 'seg_t.ds': File exists

I may be off base here, but that looks like an existing index which KS is
protecting. Try using:

Schema->clobber($index_path)

or if you plan on using stable:

create => 1,

Regards
Henry

_______________________________________________
KinoSearch mailing list
KinoSearch [at] rectangular
http://www.rectangular.com/mailman/listinfo/kinosearch

marvin at rectangular

Apr 25, 2008, 12:33 AM

Post #4 of 4 (962 views)
Permalink

Re: Issue with KinoSearch [In reply to]

On Apr 24, 2008, at 6:01 PM, Ox Gibson wrote:

> When trying to create a new invIndexer, I keeping getting this
> error...

> kino_FSFolder_open_outstream at c_src/KinoSearch/Store/FSFolder.c:
> 55: Can't
> open 'seg_t.ds': File exists

OK, I've now duplicated the problem. Thanks for the report.

The last KS releases, both maint and devel, were primarily intended to
ratchet up paranoia about clobbering files for security reasons.
Throughout the KS code base, instead of clobbering, we now unlink then
perform a safe open using (O_CREAT | O_EXCL). It looks like this
problem has arisen because I missed an unlink.

seg_t.ds is an old file left behind from an aborted indexing session
-- maybe a crash, or maybe a quit prior to calling finish(). With
earlier versions, the InvIndexer would just clobber this file. Now
it's trying to perform a safe open, but the unlink hasn't happened
first.

I believe that the solution is for InvIndexer to sweep the invindex
directory for unused KS files before it tries to initialize the
SegWriter. I'll try and put together a new release in the next couple
of days that does that. In the meantime, you should be able to clear
the block by moving any files that start with 'seg_t' out of the
invindex directory.

> I am using the following code to create a new invIndexer:
>
> sub invindexer {
> my $self = shift;
> my $path = $self->get_maildir->get_config_path . "/kino";
> my $hostname = hostname();
> die "Can't get unique hostname" unless $hostname;
>
> my $invindex = GT::Maildir::KinoSearch::Schema->open($path);
> my $lock_factory = KinoSearch::Store::LockFactory->new(
> folder => $invindex->get_folder,
> agent_id => $hostname,
> );

Since you're using a lock_factory... is this invindex is on an NFS
volume?

Do you know whether an indexing session crashed before completion, and
if so what the cause was? The fix is the same regardless; I'd just
like to know what aborted the indexing session.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/

_______________________________________________
KinoSearch mailing list
KinoSearch [at] rectangular
http://www.rectangular.com/mailman/listinfo/kinosearch

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads