Gossamer Forum
Home : Products : Gossamer Links : Version 1.x :

search and/or bug fixed!

Quote Reply
search and/or bug fixed!
Hi all,

searching with links SQL really pissed me off. It took me 3 days of debugging, following the code and understanding what it does. Mainly because DBSQL.pm and Search.pm where written by different programmers using different styles.

The changes to be made are not very complicated and just some lines of code to be modified.

You can download the modification at:

http://www.silberling.com/lsql/index.htm#download

It fixes following problems:

1. filters can be used
2. substring finds all substring matches
3. and / or searches work (including substring / whole word)

I hope this will make an end to all the discussions about Links SQL not searching properly.

regards, alexander

Quote Reply
Re: search and/or bug fixed! In reply to
Thank you, Alex...So, the edited pm files that come with v1.12 do not fix the and/or search options as stated in the download area of Links SQL??

Regards,

Eliot Lee

Quote Reply
Re: search and/or bug fixed! In reply to
Hi Eliot,

I was not able to download the update yet. My request for a new pwd / username for the download area was ignored till now. (I only saved it in my emails, and due to HD crash I lost them all)

Thanks for the irony..

regards, alexander

Quote Reply
Re: search and/or bug fixed! In reply to
Sorry for that question, but why are all these discussions when there is a working update? I am following the threads in the forum with interest because I am running linksql and like the program very much. I have 1.11 running and did not know that the search "bug" was fixed - maybe I didn´t follow the discussions very well but I tried and I am interested. Anyway: If you say that some or most of the search bugs are fixed in 1.12 I am going to install the newer version instantly. Thanks for the information which I obviously missed.

Cheers

Niko

http://www.master-productions.com
Quote Reply
Re: search and/or bug fixed! In reply to
Alex404 you did a great job !!!!

What i really don't get is GS. It was a closed discussion and we had
to wait till next version come no one could solve this problem. Now they have a update (didn't even informed me?) for this ? So the new version 2.0 will take some while i quess. Hmm.....

Sometimes i really don't get those GS guys...

ALEX404 you RULE.

Bye Startpoint.

Quote Reply
Re: search and/or bug fixed! In reply to
Blush...thanks

Quote Reply
Re: search and/or bug fixed! In reply to
Alexander...

I really appreciate your valiant effort with "fixing" this problem...although the only difference I see is that the substring option now works pretty well...but I get the same result with the boolean search...no difference in the number of results.

Regards,

Eliot Lee

Quote Reply
Re: search and/or bug fixed! In reply to
Hi Eliot,

Try to find out if the search object is initialized correctly.

In Search.pm in the sub query the code forkes depending on the requested search mode. This mode was left as default in the old code (It used the default value of 4 (parse) and never changed.) To check it the correct mode was set just insert the following line in the sub query{ of Search.pm :

insert after the lines:

my $self = shift;
my $query_string = shift;

$self->_debug_message ("query: search initilized for '$query_string'", 4);

insert:
print "Search initialized for mode number =" . $self->{'mode'};

(this is an ugly way to debug, it messes up the html output but you donn't need to check the logs.)

Now do a search with bool=or or bool=and

the printout should be:

(bool=and)
Search initialized for mode number =0

(bool=or)
Search initialized for mode number =1

(bool not set)
Search initialized for mode number =4

If the printout corresponds to the parameter bool there must be someting going wrong with search.pm.

If not, the parameters are not passed correctly from search.html -> search.cgi -> dbsql.pm -> search.pm

post me your result and I can help you.

regards, alexander


Quote Reply
Re: search and/or bug fixed! In reply to
Well...I did try that...and nothing works differently. Also, why do you AND and OR for parameter values in your code hack when the values should be and and or? The values are case sensitive between the modules and search.cgi and template files, right? Anyway...I tried both AND/OR and and/or...and still no difference.

Anyway...thanks for your comments...I will try to debug it further and see what happens.

Regards,

Eliot Lee

Quote Reply
Re: search and/or bug fixed! In reply to
Hi Eliot,

The life of the bool param:

1. in your html page you pass the parameter bool=and
2. Search.cgi reads the param $in->('bool') and creates a var called $bool with the value either OR or AND

3a Serch.cgi (Ver 1.1) doesn't care about bool and forgets it..
3b. Search.cgi(Ver 1.2) calles DBSQL->query and passes bool => $bool (still uppercase)

4. DBSQL.pm sub query checks for a parameter 'ma' and sets a var $bool=OR if the param 'ma' is given (regardless of its value) or set $bool=AND if the param 'ma' is empty or not given.

5. DBSQL sub query forgets about the var $bool and the param 'ma' and ignores them.

6. DBSQL sub query (Ver 1.12) checks for the param 'bool'
and passes it to the sub query_indexed (still uppercase)

7. DBSQL sub query_index sets a var $bool with the value of 'bool'.
Now it checks if $bool eq 'AND' if yes, it sets the search object param 'mode'=1 (and search). If not, it sets the 'mode'=4 (parsed search)

8. Here we are finally in Search.pm checking what to do.
In the sub query (of Search.pm) it checks its given params ( $self->{'mode'} ) and basicly does the following:

if mode == 4 it calles _get_by_querystring (default mode always used in Ver 1.11)
if mode == 3 it calles _get_by_phrase

... here is the point where the progammer from GT gave up..

In Reply To:
use Data::Dumper;
print "Content-type: text/html\n\n<pre>: ", Dumper($self);
he just spits out the param of the search object and stops searching...and does not check further modes.

his problem is the little $o variable not being prepaired as the correct hash for..

if mode == 0 it calles _get_by_intersection($o)
if mode == 1 it calles _get_by_union($o)

Now for your primary question where does the bool go?

answer: nowere

For the question does the Ver 1.12 fix and/or searches?

answer: no

I hope this text will help you understanding the changes I made to get it working.

The main thing is:

to get $self->{'mode'}=0 when the param 'bool'=and is passed

and

get $o->{'keyword'}=0 or 1 (depending on substring/whole word (param 'ww')) set correct.

regards, alexander

Quote Reply
Re: search and/or bug fixed! In reply to
I have also fixed the search bug. But I don't know what I have done, perhaps somebody could explain it to me: I have changed the search-ni.cgi to search.cgi (renamed), for that I don't have to reindex every day (much imported links every day). And after that the boolean operators worked. Can this be? I tried Alex404 hints, but I always got a software compilation error in the search.cgi (the orginal), so I have cancel the modifications. I reinstalled the search.pm, db... and search.cgi (ni), and it worked. It's great, but I don't really know why it works. Does not matter...

Quote Reply
Re: search and/or bug fixed! In reply to
Hi Andreas,

There are three main differences between search.cgi and search-ni.cgi.

1. Seach.cgi uses Search.pm to find the indexed words by relevance. Lets say you have the records :

1 - Title: "Hello all"
Descr: "welcome to my world of links...bla, bla"

2 - Title: "Links SQL software"
Descr: "Links is a powerful software to maintain links to..bla, bla"

If you search for "links" with search.cgi it will return record No.2 first and then return record No. 1 calculating a relevance applying to the weight you set.
Search-ni.cgi doesn't care about the relevance nore about the weight. It just looks if the word "links" is in any of the fields and, If yes, returns the record.

2. Search-ni.cgi is slower because it has to query the whole recordset (Title, Description) to find a match (but stops after 1000 results)
If you have a big database Search.cgi is able to search the whole database, sort the results by relevance and return the most relevant results first. Search-ni only returns the first 1000 matches.

3. Search.cgi evaluates filters and also searches for phrases (Search-ni olny checks for each word)

Your users will reward your search-engine with using it and coming back if it returns the answers they want. If it only returns any answers regardless of relevance they will not use it again.

In Reply To:
but I always got a software compilation error in the search.cgi
send me or post the error message. I will fix it for you, or if its a general bug I will change the howto manual.

regards, alexander

Quote Reply
Re: search and/or bug fixed! In reply to
Thanks Alex! I didn't know that the search.cgi ignores the relevance. But that is ok for me, for I do not have that many links. I think I stay to the search-ni.cgi. But why does the boolean operator "and" and "or" work with the search-ni.cgi and not with the search.cgi (except for your modificated search.cgi)?

The same to Alex in German:
Und viele Grüße noch nach Karlsruhe! Ich habe erst spät gemerkt, daß du auch in Deutschland bist. Du kannst dir ja mal, wenn du zuviel Zeit haben solltest, meine Seite (der Jura-Lotse unter http://www.jura-lotse.de) ansehen. Richtig funktionieren dürfte alles erst nach einem rebuild heute abend (die Suche auf der Startseite funktioniert aber jetzt schon). Ich bin wirklich froh, daß die Verknüpfungen endlich funktionieren. Und danke für das Angebot, meine search.cgi zu modifizieren, aber ich behalte doch die nicht-indexed-Version. Da ich jeden Abend ca. 30-50 Links importiere, die ich offline editiert habe und ich nicht jedes mal einen re-index durchführen will. Ich habe auch erst 10.000 Links, daher geht das auch noch schnell genug. Also dann, noch ein schönes Wochende, Grüße aus Augsburg

Quote Reply
Re: search and/or bug fixed! In reply to
Sorry, but the search (search-ni.cgi) with the boolean operators does not function that well. Sometimes it works, sometimes not, sometimes there is a software error. I don't know what it is. Maybe I need Alex404 help to modify the search.cgi. May I send you my search.cgi via email or should I post it here?

Quote Reply
Re: search and/or bug fixed! In reply to
Hi Andreas,

In Reply To:
why does the boolean operator "and" and "or" work with the search-ni.cgi and not with the search.cgi
Well, search-ni.cgi builds a SQL query from the params given and uses the param 'bool' correctly.

search.cgi lets seach.pm do all the work (which is faster). but it ignores the param 'bool' and some other features Search.pm provides.

regards, alexander

Quote Reply
Re: search and/or bug fixed! In reply to
Hi Andreas,

In Reply To:
May I send you my search.cgi via email or should I post it here?
please contact me direcly. The modification alters three scripts: Search.cgi, DBSQL.pm and Search.pm.

Because this is a public forum it wouldn't be fair to publish the heart of this searchengine (DBSQL and Seach.pm)

In Reply To:
Sometimes it works, sometimes not, sometimes there is a software error.
search-ni.cgi simply builds an SQL query from the searchterms passed. Not checking syntax or phrases.
This can cause SQL-errors.

regards, alexander

Quote Reply
Re: search and/or bug fixed! In reply to
Thanks for your help! I contact you directly via email.

Quote Reply
Re: search and/or bug fixed! In reply to
I'd be interested to know if you got this mod working Eliot?

Quote Reply
Re: search and/or bug fixed! In reply to
In Reply To:
I was not able to download the update yet. My request for a new pwd / username for the download area was ignored till now. (I only saved it in my emails, and due to HD crash I lost them all)
Hi Alex,

Thanks for the fixes, I haven't had a chance to look at it yet, but I did send out your username and password to you on the 13th of September. It's a different email address then what you have in your forum profile, perhaps that's the problem? I'm resending it to both emails now.

Cheers,

Alex

--
Gossamer Threads Inc.
Quote Reply
Re: search and/or bug fixed! In reply to
I'm not sure if it makes any difference to the mod and how it works, but I changed the following in search.cgi from ...

($in->param('bool') =~ /^(and|or)$/i) ? ($bool = uc $1) : ($bool = 'AND');

... to ...

($in->param('bool') =~ /^(and|or)$/i) ? ($bool = uc $1) : ($bool = 'OR');

... and it seems to work find on my site.

If I remember correctly I changed this so it found 'any' words, because it was defaulting to finding only sites with 'all' words, but I don't remember having any great difficulty installing the mod.

All the best
Shaun

Quote Reply
Re: search and/or bug fixed! In reply to
Hi Quango,

It shouldn't make a difference because:
In Reply To:
($in->param('bool') =~ /^(and|or)$/i) ? ($bool = uc $1) : ($bool = 'AND');
means: If there is a param 'bool' and it is 'or' or is 'and' then $bool = uppercase of 'bool' ( OR or AND). If there is no param 'bool' or it is not 'or' or 'and' then $bool = 'AND'.

You just changed the default value of $bool = 'OR'

I have seen some sites where (in the search template) the param bool=OR | bool=AND is passed. This is a reason that search.cgi always uses the default value $bool='AND' because the param check is case sensitive.

regards, alexander