Gossamer Forum
Home : Products : Gossamer Links : Version 1.x :

search.pm sort by priority

Quote Reply
search.pm sort by priority
Hi everyone,
I have modified LinksSQL by adding a priority rating to the links display. Everything works perfectly except for the fact that in searches I cannot seem to sort the order of the display. I have studied the search.pm, search.cgi, and search-ni.cgi modules for insights on how the search results are prioritized by score, yet, I need to add a routine which will sort the outcome on the basis of its priority rating. (By the way, priority ratings are an integer value from 0-9 located in a column called isPriority in the Links table).

I have attempted to change the SQL statements with an "ORDER BY Links.isPriority" clause (where applicable), but, to no avail. I have also attempted to modify the sort commands but with no success either.

How can I access the ranking system of the returned results to add sort parameters? Or, is it possible to wait until the sort is completed and (before the output is created) add a routine to sort by priority?

Has anyone successfully done this hack? I would appreciate any insights you might have to help me solve this problem.

Thanks

Quote Reply
Re: search.pm sort by priority In reply to
Hey,
Anyone out there?

I have now searched most of the routines involved in querying, but, mostly I find myself more perplexed now than before. I am having an exceptionally hard time nailing down which routine actually sorts the results prior to output. Each time I attempt to modify these routines I get various errors -- mostly SQL join errors. I have also attempted to bind collumns to "force the results" but also to no avail.

I should note that the priority rating works perfectly if you click on a category and view the entire topic. However, my problem is in trying to sort the results returned when a search is conducted to show the priority links in descending order within their respective categories. If you would like to try it out go to http://search.viafamily.com and type in "web promotion" into the search box. Note the results. Then follow the link to the category heading "web site promotion" and note the order the links come up in -- this is the correct order (Jimworld - priority=9; Internet Resources - priority=6; Submit it - priority=4; Promotion world - priority=2).

If anyone can help me isolate the correct subroutine or could suggest a viable work around I will be eternally grateful.

TIA,

Daren

Quote Reply
Re: search.pm sort by priority In reply to
I don't know if this was specifically addressed in the 1.13 upgrades to the search routines, but prior to that, the searches could only be returned in 'score' order. This was something added in the last beta (?) where searches were ranked on relevancy.

When I asked alex about it when the 1.11 (?) release came out, he said the change was too deeply embedded to be changed easily, if at all.

The search routines were changed in the 1.13 release, but I'm not sure if that was addressed. Most of us worked around it, waiting for the next release to come out which does address some of the more important and lingering issues with the searches.

PUGDOGŪ
PUGDOGŪ Enterprises, Inc.
FAQ: http://pugdog.com/FAQ


Quote Reply
Re: search.pm sort by priority In reply to
Pugdog,
Thanks for your insights. I do have to agree with you that these routines are very deeply imbedded in the guts of the program! ;) However, I think I have finally come up with a solution that I will be implementing this morning. If I can get it to work then I will post here again and let you know. I will also create a web page for viewers of this message board so you can see what I have done in making the changes. I have been keeping extensive notes on the changes I have made and I will post that for people to see. So, as soon as I get this done I will post the URL here.

Thanks again for your help and insights! I appreciate it!

Big D

Quote Reply
Re: search.pm sort by priority In reply to
Hi,

Links SQL uses two ways of searching.

1. Indexed Search (via search.cgi)

When a link gets added by validate it is also indexed. LinksSQL has a table in which keywords are stored with information with: keyword, link-id, weight.
This information is gathered by the indexing routine. It checks if a weight is set for the columns, calculates the link-keyword-weight and adds it to the search table.
eg:
ID: 1234567
Title: how search works (weight = 3)
Description: Search.cgi is a perl script used to search (weight =2)
URL: http://www.xy.com/search/documentation (weight=1)

the weight for the keyword "search" will be 8 because
Title: found 1x weight 3 = 3
Descr: found 2x weight 2 = 4
Url: found 1x weight 1 = 1
------------------------------
Weight for this link = 8

Now a user searches via search.cgi for "search"
Search.cgi will search the search-table for "search"
and will find the link 1234567 with a weight of 8 for this keyword together with other links orderd by weight.
This list of link-ID's is returned. Now the link data is retrieved from the links table to be displayed.

To change this order you have to hack search.pm and this is
no fun at all.

in order to let some links "be more equal than others"
I would propose to ad a new field in the links table named "Keywords" and give it a extreme weight (eg. 8)
Add the relevant keywords, seperated by spaces, to the
links you want to have displayed first in your search
results. (Do not forget to re-index the database)



2. Non Indexed Search (via search-ni.cgi)
The non indexed search is a simple SQL query on the links-table, ignoring any weights you set.

To manipulate this search I would propose to add a new field to the links table named "priority", weight 1. (The weight you assign just has to be > 0 ) containing numbers from 0 to 9 where 9 is the highest priority.

Now just change the sortorder to by priority

I hope it helps

regards, Alexander




Quote Reply
Re: search.pm sort by priority In reply to
Alexander,
Thanks for your insights as well! I made the modifications to search-ni.cgi a while back and they work perfectly (simply adding the ORDER BY statement). But, thanks anyway.

As to your help on the search.cgi mess, your comments were very helpful. THANKS! The hack I intended this morning did not quite become as simple as I hoped. However, I have decided that I will probably have to intercept the returned index score and then add a routine to check the priority rating for that link, or, I will have to add my own indexing routine into the already existing routine to give the priority links more of a chance of "coming out on top". So, your suggestion about adding a keywords routine might help (except that I will have to rename it because we already have a keywords entry in the form and database which is highly indexed). I am going to try the first route now (hopefully the easiest) and, if that fails, the second.

Thanks so much for your insights.

Cheers,


Daren (aka Big D)

Quote Reply
Re: search.pm sort by priority In reply to
Hello again,
I have been unable to resolve the problem. Perhaps some of you C++ types out there can explain to me the following line of code from subroutine search in search.cgi:

my @query_results = sort { $$b[1] <=> $$a[1] } map ( [$_, $$id_results {$_} ], keys %$id_results );
$self->{total_results} = $#query_results + 1;

I understand line two (getting the total number of results). I also understand that the categories are being sorted based upon their indexed value and that they are being sorted from highest to lowest. I looked up the sorting routine being used in this line and it seems to be "a Schwartzian Transform" function mapping to a "non-existent array" for the purposes of sorting. However, I need help with the part inside the brackets. What is this? What are the comma separated elements here?

Could someone explain this to me? The only example I have of this is from a book and it is unfortunately not of sufficient detail to explain what is going on here.

I would greatly appreciate any help in this matter.

TIA!


Daren

Quote Reply
Re: search.pm sort by priority In reply to
Hey everyone,

Its me again.

I did some research on the Schwartzian Transform and now I understand it and what is going on here. Unfortunately, this is not the answer to the problem. I am barking up the wrong tree again.

What the above mentioned line does is it creates a non-existent array to store the individual keys of the id_results hash which is then sorted by the key which corresponds to the score (in this case the second a[1]) of the links which matched the search criteria. This creates a list of references to the original hash values which can be sorted by score from highest to lowest and then re-mapped into the array query_results in the proper order by highest score. The subroutines for producing the output then divide the results up by score within their individual category id numbers.

THINKING OUT LOUD:
What I need to do is intercept the scoring routine and multiply the score of the link by its priority value. However, I need to limit that to within the categories themselves. I don't think that this can be done the way that the program is written. If I changed the scoring routine that would skew the results of the search so that the matching priority links would come up first in the results even when they might not be the best match for the search. The best thing to do would be to intercept the results at the point in which they are being outputted and then sort each category by score adding in the priority value to make it come up on top but only within the category.


If anyone has any thoughts on this please let me know. I will continue hacking away and I will post again if anything changes.

Thanks!


Big D

Quote Reply
Re: search.pm sort by priority In reply to
ABANDON SHIP!!! ABANDON SHIP!!!
I have been forced to abandon my quest to sort the resulting links by their priority order. Based on the way the program works I don't think that it can be done. If anyone is interested in seeing my notes and modification logs please feel free to contact me (melson@711.net). I will be glad to e-mail you a copy.

Thanks to Alexander and Pugodg for your thoughts and insights. Happy computing!


bigd
(aka Daren Melson)

Quote Reply
Re: search.pm sort by priority In reply to
I'll bet you learned a lot about Links and PERL in the process though ;)

PUGDOGŪ
PUGDOGŪ Enterprises, Inc.
FAQ: http://pugdog.com/FAQ


Quote Reply
Re: search.pm sort by priority In reply to
No doubt about that! ;)

I am mostly a Java programmer. My company's web department liked the script and had been using Links 2.0. We upgraded to LinksSQL because of the sheer numbers of links we have. They asked me to modify the script to add certain functions, so, I got a real crash course in Perl. Yet, I love to learn, especially programming, and I enjoyed the challenge.

Thanks again for your help.

bigd

Quote Reply
Re: search.pm sort by priority In reply to
Hi,

As you probably found out, the indexed search is very difficult to modify, and the concept of returning best matches first is very tightly ingrained. If you are using version 1.x, and need to change the sort order, you should use search-ni.cgi and add ORDER BY Column to the query.

Version 2 supports changing the order of results in indexed mode.

Cheers,

Alex

--
Gossamer Threads Inc.