Gossamer Forum
Home : Products : Links 2.0 : Customization :

altavista.cgi to search dmoz

Quote Reply
altavista.cgi to search dmoz
I remember reading people who posted code to do this. I believe it was glennu or maybe junko, but i can't remember so i changed altavista.cgi to search dmoz =>
http://lh.links247.net/...-bin/Look/s-dmoz.cgi
Thats the demo, and you can also get the source on the first page just like altavista.cgi.
Adios
Lavon Russell
LookHard Mods
lavon@lh.links247.net
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Cool beans

Making it template based would be a good addition.
Quote Reply
Re: [RedRum] altavista.cgi to search dmoz In reply to
Good idea, i didn't even think of that. I'll probably do that next, and then show how to incorporate it into search.cgi, although the altavista.cgi instructions to do that should be enough. But i'll just see what happens.
Lavon Russell
LookHard Mods
lavon@lh.links247.net
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
This one is in templates:
http://lh.links247.net/...-bin/Look/d-dmoz.cgi
I'll probably come back again with a links incorporation.
Lavon Russell
LookHard Mods
lavon@lh.links247.net
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Excellent :) Thank you, thank you, thank you for your efforts !! Please do post "dummie" instructions to incorporate "dmoz" into the search.cgi with templates.

Pat
Quote Reply
Re: [Patricio] altavista.cgi to search dmoz In reply to
This may help... the "search.cgi" (based in the altavista mod) I used to pulled Excite results:
http://www.qdice.com/..._cgi-with-excite.txt

It is not working anymore but you can see it here:
http://www.qdice.com/....x=41&submit.y=3

As you can see it works with "search.cgi" and Links' templates... It just needs an EXPERT to modify it so it works with DMOZ

Thank you in advance for your help Smile
Quote Reply
Re: [Patricio] altavista.cgi to search dmoz In reply to
Would something like this => http://lh.links247.net/...=keyword&bool=or be fine? Or do you think it would be better like the old one, where you would have to pass maybe &dmoz=1 through the form input to search it and then search it by itself and not your database too?

I think the way the demo is looks better but then again it has to go through the database first to find cats and links, and then go to dmoz and get those. Any ideas on which is better?
Lavon Russell
LookHard Mods
lavon@lh.links247.net
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Uhmmm... The demo LoOkS beautiful Cool !!!

So, if I understand it.... in your demo "one search box" first searched within Links, then Dmoz, and pulled results together into one page.

How about offering a choice for the user...

*Links OR *Demoz OR *Both

Eitherway, it looks great... I wish I had your Perl know how to help more. Thank you again Smile can't wait to learn more and put it to work.

Pat


Quote Reply
Re: [Patricio] altavista.cgi to search dmoz In reply to
ok, so what i'll do is, by default, it could just search links. That is if no search=dmoz is passed in, but then when search=dmoz or something like that is selected, it will only search dmoz, but then both will produce results like the demo i have shown. Sounds good.
Lavon Russell
LookHard Mods
lavon@lh.links247.net
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Thank You, I think it sounds good.
I'm biting my nails Crazy waiting to see the mod.
Thank you again. I'm certain your contribution will be greatly appreciated by many... Smile
Quote Reply
Re: [Patricio] altavista.cgi to search dmoz In reply to
Hey Bmxer,

i think you already have a dmoz fetcher.What people is looking for now is to find a metacrawler or something like this to search a large database.A mod for Google for example will be the best mod i have ever seen and for others too.


Loucian
Quote Reply
Re: [loucian] altavista.cgi to search dmoz In reply to
Google... sounds good but let's stick to the "topic's subject" and get this DMOZ mod to work first.

Besides, a "google mod" may run into the same problems the "altavista mod" did... the search string is often changed making it VERY, VERY hard to keep it working PLUS there is also a question of "copyright issues"
Quote Reply
Re: [Patricio] altavista.cgi to search dmoz In reply to
Yeah, i thought about google first but then realized odp is open source. So anyway,
http://lh.links247.net/...-bin/Look/search.cgi When you select search links search one letter searches like c or c a or web or yahoo, things like that because i don't have many links. When you search both you can search web, yahoo, or anything because it also searches dmoz. and then when you search dmoz, search anything.
Lavon Russell
LookHard Mods
lavon@lh.links247.net
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
I selected 5 results and got hundreds :)
Quote Reply
Re: [RedRum] altavista.cgi to search dmoz In reply to
hey paul,
could you explain a little more. Dmoz only displays 20 per page. So i couldn't do anything with that, and the maxhits for links and cats, that you can set on the search pages are only for the internal links. I don't have hundreds of internal links in my database, so you must mean dmoz. Could you tell me maybe the term you searched for and what options you selected?
Lavon Russell
LookHard Mods
lavon@lh.links247.net
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Hey...!!!! It looks perfect to me Sly
I searched for "Web"

local search:
http://lh.links247.net/...=keyword&bool=or

dmoz search:
http://lh.links247.net/...=keyword&bool=or

both:
http://lh.links247.net/...=keyword&bool=or

Beautiful "Bmxer" !!! Thank you, thank you. Please tell us how to set this up.

Pat
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Hey Bmxer,
Quote:
Yeah, i thought about google first but then realized odp is open source.

Think it better! GOOGLE returns thousands of results and that make a site to look stronger with large database.Also it search not only the text but meta too. Dmoz is a case sensitive and beware you already did a dmoz mod that fetches the categories of dmoz if anyone want to use dmoz.

Think about that Bmxer.Everyone here needs Google results because its the best search engine outhere!

Loucian
Quote Reply
Re: [loucian] altavista.cgi to search dmoz In reply to
Wow! This is fantastic! If you can include Google I will be amazed!!!

Quote Reply
Re: [loucian] altavista.cgi to search dmoz In reply to
loucian,
Like i've said above, Google is not an open source database. As far as i know, they may charge for something like this. I could make this do there site, but as far as distributing it, i don't know. If you'd like to email google and give them a link to the dmoz one and ask if we could make one for there site and release it open source within the boundaries of our Links 2 community, then you can do so. I will make a google version now, but i can't release it without permission as far as i know.
Lavon Russell
LookHard Mods
lavon@lh.links247.net
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
When will the fantastic script you've just coded be available?

Smile
Quote Reply
Re: [boomboom] altavista.cgi to search dmoz In reply to
you mean the one for templates inside search.cgi using dmoz? probably wednesday or thursday.

I've converted a script to try to do this with google, but it's hell trying to parse their results. I get some results fine, but then others are screwed up and printing stuff i told it not to. It's wierd. I'm not too sure its gonna be easy. They probably have it like this for a reason. Another strange thing is when i do searches with them using their search for web, i get 124,000,000 results, but using mine, i get 126,000,000. So i think they can block stuff like this.
Lavon Russell
LookHard Mods
lavon@lh.links247.net
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
yeah, the templates version. Cool :)

Wow - you're a clever guy! With goFetch 2.0 Spider and this you've made things so cool for me....

Thank you so much!
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Bmxer.. I agree with you "Google" will never allow this. You have to pay a fee to use their results within your site, the free program they have allows for you to have your site's logo but no link back, nor you can frame them either: http://www.google.com/services/

SORRY GUYS !! "OFFICIALLY" We can't do it.... So let's STICK TO THE TOPIC AND LET "Bmxer" finish the DMOZ mod first.

Once is done then someone with "no concerns" can modify it to work with google.

For those of us who can't use google, please keep up the good woork Bmxer... please post the DMOZ mod when ready.

Pat
ps: the Fetch mod is intented for a different purpose not to pull results within Links' search.cgi

Quote Reply
Re: [qdice] altavista.cgi to search dmoz In reply to
Yes you are right. Google may need a payment option to allow us to get results from them.


Loucian
Quote Reply
Re: [loucian] altavista.cgi to search dmoz In reply to
ok, i put up very ugly, but prolific instructions. Have i ever mentioned how much i hate making instructions? Because when i make mods, i don't write down what i do so its a pain going back through the files searching for things that are relevant to the mod. Smile
I implemented instructions on how to add my category span as well. if you already have it. Disregard the additions it says to make. The additions will have a ###for cat span above them, and a ########### after them, so you know. anyway here is the code:
http://lh.links247.net/...ads/search-dmoz.html

You need to have a host with LWP::Simple
Lavon Russell
LookHard Mods
lavon@lh.links247.net

Last edited by:

Bmxer: Oct 9, 2001, 1:01 PM
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Yeahp !!Smile you're right Bmxer you are a talented coder but I think I wait 'til the weekend to concentrate on the instructions Wink Can't wait to try them out...

QUICK ??? after the catg mod the dmoz part begins at: sub dmoz right ? and that goes in search.cgi correct ? So as long as I have that in there PLUS the ##in search.html## values and the <%values%> in ##In search_results.html## it should work... ? I just want to try the dmoz mod first and then add the "cat search span mod" What do you think ??
Quote Reply
Re: [qdice] altavista.cgi to search dmoz In reply to
I put both together, b/c i felt with the dmoz mod showing 20 links at a time, it may be much if like 20 cats are shown too. But i had already thought, i'll just get rid of these instructions and put new ones up only including the dmoz search mod. Or i'll just extract the parts for the cat mod. Probably by the end of today.
Lavon Russell
LookHard Mods
lavon@lh.links247.net
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Gracias Smile I did save the other ones too !
Quote Reply
Re: [qdice] altavista.cgi to search dmoz In reply to
ok, you can go to the same place, and just have dmoz search instructions.
Lavon Russell
LookHard Mods
lavon@lh.links247.net
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Ok... I'm stock. Have reviewed every line (keep in mind I'm no Perl coder) and it seems I did it all right... but is not working and I'm getting a "500-error" Here is the search.cgi
http://www.nuzgeeks.com/search.txt

I'll keep at it...Crazy meanwhile any ideas ?

Last edited by:

qdice: Oct 12, 2001, 11:21 PM
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Is the problem here may be ???
Quote:
Code:
Now above &site_html_search_results and under # Print HTML etc...
add this:

# ------------------------------------------

if (($link_hits eq '') or (!$link_hits)) {
$link_hits = '0';
}
if (($cat_hits eq '') or (!$cat_hits)) {
$cat_hits = '0';
}
if (($total eq '') or (!$total)) {
$total = '0';
}
# ------------------------------------------

Now, after # Print out the HTML results.
&site_html_search_results;
}

# ------------------------------------------

add these 2 new subs:

# ------------------------------------------
sub dmoz {

1. You mean in between
#Print out HTML
(SO HERE)
&site_html_search_results;
}

RIGHT??

AND THEN after
&site_html_search_results;
}

Which is now after what I just added...
RIGHT??

AND NOT

"Now, after # Print out the HTML results.
&site_html_search_results;
}
"
Because I just added somehting in between...
RIGHT ??

As you can see I'm a CrazyCrazy SO I'M JUST GUESSING

Also I noticed this part...
Code:
# Store the search results here.
local (%link_results, @category_results);

Is there something missing there ? like <%link_result2%> ??

And how about this...
Code:
# The HTML used in the output is stored here.
local ($cat_hits, $link_hits, $category_results, $link_results, $next);

Again just guessing...CrazyUnsure


Quote Reply
Re: [qdice] altavista.cgi to search dmoz In reply to
Ok so whats the diff between

$link_hits eq ''

and !$link_hits Crazy
Quote Reply
Re: [RedRum] altavista.cgi to search dmoz In reply to
oops, i left that in there, i had the one way, and then just added the other and forgot to delete it
Lavon Russell
LookHard Mods
lavon@lh.links247.net
Quote Reply
Re: [qdice] altavista.cgi to search dmoz In reply to
i didn't know that me saying before this and after this would be confusing. sorry about that. but yes that is where everything should go. I tested it on my site, and it was just a missing bracket. I forgot to put it in the install instructions.

where you have:
Code:
(($in{'search'} eq "dmoz") || ($in{'search'} eq 'both') || ($in{'search'} eq '') || (!$in{'search'})) and &dmoz;
if (($in{'search'} eq "links") || ($in{'search'} eq 'both') or ($in{'search'} eq '') or (!$in{'search'})) {
my $status = &search (\@search_terms, $bool);
if ($status ne "ok") { &site_html_search_failure ($status); return; }
add another right bracket to the end

oh also, lower where you see
else{$addme=20}
make it
else{$addme=20;}


#--------------
I've also updated the mod instructions with the above changes as well as removed redundant code that Paul brought to my attention. Thanks Paul. If you find anymore problems just post them here.
Lavon Russell
LookHard Mods
lavon@lh.links247.net

Last edited by:

Bmxer: Oct 13, 2001, 9:24 AM
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Ok !!!! It's woekingSly

However, I had to make a couple of changes.

What I did:
1. Installed everything per Bmxer mod instructions
2. Made custom changes to "search_results.html" match my site.
3. Changed the "mod" so it will work with a "dmoz_link" template the way the original altavista search.cgi mod did.

Check it out here:

My links/dmoz search:
http://www.nuzgeeks.com/...;IMAGE.y=6&wnh=1

This are the changes I made to the mod.

IMPORTANT: i"m no coder, I just compared the original altavista mod and the Bmxer's Dmoz mod and through trila and error made changes

Code:

### dmozMOD WITH MODIFICATION SO IT WOULD
USE A TEMPLATE CALL DMOZ_LINKS.HTML###
FIRST: Follow all of the original DMOZ SEARCH.CGI
mod instructions then make the changes bellow.


### IMPORTANT: I did not write this code, I'm no
Perl expert. I just took parts of the altavista
search.cgi mod, compare notes and made changes
to the dmoz search.cgi mod with "smart" guesses
and trial and error :-) All I know is that it works.

######## So here it is:

#####-->In "search.cgi" take this part:

# Loop through the results pulling out each link.
if ($total) {
while ($results =~ m#<li><a href="([^"]+)">(.+?)</a> - (.+?)<br>#sog) {
$numlist= $numlist+1;
($url, $title, $description) = ($1, $2, $3);
$link_results2 .= qq~$numlist. <a href="$url">$title</a> -
$description<br><small>-- $url</small><p>~;

#####-->and replace with:
# Loop through the results pulling out each link. INCLUDES ALTAVISTA dmoz_link

if ($total) {
while ($results =~ m#<li><a href="([^"]+)">(.+?)</a> - (.+?)<br>#sog) {
$numlist= $numlist+1;
($url, $title, $description) = ($1, $2, $3);
%tmp = ("Count", $count, "URL", $url, "Title", $title, "Description", $description, "URL2", $url2, "Numlist", $numlist);
$link_results2 .= &site_html_dmoz_link (%tmp) . "\n";

#####--> Then in "site_html_templates.pl" add this routine:

sub site_html_dmoz_link {
# --------------------------------------------------------
# This routine is used to display what a link should look
# like.


Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Question about if's and end's

I have this before the Links results:
Code:
<p><font face="Georgia, Times New Roman, Times, serif" size="2">
Your search returned <strong>
<%cat_hits%>
</strong> Categories and <strong>
<%link_hits%>
</strong> nuzLinks.</font></p>
<%if next%>
<%endif%>
<%if category_results%>

How would I edit the if and end if tags so it doesn't
print the number of categories and Links when
searching DMOZ ?

Thank you for your feedback
Quote Reply
Re: [qdice] altavista.cgi to search dmoz In reply to
I think it would take to much logic to try and get that to happen and the template parser won't do things like or , and. If you go to mine, you'll see i put how many links and cats and webpages there are inside the
<%if link_results%> call.
Actually, you could do something like this:
Your search returned
<%if link_hits > '0'%><%link_hits%> websites : <%endif%> <%if cat_hits > '0'%> <%cat_hits%> categories : <%endif%> <%if total > '0'%> <%total%> webpages<%endif%>

but if you do it this way, more than likely, you'll get a colon after one of the totals and there will nothing trailing
like if you only search websites, you'll get
5 websites : 2 categories :
only if you search all will you get it like this
5 websites : 2 categories : 5000 webpages
Lavon Russell
LookHard Mods
lavon@lh.links247.net
Quote Reply
Re: [Bmxer] altavista.cgi to search dmoz In reply to
Hi Bmxer... are you around ?

The DMOZ mod is dead Unsure not sure why ??

Check this out:

http://www.nuzgeeks.com/...bin/links/search.cgi

A links only search works, but a dmoz or a combine search doesn't

I'm assuming something has changed with DMOZ, may be the search query...? PLEASE HELP Pirate

Pat
Quote Reply
Re: [qdice] altavista.cgi to search dmoz In reply to
Ok.... I changed the dmoz search string...

FROM THIS:

#----------
sub dmoz {

# We are searching, so get the query, escape it, and use LWP to search.
use LWP::Simple;
$query = $in{'query'};
my $query_q = &urlencode($query);
$num = 20;

$nhst = $in{'wnh'};
if ($nhst) {$addme = $num * $nhst;}
else{$addme=20;}
$base = $addme;
$numlist = $base - $num;
$pages = $numlist +1;
$page = ($in{'wnh'} * 20) || 1;
$url = "http://search.dmoz.org/cgi-bin/search?search=$query_q&jstart=$pages";
$results = get ($url);


TO THIS:
$url = "http://search.dmoz.org/cgi-bin/search?search=$query_q";

It works now, but since I'm not code expert not sure what I'm missing by have taken the last part out but I'm guessing it added consecutive #'s to each link result found...?? I'm also assuming it stoped working becaue dmoz must have changed ?

Any instructive feedback ??

BTW, here is the error I was getting before:

http://www.nuzgeeks.com/...arch-links-error.htm

Pat

Last edited by:

qdice: May 9, 2002, 10:50 PM
Quote Reply
Re: [qdice] altavista.cgi to search dmoz In reply to
NOW THIS IS WEIRD...!!!PirateUnsureCrazy

The change above... works in one site but not in another... same exact script running in both...

Check it out:

http://www.nuzgeeks.com/cgi-bin/links/search.cgi?query=news&mh=25&search=dmoz&type=keyword&bool=and&submit.x=27&submit.y=12

http://www.qdice.com/cgi-bin/links/search.cgi?query=news&mh=25&search=dmoz&type=keyword&bool=and&submit.x=27&submit.y=12

IS THIS A SERVER SIDE PROBLEM ??
Quote Reply
Re: [qdice] altavista.cgi to search dmoz In reply to
Because the one server that it doesn't work on is missing necessary modules like warnings.pm. BTW: Can't locate means that the script cannot find the relevant module/PM file.
========================================
Buh Bye!

Cheers,
Me