Gossamer Forum
Home : Products : Links 2.0 : Customization :

Links -- Altavista style?

(Page 1 of 2)
> >
Quote Reply
Links -- Altavista style?
What are your thoughts of an Altavista style search engine? Is this something people would be interested in? I see it as:

1. An admin area where you have full control to add/remove/modify/view links.
2. A public add link routine.
3. A public search routine.

It would be much simpler then Links, as it is not category based, all the links are just in a pool. It would have an enhanced searching, probably very similiar to Altavista style syntax.

Any thoughts?

Alex
Quote Reply
Re: Links -- Altavista style? In reply to
Do you mean that it would be like a regular search engine. Because that would be pretty cool.
Michael
Quote Reply
Re: Links -- Altavista style? In reply to
Yes, that would be a great idea!

Can we have a feature where users can Add to the engine's pool automatically. That way we don't have to spend too much time validating new URLs everyday. Also, have a option of shutting down automatic add urls in case you wanted to validate by hand.

Just an idea.

John
Quote Reply
Re: Links -- Altavista style? In reply to
Alex, this would be a superb idea if it could be integrated with Links so that if I validate an entry it adds it to the directory (Yahoo style) plus crawls a set number of pages from that site (Altavista style) so that in the event of a search not being succesful it will default to the search engine results.

Obviously hard disk space will present a problem if this got too big, but there may be a way of limiting the number of pages indexed in order so save disk space ?

------------------
Craig Antill
10 Tenths Motorsport
www.Ten-Tenths.com
Quote Reply
Re: Links -- Altavista style? In reply to
How about integrate it with Links:

1) Enhance the links search routine.
2) Allow both a directory, and a straight search engine type site??



------------------
--
Ed Bownass - edward@bownnet.demon.co.uk
Links mods site - http://www.bownnet.demon.co.uk/linksmods/
Quote Reply
Re: Links -- Altavista style? In reply to
if it was an option to have the altavista style and yahoo style then go for it cuz I like the yahoo style cuz altavista is messy
Quote Reply
Re: Links -- Altavista style? In reply to
Man Alex, I wish I new haft of what you know, You written 3 of the best programs, and coming out with anther one, coming from you I would know that there had to be some good in it, I say go for it, but one qustion, will it be free to download for personal sites like Links, DBMAN, and File Man?
Thanks
Joey

------------------
Links is HOT! Get it and you will LOVE IT!
Quote Reply
Re: Links -- Altavista style? In reply to
XAV made something similar to this at www.xav.com/latest

However, I didn't like it beacuse it couldn't handle very many pages, and it took too long to re-index. I also found tat the database couldn't bebigger than a set amoun, maybe 5mb. If you can make an ALtavista search that can handle several thousand 20-30 thousand pages, then I would absolutely love you!

------------------
Nicholas Clark

Quote Reply
Re: Links -- Altavista style? In reply to
That would be wonderful....
I tried to do exactly that on my site, but some of the feature had to be taken out and it still has a few bugs...

Check it out:
http://searchme.hypermart.net

-Todd


P.S. Alex has to be a genius b/c Links is coded so well...so advanced... i would never have thought of half of this stuff... :-)

Quote Reply
Re: Links -- Altavista style? In reply to
That would be great. However, I think that would be a prime candidate for mSQL or some other DB engine (rather than text files) in order to handle the search time. - IT WOULD BE AWESOME THOUGH!
Quote Reply
Re: Links -- Altavista style? In reply to
Alex, I like the idea very much so, but I do think Sandline brings out a good point since speed is of concern at least to me.

In Addition, I would much rather install capabilities for users to search the web without having to leave my site to do it.

But, at the same time, Links 2.0 seems to be exactly what I need since it will be for a proprietary website with a closed market, and I do believe scripts like Links are better the way they are...PRIVATE/custom since not everyone wants TOO many links irrelevant to site goals.

Now let me clarify this a bit.

On our website (when its launched) the Links 2.x will be used for Jewish Sites ONLY, since its a website dedicated to the world Jewish Community.
Therefore, I need not have something too powerfull, UNLESS its a real Search Engine open to the public, which is not.

So, for sites like ours, and other individual sites that target a certain Market, Links 2.0 is PERFECT.

These are my two cents.

Sol
Quote Reply
Re: Links -- Altavista style? In reply to
Sounds interesting, however, I like the Yahoo style of links as it is now (with categories.......).

So if ever you plan to do some kind of Altavista version of LINK, do not abandon the current system. Altavista suff are good, but it can end up like a big big mess, with people spamming the database. It would be very unsuitable for very targeted search engines, which require very precise listings, and LINKS as it is right now is perfect for this.

Perhaps a combination of both ? Links are submitted by users and directly accessible by web surfers (ALTAVISTA), and then the admin may categorize some links (Yahoo).

So the users would end up with 2 search mode:
1) the pool mode: (altavista)
2) The clean mode (searching the categorized links only)

I even think that this combo can be EXCELLENT !! But it would require from you lot of work and as it may generate a very intense CGI activity, it would absolutely require SQL database treatement.

I would suggest that you finish this version 2.0 so we'll all have the best YAHOO syle search engine possible, and then in a version 3.0 you could add the "altavista" mode, in which users could submit their site, and instead of waiting for the admin to validate them, it would be added to a second database directly searchable for the users, and in which the admin would pick up the links to be categorized.

But for now, lets finish 2.0 Smile)

Paul

[This message has been edited by Paul (edited December 28, 1998).]
Quote Reply
Re: Links -- Altavista style? In reply to
Alex, Quote: Umm, perhaps you should ask the XAV author that question? I have no idea how they do their searching, and wading through 120k of code is daunting even to me. =)
when you go to write it, would you put it like that 120K? Or would you put it like Links? Like you said above from anther post 120K is a bit to big, please make it like Links where all the HTML will come from somehting like site_html.pl file, that way we can easily use/modify it, and have it work something like Links(links.def)
Thanks
Joey
Quote Reply
Re: Links -- Altavista style? In reply to
Just a thought on AltaVista and incorperating its style of design into Links. I think this boils down to the difference between context type search engines and content type engines. AltaVista in essence doesn't do alot to allow users to add their own links. They more or less do the work for you by spidering or roboting the web and writing content into the DB as it finds stuff.

Is this a good idea for Links? In some ways, it could be, but in other ways it could be a horrendous mess. For one, most people I know, don't have the server resources to catalog a few million pages like Alta Vista does. However, a couple of ideas strike me that might make this useful to the average Joe user of links:

1) As an admin for a links DB its sometimes hard to get exposure for your site due to lack of links in the DB. A program could be devised to spider through existing links or an engine like Yahoo or AltaVista to create a DB of links for Addition to the Links engine. I would work this like validation, because it could be possible to add thousands of useless links into your DB at the stroke of a hand.
2) If a user cannot find relavent info through normal DB searches, you could allow references to Lycos, AltaVista, Excite through links on the subsequent search page. This is what Yahoo does already. In fact, to my knowledge, Yahoo uses AltaVista's search engine for queries that do not turn up searches on their local DB.

These are just my takes on an AltaVista Style of links. If we are thinking of making a simple version of a Free-for-All Links program, its not a bad idea. I do not think there are any currently with "great" admin functions. I personally prefer the ability to Categorize links.

------------------
Fred Hirsch
Web Consultant & Programmer
Quote Reply
Re: Links -- Altavista style? In reply to
Yahoo now uses their own robot called "inkotomi" which i have seen robot my directory a few times.

Also, unless you have a server co-located or inhouse, no one would have enefough system resources to even try to cerate a altavista spider for their website.

------------------
-----------------------------------------------------
James Ribar
Webmaster http://ddo-rpms.hypermart.net/
Founder & Webmaster http://netace.hypermart.net

No More Searching! Find!
Use NetAce to find stuff on the internet quickly and effecently.
http://netace.hypermart.net

Get Your Free NetAce e-mail address @ http://netace.zzn.com

Quote Reply
Re: Links -- Altavista style? In reply to
Wow! Quite the responses. =) First off, I don't envision this as part of Links, but rather a separate program. The value in Links is in creating a specialized directory using your own expert knowledge, something Yahoo or the larger directories can not duplicate. For example, if you are looking for information on Windows CE, which would you like to search through CEGlobe.com or Yahoo? Smile

For this program, what I had imagined was an SQL based search engine that people could easily add thousand of URL's from a number of different sources:

1. Auto submission tools.
2. Visitors to the site.
3. Spidering URL's
4. Admin interface.

There wouldn't be any categorization. I have set up two of these before, and both now have 125,000 low quality links (mainly from auto submission tools), but do get a lot of traffic as well as exposure from the autoresponders sent out with each addition.

If there is demand for this type of tool, then I'll look into it some more. It wouldn't come close to being devloped till late Feb, but just a thought. Smile

Cheers,

Alex
Quote Reply
Re: Links -- Altavista style? In reply to
I think an option to display the other spidered pages on another page after the intital search results?


MY 2c

------------------
elitecaraudio.com
carforums.com
Quote Reply
Re: Links -- Altavista style? In reply to
Hi Alex,
I just wonder how much storage space you used to index 125000 links ?

I used Fluid dynamic II (what they call XavII) to index about 100 webpages within one hour and it works great.
But it totally used up about 600 kbytes.
So, i give up.

Could you tell me?
Thanks
Quote Reply
Re: Links -- Altavista style? In reply to
This would be great except I would like the spider to search for keywords that I list to keep out site that do not pertain to my niche market.

Jerry
Quote Reply
Re: Links -- Altavista style? In reply to
I would like to point out to all a url http://www.alta-vista.net/av/content/about_our_technology_2.htm telling aobut altavistas hardware. anyone that is trying to make a altavista is outa luck unless youy want to invest in 30 or more suped up computers.
Quote Reply
Re: Links -- Altavista style? In reply to
I must agree, instead of trying to copy after the big shots why don't every one just take and do there best, not someone elses, think Alta Vista, Hot Bot, Excite, Yahoo did like that? Im not sure if ya are into evrything I am or not, but I can say if you was plainng to get as big as the above you'll need servel if not more millions of dollars, when and if Alex comes out with a spider script it would be great, and get the job done, but it will never be Alta Vista, it takes more than just a script to get anywhere(no hard intentions intended Alex)I have had many sites over the last 2 years, and must say that its not easy, I was on a dedicated server once, and there is simpely to much to do, think about it before you try and become one of those big shots!
Thanks
Joey
Quote Reply
Re: Links -- Altavista style? In reply to
I have an idea, a spider that goes out and spiders the sites in your links database, but does not add them to links, it adds them to a diffrent database and when people search you get the main search and the spider found ones at the bottom....
Quote Reply
Re: Links -- Altavista style? In reply to
Alex, this sounds like a fabulous idea! Keep the great script coming! I think this would help us small nich sites to really do some justice Wink

------------------
Brian
wwnurse.com
Quote Reply
Re: Links -- Altavista style? In reply to
The real great use for this idea would be to search your niche sites.

Imagine: click to Add in Links (it could be a separate program Alex, but also availible to integrate into Links) - anyway there is are two checkboxes:
Add to Directory
Add to Index

perhaps not the clearest words to use.. and there could be a description. Any directory would have Categories, everything we are used to for Links and be subject to validation. The Index would also be subject to validation.

So if someone is running a Links directory for Extreme Sports, they would search for Extreme Sports websites in the directory OR have the option of searching through (ala Altavista) other various top Extreme Sports sites. It wouldn't have to be a general spider that indexes every site in the known universe but simply a site that indexes given sites in the Links directory. - and would be a GREAT addition

Quote Reply
Re: Links -- Altavista style? In reply to
If I remember correctly all the source code for Xav II is availible at www.xav.com - and it is freeware from an abandonded project. Possibly it might make sense to simply encorporate it's functionally more tightly into Links?


[This message has been edited by nabeelh (edited March 10, 1999).]
> >