HELP!

MY site got realy big and there´s no
way that I can use SQL / mySQL.
Any ideas for modifications I could do?
I´ll do them on my own, but I got no
good ideas but splitting the links.db
into smaller files.

Thanks,
Jens

I use 3 searches, different databases. Category Search, WebPage Search, and Website Search. In category, i deleted the part that brings the links so thats fast i guess. In Webpage, I deleted the bringing back of the category results up top and the categories above the links. Website, i cut out the category results up top, but i left the bringing back of what category they're in. I can't really tell if it does anything, because my databases aren't that big, but you can try. Backup though.

------------------
-----------------
Lavon Russell
LookHard! Search
www.lh.yi.org
webmaster@lh.yi.org

There is nothing you can do, but split the databases up. Flat file searches have limited ability to handle big files, since they have to be read linearly from begining to end. Because records are variable length, you can't use any easy indexing scheme, and an index would have to be rebuilt any time the indexed file changed -- this is a real problem on a file with 19,000 lines.

The reason for RDBMS is to address these problems, and one of the first things they do is assign fixed-length records. Variable length fields (blobs) are addressed by pointers (usually) outside the file.

MySQL is the answer for this type of problem, in the long term. If you are at 19,000 records now, what will happen when you add another 5,000 ?? Will any fixes you make now be able to handle them, or will you be locked into a modified nightmare that you can't maintain?

SQL databases make life a lot easier once you learn the language. They do a lot of work for you, and behind the scene, to address these problems, and others you haven't yet realized even exist.

Good luck

Switch to a provider that offers sql, or limmit your listings.

There IS a solution to this:

My original search took 25 secs for about
19,000 entries in links.db

Now I use GREP to do the searching and tests
approved that I got 400% faster searching.
There´s nothing better than GREP

Now there is one final problem:
in search.cgi look for a line like:
"Go through each category of links returned, and build the HTML..."

this thingy eats about every inprovements
you do to search.cgi!! IF one could do some
performance upgrades to this one, there´d be no need to upgrade to sql or anything else.

Look: It took me 25 secs to do a searching that grep can do in 1 (ONE!) second!
There must be a solution for this since
the part that I describe above does only do
some string-routines

ANy help ?
Floyd

thanks for your answers. I will develope
a work around by splitting the file into
smaller chunks and I`ll apply the
mod "search this category only"

I modified the search.cgi already but there´s
not real improvement with it. The big prob
at the moment is that the catalog is only
used by some of my visitors. A usual search
for the term 'internet' returns 841 Links
and takes 15 secs... but what will happen if
I have many visitors on the catalog and everyone wants to search something. Perhaps I should try to install ca cache like squid on my system. Perhaps if the file is in cache-memory all the time, it get´s better.

Floyd

The answer is not a cache, but SQL.

I installed MySQL on a Unix machine in a few hours, and took a couple of days to learn it.

This might be a dumb question, but what is GREP ?

im setting up a large database that consists of four main categories, im using Links 2 for to of them, and WebData for the other two, although i am splitting up the databse, i still need to make sure that they will perform well for the next six months or more.

if this GREP can help me further improve the speed, then please tell me how to use it

Thank you,
Laith

By the Grep i use, its not like a cgi or anything. Its a computer program, somewhat like notepad with windows or another text editor except it only searches files. I don't know how much this could help, but i do think that perl-x for windows nt, if used, would be a minor speed solution because it would already somewhat cache what it needs to do. As in the search.cgi searching the database. If new data is put in then i just think it memorizes that data. But it sucks because its only for nt.

------------------
-----------------
Lavon Russell
LookHard! Search
www.lh.yi.org
webmaster@lh.yi.org

GREP is a tool well known from unix.
It´s there to search through any kind of files. You can use regexpressions just like in perl and stuff. By using grep
I am able to speed up a search that took 25 seconds to just 1-2 seconds. That´s a search
through 19,500 Links. Sounds pretty good, eh?

At the moment I´m working on the old search.cgi to imlement all features to my search-version using grep. It now can search, but the output needs to be reorganized. When it´s complete I´ll post it, if you want me to.

Greets, Jens(floyd)

Where do you find the unix-grep? Do you know if it has been said to work on Win32 systems?

------------------
-----------------
Lavon Russell
LookHard! Search
www.lh.yi.org
webmaster@lh.yi.org

GREP is found on any unix/linux distribution
and is available to everyone under
the gnu-license.

there´s also grep for DOS or winNT etc.
just search for it on any
freeware-archive
floyd

 

I'd love to see it when it's finished, if possible.

Dan O.

could you just do somethin like:

$raw_results = `grep $searchstring links.db`;

(or however grep works again)

then parse that out to get a list of results...?? it could probably be made to say what category a link is in, too...

Is there anyway to have this work on NT?

------------------

Yes, thats the deal. I`m working
on a version using unix GREP and SORT
because there´s simply nothing faster
than those two progs. If it´s done, I´ll
post it, too

Jens(floyd)

Yes, like stated above,
GREP and SORT are available for almost
any system.


I have a simple version of my "search-with-grep"
running under
http://www.infowelt.net/nw/links/cgi-bin/search.cgi

give it a try
Floyd(Jens)

It sounds like grep is a very great mod for searching. Can't wait to see the script!

Robbie

Floyd, I want to compliment you on your persistence with this. I have about 30,000 links in my Links2 (with templates) database and I would be very glad to see your ideas and code/mods on this. I am not live as yet and still working on the R&D (research & developement) on my win/98 with Apache server. However I have had some problems with builds and searches once I put some tests online. Would also like some ideas on how you did the time/benchmark testing.

I have also done a mod (DBMan) that uses templates and also a mod that uses DBMan for links "on-the-fly updates, add, delete, and modify that require me to use the search.cgi so that is one big reason this mod of yours is important to me

TimRyan


[This message has been edited by timryan (edited September 05, 1999).]

Count me in as an interested party on the GREP subject for the non-SQL Links2 system (currently installing on a alt/electronica music portal). I use a couple of different shopping cart systems that do lookups of tab and space delimited datafiles using GREP and it's wicked fast (Dell PowerEdge/Linux/Apache and Sun/Apache/BSD) with thousands of products and related pricing, image link, descriptions, etc. So, let us know when it works!

------------------
Christopher Simmons, pres.
Mindset

I would be interested in using grep on UNIX and NT.

Regards,

------------------
Eliot Lee
Founder and Editor
Anthro TECH, L.L.C
http://www.anthrotech.com/
info@anthrotech.com
==========================
Coconino Community College
http://www.coco.cc.az.us/
Web Technology
Coordinator
elee@coco.cc.az.us

Hi Floyd,

You've finish the search script with GREP?

Can you send this script to me?

------------------
[]'s

Lucas Saud - #34750464

timryan,

=====
I have also done a mod (DBMan) that uses templates
=====
Can we get this mod? If so, how-where?

Thanks