I noticed that the more big it is going being the more database reading Links it is to accomplish a search in the database, what should not happen with a database with less than 500.000 URL's,

It would like the team of development of LinksSQL to observe that problem better, because after all Links SQL was made to support great volumes of URL's.

------------------
-=-=-=-=-=---=-=-=-=-=-=-
Elijânio - Br Company
-=-=-=-=-=---=-=-=-=-=-=-

What is your server configuration?

Most likely, it's a RAM problem on your server, not a links problem.

Could you post (or email me) your server configuration, and I'll take a guess! IF the server is under configured, that's the problem. If it's not, it might be something else --- are you running a standard links installation, or one with a lot of modifications? Are you running mod_perl?

I know there have been some posts on what machines can do in the FAQ several months ago, you might want to check for those.



------------------
POSTCARDS.COM -- Everything Postcards on the Internet www.postcards.com
LinkSQL FAQ: www.postcards.com/FAQ/LinkSQL/

One of the main issues is RAM. You'll never want to go to disk when using Links SQL, and you should have at least enough ram to hold your entire Mysql database in memory. That means if your database is 350 MB, I would recommend 512 MB of memory (because Apache, sendmail, and others use memory too).

That said, we are always looking at improving the code, and will be trying to tune it better in the future. However, you are never going to be able to run something the size of DMoz on a virtual server account.

Cheers,

Alex

Now I am just with 128 MB-RAM
I am using mod_perl 1,21
and I am not running in virtual server.
The Server is really my.
I have 140.000 links in the database and on average 50.000 page visualizations a day.

I will increase the RAM memory to see if it solves the problem.

I know that I'm not normally welcome here, but I thought I'd throw out my 2 cents here...

Just yesterday, we solved a similar problem with one of our hyperseek customers, who was having like 10 second search times (sometimes longer) on a 350,000 link database. His machine is a p300 with 256 MB Ram.

After doing a ton of testing, logging, tracking, etc. we finally nailed it down to a combination of 2 things, which together solved the problem:

http://www.iwebsupport.com/kb/reviews/952540700.html

That's a link to our knowlege base article about this which explains the root of the problem, and the solution we found for it. I hope it helps you out.

While it's always good to have as much memory, and disk space as you can afford, as well as a quality program like links or hyperseek, sometimes, there's something else on the server side that might work out. In the case of our customer, he came back up roses, with the 1 second results we expected. Hopefully, this helps you out as well.

John

Although in this case, while searching Links SQL never does a join that is big enough to cause a mysql table full message. (This does happen on a duplicate check though, see the faq).

Cheers,

Alex

mySQL creates temp tables not just on joins, but whenever you have an ORDER BY... no matter how small or simple the query, the order by generates the temp table, and if your root partition has no space, you could notice the slowdown...

The 'bigtables' setting is pretty commonplace, but the "tmpdir" setting is one that we all sort of take for granted.

----
From the mySQL Manual:
When sorting (ORDER BY or GROUP BY), MySQL normally uses one or two temporary files. The maximum disk-space needed is (length_of_what_is_sorted +
sizeof(database_pointer)) * number_of_matched_rows * 2. sizeof(database_pointer) is usually 4, but may grow in the future for really big tables.

For some SELECT queries, MySQL also creates temporary SQL tables. These are not hidden and have names of the form `SQL_*'.
----

J-

[This message has been edited by jcokos (edited March 10, 2000).]

Where is it that I alter that?
in what file? how to proceed?



-=-=-
Change your server (both) to start up the mysqld daemon with the following paramaters in addition to any it's already starting with:

--big-tables --tmpdir=/usr/local/mysql/temp

Obviously, change the tmpdir to be some directory with 777 permissions on the "big" filesystem on the servers
-=-=-=-=

------------------
-=-=-=-=-=---=-=-=-=-=-=-
Elijânio - Br Company
-=-=-=-=-=---=-=-=-=-=-=-

We changed it right in the "safe_mysqld" shell script, adding those 2 paramaters to the end of the lines where it's starting up the mysqld daemon.

John

OK..
256 MB-RAM
add mysql parameters...

and nothing!

All made the alterations presented at this forum and I didn't obtain any result.
He continues slow to accomplish researches in words that he has more than 6000 answers in the database, he/she would like to know if there is some other alternative..

And Alex,
It exists or there is some project of a version of Links SQL in language C/C++ that the can be compiled him it would turn some ten times faster... or some there is been it sorts things out of the version in Perl to be compiled....


Message edited by Elijânio in 03/11/2000 - Br Company
BrLink:: Http://www.brlink.br

 
Would you let me have a look at your system to try some benchmarks and see how things are working? Email me login info if possible.


I really doubt c/c++ would be 10 times faster. I doubt it would even be two times faster. You should look to mod_perl to keep your scripts compiled in memory and database connections pooled. We can set this up for you if you are on a dedicated server running unix + apache. Contact me for details..

Cheers,

Alex

If you look at the way Links works, all the "intensive" operations are already in C/C++ via MySQL, and is highly optimized.

All Links SQL is doing is making calls to the database, and formatting the output. There is almost no way to speed that up, and still allow for cross-platform compatibility.

Alex's estimate of not even 2x faster is probably generous! For what is being done by the perl part of the equation, perl is _very_ optimized for string/text processing, whereas C/C++ is _NOT_. The speed of any C/C++ program would depend on how well you wrote the string handling routines, and there is _NO_ way they would be any better, or even as good, as the open-source widely developed, PERL project!

Most problems are hardware/resource related.


I have a 128 meg Sun Ultra 10 Sparc, with 128 meg of ram, and 2 4-gig harddrives, running 16 IP addresses, and 100+ aliased domains, with 6 of the domains "active". We are taking at least 50,000 cgi/page views a day, and much more!

My machine rarely breaks a sweat. I got it up to 1% utiliaztion for 2 hours by running an import on the DMOZ.org database (which is now almost 100MEG compressed!)

Just to summarize, the machine still only had 64 MB of memory.

Cheers,

Alex

Alex had a mistake in the parameters of Linux...
in the boot of the lilo it lacked a parameter that was the following:
append = mem=256M "

but I didn't notice difference in the access to the database of Links, because it continues delaying a long time to accomplish a search for a word that has many results in the database, for instance the word MP3 that has about 6.000 links delays a long time to give the answer...

:c(

Is all of your site loading slow, or just Links SQL?

I had a problem similar to this. It was a router problem with a server that was causing connections to take over 1000 milli-seconds, when 200-300 is normal.

You might not have a router problem, but something along these lines.

Since nothing else seems to be to blame, try checking this out.

------------------
Robert Blackstone
Webmaster of Scato Search
www.scato.com