Gossamer Forum
Home : Products : Gossamer Links : Version 1.x :

50.000+ links and high traffic

Quote Reply
50.000+ links and high traffic
 
This info can be usefull for some guys who are working on bigger projects
and their sites will have 50k links and more.

Our site has 50k links and we are getting 70.000 unique hits per day,
without Mysql links version i doubt that our server will be able to handle
such a big traffic,we have been on virtual host ,now we have move everything
to dedicated server ,we have tweak whole system or cgi`s will eat too much
of resources ,this must be done under linux too,if you are using basic httpd
configuration (256 www client reguests per second),with 70K hits per days you
will not be able to handle that.If you have 50k links and about 1--5k hits
day ,you can still be on virtual host,but some hosting companies can delete you
because you will eat them a lot of resources ,without notice.
There is a simple solution,if you dont have much money to spend on
dedicated servers,order virtual hosting at company that uses RAID mirroring
disks,dont forget to ask admins if they allow you to run that type of site
under virtual hosting plan.If your site gets big some day,you can easily
move to dedicated server,without changing permission on each file,RAID
will create,exactly the same mirror you have on virtual hosting plan.
Sorry for my not soo briliant english,hope
this message will help someone,who is thinking about running bigger projects
with Links SQl.

Specification of our dedicated server:

OpenBSD
Pentium III 600
256MBR
20GB HD
10/100mb ethernet
apache webserver
Quote Reply
Re: 50.000+ links and high traffic In reply to
Did you mention 20 GB of HD, but doesn't mysql work very well with HD's above 4 GB, what do you make for him to work without problems?
Quote Reply
Re: 50.000+ links and high traffic In reply to
Hi

We use 20Gb HD because ,that was our choice,
yes mysql works fine on 4gb HD`s too,
50.000 links will take you around 80mbs`s of HD.

Quote Reply
Re: 50.000+ links and high traffic In reply to
 
Quote:
50.000 links will take you around 80mbs`s of HD.
Does MySQL take up more space than the equivalent flatfile database? Assuming a "standard" number of fields and description lengths, 5,000 links in Links 2 is about 1 mb. That would equate to 10 mb for 50,000 links, a far cry from 80 mb!

Am I missing something, or is it that bloated? If it takes up that much space, I might run out of diskspace sooner than expected....

Dan
Quote Reply
Re: 50.000+ links and high traffic In reply to
I just check and its 54MB`s.sorry for posting wrong info.


Quote Reply
Re: 50.000+ links and high traffic In reply to
Any relational database takes up more room than a flat file database, largely because of the indexes.

You can find a 10-30x increase in disk space usage depending on what sort of indexes you keep around, and how large and how many fields you are trying to keep indexes on.

------------------
POSTCARDS.COM -- Everything Postcards on the Internet www.postcards.com
LinkSQL FAQ: www.postcards.com/FAQ/LinkSQL/








Quote Reply
Re: 50.000+ links and high traffic In reply to
What, are you trying to scare me?! Smile 30x would put me up against my disk space limit... I checked with my host, and fortunately my Links database is using only 4-5x the space of the Links 2 flatfile db, including search indexing and everything. Whew!

Dan
Quote Reply
Re: 50.000+ links and high traffic In reply to
If you have too much traffic, you can actually use TurboCluster Server 4.0 to spread the traffic. It supports up to 25 nodes.

I know windows 2000 server edition also support cluster feature, but personal speaking, I prefer Linux much more.

Of course, you need to buy more server but it reduce the risk for putting all stuffs in one server.



------------------
Bear EveryWhere
Quote Reply
Re: 50.000+ links and high traffic In reply to
Bear we have been using linux ,redhat 6.2
but problem was that 6.2 doesnt allow you to tweak the system,my admin has try few things
it was working but server was damn lagged,now we have openBSD and you can tweak the system easily,its extremly secure,and server is working extremly fast ,BSD platforms are the best for high traffic,i dont suggest linuxland.

We are hosting at http://www.i-sphere.com
Isphere Interactive .

Their tech support is wonderfull,fast servers
i just love them.

[This message has been edited by 0liver (edited March 07, 2000).]
Quote Reply
Re: 50.000+ links and high traffic In reply to
Oh is OpenBSD better than Linux for high traffic site?

I really don't know much about OpenBSD, can the software which run in Linux also run in OpenBSD? Like apache, qmail or stuffs like that?



------------------
Bear EveryWhere
Quote Reply
Re: 50.000+ links and high traffic In reply to
i dont know if there is the same software..Everything linux has,you will find
in OpenBSD too.
yes BSD platforms are better for traffic
OpenBSD or FreeBSD .
Quote Reply
Re: 50.000+ links and high traffic In reply to
Just realize that as your database grows, and your descriptions get larger, or more detailed, the indexes will take up more room.

Links 2.0 doesn't index the fields. Links SQL indexes all the words in each of the "weighted" fields. Our descriptions are fairly long and detailed, and our index files are quite large... actually very large Smile

Just keep an eye on your data directory as your site grows, and you make changes to the database. You can weight other fields, rebuild, and see how much the space consumption changes.

There is no one answer -- you really need to just keep an eye on it.

Realize that your database size is _not_ the only factor. The size of the /usr/mysql/data/database_name directory is one you have to watch!