Hi all,

How many records are possible, I mean how many records can be administrated by db.cgi in a high-performance time and how many generelly?

Who of you has a very big database, so that I can look, how fast it is?

Pit

As I have written before, it really depends on multiple factors, including RAM, CPU (Memory), etc. of both server and client systems. In addition, it depends on the amount of data per record you have in the database. The more data and fields you have, the longer it takes to parse the information.

I have found that 1.0 MG is the upper limit (as mentioned in the README file) before experiencing delays in searching and managing the files.

Regards,

------------------
Eliot Lee....
Former Handle: Eliot
Anthro TECH, L.L.C
anthrotech.com
* Check Resource Center
* Search Forums
* Thinking out of the box (codes) is not only fun, but effective.

The question is not how many, but how large. The database is effected by the size of the database. An easy guide is:

* up to 1Meg, the database will be quick and efficient.
* from 1 to 2Meg, the database will have brief delays. At the upper end, it may take up to 10 seconds to complete a search. DBMan SQL is recommended.
* above 2Megs, the database system is unreliable. The delays may be so large that the browser will time out. The results may not be completely accurate. DBMan SQL is highly recommended.

--Lee

You've said something like this before, Lee, and I'm not quite sure where you got your info. I know that the ReadMe file says 1Meg. Is the 2Meg info from your own experience?


------------------
JPD

While testing DBMan, I had to cut down the size of my database. I have 40,000 registered Marines in my guestbook and the size of the db file is over 9MB. I can search minimal delay (about 10 seconds) and adding records takes about 30 seconds. But I can't delete or modify the db because the server runs Out of Memory! I cut the db down to 2MB (for testing) and still had the same problem. My server's cutoff point before it would run out of memory was about 1.3MB.

I will be going with the DBManSQL version. Just got MySQL installed on my server and I just need to get the $$$ to purchase the DBManSQL. What sucks is that I will have to rewrite many of the cool mods for SQL.

So there are many factors involved in how big your flat-file db can be. Server memory and speed are the top 2. Then it trickles down to what you are actually doing or allowing users to do with the db. If you allow modify/delete, you're limited to server memory!

Semper Fi,
Aj

Actually, JPD, my "2MB" info is from internal HP softcoders, and from testing.

A friend of mine is part of a team of web coders, and she's been working with flatfile databases for a score of years. There are some old farts also in her team that have been at this stuff since the beginning.

The first time I installed DBMan, I was testing it for usability for a purpose, and asked my friend to tell me the limitations of it. There were a large number of functionality issues (which I've mostly resolved since then), and the database size limitations. She did indicate the size is dependant on several physical factors, including memory and CPU, but as a general rule of thumb, the largest database you should make is twice its optimal performance.

With that, I went out to verify it. I created a program to make a database with however many records I wanted it to have, then tested it with benchmarking enabled.

At 1meg (which I had read from the forum), there was really no noticable lag (usage was never higher than 7).

At 2meg, there was a bit more lag (usage now averaged around 14), and the pages took about 6 seconds to process.

I kept going, and it just got worse above 2meg. It was still functional up to 25megs, but it timed out often. It took at least 2 minutes to get any results back, if at all.

So, to conclude, I agreed with her that 2meg should really be the cutoff point for the database.

--Lee



[This message has been edited by leisurelee (edited May 12, 2000).]

Thanks, Lee! I was just curious as to where the information came from.


------------------
JPD

I thought Pit (or anyone else interested) might like to use the code I wrote to make the test database.

Here's my code (backup your original db file first!!):


filler text is a bunch of nonsense data that makes each record what your average real database record is. (real database size divided by number of records.) You should play with this a bit. Have the script make 1 record, and see if it matches the filesize needed. This is pretty useful to see how your own database will work when you have a large number of records in it.

--Lee

Hi all,

thanx alot for your help about that question.

What can I do, my target is a database with about 50000-80000 entries and 30 fields.

If I must search for a mod, that makes different databases, for example one for each category??

Is this a possibility for me??

Or have I get the sql-version, is that the right answer for me?

Pit

BTW: I have to ask the differences of the normal and the sql-version in a new thread.

 
Upgrade to DBMAN SQL.


MySQL developers have denoted (through the mysql.com site) that databases can hold between 45,000 and 100,000 records depending on the size of fields. Script developers have given more conservative estimates around 25,000 to 50,000. DBAs from various companies I've worked with have given estimates between 100,000 and 120,000 records.

And these estimates also vary with your server's resources and configurations.

But the bottom line is that the flat file system of DBMAN can NOT handle the large number of records you anticipate to add.


Not really necessary since there are Topics in this forum that have addressed this issue.



Regards,

Regards,

------------------
Eliot Lee....
Former Handle: Eliot
Anthro TECH, L.L.C
anthrotech.com
* Check Resource Center
* Search Forums
* Thinking out of the box (codes) is not only fun, but effective.

 
Does anyone know the options after we reach approx 100-120k records in mysql?
For a DB between 1-2megs we use Dbman as-is, we then uprade to the mysql version which can hold upto 120k records with x amount of megs.
Is there a DB language that can cope with 120k to 1 million records?

rog

Oracle (which uses SQL)...but ported data from MySQL to Oracle is bit tricky.

Regards,

------------------
Eliot Lee....
Former Handle: Eliot
Anthro TECH, L.L.C
anthrotech.com
* Check Resource Center
* Search Forums
* Thinking out of the box (codes) is not only fun, but effective.