Mailing List Archive: DBMail: users

MAX_CHILDREN

Index | Next | Previous | View Threaded

Jon at host-it

Jun 16, 2009, 1:08 PM

Post #1 of 7 (432 views)
Permalink

MAX_CHILDREN

Guys,

We're now quite consistently hitting MAX_CHILDREN limit of 300 in pop3 during business hours. I understand it's hard set in code to 300 (after pushing it higher and getting warnings in logs I checked through the code).

What are the implications of increasing this limit?

Regards

Jon

_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

michael.monnerie at is

Jun 16, 2009, 1:29 PM

Post #2 of 7 (406 views)
Permalink

Re: MAX_CHILDREN [In reply to]

On Dienstag 16 Juni 2009 Jon Duggan wrote:
> MAX_CHILDREN limit of 300 in pop3

Are you sure your box has enough RAM to keep 300 pop3 daemons around,
plus the memory they need during I/O? Maybe it'd be better to install a
2nd box? If your machine is already swapping, more processes with
actually lower your throughput.

If the average pop3 user can receive with 100KB/s, your machine would
need 30.000 KB/s (about 30Mb/s) Internet connection, and of course your
database (same server? or another?) needs to be fast enough also. Sure
you don't hit an I/O limit somewhere?

Not want to offend, just help. But I don't know for the 300 limit in the
code.

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660 / 415 65 31 .network.your.ideas.
// PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net Key-ID: 1C1209B4

_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

paul at nfg

Jun 16, 2009, 1:47 PM

Post #3 of 7 (405 views)
Permalink

Re: MAX_CHILDREN [In reply to]

The hardcoded ceiling was aimed at avoiding overloading the database
server. You can change the value and recompile if you like.

It does make me curious what hardware are you on anyway?

Wish I had a good load tester for pop3...

Michael Monnerie wrote:
> On Dienstag 16 Juni 2009 Jon Duggan wrote:
>> MAX_CHILDREN limit of 300 in pop3
>
> Are you sure your box has enough RAM to keep 300 pop3 daemons around,
> plus the memory they need during I/O? Maybe it'd be better to install a
> 2nd box? If your machine is already swapping, more processes with
> actually lower your throughput.
>
> If the average pop3 user can receive with 100KB/s, your machine would
> need 30.000 KB/s (about 30Mb/s) Internet connection, and of course your
> database (same server? or another?) needs to be fast enough also. Sure
> you don't hit an I/O limit somewhere?
>
> Not want to offend, just help. But I don't know for the 300 limit in the
> code.
>
> mfg zmi

--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Jon at host-it

Jun 16, 2009, 2:37 PM

Post #4 of 7 (408 views)
Permalink

Re: MAX_CHILDREN [In reply to]

The pop3 box is a dedicated quad core with 4gb of ram.

I've watched the box looking for bottlenecks during the busy periods, there are no bandwidth/cpu/disk bottlenecks on the pop3 server, likewise ive watched the backend server which seems like it's at reasonable load but not exhausted as far as performance is concerned.

We push at peak maybe 10mbit/s so no where near the 30mb/s you quote but bandwidth is not an issue (it's as good as unlimited).

Checking through the scoreboard it seems like we are just getting alot of hits (i don't think surprising given over 6,200 accounts).

At this stage having found no bottlenecks im thinking increasing this hard 300 limit might be ok, but i wanted to seek advice on potential repercussions before going ahead.

During cleanup also becomes a problem, so the quick fix for me is to increase the limit and allow me enough time to plan a second server and loadbalance the two

Thanks.

Jon

> -----Original Message-----
> From: dbmail-bounces[at]dbmail.org [mailto:dbmail-bounces[at]dbmail.org] On
> Behalf Of Michael Monnerie
> Sent: 16 June 2009 21:30
> To: DBMail mailinglist
> Subject: Re: [Dbmail] MAX_CHILDREN
>
> On Dienstag 16 Juni 2009 Jon Duggan wrote:
> > MAX_CHILDREN limit of 300 in pop3
>
> Are you sure your box has enough RAM to keep 300 pop3 daemons around,
> plus the memory they need during I/O? Maybe it'd be better to install a
> 2nd box? If your machine is already swapping, more processes with
> actually lower your throughput.
>
> If the average pop3 user can receive with 100KB/s, your machine would
> need 30.000 KB/s (about 30Mb/s) Internet connection, and of course your
> database (same server? or another?) needs to be fast enough also. Sure
> you don't hit an I/O limit somewhere?
>
> Not want to offend, just help. But I don't know for the 300 limit in
> the
> code.
>
> mfg zmi
> --
> // Michael Monnerie, Ing.BSc ----- http://it-management.at
> // Tel: 0660 / 415 65 31 .network.your.ideas.
> // PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import"
> // Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4
> // Keyserver: wwwkeys.eu.pgp.net Key-ID: 1C1209B4
>
> _______________________________________________
> DBmail mailing list
> DBmail[at]dbmail.org
> http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Jon at host-it

Jun 16, 2009, 2:43 PM

Post #5 of 7 (405 views)
Permalink

Re: MAX_CHILDREN [In reply to]

Thanks Paul, I'll up to 450 and see how we go.

Our pop3 box is a quad xeon with 4gb of ram, the DB backend is a dual quad xeon with 16gb of ram using mysql.

Which leads me to another question. It seems for good reason that people running larger setups are preferring postgres, is there anyone out there running over 6k users (mixed pop3/imap) on a mysql setup? What sort of numbers can we expect from this combination of hardware. We're already planning to scale the setup with a separated backend for new users and perdition to map the pop3/imap services so we're not worried about scaling, but i'd be interested to hear of people running higher than 6k users in mysql and what hardware theyre using.

Jon

> -----Original Message-----
> From: dbmail-bounces[at]dbmail.org [mailto:dbmail-bounces[at]dbmail.org] On
> Behalf Of Paul J Stevens
> Sent: 16 June 2009 21:47
> To: DBMail mailinglist
> Subject: Re: [Dbmail] MAX_CHILDREN
>
>
> The hardcoded ceiling was aimed at avoiding overloading the database
> server. You can change the value and recompile if you like.
>
> It does make me curious what hardware are you on anyway?
>
> Wish I had a good load tester for pop3...
>
>
>
> Michael Monnerie wrote:
> > On Dienstag 16 Juni 2009 Jon Duggan wrote:
> >> MAX_CHILDREN limit of 300 in pop3
> >
> > Are you sure your box has enough RAM to keep 300 pop3 daemons around,
> > plus the memory they need during I/O? Maybe it'd be better to install
> a
> > 2nd box? If your machine is already swapping, more processes with
> > actually lower your throughput.
> >
> > If the average pop3 user can receive with 100KB/s, your machine would
> > need 30.000 KB/s (about 30Mb/s) Internet connection, and of course
> your
> > database (same server? or another?) needs to be fast enough also.
> Sure
> > you don't hit an I/O limit somewhere?
> >
> > Not want to offend, just help. But I don't know for the 300 limit in
> the
> > code.
> >
> > mfg zmi
>
>
> --
> ________________________________________________________________
> Paul Stevens paul at nfg.nl
> NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
> The Netherlands________________________________http://www.nfg.nl
> _______________________________________________
> DBmail mailing list
> DBmail[at]dbmail.org
> http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

michael.monnerie at is

Jun 16, 2009, 5:40 PM

Post #6 of 7 (407 views)
Permalink

Re: MAX_CHILDREN [In reply to]

On Dienstag 16 Juni 2009 Jon Duggan wrote:
> The pop3 box is a dedicated quad core with 4gb of ram.

And how much memory is used during peak hours? Just curious. Each
dbmail-pop3 daemon here is shown as having an RSS footprint of 1-2MB,
and a VSZ of 5-10MB. So with 300 users you'd have about 13MB per user
available, which should be plenty. 600 users would thus mean ~6MB/user,
I guess that would be the hard limit memory wise.

What's your average CPU usage during peaks? I guess the pop3 box is not
heavy loaded, but what about the db?

> We push at peak maybe 10mbit/s so no where near the 30mb/s you quote
> but bandwidth is not an issue (it's as good as unlimited).

I forgot that with pop3 users usually remove mail from the server
directly after receiving (we have lots of IMAP users). And with that
many users, your connect rate may be high, but actual transfer rates
not.

Do you know about how many % of users choose to "leave mail on server
for X days"? When we had flat-file storage with pop3d, that setting
killed the server, as he had to read through the full file to see if
there are new messages. You need only some users with a 100MB mailbox
connecting every minute, and your server cries for help. But with
dbmail, that is no issue anymore. :-)

mfg zmi
--
// Michael Monnerie, Ing.BSc ----- http://it-management.at
// Tel: 0660 / 415 65 31 .network.your.ideas.
// PGP Key: "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38 500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net Key-ID: 1C1209B4

_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

paul at nfg

Jun 17, 2009, 12:08 AM

Post #7 of 7 (406 views)
Permalink

Re: MAX_CHILDREN [In reply to]

Jon Duggan wrote:

> Our pop3 box is a quad xeon with 4gb of ram, the DB backend is a dual
> quad xeon with 16gb of ram using mysql.

what type of disks are you using? sas?

> Which leads me to another question. It seems for good reason that
> people running larger setups are preferring postgres, is there anyone
> out there running over 6k users (mixed pop3/imap) on a mysql setup?

I don't have that kind of load on any of my installations. Postgres
users have their own problems. Postgres doesn't handle many client
connections as well as mysql (afaik). Also, client connection startup is
(or at least was) much more expensive for postgres. In fact, that was
one of my main reasons for adapting connection pooling in 2.3.

> What sort of numbers can we expect from this combination of hardware.
> We're already planning to scale the setup with a separated backend
> for new users and perdition to map the pop3/imap services so we're
> not worried about scaling, but i'd be interested to hear of people
> running higher than 6k users in mysql and what hardware theyre using.

I think Mike O'Brian is/was running that sort of numbers and more, but I
haven't seen him around here for some time.

--
________________________________________________________________
Paul Stevens paul at nfg.nl
NET FACILITIES GROUP GPG/PGP: 1024D/11F8CD31
The Netherlands________________________________http://www.nfg.nl
_______________________________________________
DBmail mailing list
DBmail[at]dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com