Mailing List Archive: exim: users

Configuring mailbox quotas on large servers

Index | Next | Previous | View Threaded

a.smith at ukgrid

Nov 27, 2009, 2:40 AM

Post #1 of 11 (1739 views)
Permalink

Configuring mailbox quotas on large servers

Hi,

what are people doing for setting mailbox quotas on systems with
very large numbers of mailboxes (hundreds of thousands)? I would
assume UNIX filesystem quotas aren't practicle as you´d need to have
all of those users in /etc/passwd or your directory service which I
think would kill performance. Anyone with any experience in this like
to share their thoughts??

thanks in advance, Andy.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

linux4michelle at tamay-dogan

Nov 27, 2009, 6:23 AM

Post #2 of 11 (1694 views)
Permalink

Re: Configuring mailbox quotas on large servers [In reply to]

Am 2009-11-27 10:40:27, schrieb Andy Smith:
> Hi,
>
> what are people doing for setting mailbox quotas on systems with
> very large numbers of mailboxes (hundreds of thousands)? I would
> assume UNIX filesystem quotas aren't practicle as you´d need to have
> all of those users in /etc/passwd or your directory service which I
> think would kill performance. Anyone with any experience in this like
> to share their thoughts??

I can not believe, you have a singel server with "hundreds of thousands"
mailboxes, because I know some VERY BIG providers which hit the limit
with arround 50-60.000 Mailboxes per physicaly server.

I have a very fast Sun Machine and run into trouble with 20.000 accounts
if they access the mailboxes between 08:00 and 17:00. And of course, I
am using 4 Income-MTA-Proxys which distribute the messages to 86 mailbox
servers.

However, my quotas are in the PostgreSQL database and I can change it
with some clicks in less then 10 seconds.

Thanks, Greetings and nice Day/Evening
Michelle Konzack
Systemadministrator
Tamay Dogan Network
Debian GNU/Linux Consultant

--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
<http://www.tamay-dogan.net/> Michelle Konzack
<http://www.can4linux.org/> Apt. 917
<http://www.flexray4linux.org/> 50, rue de Soultz
Jabber linux4michelle [at] jabber 67100 Strabourg/France
IRC #Debian (irc.icq.com) Tel. DE: +49 177 9351947
ICQ #328449886 Tel. FR: +33 6 61925193

Attachments:

signature.pgp (0.18 KB)

nigel.metheringham at dev

Nov 27, 2009, 8:23 AM

Post #3 of 11 (1693 views)
Permalink

Re: Configuring mailbox quotas on large servers [In reply to]

On 27 Nov 2009, at 14:23, Michelle Konzack wrote:

> I can not believe, you have a singel server with "hundreds of thousands"
> mailboxes, because I know some VERY BIG providers which hit the limit
> with arround 50-60.000 Mailboxes per physicaly server.

Depends what you mean by a single server. Planet used to have the mailboxes on NAS storage (NetApp), with all the front end servers identical. That had better than 3 million users although in a time where mail usage was lower.

Nigel.
--
[ Nigel Metheringham Nigel.Metheringham [at] InTechnology ]
[. - Comments in this message are my own and not ITO opinion/policy - ]

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

a.smith at ukgrid

Nov 27, 2009, 9:59 AM

Post #4 of 11 (1691 views)
Permalink

Re: Configuring mailbox quotas on large servers [In reply to]

Hi,

yes, the NetApp model is more or less what I was thinking of. I
dont have a server implementation yet, Im looking at what is feasible.
My idea is an NFS store with the maildirs, with multiple Exim SMTP
servers connected and some POP3/IMAP/webmail servers sitting behind a
network load balancer.
As per my first mail, my concern is how I can achieve some kind of
quotas on such a large number of mailboxes....

thanks Andy.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

tlyons at ivenue

Nov 27, 2009, 10:15 AM

Post #5 of 11 (1694 views)
Permalink

Re: Configuring mailbox quotas on large servers [In reply to]

On Fri, Nov 27, 2009 at 9:59 AM, Andy Smith <a.smith [at] ukgrid> wrote:
> Hi,
>
> yes, the NetApp model is more or less what I was thinking of. I
> dont have a server implementation yet, Im looking at what is feasible.
> My idea is an NFS store with the maildirs, with multiple Exim SMTP
> servers connected and some POP3/IMAP/webmail servers sitting behind a
> network load balancer.
> As per my first mail, my concern is how I can achieve some kind of
> quotas on such a large number of mailboxes....

We use courier maildrop to do the maildir delivery. You configure
"courier-authlib" properly so that it gets its maildir and quota
information for a virtual user from one of LDAP, or MySQL, etc. The
maildrop process simply talks to the courier-authlib process to get
that info. When it then attempts to deliver, it creates/adjusts a
maildirsize file with the quota setting and current mailbox size,
refusing to deliver if the maildir is over its assigned quota.

Integration with Exim is pretty easy. I use it on an older sendmail
system, and am refining it with a development box of Exim.
--
Regards... Todd
The best thing about pair programming is that you have the perfect
audience for your genius. -- Kent Beck

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

a.smith at ukgrid

Nov 27, 2009, 11:11 AM

Post #6 of 11 (1691 views)
Permalink

Re: Configuring mailbox quotas on large servers [In reply to]

Thanks Todd,

that sounds like it could work for me. Is it smooth sailing with
this approach or do you have any issues with your current sendmail
system?

thanks Andy.

Quoting Todd Lyons <tlyons [at] ivenue>:

>
> We use courier maildrop to do the maildir delivery. You configure
> "courier-authlib" properly so that it gets its maildir and quota
> information for a virtual user from one of LDAP, or MySQL, etc. The
> maildrop process simply talks to the courier-authlib process to get
> that info. When it then attempts to deliver, it creates/adjusts a
> maildirsize file with the quota setting and current mailbox size,
> refusing to deliver if the maildir is over its assigned quota.
>
> Integration with Exim is pretty easy. I use it on an older sendmail
> system, and am refining it with a development box of Exim.
> --
> Regards... Todd
> The best thing about pair programming is that you have the perfect
> audience for your genius. -- Kent Beck
>

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

tlyons at ivenue

Nov 27, 2009, 1:57 PM

Post #7 of 11 (1683 views)
Permalink

Re: Configuring mailbox quotas on large servers [In reply to]

On Fri, Nov 27, 2009 at 11:11 AM, Andy Smith <a.smith [at] ukgrid> wrote:
> Thanks Todd,
>
> that sounds like it could work for me. Is it smooth sailing with this
> approach or do you have any issues with your current sendmail system?

Haven't really had any major issues. The code is very solid and well written.

The one minor issue is something that I am working on a solution for
WRT Exim. At the point that the MTA hands off to the LDA (maildrop),
it has already been determined that it will be accepted by Sendmail or
Exim. It is at this point that the quota check is made, so if the
account is over quota, the email gets TEMPFAIL'd (errorlevel returned
is 77) and sits in the queue. An NDR message gets sent at the end of
the queue lifetime if it can't be delivered into the home directory.

There are two possibilities AFAICS:
1. Check the quota before accepting it (ie after the RCPT phase or
after the SMTP data phase).
2. Write some kind of wrapper that handles the TEMPFAIL condition.

To me, #1 seems much better because you can reject or defer at SMTP
time so you don't have to generate an NDR. #1 seems easy at first
blush, but doing it with a perl function (for example) is a problem
because the file is (in my case) owned by a different user than exim
runs as, and is mode 700, so it can't be read from within exim.

I think I'm going to try to make a quickie daemon that runs as the
virtual user (or root if that's preferred), connect to the daemon, ask
for the info for user [at] domain, the daemon returns the quota limit
and the current usage. Then either in an ACL or in perl, figure out
if the current message would put it over the limit. Doing this check
at RCPT time means that you have to trust the SIZE= part of the SMTP
conversation (if it's provided). Doing after the DATA phase means you
(I think) have the exact size of the email. I'll have to dig to see
what expansion variables I have access to after the DATA phase.

Are there any other ways of doing this? If there are, nothing has
come to mind and I welcome suggestions.

--
Regards... Todd
The best thing about pair programming is that you have the perfect
audience for your genius. -- Kent Beck

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

exim-users at spodhuis

Nov 27, 2009, 4:11 PM

Post #8 of 11 (1679 views)
Permalink

Re: Configuring mailbox quotas on large servers [In reply to]

On 2009-11-27 at 13:57 -0800, Todd Lyons wrote:
> The one minor issue is something that I am working on a solution for
> WRT Exim. At the point that the MTA hands off to the LDA (maildrop),
> it has already been determined that it will be accepted by Sendmail or
> Exim. It is at this point that the quota check is made, so if the
> account is over quota, the email gets TEMPFAIL'd (errorlevel returned
> is 77) and sits in the queue. An NDR message gets sent at the end of
> the queue lifetime if it can't be delivered into the home directory.

> Are there any other ways of doing this? If there are, nothing has
> come to mind and I welcome suggestions.

Two Routers which accept the same email. The first is set
"verify_only", the second is the maildrop Router, which is set
"no_verify".

You have the same preconditions (domains, etc) on both Routers.

On the first, you set "no_more", so that if it fails verification, the
routing won't go on to the second. This is why it's important to have
the same preconditions.

The first Router is a "redirect" router. You use "data" to decide where
to redirect to. You use string expansion in the "data = ..." line to
decide if the recipient is within quota.

If within quota, you redirect to a holding file: in practice, because
this is verify_only, you will never deliver to that file, so could use
/dev/null; but for safety's sake, redirect to a real file. Set up your
monitoring so that if that file ever grows to have length > 0 then you
alert fast, because something has gone wrong with a config push as
someone has removed verify_only.

If not within quota, you :defer: the expansion. Perhaps :fail:. Test.

You can use the string expansion to put in whatever check you want.

Now, if Exim has access to the mail storage area directly, you might use
an "accept" Router for the verify_only step and check quotas that way
(instead of the redirect's data = ...), but unless you're using mbox (or
other single-file) format, I think this would turn out to be a bad idea
in this case. Because you'd never deliver with Exim, you'd never update
the quota files used by Exim to cache the usage within the maildrop, so
every delivery might be an expensive calculation over the existing
files.

If you're using mbox format, this might not be so bad and you could try
it.

If you have the source to the maildrop command and can update it to also
update the quota files used by Exim, you could still do it this way.
This is what I'd probably do myself. But I've got strange tastes,
sometimes.

-Phil

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

lists at timj

Nov 28, 2009, 5:28 AM

Post #9 of 11 (1687 views)
Permalink

Re: Configuring mailbox quotas on large servers [In reply to]

On 27/11/09 21:57, Todd Lyons wrote:

> The one minor issue is something that I am working on a solution for
> WRT Exim. At the point that the MTA hands off to the LDA (maildrop),
> it has already been determined that it will be accepted by Sendmail or
> Exim. It is at this point that the quota check is made, so if the
> account is over quota, the email gets TEMPFAIL'd (errorlevel returned
> is 77) and sits in the queue. An NDR message gets sent at the end of
> the queue lifetime if it can't be delivered into the home directory.

I wrote something about this when a similar issue came up - might be helpful:

http://www.timj.co.uk/linux/rcpt-time-quota-maildir.php

Tim

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

tlyons at ivenue

Nov 28, 2009, 10:48 AM

Post #10 of 11 (1679 views)
Permalink

Re: Configuring mailbox quotas on large servers [In reply to]

On Fri, Nov 27, 2009 at 4:11 PM, Phil Pennock <exim-users [at] spodhuis> wrote:
>> Are there any other ways of doing this? If there are, nothing has
>> come to mind and I welcome suggestions.
>
> Two Routers which accept the same email. The first is set
> "verify_only", the second is the maildrop Router, which is set
> "no_verify".

Groovy, I'll look into doing that.

> The first Router is a "redirect" router. You use "data" to decide where
> to redirect to. You use string expansion in the "data = ..." line to
> decide if the recipient is within quota.

This is the fuzzy part. To get the current size, either I or exim
have to parse the maildirsize file.

> If within quota, you redirect to a holding file: in practice, because
> this is verify_only, you will never deliver to that file, so could use
> /dev/null; but for safety's sake, redirect to a real file. Set up your
> monitoring so that if that file ever grows to have length > 0 then you
> alert fast, because something has gone wrong with a config push as
> someone has removed verify_only.

Makes sense.

> If not within quota, you :defer: the expansion. Perhaps :fail:. Test.

I'm partial to deferring, but an argument could be made for failing
since the sender gets an immediate notification that it did not get
delivered.

> Now, if Exim has access to the mail storage area directly, you might use
> an "accept" Router for the verify_only step and check quotas that way
> (instead of the redirect's data = ...), but unless you're using mbox (or
> other single-file) format, I think this would turn out to be a bad idea
> in this case. Because you'd never deliver with Exim, you'd never update
> the quota files used by Exim to cache the usage within the maildrop, so
> every delivery might be an expensive calculation over the existing
> files.
> If you have the source to the maildrop command and can update it to also
> update the quota files used by Exim, you could still do it this way.
> This is what I'd probably do myself. But I've got strange tastes,
> sometimes.

They do use the same file: $MAILDIR/maildirsize

It looks like Exim can do what I thought had to be provided by an
external LDA. I'll start experimenting now.

Truthfully, I'm a bit stunned. I completely overlooked Exim's
extensive quota facilities because I was scope-locked on my past
positive experiences with maildrop. I did not realize that I could
use exim to enforce quota (in a router, so I think that will allow me
to defer/reject at SMTP time, which is my ultimate goal). I thought I
had to hand it off to something else in order to get quota
enforcement.

Thanks for the redirect Phil!
--
Regards... Todd
The best thing about pair programming is that you have the perfect
audience for your genius. -- Kent Beck

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

wbh at conducive

Nov 28, 2009, 12:31 PM

Post #11 of 11 (1675 views)
Permalink

Re: Configuring mailbox quotas on large servers [In reply to]

Todd Lyons wrote:
> On Fri, Nov 27, 2009 at 4:11 PM, Phil Pennock <exim-users [at] spodhuis> wrote:
>>> Are there any other ways of doing this? If there are, nothing has
>>> come to mind and I welcome suggestions.

*snip* (how-to w/r maildrop and/or routers)

> positive experiences with maildrop. I did not realize that I could
> use exim to enforce quota (in a router, so I think that will allow me
> to defer/reject at SMTP time, which is my ultimate goal). I thought I
> had to hand it off to something else in order to get quota
> enforcement.

Verification-walks aside, (which you can use, per the redacted coverage)
.. router/transports are not otherwise really 'at smpt time'.

The 'session 'ends as you leave the DATA phase.

But you *can* know the actual size while still in acl_smtp_data and
*can* use that information to insure that your defer/deny is
communicated directly to, and ONLY to, the still-connected entity,
thereby eliminating risk that an NDR is abused for any form of
backscatter spamming.

ISTR there are examples of useful acl snippets in the list archives

... all part of why Exim & its acl's are so effective vs
MTA+[milters]+[maildrop|procmail| friends]

CAVEATS:

- I don't (need to) use quotas, as a message-size limit, large RAID1,
and finite b/w on the front-end give me days if not weeks to react to
consumption of space with my particular user community.

YMOV - BUT... a setup where you publish quota 'n' but actually provide
for space for quota 'n+20%' and trigger at quota [n+10%] before
defer|deny might reduce your admin load costs more than the extra HDD
space costs. A fair bit of slack also reduces the risk of one
child-process clearing the session and THEN hitting a router or fs
blockage because one or more OTHER children were already using-up the
last of the headspace, or conversely, rejecting if the POP/IMAP daemon
was clearing more space in the meantime.

- Be aware also that there exist sending 'entities' that do not
hang-around for 'proper' completion of DATA handshakes. All the more
reason, IMNSHO, to NOT generate out-of-session NDR's to such critters,
as they are more likely to be maliciously-coded rather than just-plain
broken.

- Note necessarily a recommended practice, but I also disallow
pipelining so as to insure nothing comes onto the box before it has
passed all tests prior to acl_smtp_data. Up to that point, there has
been nothing exchanged but ephemeral handshakes, no HDD space required
for those at all.

Do your own due diligence on that issue, as one size does not fit all.

Finally - if you are running a hot and heavy MTA, it helps to NOT use
the default /var mount-point for mailstore (or even, necessarily for
spool). If ~/var ever overflows, the system logger (among other needful
things) is stumped, which is not good.

As Exim can easily be pointed to use of a bespoke mount (separate
hardware RAID recommended) for mailstore and optionally spool, that
makes migration, upgrades, or failure recovery a bit cleaner, and helps
insure that you can at least ssh into a box that has not dropped into
single-user mode due to an 'unable to write..' panic.

HTH,

Bill

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads