Mailing List Archive: exim: users

Which hardware do you use for your installation?

Index | Next | Previous | View Threaded

gymer at odense

Jun 11, 2008, 9:02 AM

Post #1 of 11 (2306 views)
Permalink

Which hardware do you use for your installation?

Looking for experience with large exim installations. Please answer the
questions below:

What are the machine specs for you exim installation?

How many messages does it handle daily?

What scanning mechanism do you apply?

/Lasse

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

ssmeenk at freshdot

Jun 11, 2008, 12:46 PM

Post #2 of 11 (2188 views)
Permalink

Re: Which hardware do you use for your installation? [In reply to]

Quoting Lasse Birnbaum Jensen (gymer [at] odense):

> Looking for experience with large exim installations.
> Please answer the questions below:

We recently reworked our mail-setup at the company i work for. And being
an 'open' company, my boss probably won't mind me posting this rought
sketch. Also i wonder if people reading this have any comments on this
setup :)

Numbers are on a daily basis.

> What are the machine specs for you exim installation?

Our mailsetup consists of about 8 separate clusters of servers sharing a
common task dealing with mail ranging from incoming to imap servers.

On the incoming end (mx) we now have about ~5 servers with a single AMD
Opteron 2.8GHz Dual-Core CPU, 4G ram, and just one SATA-II disk as
storage isn't important.

> How many messages does it handle daily?

In a loadbalanced setup, each server handles about ~850.000 messages.
Grand totalling about ~4.300.000 messages combined on the MX'es. They do
this without breaking a sweat, but accepting mail isn't the hardest part.

> What scanning mechanism do you apply?

On a separate filtering platform, consisting of ~7 servers with 2 dual
core CPUs at 3GHz, 4GB ram and again just one disk. We use Amavis in
combination with ClamAV and 2 commercial virusscanners for holding back
the virusses and Exiscan w/ SpamAssassin on separate platform consisting
of ~10, fast CPU servers, for spamscanning.

These filterservers handle ~75% of the traffic coming in at the mx'es.
That's ~530.000 per server, grand total ~3.200.000 meaning about
1.000.000 messages get rejected at the MXes. Either because of
nonexistent addresses or dnsbls and the like.

Spam is either delivered or dropped. We don't bounce spams.

Delivery is done by yet another platform consisting of just ~4 servers
somewhat in the same order of business as above. They get just a mere
~8% of total incoming traffic, ~120.000 deliveries per server, ~350.000
total. Maildirs for the users are stored on NetApp filers.

One big fat tip for you, free of charge:
Exim is not good at dealing with large queues, partly because of the way
it keeps state in the db4 file. That's why almost all these platforms
have their spool in memory and 'message logging' is turned off (-Mvl
doesn't work).

Exim is really great at being flexible. Almost anything you want is
possible in some way or the other. The other MTA's we tried had severe
problems with certain design-specific aspects of our setup.

We call it SWAMP, short for 'Sendmail Wasn't an Acceptable Mail Product'.

I hope this helps :)

Kind regards,
Sander.
--
| For security, this message has been encrypted with ROT13 twice.
| 1024D/08CEC94D - 34B3 3314 B146 E13C 70C8 9BDB D463 7E41 08CE C94D

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

kroshka at atypon

Jun 11, 2008, 2:18 PM

Post #3 of 11 (2199 views)
Permalink

Re: Which hardware do you use for your installation? [In reply to]

Sander Smeenk wrote:

> Our mailsetup consists of about 8 separate clusters of servers sharing a
> common task dealing with mail ranging from incoming to imap servers.

Looks like a solid setup, as far as I can see.

> On the incoming end (mx) we now have about ~5 servers with a single AMD
> Opteron 2.8GHz Dual-Core CPU, 4G ram, and just one SATA-II disk as
> storage isn't important.

But why not use a raid1 instead of the one disk? It will increase
redundancy even more. Now I understand that in your load balanced setup
if one or a few servers' disks would die the system would still
function. But you would have to clone/recreate the system and bring it
back up. With a raid1 you would only have to replace the broken disk and
that's it (or if you had a hot spare you had to do nothing).

> total. Maildirs for the users are stored on NetApp filers.

Netapp uses NFS I assume? How well does it work with imap? As far as I
know using imap on anything but unix format mailboxes on a networked
filesystem like NFS can cause problems due to file locking and such.

> have their spool in memory and 'message logging' is turned off (-Mvl
> doesn't work).

I remember reading on here that having the spool in memory isn't such a
good idea, but I forgot the exact reasons.

> We call it SWAMP, short for 'Sendmail Wasn't an Acceptable Mail Product'.

;-)

Greetings,
Jeroen

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

ssmeenk at freshdot

Jun 12, 2008, 1:05 AM

Post #4 of 11 (2200 views)
Permalink

Re: Which hardware do you use for your installation? [In reply to]

Quoting Jeroen van Aart (kroshka [at] atypon):

> > On the incoming end (mx) we now have about ~5 servers with a single AMD
> > Opteron 2.8GHz Dual-Core CPU, 4G ram, and just one SATA-II disk as
> > storage isn't important.
> But why not use a raid1 instead of the one disk?
> [ .. ] you would have to clone/recreate the system and bring it back up.

All of our 'platform' servers use PXE to boot and, depending on in what
group we put the mac-address in the dhcp config, the server
automatically installs itself with cfengine to do whatever function we
need it to do. It partitions the disks, debootstraps Ubuntu, installs
the necessary programs and configures them from our SVN-repository.

This process is runs completely unmanaged. :)

So although i see your point, in our setup it's not necessary.

> Netapp uses NFS I assume? How well does it work with imap? As far as I
> know using imap on anything but unix format mailboxes on a networked
> filesystem like NFS can cause problems due to file locking and such.

Yep, we use NFS with the NetApps. It works like a charm, no locking
issues whatsoever. I'm not entirely sure how our pop3/imap servers do
their locking. It could well be that they keep state on local disk.

I'd have to dive into that if you really want to know ;)

> > have their spool in memory and 'message logging' is turned off (-Mvl
> > doesn't work).
> I remember reading on here that having the spool in memory isn't such a
> good idea, but I forgot the exact reasons.

Well, if there's a lot of mail in the queue for some reason and the
server spontaneously reboots or crashes, you will lose the mail. That's
probably the only argument i can see against having a spool in memory.
But only servers that 'transit' mail have their spools in memory.
Messages shouldn't be on those servers for longer than about a minute.

Our monitoring setup keeps a close eye on the queuesizes and queuetimes
so we get notified immediately if something is wrong.

We -had- to do this trick with tmpfs, Exim couldn't handle the load with
the spool on disk even in raid setup on SATA-II disks.

Thanks for your ideas!

Regards,
-Sndr.
--
| /dev/hda1 has been checked 20 times without being mounted, mount forced
| 1024D/08CEC94D - 34B3 3314 B146 E13C 70C8 9BDB D463 7E41 08CE C94D

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

nigel.metheringham at dev

Jun 12, 2008, 1:11 AM

Post #5 of 11 (2195 views)
Permalink

Re: Which hardware do you use for your installation? [In reply to]

On 11 Jun 2008, at 22:18, Jeroen van Aart wrote:
> But why not use a raid1 instead of the one disk? It will increase
> redundancy even more. Now I understand that in your load balanced
> setup
> if one or a few servers' disks would die the system would still
> function. But you would have to clone/recreate the system and bring it
> back up. With a raid1 you would only have to replace the broken disk
> and
> that's it (or if you had a hot spare you had to do nothing).

I have to say I am getting less convinced by RAID 1 on systems. Its a
really good idea in theory, but in practice disks appear to be able to
come up with failure modes outside of what the controller or software
expects, and of course these failure modes are less well tested.... Add
to that the controllers for hardware RAID sometimes appear to find their
own failure modes. Designing so that the complete unit (ie one box with
internal disk etc) is disposable and is just removed from service when
there is a problem certainly has its attractions.

>> total. Maildirs for the users are stored on NetApp filers.
>
> Netapp uses NFS I assume? How well does it work with imap? As far as I
> know using imap on anything but unix format mailboxes on a networked
> filesystem like NFS can cause problems due to file locking and such.

Maildir is locking free. (Well thats strictly untrue - it uses directory
level concurrency control within the kernel but thats invisible to you).

Imap on NFS (particularly NetApp NFS) using Maildir is hardly new stuff
- thats how the Planet/Freeserve mail system was implemented 10 years
ago - that scaled to something over 3 million users then. You can add
some tweaks to Maildir to reduce the NFS load (see Yann Golanski's paper
on the subject from around the turn of the millennium). NFSv3 may have
made some of these tweaks less useful.

Nigel.

--
[ Nigel Metheringham Nigel.Metheringham [at] InTechnology ]
[. - Comments in this message are my own and not ITO opinion/policy - ]

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

brian at ecmsquared

Jun 12, 2008, 5:24 PM

Post #6 of 11 (2184 views)
Permalink

Re: Which hardware do you use for your installation? [In reply to]

On Jun 11, 2008, at 2:46 PM, Sander Smeenk wrote:
> One big fat tip for you, free of charge:
> Exim is not good at dealing with large queues, partly because of
> the way
> it keeps state in the db4 file. That's why almost all these platforms
> have their spool in memory and 'message logging' is turned off (-Mvl
> doesn't work).

This is why we tend to use Postfix for managing outbound email queues.

Exim for all the intelligent stuff; Postfix for brute force SMTP
delivery.

Moya dve kopek,
Brian

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

hagen at rz

Jun 13, 2008, 1:59 AM

Post #7 of 11 (2182 views)
Permalink

Re: Which hardware do you use for your installation? [In reply to]

Brian Blood schrieb:
> On Jun 11, 2008, at 2:46 PM, Sander Smeenk wrote:
>> One big fat tip for you, free of charge:
>> Exim is not good at dealing with large queues, partly because of
>> the way
>> it keeps state in the db4 file. That's why almost all these platforms
>> have their spool in memory and 'message logging' is turned off (-Mvl
>> doesn't work).
>
>
> This is why we tend to use Postfix for managing outbound email queues.
Just for the record: could you please put it into perspecive? Which is
the queue-size where exim starts showing problems?

Some time ago at exim-users-de some admin reported his queue-problems.
His queue was incredibly big and since everbody knows about exims
large-queue-problems he asked for advice.... IIRC how to fall back to a
second outbound-host if the first one can't deliver instantly,
disk-optimizations, changing number of concurrent queue-runners etc.

Queue-size was 300. Greylisting, unreachable recipients, etc.
In my experience, that size is really no issue for exim. ;-)

--
CU,
Patrick.

Attachments:

smime.p7s (5.75 KB)

ssmeenk at freshdot

Jun 13, 2008, 7:05 AM

Post #8 of 11 (2187 views)
Permalink

Re: Which hardware do you use for your installation? [In reply to]

Quoting Patrick von der Hagen (hagen [at] rz):

> Queue-size was 300. Greylisting, unreachable recipients, etc.
> In my experience, that size is really no issue for exim. ;-)

Oh, no, 300 is peanuts for Exim. Given that it's not doing intensive
spamscanning, virusscanning etc on the same server and the server is
nice and fast...

Our platform began to show issues with queues wel over 10k. That's why
we also have mailspoolers running postfix that kick in when the mx'es
can't take the load anymore.
Also we redirect bounces to another box running postfix to deliver them.

Exactly as Brian Blood stated.
Exim for the intelligent stuff, Postfix for the brute force deliveries :)

-Sndr.
--
| My Bonnie looked into a gas tank, the height of its contents to see!
| She lit a small match to assist her, oh bring back my Bonnie to me.
| 1024D/08CEC94D - 34B3 3314 B146 E13C 70C8 9BDB D463 7E41 08CE C94D

Attachments:

signature.asc (0.18 KB)

frank at bcsi

Jun 13, 2008, 8:17 AM

Post #9 of 11 (2167 views)
Permalink

Re: Which hardware do you use for your installation? [In reply to]

Dumb question(s)...

Sander Smeenk wrote:
> Exim for the intelligent stuff, Postfix for the brute force deliveries :)
>

Why is this a problem for EXIM (someone mentioned db4)?

Can/should this be fixed? It seems silly and time consuming to have to
run/support two mta's.

None of my customers using EXIM are processing even close to 10K emails
but I'm just trying to understand the underlying technology.

Cheers all.

Attachments:

frank.vcf (0.24 KB)

wbh at conducive

Jun 14, 2008, 2:59 AM

Post #10 of 11 (2175 views)
Permalink

Re: Which hardware do you use for your installation? [In reply to]

Sander Smeenk wrote:
> Quoting Patrick von der Hagen (hagen [at] rz):
>
>> Queue-size was 300. Greylisting, unreachable recipients, etc.
>> In my experience, that size is really no issue for exim. ;-)
>
> Oh, no, 300 is peanuts for Exim. Given that it's not doing intensive
> spamscanning, virusscanning etc on the same server and the server is
> nice and fast...
>
> Our platform began to show issues with queues wel over 10k. That's why
> we also have mailspoolers running postfix that kick in when the mx'es
> can't take the load anymore.
> Also we redirect bounces to another box running postfix to deliver them.
>
> Exactly as Brian Blood stated.
> Exim for the intelligent stuff, Postfix for the brute force deliveries :)
>
> -Sndr.
>

To be fair, several other MTA, Postfix among them, can do the
'intelligent' stuff also.

Can't say as you will still be sane after sorting out how to program and
maintain the needful rules sets, OTOH. Exin places a lighter load on the
'wetware'.

;-)

Bill

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

ssmeenk at freshdot

Jun 18, 2008, 10:51 AM

Post #11 of 11 (2111 views)
Permalink

Re: Which hardware do you use for your installation? [In reply to]

Quoting Frank S. Bernhardt (frank [at] bcsi):

>> Exim for the intelligent stuff, Postfix for the brute force deliveries :)
> Why is this a problem for EXIM (someone mentioned db4)?

I have good reasons to believe Exim isn't good at handling large queues
due to the way the spool directory is managed. A quick guess would be
that the db4 file, used for storing retry and wait data, might get big
and lumpy with loads of messages in the queue and loads of hosts to
deliver to.

Though i don't have hard proof for this theory, nor did i check wether
it is possible to choose some other database backend; after we moved the
spool to /dev/shm (in memory) our platform, Exim was much faster and
could handle more messages.

Postfix otoh has a well structured spool directory. I think Postfix
keeps state in memory. It has a much bigger memoryfootprint with it's
'split functions' design...

> Can/should this be fixed?

Maybe. It has to be investigated more thoroughly first, i think.
Such a change probably affects large parts of Exim's code.

> It seems silly and time consuming to have to run/support two mta's.

Well, all it does is accept, queue and deliver. Nothing special, as
long as we keep the box up to date we don't have to 'run/support'
anything about it. ;-)

HTH
-Sndr.
--
| One nice thing about egotists: They don't talk about other people.
| 1024D/08CEC94D - 34B3 3314 B146 E13C 70C8 9BDB D463 7E41 08CE C94D

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads