Mailing List Archive: exim: users

How do you get multiple messages sent via the same smarthost SMTP connection?

Index | Next | Previous | View Threaded

adam.nielsen at uq

Jun 30, 2008, 10:32 PM

Post #1 of 14 (498 views)
Permalink

How do you get multiple messages sent via the same smarthost SMTP connection?

Hi all,

I've got a Mailman installation on a server that feeds its messages in
to Exim. Since this machine is behind a firewall, all mail is
smarthosted out to another mailserver.

The problem is that this other mailserver (beyond my control) has a rate
limit enforced. This means that after 20 connections in a row further
connections are dropped until traffic stops for a few minutes.

Once Mailman has delivered all the messages to Exim, Exim seems to make
a separate SMTP connection to the smarthost for each message. After 20
messages (20 connections) the IP gets rate limited and Exim can't
deliver any more messages (the rest of the queued messages fail with
"defer (-53): all host address lookups failed temporarily") 15 minutes
later when the next queue runner starts another 20 messages go but
that's it.

From reading the docs it seems that Exim is supposed to reuse SMTP
connections, but for some reason this isn't happening for me. Perhaps
it's because of the "mailman_verp_smtp" transport I have set up?
There's an option in there called "max_rcpt" which limits one recipient
per message which would seem to be causing this problem, but as far as I
understand I can't remove that otherwise the wrong envelope addresses
etc. would be used.

Any ideas would be much appreciated, because at the moment a mailout
takes almost an entire day to complete!

Many thanks,
Adam.

----------
2008-07-01 15:14:19 1KDSVQ-0007ki-QN == someone[at]somewhere.com
R=mailman_verp_router T=mailman_verp_smtp defer (0): SMTP error from
remote mail server after initial connection: host smart.host.name
[1.2.3.4]: 421 4.3.2 Connection rate limit exceeded.
2008-07-01 15:14:19 1KDSVQ-0007ki-QN failed to open DB file
/var/spool/exim/db/wait-mailman_verp_smtp: Success
2008-07-01 15:14:19 1KDSVQ-0007ki-QN == someone[at]somewhere.com
R=mailman_verp_router T=mailman_verp_smtp defer (-53): all host address
lookups failed temporarily
2008-07-01 15:14:19 1KDSVQ-0007ki-QN failed to open DB file
/var/spool/exim/db/wait-mailman_verp_smtp: Success
2008-07-01 15:14:19 1KDSVQ-0007ki-QN == someone[at]somewhere.com
R=mailman_verp_router T=mailman_verp_smtp defer (-53): all host address
lookups failed temporarily
2008-07-01 15:14:19 1KDSVQ-0007ki-QN failed to open DB file
/var/spool/exim/db/wait-mailman_verp_smtp: Success
2008-07-01 15:14:19 1KDSVQ-0007ki-QN == someone[at]somewhere.com
R=mailman_verp_router T=mailman_verp_smtp defer (-53): all host address
lookups failed temporarily

mailman_verp_router:
# driver = dnslookup
#Send this via a smarthost instead
driver = manualroute
route_list = * SMART_HOST
host_find_failed = freeze

# we only consider messages sent in through loopback
condition = ${if \
or{ \
{eq{$sender_host_address}{127.0.0.1}} \
{eq{$sender_host_address}{::1}} \
}{yes}{no} \
}

# we do not do this for traffic going to the local machine
domains = !+local_domains:!+mm_domains
ignore_target_hosts = <; 0.0.0.0; \
64.94.110.11; \
127.0.0.0/8; \
::1/128;fe80::/10;fe \
c0::/10;ff00::/8
# only the un-VERPed bounce addresses are handled
senders = "*-bounces@*"
transport = mailman_verp_smtp

mailman_verp_smtp:
driver = smtp
# put recipient address into return_path
return_path =
${local_part:$return_path}+$local_part=$domain@${domain:$return_path}
# must restrict to one recipient at a time
max_rcpt = 1
# Errors-To: may carry old return_path
headers_remove = Errors-To
headers_add = Errors-To: ${return_path}

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

exim-users at spodhuis

Jun 30, 2008, 11:28 PM

Post #2 of 14 (486 views)
Permalink

Re: How do you get multiple messages sent via the same smarthost SMTP connection? [In reply to]

On 2008-07-01 at 15:32 +1000, Adam Nielsen wrote:
> I've got a Mailman installation on a server that feeds its messages in
> to Exim. Since this machine is behind a firewall, all mail is
> smarthosted out to another mailserver.
>
> The problem is that this other mailserver (beyond my control) has a rate
> limit enforced. This means that after 20 connections in a row further
> connections are dropped until traffic stops for a few minutes.

You haven't set connection_max_messages = 1 ?

Look at remote_max_parallel and queue_run_max.

What's probably happening is that Exim is receiving the mail and
immediately trying to send it on. If you can tolerate a short delay in
sending email, you might try configuring Exim to queue all mail instead
of delivering immediately and running with more frequent queue-runners.

Set "queue_only" true in the main configuration. Or, if it's not *all*
mail sent out via the smarthost, look at instead:
queue_smtp_domains = *

Start exim with "exim -bd -q2m" to be a listening daemon which forks
queue-runners every 2 minutes. Or -q1m if you want.

Actually, "exim -bd -odqs -q2m" would be equivalent if you want to
change the cmdline invocation and not change the config.

Basically, you're arranging for Exim to figure out where all the mail
has to go, but only actually sending it out periodically, when there's
likely to be multiple items that can be delivered down a connection.

Regards,
-Phil

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

adam.nielsen at uq

Jun 30, 2008, 11:37 PM

Post #3 of 14 (485 views)
Permalink

Re: How do you get multiple messages sent via the same smarthost SMTP connection? [In reply to]

>> The problem is that this other mailserver (beyond my control) has a rate
>> limit enforced. This means that after 20 connections in a row further
>> connections are dropped until traffic stops for a few minutes.
>
> You haven't set connection_max_messages = 1 ?

Nope, I've tried setting it to 200 messages as well but with no difference.

> Look at remote_max_parallel and queue_run_max.
>
> What's probably happening is that Exim is receiving the mail and
> immediately trying to send it on. If you can tolerate a short delay in
> sending email, you might try configuring Exim to queue all mail instead
> of delivering immediately and running with more frequent queue-runners.

That's what I thought, but after the first run through there are about
1000 messages stuck in the queue (because they couldn't contact the
server.) Every time I start a queue runner another 20 messages go but
the rest are stuck, so I assume pre-queueing all the messages wouldn't
make any difference?

> Basically, you're arranging for Exim to figure out where all the mail
> has to go, but only actually sending it out periodically, when there's
> likely to be multiple items that can be delivered down a connection.

Are "figuring out where the mail has to go" and "sending it out"
actually two different steps? Because like I say the mail is being
queued, and flushing the queue still only sends one message per SMTP
connection - but if it's not "figuring out where the mail has to go"
until the last minute perhaps this could still explain why it's only
sending one message per connection.

Thanks for your reply!

Cheers,
Adam.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

graeme at graemef

Jul 1, 2008, 2:56 AM

Post #4 of 14 (477 views)
Permalink

Re: How do you get multiple messages sent via the same smarthost SMTP connection? [In reply to]

On Tue, 2008-07-01 at 15:32 +1000, Adam Nielsen wrote:
> From reading the docs it seems that Exim is supposed to reuse SMTP
> connections, but for some reason this isn't happening for me. Perhaps
> it's because of the "mailman_verp_smtp" transport I have set up?

Yes, that's it. Mailman and the VERP router ensures that every message
is sent to a single recipient (for bounce handling automation).

You can change the way exim is running to get around this. At the moment
you're probably calling it with options like "-bd -q20m" or similar.

If you change the daemon options to "-bd" only, to run the listening
daemon, you can then run the queue runners from cron instead with
different options.

http://www.exim.org/exim-html-current/doc/html/spec_html/ch05.html#SECID39

"-qq...

An option starting with -qq requests a two-stage queue run. In
the first stage, the queue is scanned as if the
queue_smtp_domains option matched every domain. Addresses are
routed, local deliveries happen, but no remote transports are
run.

The hints database that remembers which messages are waiting
for specific hosts is updated, as if delivery to those hosts had
been deferred. After this is complete, a second, normal queue
scan happens, with routing and delivery taking place as normal.
Messages that are routed to the same host should mostly be
delivered down a single SMTP connection because of the hints
that were set up during the first queue scan. This option may be
useful for hosts that are connected to the Internet
intermittently."

So you could have a cron job which looks like this:

*/20 * * * * /usr/sbin/exim -qq

In conjunction with a non-queue-running daemon that may get you what you
want.

Graeme

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Lena at lena

Jul 1, 2008, 9:51 AM

Post #5 of 14 (471 views)
Permalink

Re: How do you get multiple messages sent via the same smarthost SMTP connection? [In reply to]

> From: Adam Nielsen

> Exim seems to make
> a separate SMTP connection to the smarthost for each message.

> From reading the docs it seems that Exim is supposed to reuse SMTP
> connections, but for some reason this isn't happening for me.

I'm not sure, but you can try:

begin transports
remote_smtp:
driver = smtp
serialize_hosts = *

This solution is better (if it works) because messages will not wait for
next queue run.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

exim-users at spodhuis

Jul 1, 2008, 4:03 PM

Post #6 of 14 (468 views)
Permalink

Re: How do you get multiple messages sent via the same smarthost SMTP connection? [In reply to]

On 2008-07-01 at 16:37 +1000, Adam Nielsen wrote:
> Are "figuring out where the mail has to go" and "sending it out"
> actually two different steps? Because like I say the mail is being
> queued, and flushing the queue still only sends one message per SMTP
> connection - but if it's not "figuring out where the mail has to go"
> until the last minute perhaps this could still explain why it's only
> sending one message per connection.

Figuring out: Routers
Sending it out: Transports

You need more than just "it's left around"; the routing needs to have
happened, so that a given SMTP connection has a whole bunch of messages
which can go down, even if split up into one recipient each.

You want to ensure that Routing is done ahead of time; -odqs will do
this, or two-stage queueing with -qq (thanks Graeme for the reminder)
or using queue_smtp_domains.

Just using queue_only isn't sufficient, my mistake. The memory of
managing heavily loaded Exim is fading. :^/

-Phil

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

adam.nielsen at uq

Jul 1, 2008, 5:41 PM

Post #7 of 14 (465 views)
Permalink

Re: How do you get multiple messages sent via the same smarthost SMTP connection? [In reply to]

> You need more than just "it's left around"; the routing needs to have
> happened, so that a given SMTP connection has a whole bunch of messages
> which can go down, even if split up into one recipient each.

Ah ok, that makes it clearer.

> You want to ensure that Routing is done ahead of time; -odqs will do
> this, or two-stage queueing with -qq (thanks Graeme for the reminder)
> or using queue_smtp_domains.

Thanks for the suggestions. I tried changing the daemon to use only
-bd, however the messages were still being delivered immediately. I
think this is because the docs say -odb is the default, which performs
immediate delivery.

I changed the daemon to use "-bd -odqs" and this appeared to work
correctly, all the outgoing messages came up as:

...R=mailman_verp_router T=mailman_verp_smtp defer (0): SMTP delivery
explicitly queued

Which seems like what I want. The problem is when I run "exim -qq -v"
to perform the delivery, I still only get one message sent per
connection - the exim output clearly shows it disconnecting and
reconnecting to the same server:

SMTP<< 354 Enter mail, end with "." on a line by itself
SMTP>> writing message and terminating "."
SMTP<< 250 2.0.0 m620JWVh003668 Message accepted for delivery
LOG: MAIN
failed to open DB file /var/spool/exim/db/wait-mailman_verp_smtp: Success
SMTP>> QUIT
LOG: MAIN
=> next.address[at]example.com R=mailman_verp_router T=mailman_verp_smtp
H=smtp.hostname [1.2.3.4]
Connecting to smtp.hostname [1.2.3.4]:25 ... connected
SMTP<< 220 smtp.hostname ESMTP Sendmail 8.13.8/8.13.8; Wed, 2 Jul
2008 10:19:32 +1000
SMTP>> EHLO my.hostname

I've been wondering what that "failed to open DB file - Success" error
was for - given that it checks the database *before* disconnecting from
the server, it this where Exim stores the list of other messages waiting
to use the same connection? According to the FAQ it looks like there's
some problem with my DBM library? The DB file doesn't exist.

Thanks,
Adam.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

adam.nielsen at uq

Jul 1, 2008, 7:24 PM

Post #8 of 14 (465 views)
Permalink

Re: How do you get multiple messages sent via the same smarthost SMTP connection? [In reply to]

> I'm not sure, but you can try:
>
> begin transports
> remote_smtp:
> driver = smtp
> serialize_hosts = *
>
> This solution is better (if it works) because messages will not wait for
> next queue run.

I tried this, but it seems to have a slightly different result. It does
only make one connection to the smarthost, but once it has sent that
message it skips all remaining messages in the queue because the single
allowed smarthost connection is in use. (Perhaps because it was doing
it in parallel.)

This means that only one message is sent each time the queue runs, so
for 1000 messages the queue will need to be run through 1000 times!
Even if it was changed so that no parallel deliveries were permitted, it
looks like it will still disconnect from the smarthost after each message.

Thanks for the suggestion though!

Cheers,
Adam.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

exim-users at spodhuis

Jul 1, 2008, 9:34 PM

Post #9 of 14 (465 views)
Permalink

Re: How do you get multiple messages sent via the same smarthost SMTP connection? [In reply to]

On 2008-07-02 at 10:41 +1000, Adam Nielsen wrote:
> SMTP<< 354 Enter mail, end with "." on a line by itself
> SMTP>> writing message and terminating "."
> SMTP<< 250 2.0.0 m620JWVh003668 Message accepted for delivery
> LOG: MAIN
> failed to open DB file /var/spool/exim/db/wait-mailman_verp_smtp: Success
> SMTP>> QUIT
> LOG: MAIN
> => next.address[at]example.com R=mailman_verp_router T=mailman_verp_smtp
> H=smtp.hostname [1.2.3.4]
> Connecting to smtp.hostname [1.2.3.4]:25 ... connected
> SMTP<< 220 smtp.hostname ESMTP Sendmail 8.13.8/8.13.8; Wed, 2 Jul
> 2008 10:19:32 +1000
> SMTP>> EHLO my.hostname
>
> I've been wondering what that "failed to open DB file - Success" error
> was for - given that it checks the database *before* disconnecting from
> the server, it this where Exim stores the list of other messages waiting
> to use the same connection? According to the FAQ it looks like there's
> some problem with my DBM library? The DB file doesn't exist.

Yes, that's where Exim stores the list of mails pending for the
destination.

You can "rm -rf" the db directory and Exim should recreate it with
correct permissions, provided that it can write to /var/spool/exim/.
Just don't remove the parent of the db directory! The mails in the
spool are elsewhere under /var/spool/exim/.

-Phil

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

adam.nielsen at uq

Jul 1, 2008, 9:42 PM

Post #10 of 14 (462 views)
Permalink

Re: How do you get multiple messages sent via the same smarthost SMTP connection? [In reply to]

> You can "rm -rf" the db directory and Exim should recreate it with
> correct permissions, provided that it can write to /var/spool/exim/.
> Just don't remove the parent of the db directory! The mails in the
> spool are elsewhere under /var/spool/exim/.

Hmm, well I deleted it and Exim recreated it, but it still gives me the
same error message. The db/ folder only contains retry.lockfile and
wait-remote_smtp.lockfile. Should there be other non-lockfiles in there?

Thanks,
Adam.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

exim-users at spodhuis

Jul 1, 2008, 11:47 PM

Post #11 of 14 (458 views)
Permalink

Re: How do you get multiple messages sent via the same smarthost SMTP connection? [In reply to]

On 2008-07-02 at 14:42 +1000, Adam Nielsen wrote:
> > You can "rm -rf" the db directory and Exim should recreate it with
> > correct permissions, provided that it can write to /var/spool/exim/.
> > Just don't remove the parent of the db directory! The mails in the
> > spool are elsewhere under /var/spool/exim/.
>
> Hmm, well I deleted it and Exim recreated it, but it still gives me the
> same error message. The db/ folder only contains retry.lockfile and
> wait-remote_smtp.lockfile. Should there be other non-lockfiles in there?

Yes. Various db files, whose maintenance is described in section 50 of
spec.txt and some wait-* files for each transport in use. Sounds as
though your Exim install is broken, permissions-wise. Did you remove
the setuid-root permission from the binary or something?

If you can't find the problem your end, you'll need to post an "ls -ld"
of the db folder, the output of:
$ exim -bP exim_user exim_group
the Local/Makefile (you can skip the comments for the list, so:
$ grep '^[^#]' Local/Makefile
and preferably as much of your configuration as you're comfortable
posting.

-Phil

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

adam.nielsen at uq

Jul 2, 2008, 4:49 PM

Post #12 of 14 (443 views)
Permalink

Re: How do you get multiple messages sent via the same smarthost SMTP connection? [In reply to]

> Yes. Various db files, whose maintenance is described in section 50 of
> spec.txt and some wait-* files for each transport in use. Sounds as
> though your Exim install is broken, permissions-wise. Did you remove
> the setuid-root permission from the binary or something?

Nope, this all looks fine. The users match up and the permissions seem
ok (after all, Exim itself successfully creates the db/ folder if I
delete it.)

After reading section 50, I think the problem lies with the DB library:

# exim_dbmbuild /etc/aliases test.db
exim_dbmbuild: unable to create test.db.dbmbuild_temp: Success

This is as root, in a folder I can create other files in. I'm a bit
confused about what DBM is, as far as I can make out it's an interface
for creating database files, but with different implementations
depending on the OS. There seem to be so many variations on it that
Google isn't much help.

It would appear that I'm using the wrong/old version - what version does
Exim require?

Thanks for all this help :-)

Cheers,
Adam.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

adam.nielsen at uq

Jul 2, 2008, 7:24 PM

Post #13 of 14 (440 views)
Permalink

Re: How do you get multiple messages sent via the same smarthost SMTP connection? [In reply to]

> It would appear that I'm using the wrong/old version - what version does
> Exim require?

Ah, well I found the very helpful doc/dbm.discuss.txt. I'm not sure
what version of the Berkeley DB I have installed, but I thought I'd try
to go with the native interface to GDBM instead.

Unfortunately after recompiling and reinstalling the latest version I
could find (gdbm-1.8.3) I can't compile Exim (this is with USE_GDBM=yes)

gcc exim_dbmbuild.c
gcc -o exim_dbmbuild
exim_dbmbuild.o(.text+0x458): In function `main':
: undefined reference to `gdbm_open'
exim_dbmbuild.o(.text+0x62f): In function `main':
: undefined reference to `gdbm_store'
exim_dbmbuild.o(.text+0x6bc): In function `main':
: undefined reference to `gdbm_store'
exim_dbmbuild.o(.text+0x94c): In function `main':
: undefined reference to `gdbm_store'
exim_dbmbuild.o(.text+0x9e0): In function `main':
: undefined reference to `gdbm_store'
exim_dbmbuild.o(.text+0xa63): In function `main':
: undefined reference to `gdbm_close'
collect2: ld returned 1 exit status
make[1]: *** [exim_dbmbuild] Error 1

It looks like it's not picking up the library file for some reason, but
I've erased all traces of the old gdbm files from /usr/lib (and
/usr/local/lib) but I still get that error. I'm not sure sure how to
find out what the full command line is so I'm not sure exactly what's
happening. Perhaps it's not even trying to link in the gdbm library?

Any ideas? If I put USE_DB=yes or leave that out entirely in the
Makefile then it compiles without problems but I get that failed/success
error.

Thanks,
Adam.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

adam.nielsen at uq

Jul 2, 2008, 8:35 PM

Post #14 of 14 (435 views)
Permalink

Re: Solved compilation failure with gdbm library (was: How do you get multiple messages sent via the same smarthost SMTP connection?) [In reply to]

> Perhaps it's not even trying to link in the gdbm library?

I've done some more digging and found the command it's using to compile
and that is indeed the case:

$ gcc -o exim_dbmbuild exim_dbmbuild.o -lnsl -lcrypt -lm -ldb
exim_dbmbuild.o(.text+0x458): In function `main':
: undefined reference to `gdbm_open'
exim_dbmbuild.o(.text+0x62f): In function `main':
: undefined reference to `gdbm_store'
exim_dbmbuild.o(.text+0x6bc): In function `main':
: undefined reference to `gdbm_store'
exim_dbmbuild.o(.text+0x94c): In function `main':
: undefined reference to `gdbm_store'
exim_dbmbuild.o(.text+0x9e0): In function `main':
: undefined reference to `gdbm_store'
exim_dbmbuild.o(.text+0xa63): In function `main':
: undefined reference to `gdbm_close'
collect2: ld returned 1 exit status

If I change it from the "db" library to "gdbm" then it works:

$ gcc -o exim_dbmbuild exim_dbmbuild.o -lnsl -lcrypt -lm -lgdbm
$ ./exim_dbmbuild /etc/aliases test.db
18 entries written

So it looks like the Makefile isn't pulling in the correct library when
USE_GDBM=yes.

I edited the generated Makefile by hand and changed the library:

DBMLIB = -lgdbm

And this appears to have worked fine - I recompiled Exim and installed
it and I don't get any of those failed/success errors any more! Now I
can go back to investigating the original problem :-)

Cheers,
Adam.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com