Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: exim: users

Exim outgoing emails throttling

 

 

exim users RSS feed   Index | Next | Previous | View Threaded


mirfan1981 at gmail

Jul 16, 2012, 11:21 PM

Post #1 of 13 (3167 views)
Permalink
Exim outgoing emails throttling

I normally sends mass no. of emails via script twice per day (around 40000
emails) each email contains one recipient.
And my script will sleep for 5 seconds after every 1000 emails. I noticed
many exim processes running when script exeuctes
and around 20-30% emails stucks in queue and server load is quite high
during mass mailing.
What are the specific parameters to optimize outgoing mass emails ? Is
there any way to throttle outgoing emails ?

Furhter, i noticed stuck(queued) emails logs describes.

* Recipient address rejected: Greylisted for 2 minutes
* 451 Temporary local problem - please try later
* R=dkim_lookuphost defer (-1): host lookup did not complete

To me it's look likes recipient addresses are not accepting my emails.
Probably DNS, RDNS or SPF issue. right or wrong ?

and for successfull outgoing emails normally log describes
* SMTP Outbound Connection ..
what does that means ?
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


Chris.Knadle at coredump

Jul 17, 2012, 8:19 AM

Post #2 of 13 (3108 views)
Permalink
Re: Exim outgoing emails throttling [In reply to]

On Tuesday, July 17, 2012 02:21:43, Muhammad Irfan wrote:
> I normally sends mass no. of emails via script twice per day (around 40000
> emails) each email contains one recipient.
> And my script will sleep for 5 seconds after every 1000 emails. I noticed
> many exim processes running when script exeuctes
> and around 20-30% emails stucks in queue and server load is quite high
> during mass mailing.
> What are the specific parameters to optimize outgoing mass emails ? Is
> there any way to throttle outgoing emails ?
>
> Furhter, i noticed stuck(queued) emails logs describes.
>
> * Recipient address rejected: Greylisted for 2 minutes
> * 451 Temporary local problem - please try later
> * R=dkim_lookuphost defer (-1): host lookup did not complete
>
> To me it's look likes recipient addresses are not accepting my emails.

Sort of, and not exactly.

> Probably DNS, RDNS or SPF issue. right or wrong ?

I don't think this is RDNS or SPF is related.


* Greylisting:
https://en.wikipedia.org/wiki/Greylisting



* "451 Temporary local problem - please try later" is usually the response
when a local configuration error has been reached. So for instance if you
have written an ACL rule that has an error in it and an incoming SMTP
connection reaches it, a "Temporary local problem" will result.



* With DKIM the verifier uses DNS to retrieve the signer's public key to
verify the signed mail.

DKIM:
https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail

The lookup failing likely means the DNS lookup timed out. DNS [normally] uses
UDP which means the packets are not guaranteed to arrive, so these can get
lost if (for instance) the packet queue on the router or switch is full such
that some incoming packets have to be dropped.

> and for successfull outgoing emails normally log describes
> * SMTP Outbound Connection ..
> what does that means ?

Without context it's hard to know, but it sounds like what it says -- that an
outbound SMTP connection is/was made.

-- Chris

--
Chris Knadle
Chris.Knadle [at] coredump

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


mirfan1981 at gmail

Jul 19, 2012, 3:26 AM

Post #3 of 13 (3110 views)
Permalink
Re: Exim outgoing emails throttling [In reply to]

I am concerning more specifically for what are the specific parameters to
optimize outgoing mass emails in exim.conf ?
Is there any way to throttle outgoing emails ?

Thank you.

On Tue, Jul 17, 2012 at 8:19 PM, Chris Knadle <Chris.Knadle [at] coredump>wrote:

> On Tuesday, July 17, 2012 02:21:43, Muhammad Irfan wrote:
> > I normally sends mass no. of emails via script twice per day (around
> 40000
> > emails) each email contains one recipient.
> > And my script will sleep for 5 seconds after every 1000 emails. I noticed
> > many exim processes running when script exeuctes
> > and around 20-30% emails stucks in queue and server load is quite high
> > during mass mailing.
> > What are the specific parameters to optimize outgoing mass emails ? Is
> > there any way to throttle outgoing emails ?
> >
> > Furhter, i noticed stuck(queued) emails logs describes.
> >
> > * Recipient address rejected: Greylisted for 2 minutes
> > * 451 Temporary local problem - please try later
> > * R=dkim_lookuphost defer (-1): host lookup did not complete
> >
> > To me it's look likes recipient addresses are not accepting my emails.
>
> Sort of, and not exactly.
>
> > Probably DNS, RDNS or SPF issue. right or wrong ?
>
> I don't think this is RDNS or SPF is related.
>
>
> * Greylisting:
> https://en.wikipedia.org/wiki/Greylisting
>
>
>
> * "451 Temporary local problem - please try later" is usually the response
> when a local configuration error has been reached. So for instance if you
> have written an ACL rule that has an error in it and an incoming SMTP
> connection reaches it, a "Temporary local problem" will result.
>
>
>
> * With DKIM the verifier uses DNS to retrieve the signer's public key to
> verify the signed mail.
>
> DKIM:
> https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail
>
> The lookup failing likely means the DNS lookup timed out. DNS [normally]
> uses
> UDP which means the packets are not guaranteed to arrive, so these can get
> lost if (for instance) the packet queue on the router or switch is full
> such
> that some incoming packets have to be dropped.
>
> > and for successfull outgoing emails normally log describes
> > * SMTP Outbound Connection ..
> > what does that means ?
>
> Without context it's hard to know, but it sounds like what it says -- that
> an
> outbound SMTP connection is/was made.
>
> -- Chris
>
> --
> Chris Knadle
> Chris.Knadle [at] coredump
>
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/
>
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


Chris.Knadle at coredump

Jul 19, 2012, 9:46 AM

Post #4 of 13 (3097 views)
Permalink
Re: Exim outgoing emails throttling [In reply to]

On Thursday, July 19, 2012 06:26:16, Muhammad Irfan wrote:
> I am concerning more specifically for what are the specific parameters to
> optimize outgoing mass emails in exim.conf ?
> Is there any way to throttle outgoing emails ?

I don't know offhand but I can give you a quick hint.
In the online documentation for Exim:

- Look at items in "Resource Control" in Section 14, subsections 10 and 21
- Have a look at the "rate limiting" in section 42, subsections 36 - 41.

-- Chris

--
Chris Knadle
Chris.Knadle [at] coredump

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


dwmw2 at infradead

Jul 23, 2012, 8:58 AM

Post #5 of 13 (3073 views)
Permalink
Re: Exim outgoing emails throttling [In reply to]

On Tue, 2012-07-17 at 11:19 -0400, Chris Knadle wrote:
> > * Recipient address rejected: Greylisted for 2 minutes
> > ...
> > Probably DNS, RDNS or SPF issue. right or wrong ?
>
> I don't think this is RDNS or SPF is related.

It might be. Any half-sane greylisting implementation will have to have
some *reason* for greylisting the messages. Lack of reverse DNS is
certainly something that triggers greylisting on my systems.

You could also imagine people using SPF as a trigger — either an SPF
failure causing greylisting to happen, or an SPF 'pass' causing it to be
avoided. It's one of the few things that SPF might actually be useful
for.

--
dwmw2
Attachments: smime.p7s (6.03 KB)


Chris.Knadle at coredump

Jul 23, 2012, 12:10 PM

Post #6 of 13 (3068 views)
Permalink
Re: Exim outgoing emails throttling [In reply to]

On Monday, July 23, 2012 11:58:58, David Woodhouse wrote:
> On Tue, 2012-07-17 at 11:19 -0400, Chris Knadle wrote:
> > > * Recipient address rejected: Greylisted for 2 minutes
> > > ...
> > > Probably DNS, RDNS or SPF issue. right or wrong ?
> >
> > I don't think this is RDNS or SPF is related.
>
> It might be. Any half-sane greylisting implementation will have to have
> some *reason* for greylisting the messages. Lack of reverse DNS is
> certainly something that triggers greylisting on my systems.
>
> You could also imagine people using SPF as a trigger — either an SPF
> failure causing greylisting to happen, or an SPF 'pass' causing it to be
> avoided. It's one of the few things that SPF might actually be useful
> for.

The two greylisting implementations I'm familiar with, greylistd and postgrey,
don't seem to have any configuration options related to RDNS or SPF. As far
as I can tell they both operate solely on whether there is already an entry
for the sender emailing the recipient, otherwise a temporary rejection is done
to force the sending mailserver to retry. There are configuration options for
the minimum wait time for accepting a retry and the length of time to hold
greylist entries before they expire.

-- Chris

--
Chris Knadle
Chris.Knadle [at] coredump

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


dwmw2 at infradead

Jul 24, 2012, 5:24 AM

Post #7 of 13 (3062 views)
Permalink
Re: Exim outgoing emails throttling [In reply to]

On Mon, 2012-07-23 at 15:10 -0400, Chris Knadle wrote:
> The two greylisting implementations I'm familiar with, greylistd and postgrey,
> don't seem to have any configuration options related to RDNS or SPF. As far
> as I can tell they both operate solely on whether there is already an entry
> for the sender emailing the recipient, otherwise a temporary rejection is done

I'm not sure about greylistd, but my encounters with postgrey have led
me to believe that if there's *anything* you can get wrong when
implementing greylisting, that's the design choice that postgrey will
have made.

cf. http://wiki.exim.org/SimpleGreylisting

--
dwmw2
Attachments: smime.p7s (6.03 KB)


Chris.Knadle at coredump

Jul 24, 2012, 6:55 AM

Post #8 of 13 (3081 views)
Permalink
Re: Exim outgoing emails throttling [In reply to]

On Tuesday, July 24, 2012 08:24:42, David Woodhouse wrote:
> On Mon, 2012-07-23 at 15:10 -0400, Chris Knadle wrote:
> > The two greylisting implementations I'm familiar with, greylistd and
> > postgrey, don't seem to have any configuration options related to RDNS
> > or SPF. As far as I can tell they both operate solely on whether there
> > is already an entry for the sender emailing the recipient, otherwise a
> > temporary rejection is done
>
> I'm not sure about greylistd, but my encounters with postgrey have led
> me to believe that if there's *anything* you can get wrong when
> implementing greylisting, that's the design choice that postgrey will
> have made.

Re: postgrey -- in what sense?

> cf. http://wiki.exim.org/SimpleGreylisting

I had a look through the implementation at the link above. I like the fact
that remote mail servers get whitelisted after they've been found to retry
sending mail, as that makes sense and is something that most greylisting
implementations don't do.

However I don't see anything in the implementation about RDNS or SPF. I
personally wouldn't trust RDNS or SPF in a greylisting implementation, because
both are something that are in the control of the DNS server for the sending
domain.

BTW let me know where I can get your GPG key.
You can get mine from hkps://zimmermann.mayfirst.org

-- Chris

--
Chris Knadle
Chris.Knadle [at] coredump
GPG Key: 4096R/0x1E759A726A9FDD74
Attachments: signature.asc (0.82 KB)


dwmw2 at infradead

Jul 24, 2012, 7:20 AM

Post #9 of 13 (3071 views)
Permalink
Re: Exim outgoing emails throttling [In reply to]

On Tue, 2012-07-24 at 09:55 -0400, Chris Knadle wrote:
> However I don't see anything in the implementation about RDNS or SPF.

It's not in the implementation. If it appeared, it would be one of the
examples in the 'Setting the conditions for "suspicious" mail' section.

As it says, you can use whatever conditions you see fit. Perhaps we
should add rDNS to the examples.

> personally wouldn't trust RDNS or SPF in a greylisting implementation, because
> both are something that are in the control of the DNS server for the sending
> domain.

I think you're missing the point of the 'suspicious' trigger. It's not
about *trusting* rDNS or SPF. If an email arrives which is not
considered suspicious in any other way — it has no SpamAssassin points,
isn't HTML, it has a Message-Id: and Date: header as it should, and
doesn't match any of your other triggers, then normally you'd accept
that message *without* the gratuitous delay that greylisting would add.

But if that same mail comes from a host which lacks reverse DNS, or has
an SPF 'fail' result, *then* you might want to greylist the mail.
Because now you *do* have a reason to consider it 'suspicious'.

--
dwmw2
Attachments: smime.p7s (6.03 KB)


dwmw2 at infradead

Jul 24, 2012, 7:33 AM

Post #10 of 13 (3060 views)
Permalink
Re: Exim outgoing emails throttling [In reply to]

On Tue, 2012-07-24 at 09:55 -0400, Chris Knadle wrote:
> BTW let me know where I can get your GPG key.
> You can get mine from hkps://zimmermann.mayfirst.org

Yours is also (sensibly) available from the standard keyservers; I got
it from keys.gnupg.net. Mine can be obtained there too. But my messages
aren't signed with PGP; they're signed with S/MIME. And I believe the
full trust chain is included therein.

--
dwmw2
Attachments: smime.p7s (6.03 KB)


wbh at conducive

Jul 25, 2012, 12:57 AM

Post #11 of 13 (3062 views)
Permalink
Re: Exim outgoing emails throttling [In reply to]

David Woodhouse wrote:
> On Tue, 2012-07-24 at 09:55 -0400, Chris Knadle wrote:
>> However I don't see anything in the implementation about RDNS or SPF.
>
> It's not in the implementation. If it appeared, it would be one of the
> examples in the 'Setting the conditions for "suspicious" mail' section.
>
> As it says, you can use whatever conditions you see fit. Perhaps we
> should add rDNS to the examples.
>
>> personally wouldn't trust RDNS or SPF in a greylisting implementation, because
>> both are something that are in the control of the DNS server for the sending
>> domain.
>
> I think you're missing the point of the 'suspicious' trigger. It's not
> about *trusting* rDNS or SPF. If an email arrives which is not
> considered suspicious in any other way — it has no SpamAssassin points,
> isn't HTML, it has a Message-Id: and Date: header as it should, and
> doesn't match any of your other triggers, then normally you'd accept
> that message *without* the gratuitous delay that greylisting would add.
>
> But if that same mail comes from a host which lacks reverse DNS, or has
> an SPF 'fail' result, *then* you might want to greylist the mail.
> Because now you *do* have a reason to consider it 'suspicious'.
>

If one applies rDNS sanely there isn't much need for the rest.

'The percentages' have it. If there is nothing obviously and
fundamentally 'wrong' right up-front, survivors carrying garbage are a
minority.

Still deserves further filtering, but if passing rDNS, all too often
everything else about the 'carrier' is legit as well - it is their
end-user who has been compromised.

Or is simply a professional advertiser studious enough to get all the
dots connected. Dots we lay out for them right here. And they get it
'right enough' that a specific LBL in response to a client complaint is
all that's left, as even their content is crafted to pass.

Meanwhile, layer upon layer of 'features' are piled-onto basic smtp that
focus on progressively less productive minutiae. SPF, DK, DKIM just burn
cycles and pollute headers for the vast majority of traffic, though
their delays and overhead are at least relatively transparent.

Greylisting, OTOH, gets 'in your face' when broadly applied.

I can't see playing the game where one smacks legit arrivals on first
sight just on general principle, then - 'maybe' - is kind enough to
whitelist those who ... actually hadn't set a foot wrong in the first
damned place.

Somebody, somewhere, is larfin' their arse off at all the runnin' in
circles they've got us doing!

Might it be an employment agency? One that places mailadmins into an
expanding need?

Doubtful. I don't see even that sort of gain being had.

Bill
--
韓家標

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


dwmw2 at infradead

Jul 25, 2012, 9:41 AM

Post #12 of 13 (3044 views)
Permalink
Re: Exim outgoing emails throttling [In reply to]

On Wed, 2012-07-25 at 07:57 +0000, W B Hacker wrote:
> Greylisting, OTOH, gets 'in your face' when broadly applied.
>
> I can't see playing the game where one smacks legit arrivals on first
> sight just on general principle, then - 'maybe' - is kind enough to
> whitelist those who ... actually hadn't set a foot wrong in the first
> damned place.

Sounds like you haven't read the wiki page that was referenced. It
covers this fairly comprehensively.

--
dwmw2
Attachments: smime.p7s (6.03 KB)


wbh at conducive

Jul 26, 2012, 7:56 AM

Post #13 of 13 (3041 views)
Permalink
Re: Exim outgoing emails throttling [In reply to]

David Woodhouse wrote:
> On Wed, 2012-07-25 at 07:57 +0000, W B Hacker wrote:
>> Greylisting, OTOH, gets 'in your face' when broadly applied.
>>
>> I can't see playing the game where one smacks legit arrivals on first
>> sight just on general principle, then - 'maybe' - is kind enough to
>> whitelist those who ... actually hadn't set a foot wrong in the first
>> damned place.
>
> Sounds like you haven't read the wiki page that was referenced. It
> covers this fairly comprehensively.
>
>
>
>
Did that research. And more. Implemented and tested more than one
method. Many more.

Yes, there are 'shades of grey' so to speak. 'Good' implementations as
well as careless ones.

But I still don't see those as 'right way and wrong way' to implement
greylisting.

I see them as 'unproductive' and 'less unproductive'.

Too little gain for the effort at my end, too much nuisance to others at
the far end.

Why so? You had stated:

> But if that same mail comes from a host which lacks reverse DNS, or has
> an SPF 'fail' result, *then* you might want to greylist the mail.
> Because now you *do* have a reason to consider it 'suspicious'.

I CANNOT consider those as 'suspicious'. Or anything else.

If a submission failed rDNS it was kicked off the TCP/IP teat in
acl_smtp_connect.

Gone. End of story. Nothing left to anal-ize but a brief log entry.

rDNS is a test that Exim does so well - permitting DNS records that are
convoluted, arguably imperfect, but 'close enough' - that it basically
just *does not* give rise to 'false-positives'.

A small - VERY small - LWL covers the few correspondents with badly
broken DNS that one 'just has to' accept anyway 'coz end-users demand
traffic form their kinder with badly configured servers.

Most of those would fail SPF, DK, DKIM, even SA ALSO, so the LWL
maintenance (peak was 16 lines of it here) is unavoidable.

And far less work than trafficking in multiple successive layers of
complex band-aids.

YMMV of course.

Some folks (Perkel-San) make their bones, even their living, on stats as
to how much spam they 'deal with' instead of how little they ever see at
all.

Not only have the criminal and lazy forced the complexity upon us, they
have caused us to condition ourselves to defend it as normal, necessary,
and in need of expansion.

'Blessed are they who run round in circles, for they shall be known as
mailadmins'.

Bill
--
韓家標

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

exim users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.