Mailing List Archive: exim: users

Exim 5.x

1 2

View All

Index | Next | Previous | View Threaded

wbh at conducive

May 19, 2011, 7:43 AM

Post #26 of 41 (699 views)
Permalink

Re: Exim 5.x [In reply to]

W B Hacker wrote:
> Phil Pennock wrote:
>> On 2011-05-19 at 12:35 +0100, Dominic Benson wrote:
>>> On 19/05/11 11:50, Phil Pennock wrote:
>>>>
>>>> New queuing system refers to an approach to scale up the spool
>>>> directory
>>>> to something more queue-like, with segregated admin-defined queues (eg,
>>>> "big_freemail_provider_x"). This is because while Exim is excellent at
>>>> inbound mail, it doesn't always scale as well as some would like for
>>>> outbound mail which can't be immediately delivered. Nothing has been
>>>> done on this. Patches welcome.
>>>>
>>> Is this AKA bug 336? It sounds quite interesting, so I think I might
>>> have a look at making some inroads into the problem.
>>>
>>> If there are are any notes/thoughts about behaviour/config/use it would
>>> be handy to hear them.
>>
>> I wasn't aware of 336, but yes it's related.
>>
>> At present, there's split_spool_directory, which divides things up with
>> one level of hashing, and then some people script their own queue-runner
>> launchers, running in parallel over sub-trees of the split spool instead
>> of having the Exim daemon launch runners over everything, which compete
>> with each other.
>>
>> Nothing more specific was discussed, that I either recall or find in the
>> minutes; we all understood the general problem.
>
> Known bound, yes.
>
> Problem?
>
> Not sure.
>
> I stand on the position that WHEN one is loading Exim so heavily that it
> HITS that sort of bound, one has far too many eggs in one basket
> downtime-risk-wise, and should split that load over multiple Exim
> (free)....
>
> ... on multiple separate boxen (generally cheap). Or at least usually
> 'cheapER' than answering complaints from such a huge user base - or
> rushing to a remote data center.
>
>>
>> If we use a new sub-directory of spool_directory to hold named queues,
>> then previous Exim installs won't know of the content, but it should be
>> fairly easy to script a rollback tool which recombines queues into the
>> original queue.

*snip*

..or doesn't need to do so at all...

Here's a use-what-we-have scenario for high load and costly (leased?)
single-box.

Simulate a cluster of virtual servers, but w/o the noise and nuisance of
jails or actual VDM's:

- create a directory tree, preferably on separate devices/mounts, ELSE
SSDD (Disk Drive, not Different Day) - optionally [sub] named per
listener-IP.

- install or clone exim1 thru exim(n), each with its executables and
~/configure renamed to suit, and its own listener IP, master script to
control sequence & time-spacing of startup.

These CAN (with same-group privs) all write to a common mailstore on
yet-another mount to make life easier for POP/IMAP. Or those can ALSO
remain separate - for similar load management reasons.

Each of these umm, 'partitions'..? .. will have its own queue - hints
db, logs (or not) .. everything else.

So long as the BOX and OS have the guts to pull it anyway, the Exim
queue issue - and all else - is compartmentalized to smaller chunks of
the load...

CAVEAT: Separate HDD spindles and head positioners, eg whole drives - de
riguer. ELSE, solid-state aside - r/w load creates the enviable, but
ultimately costly, sound of self-fornicating machinery of the sewing
machine phylum.

I'd even bet this has been done already...

;-)

Bill

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

dominic at lenny

May 19, 2011, 8:49 AM

Post #27 of 41 (702 views)
Permalink

Re: Exim 5.x [In reply to]

On 19/05/11 15:43, W B Hacker wrote:
> W B Hacker wrote:
>> Phil Pennock wrote:
>>> On 2011-05-19 at 12:35 +0100, Dominic Benson wrote:
>>>> On 19/05/11 11:50, Phil Pennock wrote:
>>>>>
>>>>> New queuing system refers to an approach to scale up the spool
>>>>> directory
>>>>> to something more queue-like, with segregated admin-defined queues
>>>>> (eg,
>>>>> "big_freemail_provider_x"). This is because while Exim is
>>>>> excellent at
>>>>> inbound mail, it doesn't always scale as well as some would like for
>>>>> outbound mail which can't be immediately delivered. Nothing has been
>>>>> done on this. Patches welcome.
>>>>>
>>>> Is this AKA bug 336? It sounds quite interesting, so I think I might
>>>> have a look at making some inroads into the problem.
>>>>
>>>> If there are are any notes/thoughts about behaviour/config/use it
>>>> would
>>>> be handy to hear them.
>>>
>>> I wasn't aware of 336, but yes it's related.
>>>
>>> At present, there's split_spool_directory, which divides things up with
>>> one level of hashing, and then some people script their own
>>> queue-runner
>>> launchers, running in parallel over sub-trees of the split spool
>>> instead
>>> of having the Exim daemon launch runners over everything, which compete
>>> with each other.
>>>
>>> Nothing more specific was discussed, that I either recall or find in
>>> the
>>> minutes; we all understood the general problem.
>>
>> Known bound, yes.
>>
>> Problem?
>>
>> Not sure.
>>
>> I stand on the position that WHEN one is loading Exim so heavily that it
>> HITS that sort of bound, one has far too many eggs in one basket
>> downtime-risk-wise, and should split that load over multiple Exim
>> (free)....
>>
>> ... on multiple separate boxen (generally cheap). Or at least usually
>> 'cheapER' than answering complaints from such a huge user base - or
>> rushing to a remote data center.
>>
>>>
>>> If we use a new sub-directory of spool_directory to hold named queues,
>>> then previous Exim installs won't know of the content, but it should be
>>> fairly easy to script a rollback tool which recombines queues into the
>>> original queue.
>
> *snip*
>
> ..or doesn't need to do so at all...
>
> Here's a use-what-we-have scenario for high load and costly (leased?)
> single-box.
>
> Simulate a cluster of virtual servers, but w/o the noise and nuisance
> of jails or actual VDM's:
>
> - create a directory tree, preferably on separate devices/mounts, ELSE
> SSDD (Disk Drive, not Different Day) - optionally [sub] named per
> listener-IP.
>
> - install or clone exim1 thru exim(n), each with its executables and
> ~/configure renamed to suit, and its own listener IP, master script to
> control sequence & time-spacing of startup.
>
> These CAN (with same-group privs) all write to a common mailstore on
> yet-another mount to make life easier for POP/IMAP. Or those can ALSO
> remain separate - for similar load management reasons.
>
> Each of these umm, 'partitions'..? .. will have its own queue - hints
> db, logs (or not) .. everything else.
>
> So long as the BOX and OS have the guts to pull it anyway, the Exim
> queue issue - and all else - is compartmentalized to smaller chunks of
> the load...
>
> CAVEAT: Separate HDD spindles and head positioners, eg whole drives -
> de riguer. ELSE, solid-state aside - r/w load creates the enviable,
> but ultimately costly, sound of self-fornicating machinery of the
> sewing machine phylum.
>
> I'd even bet this has been done already...
>
> ;-)
>
>
> Bill
>

I think the interesting use of multiple queues comes in when policy is
involved in deciding where to go: the simplest case where they are all
equal 'bins' is, as you say, readily served by having multiple
independent instances.

Obvious candidates would be a queue for messages that are relatively
time-sensitive (SMS-Email, host status alerts etc.), which should be
delivered ASAP even if there are 50,000 in the default queue. Or the
inverse, where there is a source of an occasional torrent of messages,
which should be delayed as necessary to avoid impacting the flow of
routine mail - say allowing queue_only in a per-spool context.

Another one would be a queue for relaying to a [local] server, which can
and is prepared to accept mail at a substantially higher rate than a
typical recipient, or where the same rate to remote domains would
negatively impact WAN performance. Or its inverse where there is a
particular recipient or set thereof that require a lower rate limit.

Another again might be to put messages with questionable spam status in
a queue with lower priority/frequency.

Dom

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

iane at sussex

May 19, 2011, 9:41 AM

Post #28 of 41 (715 views)
Permalink

Re: Exim 5.x [In reply to]

On 19 May 2011, at 13:35, W B Hacker wrote:

>>
>> part allows integration with an IMAP server, a message is submitted
>> with a an IMAP url to allow forward without download, etc.
>
> ??
>
> IF the content is already located at a URI, all that is needed is the URI. We all get such - mostly as advertising.

No, the point is not to put a URI in a message, but to put it in the SMTP conversation. The URI refers to some content to be found on an IMAP server. The SMTP server fetches that content, and appends it to the message. The point is to avoid having your mobile device upload a large attachment that's already available on the server.

> Looking just at the ages of RFC's from six to seventeen years old - it seems what was found useful enough to gain traction, did so ... and has been actioned.
>
> The rest?
>
> Seems the world has been in no hurry to drink that kool-aide, er scratch that

Well 8bitmime is implemented almost everywhere. The thing is, I can't turn it on without installing a non-Exim smart host to downgrade messages that I send to other Exim sites! I'd have to do this with all my outbound 8bit mail. I guess I could use $smtp_command in the MAIL ACL to work out which mail to send, and perhaps route that mail through our Exchange server. (sorry!)

Sites with significant numbers of mobile users ought to be considering whether their MSA servers support LEMONADE features that are supported by mobile clients.

The rest, we'd want to see how mobile providers are getting on. Here are some findings from Apple (not huge, but a serious player in the mobile market), MSN and AOL

Apple's ME.COM (Oracle Communications Messaging Exchange Server) provides this

250-8BITMIME *
250-PIPELINING
250-CHUNKING *
250-DSN *
250-ENHANCEDSTATUSCODES *
250-EXPN
250-HELP
250-XADR
250-XSTA
250-XCIR
250-XGEN
250-XLOOP EBAC943BF078DE96864E96CFA8E61582
250-STARTTLS
250-AUTH PLAIN LOGIN
250-AUTH=LOGIN PLAIN
250-ETRN
250-NO-SOLICITING
250 SIZE 0

Now, I don't know whether Apple's clients implement any of that extra LEMONADE goodness, but they certainly don't when they connect to Exim servers.

Microsoft's live.com service advertises:
250-TURN
250-SIZE 41943040
250-ETRN
250-PIPELINING
250-DSN *
250-ENHANCEDSTATUSCODES *
250-8bitmime *
250-BINARYMIME *
250-CHUNKING *
250-VRFY
250-TLS
250-STARTTLS
250 OK

AOL:
250-PIPELINING
250-SIZE 36700160
250-ETRN
250-STARTTLS
250-AUTH XAOL-UAS-MB PLAIN LOGIN
250-AUTH=XAOL-UAS-MB PLAIN LOGIN
250-ENHANCEDSTATUSCODES *
250-8BITMIME *
250 DSN *

Yahoo and GMail aren't so advanced. But, if there are clients out there that can benefit from these technologies, then Exim should support them.

Exchange 2010 SP1 supports:
250-SIZE 52428800
250-PIPELINING
250-DSN *
250-ENHANCEDSTATUSCODES *
250-STARTTLS
250-AUTH GSSAPI NTLM
250-8BITMIME *
250-BINARYMIME *
250 CHUNKING *

Bizarrely, Exchange 2010 supports lots of LEMONADE features, even though their IMAP implementation only supports a single extension.

--
Ian Eiloart
Postmaster, University of Sussex
+44 (0) 1273 87-3148

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

wbh at conducive

May 19, 2011, 10:27 AM

Post #29 of 41 (700 views)
Permalink

Re: Exim 5.x [In reply to]

Ian Eiloart wrote:
>
> On 19 May 2011, at 13:35, W B Hacker wrote:
>
>>>
>>> part allows integration with an IMAP server, a message is
>>> submitted with a an IMAP url to allow forward without download,
>>> etc.
>>
>> ??
>>
>> IF the content is already located at a URI, all that is needed is
>> the URI. We all get such - mostly as advertising.
>
> No, the point is not to put a URI in a message, but to put it in the
> SMTP conversation. The URI refers to some content to be found on an
> IMAP server. The SMTP server fetches that content, and appends it to
> the message. The point is to avoid having your mobile device upload a
> large attachment that's already available on the server.
>

But that protects only the *senders* handheld from overload.

The recipient - who may ALSO be device, time, b/w, or all of the above
constrained, gets the whole shebang. Like it or not.

Think also multiple recipients, which is why I said advertising.

MLM's and other broadcast/expansion critters already have toolsets to
'build' messages from templates.

But most often folks on the receiving end don't WANT more - they want
less, so the most beloved of MLM are tjose configured to *strip*
attachments, rather than add them, let aloe inline them.

Whereas with the 'bare' URI conveyance, BOTH parties benefit.

>> Looking just at the ages of RFC's from six to seventeen years old -
>> it seems what was found useful enough to gain traction, did so ...
>> and has been actioned.
>>
>> The rest?
>>
>> Seems the world has been in no hurry to drink that kool-aide, er
>> scratch that
>
> Well 8bitmime is implemented almost everywhere.

That one 'has traction', yes. And we should see what still needs doing...

Such as MTA being able to detect lack of ability at the far-end, and
reformat, or perhaps wrap - the whole of an often massive message.

Preferably on the fly and in-session rather than back-off, remember, and
retry - so as to convey it in a manner the other MTA CAN accept.

No mean feat in some instances, but the available horsepower is far more
likely today than yesteryear.

And it is being done SOMEHOW already, so perhaps is not so difficult.

> The thing is, I can't
> turn it on without installing a non-Exim smart host to downgrade
> messages that I send to other Exim sites! I'd have to do this with
> all my outbound 8bit mail. I guess I could use $smtp_command in the
> MAIL ACL to work out which mail to send, and perhaps route that mail
> through our Exchange server. (sorry!)
>
> Sites with significant numbers of mobile users ought to be
> considering whether their MSA servers support LEMONADE features that
> are supported by mobile clients.
>
> The rest, we'd want to see how mobile providers are getting on. Here
> are some findings from Apple (not huge, but a serious player in the
> mobile market), MSN and AOL
>
> Apple's ME.COM (Oracle Communications Messaging Exchange Server)
> provides this
>
> 250-8BITMIME * 250-PIPELINING 250-CHUNKING * 250-DSN *
> 250-ENHANCEDSTATUSCODES * 250-EXPN 250-HELP 250-XADR 250-XSTA
> 250-XCIR 250-XGEN 250-XLOOP EBAC943BF078DE96864E96CFA8E61582
> 250-STARTTLS 250-AUTH PLAIN LOGIN 250-AUTH=LOGIN PLAIN 250-ETRN
> 250-NO-SOLICITING 250 SIZE 0
>
> Now, I don't know whether Apple's clients implement any of that extra
> LEMONADE goodness, but they certainly don't when they connect to Exim
> servers.

Aye - the very point of advertising - and adapting to what is advertised...

>
> Microsoft's live.com service advertises: 250-TURN 250-SIZE 41943040
> 250-ETRN 250-PIPELINING 250-DSN * 250-ENHANCEDSTATUSCODES *
> 250-8bitmime * 250-BINARYMIME * 250-CHUNKING * 250-VRFY 250-TLS
> 250-STARTTLS 250 OK
>
> AOL: 250-PIPELINING 250-SIZE 36700160 250-ETRN 250-STARTTLS 250-AUTH
> XAOL-UAS-MB PLAIN LOGIN 250-AUTH=XAOL-UAS-MB PLAIN LOGIN
> 250-ENHANCEDSTATUSCODES * 250-8BITMIME * 250 DSN *
>
> Yahoo and GMail aren't so advanced. But, if there are clients out
> there that can benefit from these technologies, then Exim should
> support them.
>
> Exchange 2010 SP1 supports: 250-SIZE 52428800 250-PIPELINING 250-DSN
> * 250-ENHANCEDSTATUSCODES * 250-STARTTLS 250-AUTH GSSAPI NTLM
> 250-8BITMIME * 250-BINARYMIME * 250 CHUNKING *
>
> Bizarrely, Exchange 2010 supports lots of LEMONADE features, even
> though their IMAP implementation only supports a single extension.
>

Uh, 'Bizarrely, Exchange 2010....' Timeout. Language tidbit.

For any non-native English speakers still awake, this is called a
'redundancy'.

;-)

Bill

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

wbh at conducive

May 19, 2011, 11:05 AM

Post #30 of 41 (701 views)
Permalink

Re: Exim 5.x [In reply to]

Ian Eiloart wrote:
>
*snip*
>
> Well 8bitmime is implemented almost everywhere. The thing is, I can't
> turn it on without installing a non-Exim smart host to downgrade
> messages that I send to other Exim sites!

Are you SURE about Exim not being able to handle it on the inbound if we
were to switch to defaulting it ON?

See Phil Hazel's response back in 2003 in this old thread:

http://www.gossamer-threads.com/lists/exim/users/22207

If nothing has reverted, I read that as a U Cambridge server having
accepted 8BITMIME with a 250-Ok many years and versions ago.

In which case it is simply a matter of defaults and educating sysadmins
(oxymoron? impossibility?) ... but not coding.

OTOH, the Exim *3.x* docs say '....not a protocol converter'.

Reasonable. But given all the OTHER stuff we now do, one wonders if THAT
still stands..

I'm now at Exim 4.73, have NOT yet rolled-in my source-code mods, just
switched ON 'accept_8bitmime', BUT this server's normal twin is offline
whilst I convert from FreeBSD to OpenBSD so I don't presently have
another Exim to test against.

Can someone running Exim find a sample of some sort of 8BITMIME and hit
me with it so we can se if BOTH ends are au fait with that?

Thanks

Bill

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

wbh at conducive

May 19, 2011, 12:39 PM

Post #31 of 41 (701 views)
Permalink

Re: Exim 5.x [In reply to]

W B Hacker wrote:
> Ian Eiloart wrote:
>>
> *snip*
>>
>> Well 8bitmime is implemented almost everywhere. The thing is, I can't
>> turn it on without installing a non-Exim smart host to downgrade
>> messages that I send to other Exim sites!
>
> Are you SURE about Exim not being able to handle it on the inbound if we
> were to switch to defaulting it ON?

"Blessed are they who go 'round in circles,
for they shall be known as 'wheels'"

Never mind. Seems 8BITMIME is largely *irrelevant*.

http://cr.yp.to/smtp/8bitmime.html

Sometimes even a blind hog (Ich not DJB) finds an acorn (DJB's pragmatic
note).

I've been trafficing happily in bothway UTF8 AND Chinese plus various
other non-ASCII encoding from Day ONE (4.4X), ditto QMail for many years
before that..

Even tested Chinese UID:PWD (PostgreSQL-resident)

Now and then an issue reading some weird or mixed non-UTF encoding in an
MUA.

Otherwise seamless.

No - that isn't 8BITMIME. Just 8-bit transparent.

Rest is up to the IMAP/POP and MUA.

But seems good enough that there doesn't seem to be a remaining NEED for
8BITMIME, either.

Trying to learn and grown anyway .. just optioned-on:

allow_utf8_domains

Nap time .....

Bill

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

iane at sussex

May 20, 2011, 2:36 AM

Post #32 of 41 (698 views)
Permalink

Re: Exim 5.x [In reply to]

On 19 May 2011, at 19:05, W B Hacker wrote:

> Ian Eiloart wrote:
>>
> *snip*
>>
>> Well 8bitmime is implemented almost everywhere. The thing is, I can't
>> turn it on without installing a non-Exim smart host to downgrade
>> messages that I send to other Exim sites!
>
> Are you SURE about Exim not being able to handle it on the inbound if we were to switch to defaulting it ON?

If (a) all exim sites switch it on at the same time, and (b) all other sites are 8bitmime enabled, then we'll be OK. Big IFs.

> See Phil Hazel's response back in 2003 in this old thread:
>
> http://www.gossamer-threads.com/lists/exim/users/22207
>
> If nothing has reverted, I read that as a U Cambridge server having accepted 8BITMIME with a 250-Ok many years and versions ago.

It doesn't advertise 8bitmime now. I'm not sure what would happen if I ignored that and sent 8bitmime anyway. Maybe it would be OK, but there may be other sites that would simply bounce the messages.

> In which case it is simply a matter of defaults and educating sysadmins (oxymoron? impossibility?) ... but not coding.
>
> OTOH, the Exim *3.x* docs say '....not a protocol converter'.
>
> Reasonable. But given all the OTHER stuff we now do, one wonders if THAT still stands..
>
> I'm now at Exim 4.73, have NOT yet rolled-in my source-code mods, just switched ON 'accept_8bitmime', BUT this server's normal twin is offline whilst I convert from FreeBSD to OpenBSD so I don't presently have another Exim to test against.
>
> Can someone running Exim find a sample of some sort of 8BITMIME and hit me with it so we can se if BOTH ends are au fait with that?
>
> Thanks
>
> Bill
>
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/

--
Ian Eiloart
Postmaster, University of Sussex
+44 (0) 1273 87-3148

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

iane at sussex

May 20, 2011, 2:43 AM

Post #33 of 41 (705 views)
Permalink

Re: Exim 5.x [In reply to]

On 19 May 2011, at 19:05, W B Hacker wrote:

> Ian Eiloart wrote:
>>
> *snip*
>>
>> Well 8bitmime is implemented almost everywhere. The thing is, I can't
>> turn it on without installing a non-Exim smart host to downgrade
>> messages that I send to other Exim sites!
>
> Are you SURE about Exim not being able to handle it on the inbound if we were to switch to defaulting it ON?
>
> See Phil Hazel's response back in 2003 in this old thread:
>
> http://www.gossamer-threads.com/lists/exim/users/22207
>
> If nothing has reverted, I read that as a U Cambridge server having accepted 8BITMIME with a 250-Ok many years and versions ago.

Yes, but it was advertising 8bitmime, so you'd expect it not to complain:

250-xoanon.csi.cam.ac.uk Hello i [193.193.193.107]
250-SIZE 20971520
250-8BITMIME
250-EXPN
250-PIPELINING
250-STARTTLS
250 HELP
mail from:<> size=5000 body=8bitmime
250 OK

Without 8BITMIME advertised, this happens:

250-ppsw-50.csi.cam.ac.uk Hello chip.uscs.susx.ac.uk [139.184.14.86]
250-SIZE 104857600
250-PIPELINING
250 HELP
mail from:<> size=5000 body=8bitmime
501 <> size=5000 body=8bitmime: malformed address: size=5000 body=8bitmime may not follow <>

You get the same result if you provide a return path.
--
Ian Eiloart
Postmaster, University of Sussex
+44 (0) 1273 87-3148

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

iane at sussex

May 20, 2011, 2:49 AM

Post #34 of 41 (700 views)
Permalink

Re: Exim 5.x [In reply to]

On 19 May 2011, at 18:27, W B Hacker wrote:

> Ian Eiloart wrote:
>>
>> On 19 May 2011, at 13:35, W B Hacker wrote:
>>
>>>>
>>>> part allows integration with an IMAP server, a message is
>>>> submitted with a an IMAP url to allow forward without download,
>>>> etc.
>>>
>>> ??
>>>
>>> IF the content is already located at a URI, all that is needed is
>>> the URI. We all get such - mostly as advertising.
>>
>> No, the point is not to put a URI in a message, but to put it in the
>> SMTP conversation. The URI refers to some content to be found on an
>> IMAP server. The SMTP server fetches that content, and appends it to
>> the message. The point is to avoid having your mobile device upload a
>> large attachment that's already available on the server.
>>
>
> But that protects only the *senders* handheld from overload.
>
> The recipient - who may ALSO be device, time, b/w, or all of the above constrained, gets the whole shebang. Like it or not.

Yes, it does. But that's fine. At worst, it means the benefit is halved. But there are other factors that improve the situation:

1. The recipient will often have better bandwidth.
2. Mobile devices usually don't download attachments without asking first.
3. I'm not an expert in such matters, but I believe that mobile connections are assymetric: sending data uses more power than receiving. Therefore one has less bandwidth to send than to receive. And, more power means less battery life.

And, yes, there are other ways of reducing bandwidth. But none of them apply in the case that I've received an email that I need to forward to a third party - perhaps so that they can deal with it while I'm away from the office.
>
>> Well 8bitmime is implemented almost everywhere.
>
> That one 'has traction', yes. And we should see what still needs doing...
>
> Such as MTA being able to detect lack of ability at the far-end, and reformat, or perhaps wrap - the whole of an often massive message.

That is the plan, apparently.

> Preferably on the fly and in-session rather than back-off, remember, and retry - so as to convey it in a manner the other MTA CAN accept.
>
> No mean feat in some instances, but the available horsepower is far more likely today than yesteryear.

Yes, I don't think horsepower is a serious issue for anyone with fewer than a million subscribers. Given that most large sites are fine with 8bitmime, translation will only be required for a minority of messages.

--
Ian Eiloart
Postmaster, University of Sussex
+44 (0) 1273 87-3148

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

pdp at exim

May 20, 2011, 3:16 AM

Post #35 of 41 (700 views)
Permalink

Re: Exim 5.x [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

On 2011-05-20 at 09:36 +0000, Ian Eiloart wrote:
> On 19 May 2011, at 19:05, W B Hacker wrote:
> > If nothing has reverted, I read that as a U Cambridge server having
> > accepted 8BITMIME with a 250-Ok many years and versions ago.
>
> It doesn't advertise 8bitmime now. I'm not sure what would happen if I
> ignored that and sent 8bitmime anyway. Maybe it would be OK, but there
> may be other sites that would simply bounce the messages.

If you happened to just send email which contains characters from
outside the 7-bit ASCII range, then I would expect the server to accept
it and process it cleanly. Exim is 8-bit clean and always has been (as
far as I know; perhaps Exim 1 or 2 wasn't?).

8BITMIME is distinct from "having content making full use of the
available bits". 8BITMIME is a protocol extension which establishes a
contract for handling of email when forwarding it on.

If an Exim MTA does not have accept_8bitmime turned on, then it will not
implement the protocol extensions (adverbs on the MAIL command) and
8BITMIME attempts will be rejected as protocol errors -- as they should
be. This recently was encountered by Ted Cooper, posting on exim-users:
http://www.exim.org/lurker/thread/20110313.035235.07b8c3b8.en.html#i20110313.035235.07b8c3b8
wherein a spambot was ignoring the non-presence of 8BITMIME in the
announced capabilities (EHLO keywords) and was using BODY=7BIT on the
MAIL line.

If an Exim MTA *does* have accept_8bitmime turned on, then:
* it will advertise the keyword in the EHLO response
* if EHLO was used, then it will accept the extra adverbs on MAIL
* it will do nothing else special with the message

Note that this means that turning on accept_8bitmime:
* for a system handling only in-bound mail is fine, and probably a
_very_ good idea (unless your POP3/IMAP servers are not 8-bit clean)
* for a system which can forward mail outwards again will result in
your install violating the RFC 1652 protocol's contract

Now, how bad it is to break the contract of RFC 1652 is debatable; email
protocol debaters being what they are, any debate would become boringly
contentious with vitriol, character assassination and flame-wars on both
sides. See, I just performed an act of character assassination!

If you wish to implement DJB's recommended behaviour per
http://cr.yp.to/smtp/8bitmime.html then just turn on "accept_8bitmime"
in Exim and you'll get exactly that.

I suspect that what we'll end up doing is implementing protocol
conversion, defaulting accept_8bitmime to true and having an option
"8bitmime_ignore" which defaults to "off" for strict RFC adherence.
Then about 7 years later, when everyone's using raw UTF-8 mail domains
and LHSs (ignoring punycode and the other baroque pieces of
infrastructure dreamed up by people who hate simplicity), we'll default
"8bitmime_ignore" to "on" before removing the protocol conversion
functionality and heaving a huge sigh of relief.

That, or at some point in the next year I'll drink enough whiskey to
decide to tell some people where they can stick their idiot 7-bit
systems and just commit a change defaulting accept_8bitmime to on, then
after the next release of Exim take a month-long holiday from the
Internet to avoid the flame-wars until they settle down and everyone
realises that in fact, DJB is right, Phil Hazel was right to refuse to
implement protocol conversion and people are making a big fuss about a
non-issue.

- -Phil
-----BEGIN PGP SIGNATURE-----

iEYEAREDAAYFAk3WP4YACgkQQDBDFTkDY38rcACgkfNqNaavuXo6LgKdWIRwb8R8
n4MAn18SFTc9PJ16TznN3qT88ra8SMgd
=Qbdj
-----END PGP SIGNATURE-----

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

pdp at exim

May 20, 2011, 3:26 AM

Post #36 of 41 (697 views)
Permalink

Re: Exim 5.x [In reply to]

On 2011-05-19 at 14:10 +0000, W B Hacker wrote:
> Phil Pennock wrote:
> > At present, there's split_spool_directory, which divides things up with
> > one level of hashing, and then some people script their own queue-runner
> > launchers, running in parallel over sub-trees of the split spool instead
> > of having the Exim daemon launch runners over everything, which compete
> > with each other.
> >
> > Nothing more specific was discussed, that I either recall or find in the
> > minutes; we all understood the general problem.
>
> Known bound, yes.
>
> Problem?
>
> Not sure.
>
> I stand on the position that WHEN one is loading Exim so heavily that it
> HITS that sort of bound, one has far too many eggs in one basket
> downtime-risk-wise, and should split that load over multiple Exim (free)....

Ah yes, thanks for the reminder:

The queue system should not assume that the spool_directory is one
file-system; a reasonable design-choice would be to have a "deferred"
queue for mails which haven't gone out in a timely manner, and have the
deferred queue be on shared storage (DRBD, etc), avoiding the shared
storage and its overhead for short-lived messages.

FWIW: I now work for a company which sends out "lots" of emails each
day (unit of measurement is "million"), mails to account-holders
notifying them of particular updates to their *cough* timeline. I'm
*not* rushing to put Exim into handling that outbound email as there's a
lot of Unicode characters, we don't support 8BITMIME conversion, we're
not fantastic at handling backlogs of millions of messages and generally
I want to get a lot more stats and perhaps implement some new features
before I even try putting Exim in-place.

-Phil, https://twitter.com/syscomet

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

wbh at conducive

May 20, 2011, 3:48 AM

Post #37 of 41 (712 views)
Permalink

Re: Exim 5.x [In reply to]

Ian Eiloart wrote:
>
> On 19 May 2011, at 19:05, W B Hacker wrote:
>
>> Ian Eiloart wrote:
>>>
>> *snip*

*snip* again

>>
>> If nothing has reverted, I read that as a U Cambridge server having
>> accepted 8BITMIME with a 250-Ok many years and versions ago.
>
> Yes, but it was advertising 8bitmime, so you'd expect it not to
> complain:
>
> 250-xoanon.csi.cam.ac.uk Hello i [193.193.193.107] 250-SIZE 20971520
> 250-8BITMIME 250-EXPN 250-PIPELINING 250-STARTTLS 250 HELP mail
> from:<> size=5000 body=8bitmime 250 OK
>

See below...

> Without 8BITMIME advertised, this happens:
>
> 250-ppsw-50.csi.cam.ac.uk Hello chip.uscs.susx.ac.uk [139.184.14.86]
> 250-SIZE 104857600 250-PIPELINING 250 HELP mail from:<> size=5000
> body=8bitmime 501<> size=5000 body=8bitmime: malformed address:
> size=5000 body=8bitmime may not follow<>
>
> You get the same result if you provide a return path.

First off, just sent you directly a test message so we can set up to do
some off-list testing - hopefully come back with better info, less chatter.

Because ... Optioned ON or OFF, I am having a hard time getting my arms
around seeing 8BITMIME as a noteworthy problem.

A) RFC permissives aside, 8BITMIME doesn't seem to be all that common in
actual use. Most 'trouble' reports are stale, and I see that as improved
MUA handling, not MTA changes.

Exim may be odd-man out with 8BITMIME optioned OFF - but it has NOT so
far become a global pariah for that behaviour.

B) I can't find much in the way of reports as to what real-world app
even *creates* 8BITMIME. Or how often.

I can't find a sample to test with, either.

C) Most current MUA even if 'fed' 8 bit-anything seem to be predisposed
to ass u me the worst, and wrap it as quoted-printable.
In the MUA, not the MTA.

D) Taken as STIPULATED that QP conversion may or may not suit the
recipient's MUA environment, it should still pass transparently THROUGH
the bowels of both MTA, if so optioned. Spam filtering against know
incidents of abuse remain separate issues.

Or perhaps not.

Optioning 8BITMIME OFF by default may have already saved many Exim
systems from extra WinCrobe/phish grief. And with no harm.

Hold THAT thought...

Otherwise, when PH said that Exim was 'not a protocol converter' I take
that as guidance as well as observation.

IMNSHO, Exim should remain transparent AND NOT *become* a protocol
converter.

So long as that is the case, MTA functionality is less vulnerable AND
arm's length disinterested third-party aloof positioning enhances the
reputation of said MTA for NOT munging stuff.

Good stuff, both.

Any conversion/encoding problems are placed back onto the shoulders of
the end-point correspondents - just the same as a .doc file created in
MS office is not expected to be readable on a MAJMAP-1100 system.
That was, after all, one of the aims of MIME, was it not?

One has always needed more than just an undamaged file.
One needs the tools to utilize it.

Except for headers, those are not really IN the MTA. It is mere
plumbing. Intelligent plumbing, yes. But nonetheless a transport, not a
consumer.

Other than tools to convert between and among various text encodings,
and perhaps html, extra tools aren't ordinarily in the MUA, either. One
detaches an attachment and onpasses to outboard editors or display tools
to USE it.

So - still trying to find the beef' on this issue.

Where else should I look?

Bill

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

wbh at conducive

May 20, 2011, 4:18 AM

Post #38 of 41 (700 views)
Permalink

Re: Exim 5.x [In reply to]

Phil Pennock wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: RIPEMD160
>
*snip*

Phil, we are on the same song-sheet on this one!

Sitting (mostly) in Asia, 8-bit, especially UTF-8, is dear to us, and
has not been a problem AT ALL with defaults. Nor with Qmail before Exim.
nor Courier-MTA. So it has been over a dozen years and 8BITMIME has not
bitten OURazz.

But after turning ON 8BITMIME, experimenting with DJB's site and a few
others, I came to the conclusion that punycode et al aren't just
'...baroque..'.

They just have to be the byproduct of a collective misuse of banned
substances, as the 'cure' is worse than the disease!

FAR worse..

> That, or at some point in the next year I'll drink enough whiskey to
> decide to tell some people where they can stick their idiot 7-bit
> systems

Fact is, hardware AND software-wise, most have already made the trip.
And long since. Any remaining damn well SHOULD be pushed off their perch.

> and just commit a change defaulting accept_8bitmime to on, then
> after the next release of Exim take a month-long holiday from the
> Internet to avoid the flame-wars until they settle down and everyone
> realises that in fact, DJB is right, Phil Hazel was right to refuse to
> implement protocol conversion and people are making a big fuss about a
> non-issue.
>

Why wait?

Give me a physical ship-to off-list, and I'll furnish the first bottle
or two to hasten the process.

Hope you don't mind single-malts. I am woefully ignorant about blends,
and wouldn't want to *harm* anyone...

;-)

Best,

Bill

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

iane at sussex

May 20, 2011, 7:38 AM

Post #39 of 41 (704 views)
Permalink

Re: Exim 5.x [In reply to]

>>
>
> First off, just sent you directly a test message so we can set up to do some off-list testing - hopefully come back with better info, less chatter.
>
> Because ... Optioned ON or OFF, I am having a hard time getting my arms around seeing 8BITMIME as a noteworthy problem.
>
> A) RFC permissives aside, 8BITMIME doesn't seem to be all that common in actual use. Most 'trouble' reports are stale, and I see that as improved MUA handling, not MTA changes.
>
> Exim may be odd-man out with 8BITMIME optioned OFF - but it has NOT so far become a global pariah for that behaviour.

Well, that'll be because mobile email is still a minority pursuit, users don't realise that a good proportion of their wait when sending email is actually unnecessary, and it degrades gracefully (the email still gets sent). That doesn't mean that Exim admins shouldn't be looking to improve the user experience.

It's not just a performance issue, it's a storage issue, too.

> B) I can't find much in the way of reports as to what real-world app even *creates* 8BITMIME. Or how often.
>
> I can't find a sample to test with, either.

The way to find out would be to enable 8bitmime in Exim, and log $smtp_command in the MAIL command.

>
> C) Most current MUA even if 'fed' 8 bit-anything seem to be predisposed to ass u me the worst, and wrap it as quoted-printable.
> In the MUA, not the MTA.

They'll certainly do that when an MTA fails to advertise 8bitmime.

> D) Taken as STIPULATED that QP conversion may or may not suit the recipient's MUA environment, it should still pass transparently THROUGH the bowels of both MTA, if so optioned. Spam filtering against know incidents of abuse remain separate issues.

As for the rest of this, you seem to be arguing that YOU see some benefit in not advertising 8bitmime. So don't. But please don't hold the rest of us back with virtually the only popular smtp software the doesn't support it.

> Or perhaps not.
>
> Optioning 8BITMIME OFF by default may have already saved many Exim systems from extra WinCrobe/phish grief. And with no harm.
>
> Hold THAT thought...
>
> Otherwise, when PH said that Exim was 'not a protocol converter' I take that as guidance as well as observation.
>
> IMNSHO, Exim should remain transparent AND NOT *become* a protocol converter.
>
> So long as that is the case, MTA functionality is less vulnerable AND arm's length disinterested third-party aloof positioning enhances the reputation of said MTA for NOT munging stuff.
>
> Good stuff, both.
>
> Any conversion/encoding problems are placed back onto the shoulders of the end-point correspondents - just the same as a .doc file created in MS office is not expected to be readable on a MAJMAP-1100 system.
> That was, after all, one of the aims of MIME, was it not?
>
> One has always needed more than just an undamaged file.
> One needs the tools to utilize it.
>
> Except for headers, those are not really IN the MTA. It is mere plumbing. Intelligent plumbing, yes. But nonetheless a transport, not a consumer.
>
> Other than tools to convert between and among various text encodings, and perhaps html, extra tools aren't ordinarily in the MUA, either. One detaches an attachment and onpasses to outboard editors or display tools to USE it.
>
> So - still trying to find the beef' on this issue.
>
> Where else should I look?
>
> Bill
>
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/

--
Ian Eiloart
Postmaster, University of Sussex
+44 (0) 1273 87-3148

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

wbh at conducive

May 20, 2011, 9:05 AM

Post #40 of 41 (699 views)
Permalink

Re: Exim 5.x [In reply to]

Ian Eiloart wrote:
>>>
>>
*snip*

>>
>> Exim may be odd-man out with 8BITMIME optioned OFF - but it has NOT
>> so far become a global pariah for that behaviour.
>
> Well, that'll be because mobile email is still a minority pursuit,

Meaning mobiles are the main source of 8BITMIME?

For Wot? Photos taken with handhelds?

Not in my ken. Not over smpt, anyway.

But that genre has otherwise been a significant part of our traffic
since well before Blackberry, i-whatever, chrome-whatever.

I went through three HP-200LX that we used with PCMCIA modems before
there was anything lighter or smaller.

> users don't realise that a good proportion of their wait when sending
> email is actually unnecessary, and it degrades gracefully (the email
> still gets sent). That doesn't mean that Exim admins shouldn't be
> looking to improve the user experience.
>

I don't think that part has SQRT Fine Ambitions to do with an MTA.

> It's not just a performance issue, it's a storage issue, too.
>

Basically smtp is simply not the proper tool for that whole class of job.

And smtp - based to a degree on extensions to telegraphy and teletype
networking protocols - is in fact being less and less USED for that sort
of work.

In some parts of the world, SMS overtook smtp in message traffic several
years ago.

And SMS is IN TURN being supplanted. Or paralleled.

IRC, Tweets, file and photo sharing ... yadda, yadda..

Horses for courses.

Turning a McCelland saddle into a motorcycle seat might be *possible*,
but as one who has used both - trust me - also a pain in the anatomy.

As the need changes, so also the technical solutions.

So long as smtp serves, it lives. If not, not.

I gave up my ASR33 and international cable address WBHACKER ages ago.

If/as and WHEN smtp is obsoleted, I'll change again.

>> B) I can't find much in the way of reports as to what real-world
>> app even *creates* 8BITMIME. Or how often.
>>
>> I can't find a sample to test with, either.
>
> The way to find out would be to enable 8bitmime in Exim, and log
> $smtp_command in the MAIL command.
>

Did that.

tailed -f current log 'til my mew acrylic eyes were tired,

grep'ed older logs

grep -R on 10GB and six years worth of what hasn't been deleted from
IMAP mailstore.

This thread and similar aside, such as a few hits from cut and paste of
folks on the list reporting server advertising in general - no traffic,
no rejections, and no complaints.

>>
>> C) Most current MUA even if 'fed' 8 bit-anything seem to be
>> predisposed to ass u me the worst, and wrap it as
>> quoted-printable. In the MUA, not the MTA.
>
> They'll certainly do that when an MTA fails to advertise 8bitmime.
>

Sounds like a cheap and cheerful solution to me...

But the MUA hasn't a clue wot its OWN host is going to offer at
composition time, far less view of the far-end or intervening relays.

So MUA coders must have the problem - or avoidance of problem - already
in-view, factored in, and defaulted 'safe'.

Will turning on *BITMIME on every Exim MTA on the planet change that?

Not that it is a bad idea, but doubt it would change a thing.

>> D) Taken as STIPULATED that QP conversion may or may not suit the
>> recipient's MUA environment, it should still pass transparently
>> THROUGH the bowels of both MTA, if so optioned. Spam filtering
>> against know incidents of abuse remain separate issues.
>
> As for the rest of this, you seem to be arguing that YOU see some
> benefit in not advertising 8bitmime.

Not that at all. Just can't see why it MATTERS. No traffic.

> So don't. But please don't hold
> the rest of us back with virtually the only popular smtp software the
> doesn't support it.
>
>

I'll leave it on. No problem there.

And 'To Be Determined', but I suspect that Exim supports it exactly as
I'd *want* it supported: 'KISS'

Accept. Transfer.

Feel free to send a sample. Even a challenging one. Or 'many' such.

Otherwise, I doubt I'll even see much 8BITMIME.

Regards,

Bill

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

pdp at exim

May 7, 2012, 4:20 AM

Post #41 of 41 (232 views)
Permalink

Re: Exim 5.x [In reply to]

On 2011-05-20 at 11:18 +0000, W B Hacker wrote:
> Phil Pennock wrote:
> > and just commit a change defaulting accept_8bitmime to on, then
> > after the next release of Exim take a month-long holiday from the
> > Internet to avoid the flame-wars until they settle down and everyone
> > realises that in fact, DJB is right, Phil Hazel was right to refuse to
> > implement protocol conversion and people are making a big fuss about a
> > non-issue.
> >
>
> Why wait?
>
> Give me a physical ship-to off-list, and I'll furnish the first bottle
> or two to hasten the process.
>
> Hope you don't mind single-malts. I am woefully ignorant about blends,
> and wouldn't want to *harm* anyone...

Bill, thank for for the whiskey a few months back. Please accept my
apologies for forgetting about this email.

You'll be delighted to know that I've committed the change to default
accept_8bitmime to true. You may, or may not, be less happy to know
that I did this independently of your offer; I found this while checking
for references to past discussion of the issue, after pushing the change
to the master git repo.

That small bottle is calling out to me. I think I shall heed its siren
call now. It's a crime that there's still some left.

-Phil

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads