Mailing List Archive: exim: users

Message duplication if To/Cc contain alias and one of addresses it's expanded into

Index | Next | Previous | View Threaded

temmokan at gmail

May 6, 2008, 12:25 AM

Post #1 of 9 (277 views)
Permalink

Message duplication if To/Cc contain alias and one of addresses it's expanded into

Hello,

After defeating message duplication yesterday, I encountered the
similar problem. Namely:

The record in /etc/aliases:

userlist: userlist,user1

The message sent:

To: user1[at]example.com
Cc: userlist[at]example.com

Result: two copies received by user1, one by userlist. The records in main.log:

2008-05-06 14:05:38 1JtHF4-00017m-2L <= user1[at]example.com
H=diamond.local [10.1.0.54] P=esmtpsa X=TLSv1:AES256-SHA:256
A=PLAIN:user1 S=649 id=48200340.6020408[at]example.com
2008-05-06 14:05:38 1JtHF4-00017t-8g <= user1[at]example.com U=exim
P=local S=1921 id=48200340.6020408[at]example.com
2008-05-06 14:05:38 1JtHF4-00017m-2L => user1 <user1[at]example.com>
R=dspam_router T=dspam_spamcheck
2008-05-06 14:05:38 1JtHF4-00017t-8g => user1 <user1[at]itelsib.com>
R=procmail T=procmail
2008-05-06 14:05:38 1JtHF4-00017t-8g Completed
2008-05-06 14:05:38 1JtHF4-000181-Gt <= user1[at]example.com U=exim
P=local S=1904 id=48200340.6020408[at]example.com
2008-05-06 14:05:38 1JtHF4-00017m-2L => userlist
<userlist[at]example.com> R=dspam_router T=dspam_spamcheck
2008-05-06 14:05:38 1JtHF4-00017m-2L Completed
2008-05-06 14:05:38 1JtHF4-000181-Gt => userlist
<userlist[at]example.com> R=procmail T=procmail
2008-05-06 14:05:38 1JtHF4-000181-Gt => user1 <userlist[at]example.com>
R=procmail T=procmail
2008-05-06 14:05:38 1JtHF4-000181-Gt Completed

Routers/transports records from exim.conf:

====================== exim.conf routers.transport start
######################################################################
# ROUTERS CONFIGURATION #
# Specifies how addresses are handled #
######################################################################
# THE ORDER IN WHICH THE ROUTERS ARE DEFINED IS IMPORTANT! #
# An address is passed to each router in turn until it is accepted. #
######################################################################

begin routers

dnslookup:
driver = dnslookup
domains = ! +local_domains
transport = remote_smtp
ignore_target_hosts = 0.0.0.0 : 127.0.0.0/8
no_more

#
# DSPAM-related routers start
#

dspam_router:
no_verify
check_local_user
# When to scan a message :
# - it isn't already flagged as spam from Spamassassin
# - it isn't already flagged as spam from DSPAM
# - it isn't already scanned
# - it isn't local
# - it isn't from one internal domain user to another
# - it is less than 2048k in size
condition = "${if and { \
{!def:h_X-Spam-Flag:} \
{!def:h_X-FILTER-DSPAM:} \
{!eq {$received_protocol}{local}} \
{ <= {$message_size}{2048k}} \
}\
{1}{0}}"
headers_add = "X-FILTER-DSPAM: by $primary_hostname on $tod_full"
driver = accept
transport = dspam_spamcheck
# Which users to run dspam for.
#local_parts = /usr/local/etc/exim/dspam-testers

dspam_addspam_router:
driver = accept
local_part_prefix = spam-
transport = dspam_addspam

dspam_falsepositive_router:
driver = accept
local_part_prefix = nospam-
transport = dspam_falsepositive

#
# DSPAM-related routers end
#

system_aliases:
driver = redirect
allow_fail
allow_defer
data = ${lookup{$local_part}lsearch{/etc/aliases}}
# user = exim
check_ancestor = true
# repeat_use = false
file_transport = address_file
pipe_transport = address_pipe

userforward:
driver = redirect
check_local_user
# local_part_suffix = +* : -*
# local_part_suffix_optional
file = $home/.forward
allow_filter
no_verify
no_expn
check_ancestor = true
file_transport = address_file
pipe_transport = address_pipe
reply_transport = address_reply

procmail:
driver = accept
check_local_user
require_files = ${local_part}:+${home}/.procmailrc:/usr/bin/procmail
transport = procmail
no_verify

localuser:
driver = accept
check_local_user
# local_part_suffix = +* : -*
# local_part_suffix_optional
transport = local_delivery
cannot_route_message = Unknown user

######################################################################
# TRANSPORTS CONFIGURATION #
######################################################################
# ORDER DOES NOT MATTER #
# Only one appropriate transport is called for each delivery. #
######################################################################

begin transports

remote_smtp:
driver = smtp
transport_filter = /etc/mail/headers-rewrite

# This transport invokes procmail to deliver mail
procmail:
driver = pipe
command = "/usr/bin/procmail -d $local_part"
return_path_add
delivery_date_add
envelope_to_add
user = $local_part
initgroups
return_output

local_delivery:
driver = appendfile
file = /var/mail/$local_part
delivery_date_add
envelope_to_add
return_path_add
group = mail
mode = 0660

address_pipe:
driver = pipe
return_output

address_file:
driver = appendfile
group = mail
delivery_date_add
envelope_to_add
return_path_add

address_reply:
driver = autoreply

#
# DSPAM transports start
#

dspam_spamcheck:
driver = pipe
command = "/usr/local/dspam-3.8.0/bin/dspam --mode=teft
--feature=noise,whitelist --debug --deliver=innocent,spam --user
${lc:$local_part} -f \
'$sender_address' -- %u"
home_directory = "/tmp"
current_directory = "/tmp"
user = exim
group = mail
log_output = true
return_fail_output = true
return_path_add = false
message_prefix =
message_suffix =

dspam_addspam:
driver = pipe
command = "/usr/local/dspam-3.8.0/bin/dspam --debug --class=spam
--source=error --user \
${lc:$local_part} -f '$sender_address' -- %u"
home_directory = "/tmp"
current_directory = "/tmp"
user = exim
group = mail
log_output = true
return_fail_output = true
return_path_add = false
message_prefix =
message_suffix =

dspam_falsepositive:
driver = pipe
command = "/usr/local/dspam-3.8.0/bin/dspam --debug
--class=innocent --source=error \
--deliver=innocent,spam --user ${lc:$local_part}\
-f '$sender_address' -- %u"
home_directory = "/tmp"
current_directory = "/tmp"
user = exim
group = mail
log_output = true
return_fail_output = true
return_path_add = false
message_prefix =
message_suffix =
====================== exim.conf routers.transport end

Is the mentioned situation expected?

Thanks.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

hs at schlittermann

May 8, 2008, 4:39 AM

Post #2 of 9 (249 views)
Permalink

Re: Message duplication if To/Cc contain alias and one of addresses it's expanded into [In reply to]

Hello,

Konstantin Boyandin <temmokan[at]gmail.com> (Di 06 Mai 2008 09:25:15 CEST):
> Hello,
>
> After defeating message duplication yesterday, I encountered the
> similar problem. Namely:
>
> The record in /etc/aliases:
>
> userlist: userlist,user1
>
> The message sent:
>
> To: user1[at]example.com
> Cc: userlist[at]example.com

exim should not care about mail headers ad all. (except in some ACL
conditions). Important is, what exim sees in "RCPT TO" lines.

Best regards from Dresden
Viele Grüße aus Dresden
Heiko Schlittermann
--
SCHLITTERMANN.de ---------------------------- internet & unix support -
Heiko Schlittermann HS12-RIPE -----------------------------------------
gnupg encrypted messages are welcome - key ID: 48D0359B ---------------
gnupg fingerprint: 3061 CFBF 2D88 F034 E8D2 7E92 EE4E AC98 48D0 359B -

Attachments:

signature.asc (0.18 KB)

temmokan at gmail

May 8, 2008, 4:11 PM

Post #3 of 9 (244 views)
Permalink

Re: Message duplication if To/Cc contain alias and one of addresses it's expanded into [In reply to]

Hello,

2008/5/8 Heiko Schlittermann <hs[at]schlittermann.de>:
>> After defeating message duplication yesterday, I encountered the
>> similar problem. Namely:
>>
>> The record in /etc/aliases:
>>
>> userlist: userlist,user1
>>
>> The message sent:
>>
>> To: user1[at]example.com
>> Cc: userlist[at]example.com
>
> exim should not care about mail headers ad all. (except in some ACL
> conditions). Important is, what exim sees in "RCPT TO" lines.

The result, from all my experience of harnessing DSPAM, is that as
soon as pipe transport is used to preprocess the message, Exim's logic
of eliminating duplicates cease to function. I had to use the
preprocessing at the stage of actual delivery, inserting
transport_filter
to all the transports used, to keep that logic working.

With all respect,

Konstantin

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

hs at schlittermann

May 9, 2008, 12:35 AM

Post #4 of 9 (242 views)
Permalink

Re: Message duplication if To/Cc contain alias and one of addresses it's expanded into [In reply to]

Konstantin Boyandin <temmokan[at]gmail.com> (Fr 09 Mai 2008 01:11:36 CEST):
>
> The result, from all my experience of harnessing DSPAM, is that as
> soon as pipe transport is used to preprocess the message, Exim's logic
> of eliminating duplicates cease to function. I had to use the
> preprocessing at the stage of actual delivery, inserting
> transport_filter

If the pipe injects the message again into exim it cannot know about
duplicates already seen, because it's an entirely new message.

You should decide at which stage you want to "spread" your message,
before or after DSPAM. (Before: you can't apply "user policies", After:
you're probably doing duplicated work.)

--
Heiko

Attachments:

signature.asc (0.18 KB)

temmokan at gmail

May 9, 2008, 1:16 AM

Post #5 of 9 (238 views)
Permalink

Re: Message duplication if To/Cc contain alias and one of addresses it's expanded into [In reply to]

>> The result, from all my experience of harnessing DSPAM, is that as
>> soon as pipe transport is used to preprocess the message, Exim's logic
>> of eliminating duplicates cease to function. I had to use the
>> preprocessing at the stage of actual delivery, inserting
>> transport_filter
>
> If the pipe injects the message again into exim it cannot know about
> duplicates already seen, because it's an entirely new message.
>
> You should decide at which stage you want to "spread" your message,
> before or after DSPAM. (Before: you can't apply "user policies", After:
> you're probably doing duplicated work.)

I'm sorry, what do you mean by 'spread'? If the DSPAM is invoked via a
router, then the described situation will result in multiple copies of
the same messages - regardless of whether DSPAM router is located
before or after system_aliases one.

Thanks.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

maillist at ovb

May 11, 2008, 1:31 AM

Post #6 of 9 (206 views)
Permalink

Re: Message duplication if To/Cc contain alias and one of addresses it's expanded into [In reply to]

Konstantin Boyandin wrote:
>>> The result, from all my experience of harnessing DSPAM, is that as
>>> soon as pipe transport is used to preprocess the message, Exim's logic
>>> of eliminating duplicates cease to function. I had to use the
>>> preprocessing at the stage of actual delivery, inserting
>>> transport_filter
>>>
>> If the pipe injects the message again into exim it cannot know about
>> duplicates already seen, because it's an entirely new message.
>>
>> You should decide at which stage you want to "spread" your message,
>> before or after DSPAM. (Before: you can't apply "user policies", After:
>> you're probably doing duplicated work.)
>>
>
> I'm sorry, what do you mean by 'spread'? If the DSPAM is invoked via a
> router, then the described situation will result in multiple copies of
> the same messages - regardless of whether DSPAM router is located
> before or after system_aliases one.
>
>
What he said is, that exim can not control any duplication of message in
the setup you have at the moment because the message gets feed to exim
as separate messages. From your log file, you feed exim three (3)
messages. To illustrate this, I've changed the order of the log lines into
the order exim sees as a session each. (stripped time stamps and SMTP
info for better readability)

1. That part is the delivery of the message from the net with authenticated esmtp and a delivery to two addresses, user1 and userlist, both delivered through the "method" dspam_spamcheck.
For this session I assume the the following SMTP envelope sender (aka RCPT TO):
user1[at]example.com and userlist[at]example.com
You see, no duplication, it's delivered to each address once. The alias expansion and duplicate control detect that user1 is already in and only one delivery to it is done.

1JtHF4-00017m-2L <= user1[at]example.com P=esmtpsa S=649 id=48200340.6020408[at]example.com
1JtHF4-00017m-2L => user1 <user1[at]example.com> R=dspam_router T=dspam_spamcheck
1JtHF4-00017m-2L => userlist <userlist[at]example.com> R=dspam_router T=dspam_spamcheck
1JtHF4-00017m-2L Completed

2. An entirely independent SMTP session to exim with the message from user1 to user1.
The RCPT TO here will be user1[at]example.com

1JtHF4-00017t-8g <= user1[at]example.com U=exim P=local S=1921 id=48200340.6020408[at]example.com
1JtHF4-00017t-8g => user1 <user1[at]itelsib.com> R=procmail T=procmail
1JtHF4-00017t-8g Completed

3. Yet another completely independent SMTP session to exim with a message with two end deliveries.
The RCPT TO here will be userlist[at]example.com, which is expanded according to the alias with an additional recipient of user1[at]example.com

1JtHF4-000181-Gt <= user1[at]example.com U=exim P=local S=1904 id=48200340.6020408[at]example.com
1JtHF4-000181-Gt => userlist <userlist[at]example.com> R=procmail T=procmail
1JtHF4-000181-Gt => user1 <userlist[at]example.com> R=procmail T=procmail
1JtHF4-000181-Gt Completed

What you expect, which can't be done, is that exim takes care of
duplicate elimination across session 2 and 3.

What you have to do is find a way that your dspam-setup does not create
an smtp session for each recipient to feed the messages back into exim,
meaning session 2 and 3 should really be one with RCPT TO of
user1[at]example.com and userlist[at]example.com. In that case, exim would
detect the condition and deliver the message once to user1 and once to
userlist. Or, you have to make sure that the alias expansion does not
take place for session 2 and 3.

Oliver

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

temmokan at gmail

May 11, 2008, 3:15 AM

Post #7 of 9 (206 views)
Permalink

Re: Message duplication if To/Cc contain alias and one of addresses it's expanded into [In reply to]

2008/5/11 Oliver von Bueren <maillist[at]ovb.ch>:
>>>> The result, from all my experience of harnessing DSPAM, is that as
>>>> soon as pipe transport is used to preprocess the message, Exim's logic
>>>> of eliminating duplicates cease to function. I had to use the
>>>> preprocessing at the stage of actual delivery, inserting
>>>> transport_filter
>>>>
>>> If the pipe injects the message again into exim it cannot know about
>>> duplicates already seen, because it's an entirely new message.
>>>
>>> You should decide at which stage you want to "spread" your message,
>>> before or after DSPAM. (Before: you can't apply "user policies", After:
>>> you're probably doing duplicated work.)
>>>
>>
>> I'm sorry, what do you mean by 'spread'? If the DSPAM is invoked via a
>> router, then the described situation will result in multiple copies of
>> the same messages - regardless of whether DSPAM router is located
>> before or after system_aliases one.
>>
>>
> What he said is, that exim can not control any duplication of message in
> the setup you have at the moment because the message gets feed to exim
> as separate messages. From your log file, you feed exim three (3)
> messages. To illustrate this, I've changed the order of the log lines into
> the order exim sees as a session each. (stripped time stamps and SMTP
> info for better readability)
>
> 1. That part is the delivery of the message from the net with authenticated esmtp and a delivery to two addresses, user1 and userlist, both delivered through the "method" dspam_spamcheck.
> For this session I assume the the following SMTP envelope sender (aka RCPT TO):
> user1[at]example.com and userlist[at]example.com
> You see, no duplication, it's delivered to each address once. The alias expansion and duplicate control detect that user1 is already in and only one delivery to it is done.
>
> 1JtHF4-00017m-2L <= user1[at]example.com P=esmtpsa S=649 id=48200340.6020408[at]example.com
> 1JtHF4-00017m-2L => user1 <user1[at]example.com> R=dspam_router T=dspam_spamcheck
> 1JtHF4-00017m-2L => userlist <userlist[at]example.com> R=dspam_router T=dspam_spamcheck
> 1JtHF4-00017m-2L Completed
>
> 2. An entirely independent SMTP session to exim with the message from user1 to user1.
> The RCPT TO here will be user1[at]example.com
>
> 1JtHF4-00017t-8g <= user1[at]example.com U=exim P=local S=1921 id=48200340.6020408[at]example.com
> 1JtHF4-00017t-8g => user1 <user1[at]itelsib.com> R=procmail T=procmail
> 1JtHF4-00017t-8g Completed
>
> 3. Yet another completely independent SMTP session to exim with a message with two end deliveries.
> The RCPT TO here will be userlist[at]example.com, which is expanded according to the alias with an additional recipient of user1[at]example.com
>
> 1JtHF4-000181-Gt <= user1[at]example.com U=exim P=local S=1904 id=48200340.6020408[at]example.com
> 1JtHF4-000181-Gt => userlist <userlist[at]example.com> R=procmail T=procmail
> 1JtHF4-000181-Gt => user1 <userlist[at]example.com> R=procmail T=procmail
> 1JtHF4-000181-Gt Completed
>
> What you expect, which can't be done, is that exim takes care of
> duplicate elimination across session 2 and 3.
>
> What you have to do is find a way that your dspam-setup does not create
> an smtp session for each recipient to feed the messages back into exim,
> meaning session 2 and 3 should really be one with RCPT TO of
> user1[at]example.com and userlist[at]example.com. In that case, exim would
> detect the condition and deliver the message once to user1 and once to
> userlist. Or, you have to make sure that the alias expansion does not
> take place for session 2 and 3.

That brings me back to the original question: is it really DSPAM at
fault in this case? It is called through pipe transport. All DSPAM
does is to read data from stdin and send modified output to stdout.
Any piece of software that behaves in the asme manner will beget
duplicates of exactly the same kind.

If the pipe transport wouldn't open a new Exim session for every
message, the problem could go away, couldn't it?

I had to incorporate DSPAM as transport filter for the transports
performing actual delivery.

Could someone tell me whether it is at all possible to set up 'message
preprocessing' via pipe transports without causing the message
duplicates problem?

Thanks.

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

maillist at ovb

May 12, 2008, 2:43 AM

Post #8 of 9 (193 views)
Permalink

Re: Message duplication if To/Cc contain alias and one of addresses it's expanded into [In reply to]

Hi Konstantin
>> What you have to do is find a way that your dspam-setup does not create
>> an smtp session for each recipient to feed the messages back into exim,
>> meaning session 2 and 3 should really be one with RCPT TO of
>> user1[at]example.com and userlist[at]example.com. In that case, exim would
>> detect the condition and deliver the message once to user1 and once to
>> userlist. Or, you have to make sure that the alias expansion does not
>> take place for session 2 and 3.
>>
>
> That brings me back to the original question: is it really DSPAM at
> fault in this case? It is called through pipe transport. All DSPAM
> does is to read data from stdin and send modified output to stdout.
> Any piece of software that behaves in the asme manner will beget
> duplicates of exactly the same kind.
>
>
The problem in your setup is the usage of a transport for that purpose.
A transport, like SMTP, is meant to get a message and send it OUT of
your system and that is it. It's not designed to have a transport
receiving a message for sending out which in fact gets feed right back
into exim again.

There exist content scanning facilities for this kind of things in exim,
which don't suffer the same problem because the message is never "sent
out" of exim and re-submitted to it.

As it was said before, the only way to avoid this would probably be to
have all alias processing and the like done before the messages go out
the pipe to dspam and having no processing of the destination address,
notably no alias expansion, after it is feed back to exim. This means, a
message feed back from dspam should go out through the final delivery
transport without any other processing immediately.

I guess this can be done in exim, but I didn't have the need for this up
till now, as I use content scanning for antispam implementation and I
don't use dspam at the moment.

So primarily, I don't think it's a exim fault, but rather the method in
which dspam is integrated into your system.
Perhaps others have an example on how to do it properly, I don't.

Oliver

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

temmokan at gmail

May 12, 2008, 7:47 AM

Post #9 of 9 (191 views)
Permalink

Re: Message duplication if To/Cc contain alias and one of addresses it's expanded into [In reply to]

Hi Oliver,

2008/5/12 Oliver von Bueren <maillist[at]ovb.ch>:
> >> What you have to do is find a way that your dspam-setup does not create
> >> an smtp session for each recipient to feed the messages back into exim,
> >> meaning session 2 and 3 should really be one with RCPT TO of
> >> user1[at]example.com and userlist[at]example.com. In that case, exim would
> >> detect the condition and deliver the message once to user1 and once to
> >> userlist. Or, you have to make sure that the alias expansion does not
> >> take place for session 2 and 3.
> >>
> >
> > That brings me back to the original question: is it really DSPAM at
> > fault in this case? It is called through pipe transport. All DSPAM
> > does is to read data from stdin and send modified output to stdout.
> > Any piece of software that behaves in the asme manner will beget
> > duplicates of exactly the same kind.
> >
> >
> The problem in your setup is the usage of a transport for that purpose.
> A transport, like SMTP, is meant to get a message and send it OUT of
> your system and that is it. It's not designed to have a transport
> receiving a message for sending out which in fact gets feed right back
> into exim again.
>
> There exist content scanning facilities for this kind of things in exim,
> which don't suffer the same problem because the message is never "sent
> out" of exim and re-submitted to it.
>
> As it was said before, the only way to avoid this would probably be to
> have all alias processing and the like done before the messages go out
> the pipe to dspam and having no processing of the destination address,
> notably no alias expansion, after it is feed back to exim. This means, a
> message feed back from dspam should go out through the final delivery
> transport without any other processing immediately.
>
> I guess this can be done in exim, but I didn't have the need for this up
> till now, as I use content scanning for antispam implementation and I
> don't use dspam at the moment.
>
> So primarily, I don't think it's a exim fault, but rather the method in
> which dspam is integrated into your system.
> Perhaps others have an example on how to do it properly, I don't.

Well, the initial DSPAM under Exim setup I tried was taken out of
DSPAM Wiki. Finally, it turned out that 'transport filter' is exactly
what serves the purpose. It guarantees the filtering/preprocessing is
done exactly before the actual delivery takes place, after all the
alias expansion and duplicates elimination have occurred.

If there are better ways, better following the Exim philosophy, well,
I'll be glad to know of them. For now, the problem is solved and I
suppose I suggest it to DSPAM community as well, since the initial one
fails in cases like mine.

Thanks for all your comments.

With all respect,

Konstantin

--
## List details at http://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com