Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: NANOG: users

REJECT-ON-SMTP-DATA (Re: Mail Server best practices - was: Pandora's Box of new TLDs)

 

 

NANOG users RSS feed   Index | Next | Previous | View Threaded


jeroen at unfix

Jul 1, 2008, 2:54 AM

Post #1 of 8 (327 views)
Permalink
REJECT-ON-SMTP-DATA (Re: Mail Server best practices - was: Pandora's Box of new TLDs)

Chris Owen <owenc[at]hubris.net> wrote

> It is because, if someone reports (by telephone, IRC or IRL) that he
> sent an email and I did not receive it, I regard as VERY IMPORTANT to
> be able to check the spam folder (with a search tool, not by hand) and
> go back to him saying "No, we really did not receive it".

The magic keyword: REJECT-ON-SMTP-DATA.

Aka during the "DATA" phase of the email, also directly scan it, then
when the spam/virus tool thinks it is spam/virus, you just reject it.

This solves a couple of things in one go:

- You don't have to handle bounces if you would send a bounce back
to the sender "hey you had a spam/virus" because you already accepted
the mail, thus less overhead at least there

- The sending SMTP server will send a bounce back to the sender.
The sender thus gets a SMTP rejection mail, with the rejection
reason and knows that you didn't accept the message and why.
They can then opt to change the content of the mail or contact you
differently.

- No more 'spam' folder, as the stuff that is spam is already rejected.
You might get a few mails through that are actually spam, but this is
mostly marginal.

This thus solves completely that email "gets lost". Also note that if a
virus/bot or something else silly is trying to send these mails it
either has to handle the bounces, which they generally (afaik ;) don't,
thus the faked sender doesn't get a swamp of failed deliveries either.
This is a Win-Win situation in my ears.

Unfortunately there is also a side-effect, partially, one has to have
all inbound servers use this trick, and it might be that they need to be
a bit heavier to process and scan all that mail. Then again, you can
better let sending SMTP servers queue a bit (or throttle them) then
having to process them anyway later.

Of course not all mail platforms can be fitted in this way, but people
who have such systems probably already have other ways to handle things.

For the excellent Spamassassin, read:
http://wiki.apache.org/spamassassin/IntegratedInMta

The 'milter' options (originally sendmail, but nowadays also available
for other mailers) can do this for you. Other anti-spam tools might also
be able to do similar things. YMMV.

Oh and of course as a access-ISP, one should also employ this trick to
the customers, that way they know that they are sending spam ;)

Greets,
Jeroen
Attachments: signature.asc (0.18 KB)


owenc at hubris

Jul 1, 2008, 7:28 AM

Post #2 of 8 (299 views)
Permalink
Re: REJECT-ON-SMTP-DATA (Re: Mail Server best practices - was: Pandora's Box of new TLDs) [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jul 1, 2008, at 4:54 AM, Jeroen Massar wrote:

> Chris Owen <owenc[at]hubris.net> wrote

I did not write this FYI.

>> It is because, if someone reports (by telephone, IRC or IRL) that he
>> sent an email and I did not receive it, I regard as VERY IMPORTANT to
>> be able to check the spam folder (with a search tool, not by hand)
>> and
>> go back to him saying "No, we really did not receive it".
>
> The magic keyword: REJECT-ON-SMTP-DATA.
>
> Aka during the "DATA" phase of the email, also directly scan it,
> then when the spam/virus tool thinks it is spam/virus, you just
> reject it.
>
> This solves a couple of things in one go:
>
> - No more 'spam' folder, as the stuff that is spam is already
> rejected.
> You might get a few mails through that are actually spam, but this
> is
> mostly marginal.

The lack of a spam folder is one of the problems with such a
solution. Having a middle ground quarantine is actually quite nice.

However, the biggest problem is these solutions are global in nature.
We let individual customers considerable control over the process.
They can each set their own block and quarantine levels, configure
their own white and blacklists and even turn the spam controls
completely off. For various reasons none of that would be possible
with this solution and all the implementations you link to all run
with a single global configuration.

Chris

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Chris Owen ~ Garden City (620) 275-1900 ~ Lottery (noun):
President ~ Wichita (316) 858-3000 ~ A stupidity tax
Hubris Communications Inc www.hubris.net
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Public Key: http://home.hubris.net/owenc/pgpkey.txt
Comment: Public Key ID: 0xB513D9DD

iEYEARECAAYFAkhqPvIACgkQElUlCLUT2d2nTQCfVq/dXvpBSVZnbgMyblgwhSp2
hD8AoIBxoz9UupxznPpZ9cC4FJ6fMc1y
=Ze+j
-----END PGP SIGNATURE-----


justin at justinshore

Jul 1, 2008, 2:17 PM

Post #3 of 8 (293 views)
Permalink
Re: REJECT-ON-SMTP-DATA (Re: Mail Server best practices - was: Pandora's Box of new TLDs) [In reply to]

Chris Owen wrote:
> The lack of a spam folder is one of the problems with such a solution.
> Having a middle ground quarantine is actually quite nice.
>
> However, the biggest problem is these solutions are global in nature.
> We let individual customers considerable control over the process. They
> can each set their own block and quarantine levels, configure their own
> white and blacklists and even turn the spam controls completely off.
> For various reasons none of that would be possible with this solution
> and all the implementations you link to all run with a single global
> configuration.

Chris,

I can think of one spam filter that does give both you and your users
individual control over all of these settings while still rejecting mail
during the SMTP dialog including the DATA phase: CanIt-Pro.

http://www.roaringpenguin.com/

CanIt-Pro is a mail filter or 'milter' in Sendmail-speak. It
essentially connects into Sendmail from the side. Sendmail calls on it
during the SMTP dialog with the remote MTA, giving CanIt-Pro the
opportunity to work its magic before the message is accepted for
delivery which allows from rejecting mail right up until the last second
RFC 2821 permits it. I use CanIt-Pro for this very reason. Each user
can have their own individual mail "stream" in CanIt terminology. Each
user can define white/blacklists by senders, domains and hosts. Users
can block or permit by MIME types or perform actions based on attachment
suffixes. They can write their own rules with regexs against the
headers or body as well as checking to see if a sending domain matches
that of the relaying MTA (not always accurate but often is; ebay.com is
a good example). Users can enable or disable individually configured
DNSBLs or change the score. They can even define rules based on SPF
values. Each user gets their own bayesian DB as well.

You as an admin can disable any of the above features on a per-user
basis so you can make it as simple or as complex as you want. You can
also pre-define streams with specific settings that users can subscribe
to if they don't want the more fine-grained control. I created a stream
that only tags suspect spam. I also created 3 streams with varying
levels of aggressiveness.

Have you ever heard the phrase "a pilot's plane"? Well I would liken
CanIt to being the equivalent for mail admins and their spam filters. I
first started using the OSS predecessor to CanIt back in late 2000 or so
called MIMEDefang. MD is still the underpinnings of CanIt. When you
buy CanIt you also get the source code so you have the ability to code
in custom things if you have the need and desire. It's perfect for SPs.

BTW, I'm not a Roaring Penguin employee. I'm just an impressed user of
their products so they've earned my loyalty.

Justin


vandry at TZoNE

Jul 4, 2008, 2:01 PM

Post #4 of 8 (274 views)
Permalink
Re: REJECT-ON-SMTP-DATA (Re: Mail Server best practices - was: Pandora's Box of new TLDs) [In reply to]

On Tue, Jul 01, 2008 at 11:54:46AM +0200, Jeroen Massar wrote:
> The magic keyword: REJECT-ON-SMTP-DATA.
[snip description on how to reject during DATA phase]
>
> Unfortunately there is also a side-effect, partially, one has to have
> all inbound servers use this trick, and it might be that they need to be
> a bit heavier to process and scan all that mail. Then again, you can

More than that: you also need to have all users in the domain (indeed
all users who share an MX server) agree on the accept/reject policy.
If users are free to use different spam filtering techniques and tune
them to their liking (e.g. someone uses SpamAssassin with a low threshold,
someone else uses it with a high threshold, someone else uses bogofilter
instead) then what do you do with mails that are addresses to more than
one user? You can have some users reject the message during the RCPT
phase and others accept it, but if you've waited until the DATA phase,
it's too late for that.

-Phil


justin at justinshore

Jul 4, 2008, 11:05 PM

Post #5 of 8 (267 views)
Permalink
Re: REJECT-ON-SMTP-DATA (Re: Mail Server best practices - was: Pandora's Box of new TLDs) [In reply to]

Phil Vandry wrote:
> On Tue, Jul 01, 2008 at 11:54:46AM +0200, Jeroen Massar wrote:
>> The magic keyword: REJECT-ON-SMTP-DATA.
> [snip description on how to reject during DATA phase]
>> Unfortunately there is also a side-effect, partially, one has to have
>> all inbound servers use this trick, and it might be that they need to be
>> a bit heavier to process and scan all that mail. Then again, you can
>
> More than that: you also need to have all users in the domain (indeed
> all users who share an MX server) agree on the accept/reject policy.
> If users are free to use different spam filtering techniques and tune
> them to their liking (e.g. someone uses SpamAssassin with a low threshold,
> someone else uses it with a high threshold, someone else uses bogofilter
> instead) then what do you do with mails that are addresses to more than
> one user? You can have some users reject the message during the RCPT
> phase and others accept it, but if you've waited until the DATA phase,
> it's too late for that.

Phil,

This is a non-problem if you use the right spam filter. I mentioned
CanIt earlier in the thread. It individually applies filtering rules to
incoming mail and can apply different rules and take actions on a
per-user basis. It handles messages with multiple recipients by feeding
copies of the message into an individual user's stream where that user's
settings dictate what actions are taken. A user may have an aggressive
spam score or an extremely conservative score, message rejection with
SpamHaus and SORBS or no DNSBLs at all, tons of custom rules and lots of
bells and whistles or spam filtering disabled completely. They've
already anticipated all the possible problems that have been brought up
in this thread. Arrange for a demo and give it a try. I don't think
you'd be disappointed.

http://mailman.nanog.org/pipermail/nanog/2008-July/001884.html

Justin


Skywing at valhallalegends

Jul 4, 2008, 11:25 PM

Post #6 of 8 (266 views)
Permalink
RE: REJECT-ON-SMTP-DATA (Re: Mail Server best practices - was: Pandora's Box of new TLDs) [In reply to]

I think the problem that was being raised here was that past the DATA phase, if one recipient is going to receive the message and another is going to reject it, you have lost the ability to communicate this back to the sender (at least without an NDR). Thus the problem of mails disappearing into spam folder black holes is back in the multirecipient case when one is dealing with DATA and recipients have differing spam policies.

- S

-----Original Message-----
From: Justin Shore [mailto:justin[at]justinshore.com]
Sent: Saturday, July 05, 2008 2:05 AM
To: Phil Vandry
Cc: nanog[at]merit.edu
Subject: Re: REJECT-ON-SMTP-DATA (Re: Mail Server best practices - was: Pandora's Box of new TLDs)

Phil,

This is a non-problem if you use the right spam filter. I mentioned
CanIt earlier in the thread. It individually applies filtering rules to
incoming mail and can apply different rules and take actions on a
per-user basis. It handles messages with multiple recipients by feeding
copies of the message into an individual user's stream where that user's
settings dictate what actions are taken. A user may have an aggressive
spam score or an extremely conservative score, message rejection with
SpamHaus and SORBS or no DNSBLs at all, tons of custom rules and lots of
bells and whistles or spam filtering disabled completely. They've
already anticipated all the possible problems that have been brought up
in this thread. Arrange for a demo and give it a try. I don't think
you'd be disappointed.

http://mailman.nanog.org/pipermail/nanog/2008-July/001884.html

Justin


jfmezei at vaxination

Jul 5, 2008, 1:52 AM

Post #7 of 8 (264 views)
Permalink
Re: REJECT-ON-SMTP-DATA (Re: Mail Server best practices - was: Pandora's Box of new TLDs) [In reply to]

one note about whether to filter at receiving SMTP server or later.

The receiving SMTP server is the one that has the conversation with the
sender.

Rejecting mail from servers having an un-backtranslatable IP is best
done right away by the receiving server right after the HELO command by
issuing error message about the IP being unbacktranslatable. Reduces the
load.

later on (for instance at the client level), you need to parse the
RFC822 text header and there are some bits that are missing, notably the
RCPT TO: commands. This is especially true when the "TO" in the 822
header is faked.

Blocking messages as early as possible also greatly reduces the load on
your system, disk storage requirements etc.


justin at justinshore

Jul 5, 2008, 8:50 AM

Post #8 of 8 (256 views)
Permalink
Re: REJECT-ON-SMTP-DATA (Re: Mail Server best practices - was: Pandora's Box of new TLDs) [In reply to]

Jean-François Mezei wrote:
> Blocking messages as early as possible also greatly reduces the load on
> your system, disk storage requirements etc.

Rejecting during the SMTP dialog but before you signal that you've
accepted the DATA output also also pushes the responsibility for sending
a DSN to the sending MTA. It's is a spammer then they'll drop the DSN.
If it's a compromised PC running Storm Worm or the like it won't
generate DSNs anyway. If it's a legit but poorly-configured MTA acting
as an open relay it will generate the DSN and eventually get itself
blacklisted. Sending a DSN to a spoofed envelope From is considered
spam in and of itself and will get an MTA blacklisted. You could always
not send DSNs in which case the sender of a legit message that had a few
to many !!!s in it will not get a bounce and will not know that there
message was blocked. It disappears into an email blackhole. Few things
piss off users like disappearing email.

It's best all around to force the sending MTA to send the bounce. Your
MTA doesn't get blacklisted, spammers' relays are forced to do a little
extra work, and senders of legit mail that's a false-positive get a DSN
telling them that their message didn't go through (and hopefully why).
Everyone wins. Block early and block often.

Justin

NANOG users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact lists@gossamer-threads.com
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.