Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: exim: users

Spam control via ratelimiting.

 

 

exim users RSS feed   Index | Next | Previous | View Threaded


sd at msu

Dec 16, 2011, 10:48 AM

Post #1 of 8 (621 views)
Permalink
Spam control via ratelimiting.

We have been toying with the concept of ratelimiting for some time. We
use it create log entries and in some cases we use the control=freeze
option to stop outgoing spam. I would use the defer option if email
clients reacted sanely to the event, but since they don't we are really
hesitant to use it. Apple Mail is the worst - it opens up a dialogue
window to the configuration pane where the user can re enter their
password (which of course many of them no longer know) or pick different
port numbers. It's all a recipe for a run on our friendly Help Desk people.

Enough complaining, here is what I and my Director would like to be able
to do.

-- Identify likely spammers via ratelimiting
-- Freeze these messages using control=freeze
-- Allow them to be delivered very slowly. And this is where it gets
ugly - I could just let these messages sit in the queue ( I don't have
autothaw enabled so now thats what they do.) I could collect message ids
and deliver them via a script one every 90 seconds or so, but that
doesn't solve the problem of one messages to 100 recipients. As soon as
I release one message hotmail or yahoo gets hit with a hundred incoming
chunks of spam and then we are on the blocklist. Is there anyway
currently to control the release of a message (addressed to multiple
recipients ) one recipient at a time?
I expect the short answer is no - probably violates a couple of RFC's,
but I wanted to ask before I moved on to other ideas. I've been
trolling the docs and haven't seen anything yet.
Thanks
/sd

--
Steve Devine
Systems& Infrastructure
Academic Technology Services
Michigan State University




--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


Lena at lena

Dec 18, 2011, 5:11 AM

Post #2 of 8 (609 views)
Permalink
Re: Spam control via ratelimiting. [In reply to]

> From: Steve Devine
>
> We have been toying with the concept of ratelimiting for some time. We
> use it create log entries and in some cases we use the control=freeze
> option to stop outgoing spam.

> here is what I and my Director would like to be able
> to do.
>
> -- Identify likely spammers via ratelimiting
> -- Freeze these messages using control=freeze
> -- Allow them to be delivered very slowly. And this is where it gets
> ugly

Why allow them to be delivered? If your software detected a likely spammer,
the only way to distinguish a honest user from a spammer/spambot
is manual inspection of content of few frozen messages (using `exipick`).
If it's spam, you don't want it delivered even very slowly;
you want to shut the user down. What's the alternative?
A spambot will continue to fill your queue with frozen messages
faster than you send them out slowly; slow spam will cause you
blacklisted after a while even if you don't care about outgoing spam
otherwise.

I like the idea from this list to detect spammers/spambots not by rate
of sending of all mail, but by rate of attempts to send to nonexistent
recipients. Spammers and spambots send to huge lists of email addresses.
Large part of email addresses in such lists don't exist anymore or
never existed (Message-Ids and corrupted strings in memory taken by
address harvesters as email addresses).

My implementation:

LIM = 100
PERIOD = 1h
WARNTO = abuse [at] example
EXIMBINARY = /usr/local/sbin/exim -f root
SHELL = /bin/sh
...
begin acl
acl_check_rcpt:
...
accept hosts = !@[] : +relay_from_hosts
set acl_m_user = $sender_host_address
# or an userid from RADIUS
condition = ${if exists{$spool_directory/blocked_relay_users}}
condition = ${lookup{$acl_m_user}lsearch\
{$spool_directory/blocked_relay_users}{1}{0}}
control = freeze/no_tell
control = submission/domain=
add_header = X-Relayed-From: $acl_m_user

accept hosts = !@[] : +relay_from_hosts
!verify = recipient/defer_ok/callout=10s,defer_ok,use_sender
ratelimit = LIM / PERIOD / per_rcpt / relayuser-$acl_m_user
continue = ${run{SHELL -c "echo $acl_m_user \
>>$spool_directory/blocked_relay_users; \
\N{\N echo Subject: relay user $acl_m_user blocked; echo; echo \
because has sent mail to LIM invalid recipients during PERIOD.; \
\N}\N | EXIMBINARY WARNTO"}}
control = freeze/no_tell
control = submission/domain=
add_header = X-Relayed-From: $acl_m_user

accept hosts = +relay_from_hosts
control = submission/domain=

accept authenticated = *
set acl_m_user = $authenticated_id
# in case of mailboxes in /var/mail: ${sg{$authenticated_id}{\N\W.*$\N}{}}
condition = ${if exists{$spool_directory/blocked_authenticated_users}}
condition = ${lookup{$acl_m_user}lsearch\
{$spool_directory/blocked_authenticated_users}{1}{0}}
control = freeze/no_tell
control = submission/domain=
add_header = X-Authenticated-As: $acl_m_user

accept authenticated = *
!verify = recipient/defer_ok/callout=10s,defer_ok,use_sender
ratelimit = LIM / PERIOD / per_rcpt / user-$acl_m_user
continue = ${run{SHELL -c "echo $acl_m_user \
>>$spool_directory/blocked_authenticated_users; \
\N{\N echo Subject: user $acl_m_user blocked; echo; echo because \
has sent mail to LIM invalid recipients during PERIOD.; \
\N}\N | EXIMBINARY WARNTO"}}
control = freeze/no_tell
control = submission/domain=
add_header = X-Authenticated-As: $acl_m_user

accept authenticated = *
control = submission/domain=

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


sd at msu

Dec 18, 2011, 5:51 AM

Post #3 of 8 (607 views)
Permalink
Re: Spam control via ratelimiting. [In reply to]

On 12/18/11 8:11 AM, Lena [at] lena wrote:
>> From: Steve Devine
>>
>> We have been toying with the concept of ratelimiting for some time. We
>> use it create log entries and in some cases we use the control=freeze
>> option to stop outgoing spam.
>
>> here is what I and my Director would like to be able
>> to do.
>>
>> -- Identify likely spammers via ratelimiting
>> -- Freeze these messages using control=freeze
>> -- Allow them to be delivered very slowly. And this is where it gets
>> ugly
>
> Why allow them to be delivered? If your software detected a likely spammer,
> the only way to distinguish a honest user from a spammer/spambot
> is manual inspection of content of few frozen messages (using `exipick`).
> If it's spam, you don't want it delivered even very slowly;
> you want to shut the user down. What's the alternative?
> A spambot will continue to fill your queue with frozen messages
> faster than you send them out slowly; slow spam will cause you
> blacklisted after a while even if you don't care about outgoing spam
> otherwise.
>
Agreed. I wasn't clear .. we freeze suspected spammers and when enough
evidence is collected we reset the password - trash the spam etc. One of
the issues we have are - we are not 24 X 7 so overnight and weekends we
are exposed. We can freeze anything that looks suspicious but then some
legit email gets delayed. Politically that's troublesome.
The idea of slow delivery is mostly to try and discourage faculty or
staff who believe they have the need to seen these kinds of emails but
don't want to use a listserv. As I said there are University politics at
work here.

Your idea is intriguing, does this work even when the recipients are on
non-local domains? Currently we freeze based on IP's and spammy
signatures.
/sd

> I like the idea from this list to detect spammers/spambots not by rate
> of sending of all mail, but by rate of attempts to send to nonexistent
> recipients. Spammers and spambots send to huge lists of email addresses.
> Large part of email addresses in such lists don't exist anymore or
> never existed (Message-Ids and corrupted strings in memory taken by
> address harvesters as email addresses).
>
> My implementation:
>
> LIM = 100
> PERIOD = 1h
> WARNTO = abuse [at] example
> EXIMBINARY = /usr/local/sbin/exim -f root
> SHELL = /bin/sh
> ...
> begin acl
> acl_check_rcpt:
> ...
> accept hosts = !@[] : +relay_from_hosts
> set acl_m_user = $sender_host_address
> # or an userid from RADIUS
> condition = ${if exists{$spool_directory/blocked_relay_users}}
> condition = ${lookup{$acl_m_user}lsearch\
> {$spool_directory/blocked_relay_users}{1}{0}}
> control = freeze/no_tell
> control = submission/domain=
> add_header = X-Relayed-From: $acl_m_user
>
> accept hosts = !@[] : +relay_from_hosts
> !verify = recipient/defer_ok/callout=10s,defer_ok,use_sender
> ratelimit = LIM / PERIOD / per_rcpt / relayuser-$acl_m_user
> continue = ${run{SHELL -c "echo $acl_m_user \
> >>$spool_directory/blocked_relay_users; \
> \N{\N echo Subject: relay user $acl_m_user blocked; echo; echo \
> because has sent mail to LIM invalid recipients during PERIOD.; \
> \N}\N | EXIMBINARY WARNTO"}}
> control = freeze/no_tell
> control = submission/domain=
> add_header = X-Relayed-From: $acl_m_user
>
> accept hosts = +relay_from_hosts
> control = submission/domain=
>
> accept authenticated = *
> set acl_m_user = $authenticated_id
> # in case of mailboxes in /var/mail: ${sg{$authenticated_id}{\N\W.*$\N}{}}
> condition = ${if exists{$spool_directory/blocked_authenticated_users}}
> condition = ${lookup{$acl_m_user}lsearch\
> {$spool_directory/blocked_authenticated_users}{1}{0}}
> control = freeze/no_tell
> control = submission/domain=
> add_header = X-Authenticated-As: $acl_m_user
>
> accept authenticated = *
> !verify = recipient/defer_ok/callout=10s,defer_ok,use_sender
> ratelimit = LIM / PERIOD / per_rcpt / user-$acl_m_user
> continue = ${run{SHELL -c "echo $acl_m_user \
> >>$spool_directory/blocked_authenticated_users; \
> \N{\N echo Subject: user $acl_m_user blocked; echo; echo because \
> has sent mail to LIM invalid recipients during PERIOD.; \
> \N}\N | EXIMBINARY WARNTO"}}
> control = freeze/no_tell
> control = submission/domain=
> add_header = X-Authenticated-As: $acl_m_user
>
> accept authenticated = *
> control = submission/domain=
>


--
Steve Devine
Systems & Infrastructure
Academic Technology Services
Michigan State University

313 Computer Center
East Lansing, MI 48824-1042
1-517-432-7327

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


Lena at lena

Dec 19, 2011, 6:20 AM

Post #4 of 8 (599 views)
Permalink
Re: Spam control via ratelimiting. [In reply to]

> From: Steve Devine

> We can freeze anything that looks suspicious but then some
> legit email gets delayed.

I think that delaying of legitimate messages is unlikely
if the criteria is 100 nonexistent recipients during 1 hour.

> does this work even when the recipients are on
> non-local domains?

Yes, because of the "callout" in "verify=recipient".

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


sd at msu

Dec 19, 2011, 6:42 AM

Post #5 of 8 (601 views)
Permalink
Re: Spam control via ratelimiting. [In reply to]

On 12/19/11 09:20, Lena [at] lena wrote:
>> From: Steve Devine
>
>> We can freeze anything that looks suspicious but then some
>> legit email gets delayed.
>
> I think that delaying of legitimate messages is unlikely
> if the criteria is 100 nonexistent recipients during 1 hour.
>

That's a good point.

>> does this work even when the recipients are on
>> non-local domains?
>
> Yes, because of the "callout" in "verify=recipient".
>
I'm going to experiment with this - thanks.


--
Steve Devine
Systems & Infrastructure
Academic Technology Services
Michigan State University


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


ben at beadynet

Dec 19, 2011, 9:26 AM

Post #6 of 8 (603 views)
Permalink
Re: Spam control via ratelimiting. [In reply to]

On Friday, December 16, 2011 06:48:24 pm Steve Devine wrote:
> Enough complaining, here is what I and my Director would like to be able
> to do.
>
> -- Identify likely spammers via ratelimiting
> -- Freeze these messages using control=freeze
> -- Allow them to be delivered very slowly. And this is where it gets
> ugly - I could just let these messages sit in the queue ( I don't have

The configuration we're using does this pretty well. In essence we've got 2
levels of ratelimiting (keyed off of the envelope sender address). The first
one is quite low (defaults to about 30 per hour - an indivual can send
messages to 30 email addresses per hour without any impact at all). Above 30,
and every message we recieve gets "queueonly" control applied to it (to
prevent immediate delivery). There's also an SQL database which is updated
with a "QUEUE" status (see later).

As far as the client is concerned all messages have been accepted, but for the
moment they're not going anywhere.

We then have a system filter which checks the sender's status in the database,
and if it's set to "QUEUE", it checks the time the message has been waiting
($message_age variable) and if it's less than a set amount (20 minutes seems
to work well), delivery is deferred. If it's older than 20 minutes then
delivery proceeds as normal. We run the queue at 5 minute intervals.

If a sender continues to blast messages at us they eventually reach the "high"
limit (say 300-400 within an hour). At this point they get a "FREEZE" entry in
the database, and when the system filter runs it freezes the message outright.
No deliveries happen, and it's up to manual intervention to either unfreeze or
delete (with the subsequent account disabling etc).

Also "FREEZE" beats "QUEUE", so any initial messages currently sat on the
queue are automatically frozen too during subsequent trips through the system
filter.

In the year this's been running, we've had a total of 3 complaints about
"delays" in sending email. People for the most part expect that if they're
sending messages to a bunch of people it won't be absolutely instantaneous (or
they don't care). Also, it's per sender configurable (i.e. if geoff [at] domain
consistently sends batches of emails to 500 people, we just set his "FREEZE"
limit a bit higher).

Spammers will usually hit the "QUEUE" limit in the first message. They then
continue to fire them out as quickly as possible, so for the most part we
catch the spam run before it even happens. Even if they don't, they don't get
many messages out before we get them. Compromised accounts used to send around
500,000 spams a month from our servers. This last year it's been cut to 2000
in total (and most of that was a single config error on our part).

The majority of "legit" big senders are only delayed for around 30 minutes max
and we either point them at our list server or increase their limit.

Hope that gives you some more ideas.

Regards

Ben
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


Lena at lena

Dec 20, 2011, 4:16 AM

Post #7 of 8 (596 views)
Permalink
Re: Spam control via ratelimiting. [In reply to]

> From: Ben

> ratelimiting (keyed off of the envelope sender address)

Perhaps not envelope sender, but $authenticated_id ?
(At least some) spambots specify random envelope sender in each message.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/


ben at beadynet

Dec 20, 2011, 5:35 AM

Post #8 of 8 (592 views)
Permalink
Re: Spam control via ratelimiting. [In reply to]

On Tuesday, December 20, 2011 12:16:51 pm Lena [at] lena wrote:
> > From: Ben
> >
> > ratelimiting (keyed off of the envelope sender address)
>
> Perhaps not envelope sender, but $authenticated_id ?
> (At least some) spambots specify random envelope sender in each message.

Agreed, that'd be better in most situations. However we have several machines
(Exchange boxes) that use these gateways as their smarthost. We don't have an
"authenticated id" for them so we're stuck with sender address. That said,
users aren't allowed to set their email address within Exchange, so it amounts
to the same thing anyway.

Ben

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

exim users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.