Mailing List Archive: SpamAssassin: users

Creating new rules

Index | Next | Previous | View Threaded

mls at vp44

May 7, 2011, 11:38 AM

Post #1 of 22 (351 views)
Permalink

Creating new rules

Hi guys.
I need some help in setting up effective rules to counter a spam wave that
has been hitting my server lately.
Most of the messages come from hotmail.com accounts and for obvious reasons
I can't block the whole domain.
All the emails have a common pattern (HTML_LINK + JUNK_TEXT):

http://pastebin.com/sWahQEjx

http://pastebin.com/aFjnyi6f

http://pastebin.com/bdeb5p9K

Any tips?
Thanks.

Andrea

ned at unixmail

May 7, 2011, 12:05 PM

Post #2 of 22 (331 views)
Permalink

Re: Creating new rules [In reply to]

On 07/05/11 19:38, Andrea Gozzi wrote:
> Hi guys.
> I need some help in setting up effective rules to counter a spam wave that
> has been hitting my server lately.
> Most of the messages come from hotmail.com accounts and for obvious reasons
> I can't block the whole domain.
> All the emails have a common pattern (HTML_LINK + JUNK_TEXT):
>
> http://pastebin.com/sWahQEjx
>
> http://pastebin.com/aFjnyi6f
>
> http://pastebin.com/bdeb5p9K
>
> Any tips?
> Thanks.
>
> Andrea
>
>
>

Yep, they are darn hard to catch.

For starters, make sure you are training them in Bayes.

Your best bet of catching these particular examples is probably through
the URIs in the spam. When I scan them now with SA they are caught by
many URIBLs. Greylisting can help here too as delaying accepting of the
mail gives the URIBLs time to blacklist the spammed URIs (although I
appreciate greylisting isn't to everyone's taste).

Personally, I've got so fed up with gorilla freemail spam I score
hotmail et al., with 3 points for starters just to give them less
wriggle room to mess up. Adjust accordingly and/or look at rules that
then whitelist legitimate senders.

I just don't believe these guys are too big to block. I've found it far
less time consuming to block and whitelist the ham than I have to deal
with all the spam that makes it through otherwise. I don't see a huge
amount of spam from gmail users, so if they can deal with it that just
tells me that hotmail, yahoo et al. just don't care.

me at junc

May 7, 2011, 1:24 PM

Post #3 of 22 (330 views)
Permalink

Re: Creating new rules [In reply to]

All the emails have a common pattern (HTML_LINK + JUNK_TEXT):

meta on info tld && !user_in_whitelist_from_spf

train bayes, adjust autolearnthreshold to less then default -0.2

why have none devs maked a policyd that make sure sender is known to the
recipient ?, (i got a new email address blocking in outlook) this is
sure working from freemail domains

jarif at iki

Jul 31, 2013, 10:43 AM

Post #4 of 22 (134 views)
Permalink

Re: Creating new rules [In reply to]

31.07.2013 20:08, Franck Martin kirjoitti:
> Hi all,
>
> I noticed there is no rules to check if the domain in various emails fields are on blocking lists like DBL at spamhaus. I'm willing to work on some of these rules, but I would appreciate any advice to bootstrap the process. If you can reference documents or say something like, look at this rule and this rule, this is close to what you need to do.
>
SpamAssassin will and does check those RBL:s. They are NET rules, and
not active when doing -D

This may be the issue now, you have tried -D to a message and see what
triggers? When run normally the network checks will be executed.

--
jarif.bit

Attachments:

signature.asc (0.26 KB)

fmartin at linkedin

Jul 31, 2013, 10:50 AM

Post #5 of 22 (134 views)
Permalink

Re: Creating new rules [In reply to]

On Jul 31, 2013, at 7:43 PM, Jari Fredriksson <jarif [at] iki> wrote:

> 31.07.2013 20:08, Franck Martin kirjoitti:
>> Hi all,
>>
>> I noticed there is no rules to check if the domain in various emails fields are on blocking lists like DBL at spamhaus. I'm willing to work on some of these rules, but I would appreciate any advice to bootstrap the process. If you can reference documents or say something like, look at this rule and this rule, this is close to what you need to do.
>>
> SpamAssassin will and does check those RBL:s. They are NET rules, and
> not active when doing -D
>
Hmm, thanks

I looked at http://spamassassin.apache.org/tests_3_3_x.html could not find any rule that do the above. Please help.

Attachments:

signature.asc (0.48 KB)

Ralf.Hildebrandt at charite

Jul 31, 2013, 10:56 AM

Post #6 of 22 (134 views)
Permalink

Re: Creating new rules [In reply to]

* Franck Martin <fmartin [at] linkedin>:

> I looked at http://spamassassin.apache.org/tests_3_3_x.html could not find any rule that do the above. Please help.

That's a bit odd. I found it being mentioned here:

http://www.spamhaus.org/faq/section/Spamhaus%20DBL#287
http://spamassassin.1065346.n5.nabble.com/enabling-SpamHaus-DBL-td55862.html
http://svn.apache.org/repos/asf/spamassassin/trunk/rules/25_uribl.cf

and by all means it should be enabled by default.

--
Ralf Hildebrandt Charite Universitätsmedizin Berlin
ralf.hildebrandt [at] charite Campus Benjamin Franklin
http://www.charite.de Hindenburgdamm 30, 12203 Berlin
Geschäftsbereich IT, Abt. Netzwerk fon: +49-30-450.570.155

fmartin at linkedin

Jul 31, 2013, 11:05 AM

Post #7 of 22 (134 views)
Permalink

Re: Creating new rules [In reply to]

On Jul 31, 2013, at 7:56 PM, Ralf Hildebrandt <Ralf.Hildebrandt [at] charite>
wrote:

> * Franck Martin <fmartin [at] linkedin>:
>
>> I looked at http://spamassassin.apache.org/tests_3_3_x.html could not find any rule that do the above. Please help.
>
> That's a bit odd. I found it being mentioned here:
>
> http://www.spamhaus.org/faq/section/Spamhaus%20DBL#287
> http://spamassassin.1065346.n5.nabble.com/enabling-SpamHaus-DBL-td55862.html
> http://svn.apache.org/repos/asf/spamassassin/trunk/rules/25_uribl.cf
>
> and by all means it should be enabled by default.
>

Ah yes, I saw these rules, but this is to check the domains of urls in the messages, not to check for instance that the domain used in the From: header is on the DBL.

jarif at iki

Jul 31, 2013, 1:00 PM

Post #8 of 22 (134 views)
Permalink

Re: Creating new rules [In reply to]

31.07.2013 21:05, Franck Martin kirjoitti:
> On Jul 31, 2013, at 7:56 PM, Ralf Hildebrandt <Ralf.Hildebrandt [at] charite>
> wrote:
>
>> * Franck Martin <fmartin [at] linkedin>:
>>
>>> I looked at http://spamassassin.apache.org/tests_3_3_x.html could not find any rule that do the above. Please help.
>> That's a bit odd. I found it being mentioned here:
>>
>> http://www.spamhaus.org/faq/section/Spamhaus%20DBL#287
>> http://spamassassin.1065346.n5.nabble.com/enabling-SpamHaus-DBL-td55862.html
>> http://svn.apache.org/repos/asf/spamassassin/trunk/rules/25_uribl.cf
>>
>> and by all means it should be enabled by default.
>>
> Ah yes, I saw these rules, but this is to check the domains of urls in the messages, not to check for instance that the domain used in the From: header is on the DBL.
>

http://svn.apache.org/repos/asf/spamassassin/trunk/rules/25_uribl.cf

--
jarif.bit

Attachments:

signature.asc (0.26 KB)

jarif at iki

Jul 31, 2013, 1:04 PM

Post #9 of 22 (134 views)
Permalink

Re: Creating new rules [In reply to]

31.07.2013 21:05, Franck Martin kirjoitti:
> Ah yes, I saw these rules, but this is to check the domains of urls in the messages, not to check for instance that the domain used in the From: header is on the DBL.
Address in From: is usually always forged in Spam nowadays. There is not
much use for checking these.

AWL plugin does it anyway, if enabled. But it does not use any external
backlists for it...

--
jarif.bit

Attachments:

signature.asc (0.26 KB)

Kevin_Miller at ci

Jul 31, 2013, 1:08 PM

Post #10 of 22 (134 views)
Permalink

RE: Creating new rules [In reply to]

Problem is, the from adddress is often a "Joe job" - i.e., a forged address, so the domain mentioned there likely doesn't have anything to do with the actual source of the mail. It seems to me that if the domain isn't the actual source of he spam, it can be detrimental to be filtering on it, particularly if Bayes is learning from it or your MTA auto-reports it to RBLs.

YMMV...

...Kevin
________________________________________
From: Franck Martin [fmartin [at] linkedin]
Sent: Wednesday, July 31, 2013 10:05 AM
To: Ralf Hildebrandt
Cc: <users [at] spamassassin>
Subject: Re: Creating new rules

On Jul 31, 2013, at 7:56 PM, Ralf Hildebrandt <Ralf.Hildebrandt [at] charite>
wrote:

> * Franck Martin <fmartin [at] linkedin>:
>
>> I looked at http://spamassassin.apache.org/tests_3_3_x.html could not find any rule that do the above. Please help.
>
> That's a bit odd. I found it being mentioned here:
>
> http://www.spamhaus.org/faq/section/Spamhaus%20DBL#287
> http://spamassassin.1065346.n5.nabble.com/enabling-SpamHaus-DBL-td55862.html
> http://svn.apache.org/repos/asf/spamassassin/trunk/rules/25_uribl.cf
>
> and by all means it should be enabled by default.
>

Ah yes, I saw these rules, but this is to check the domains of urls in the messages, not to check for instance that the domain used in the From: header is on the DBL.

fmartin at linkedin

Jul 31, 2013, 2:06 PM

Post #11 of 22 (134 views)
Permalink

Re: Creating new rules [In reply to]

On Jul 31, 2013, at 10:08 PM, Kevin Miller <Kevin_Miller [at] ci> wrote:

> Problem is, the from adddress is often a "Joe job" - i.e., a forged address, so the domain mentioned there likely doesn't have anything to do with the actual source of the mail. It seems to me that if the domain isn't the actual source of he spam, it can be detrimental to be filtering on it, particularly if Bayes is learning from it or your MTA auto-reports it to RBLs.
>

Why would they use a forged domain which is on a blacklist? I think they would tend to use a domain which is well known with good reputation. As well known domains are getting protected, then they have to move to use their own domain, which happens to appear on blacklist...

Now as we move to IPv6, reputation will shift from an IP based type reputation, to a domain based type reputation. Unfortunately, spam assassin seems to be lacking some rules.

Nevertheless, it does not matter, if it is the right or wrong direction, my question remains: how do I create such a rule?

zauschneria at gmail

Jul 31, 2013, 2:19 PM

Post #12 of 22 (134 views)
Permalink

Re: Creating new rules [In reply to]

On Wed, Jul 31, 2013 at 2:06 PM, Franck Martin <fmartin [at] linkedin> wrote:

>
> On Jul 31, 2013, at 10:08 PM, Kevin Miller <Kevin_Miller [at] ci>
> wrote:
>
> > Problem is, the from adddress is often a "Joe job" - i.e., a forged
> address, so the domain mentioned there likely doesn't have anything to do
> with the actual source of the mail. It seems to me that if the domain
> isn't the actual source of he spam, it can be detrimental to be filtering
> on it, particularly if Bayes is learning from it or your MTA auto-reports
> it to RBLs.
> >
>
> Why would they use a forged domain which is on a blacklist?
>

Indeed, if someone uses a forged domain which is on a blacklist in the
header of their mail, we want to block that email too.

Some smart B2B spammers know about this loophole in SpamAssassin and don't
use their domain name in the message body, using it only in the header
where the URI checks aren't done.

Kevin_Miller at ci

Jul 31, 2013, 2:39 PM

Post #13 of 22 (134 views)
Permalink

RE: Creating new rules [In reply to]

Because some spammers are pretty dumb. Not all of course. Addresses are constantly being harvested. If you got a list of half a million addresses, are you going to vet all those? Oft times they'll just blast them out with a botnet and the ones that fail are just collateral damage. I think the goal is usually quantity over quality. Not being a spammer though, i could be wrong. <g>

Also, it may be that the domain wasn't in a blacklist when they botted it but gets put in pretty quickly via razor, pyzor, and various MTAs that report to RBLs. I've seen a dozen or so spam hit or server and w/in 15 - 20 minutes it'll be on someone's RBL. If it works for you, live it up. Those are just my thoughts - others here have a much more informed opinion I expect....

...Kevin
________________________________________
From: Franck Martin [fmartin [at] linkedin]
Sent: Wednesday, July 31, 2013 1:06 PM
To: Kevin Miller
Cc: Ralf Hildebrandt; <users [at] spamassassin>
Subject: Re: Creating new rules

On Jul 31, 2013, at 10:08 PM, Kevin Miller <Kevin_Miller [at] ci> wrote:

> Problem is, the from adddress is often a "Joe job" - i.e., a forged address, so the domain mentioned there likely doesn't have anything to do with the actual source of the mail. It seems to me that if the domain isn't the actual source of he spam, it can be detrimental to be filtering on it, particularly if Bayes is learning from it or your MTA auto-reports it to RBLs.
>

Why would they use a forged domain which is on a blacklist? I think they would tend to use a domain which is well known with good reputation. As well known domains are getting protected, then they have to move to use their own domain, which happens to appear on blacklist...

Now as we move to IPv6, reputation will shift from an IP based type reputation, to a domain based type reputation. Unfortunately, spam assassin seems to be lacking some rules.

Nevertheless, it does not matter, if it is the right or wrong direction, my question remains: how do I create such a rule?

fmartin at linkedin

Jul 31, 2013, 2:56 PM

Post #14 of 22 (134 views)
Permalink

Re: Creating new rules [In reply to]

On Jul 31, 2013, at 11:19 PM, RGB Camera <zauschneria [at] gmail<mailto:zauschneria [at] gmail>>
wrote:

On Wed, Jul 31, 2013 at 2:06 PM, Franck Martin <fmartin [at] linkedin<mailto:fmartin [at] linkedin>> wrote:

On Jul 31, 2013, at 10:08 PM, Kevin Miller <Kevin_Miller [at] ci<mailto:Kevin_Miller [at] ci>> wrote:

> Problem is, the from adddress is often a "Joe job" - i.e., a forged address, so the domain mentioned there likely doesn't have anything to do with the actual source of the mail. It seems to me that if the domain isn't the actual source of he spam, it can be detrimental to be filtering on it, particularly if Bayes is learning from it or your MTA auto-reports it to RBLs.
>

Why would they use a forged domain which is on a blacklist?

Indeed, if someone uses a forged domain which is on a blacklist in the header of their mail, we want to block that email too.

Some smart B2B spammers know about this loophole in SpamAssassin and don't use their domain name in the message body, using it only in the header where the URI checks aren't done.

Let me give some background...

I'm part of the people that adopted DMARC (cf www.dmarc.org<http://www.dmarc.org>) this provide protection for the domains that are heavily spoofed.

Why I'm not keen in reproducing the DMARC checks in spamassassin, because it is better handled via a milter like opendmarc (because of reporting capabilities which is important), I would not make a fuss if I see something like DMARC in spamassassin.

During the development of DMARC, we realized that there are a few holes to plug for it to be more effective, as well as realizing, in general, domain reputation will become more and more important.

One of this rule, is to check how the From: header is formed cf http://tools.ietf.org/html/draft-ietf-appsawg-malformed-mail-07 and rate negatively when some headers are malformed
The other is to extract all the domains from the following fields: envelope from, from: sender, reply-to and helo/ehlo, and check them against DNSBL

There may be other rules, but this is what comes to mind, last one is suggested on spamhaus FAQ but does not seem to have made it in spamassassin.

While at the moment DMARC is for domains heavily spoofed, the above rules should benefit everyone

me at junc

Aug 1, 2013, 3:34 AM

Post #15 of 22 (130 views)
Permalink

Re: Creating new rules [In reply to]

Jari Fredriksson skrev den 2013-07-31 22:04:
> 31.07.2013 21:05, Franck Martin kirjoitti:
>> Ah yes, I saw these rules, but this is to check the domains of urls
>> in
>> the messages, not to check for instance that the domain used in the
>> From: header is on the DBL.
> Address in From: is usually always forged in Spam nowadays. There is
> not
> much use for checking these.

http://blog.returnpath.com/blog/ken-takahashi/demystifying-spf-dkim-and-dmarc

> AWL plugin does it anyway, if enabled. But it does not use any
> external
> backlists for it...

if its runs with default /16 is just a joke

change it to /24 or /32 then its more no joke

when this is done, add sagrey plugin, stops one time senders

sa 3.4.0 should have history plugin but its missing in 3.4.0-rc2

me at junc

Aug 1, 2013, 3:44 AM

Post #16 of 22 (130 views)
Permalink

Re: Creating new rules [In reply to]

Franck Martin skrev den 2013-07-31 23:06:

> Why would they use a forged domain which is on a blacklist? I think
> they would tend to use a domain which is well known with good
> reputation. As well known domains are getting protected, then they
> have to move to use their own domain, which happens to appear on
> blacklist...

agre with that, here i blacklist_from that have spf_pass and spamming
sender, and also just spamming domain that is not dkim signed or get spf
results, eg score on spf_none :)

> Now as we move to IPv6, reputation will shift from an IP based type
> reputation, to a domain based type reputation. Unfortunately, spam
> assassin seems to be lacking some rules.

still missing dmarc spamassassin plugin, there is a dkim_reput but i
dont see much help there, it could be bootstrapped if one have own
dkim_repution server and reporting based on opendkim

and it failed for me with http://www.dkim-reputation.org/ it might
work, but would work better if more used it

> Nevertheless, it does not matter, if it is the right or wrong
> direction, my question remains: how do I create such a rule?

rule for ?

fmartin at linkedin

Aug 1, 2013, 3:59 AM

Post #17 of 22 (130 views)
Permalink

Re: Creating new rules [In reply to]

On Aug 1, 2013, at 12:44 PM, Benny Pedersen <me [at] junc> wrote:

> Franck Martin skrev den 2013-07-31 23:06:
>
>> Now as we move to IPv6, reputation will shift from an IP based type
>> reputation, to a domain based type reputation. Unfortunately, spam
>> assassin seems to be lacking some rules.
>
> still missing dmarc spamassassin plugin, there is a dkim_reput but i dont see much help there, it could be bootstrapped if one have own dkim_repution server and reporting based on opendkim
>
> and it failed for me with http://www.dkim-reputation.org/ it might work, but would work better if more used it

While interesting, I think this is a dead end... There is some IETF work to do some reputation system... not sure exactly what

>
>> Nevertheless, it does not matter, if it is the right or wrong
>> direction, my question remains: how do I create such a rule?
>
> rule for ?

grabbing a domain from some headers and checking it with a DNSBL.

rwmaillists at googlemail

Aug 1, 2013, 5:39 AM

Post #18 of 22 (130 views)
Permalink

Re: Creating new rules [In reply to]

On Thu, 01 Aug 2013 12:34:26 +0200
Benny Pedersen wrote:

> Jari Fredriksson skrev den 2013-07-31 22:04:

> > AWL plugin does it anyway, if enabled. But it does not use any
> > external
> > backlists for it...
>
> if its runs with default /16 is just a joke
>
> change it to /24 or /32 then its more no joke

This would make sense if the IP address were the the first trusted
address or last external, but AWL uses the first routable address which
is commonly dynamic.

me at junc

Aug 1, 2013, 7:36 AM

Post #19 of 22 (130 views)
Permalink

Re: Creating new rules [In reply to]

RW skrev den 2013-08-01 14:39:

> This would make sense if the IP address were the the first trusted
> address or last external, but AWL uses the first routable address
> which
> is commonly dynamic.

why is this in error ?

rwmaillists at googlemail

Aug 1, 2013, 9:00 AM

Post #20 of 22 (130 views)
Permalink

Re: Creating new rules [In reply to]

On Thu, 01 Aug 2013 16:36:22 +0200
Benny Pedersen wrote:

> RW skrev den 2013-08-01 14:39:
>
> > This would make sense if the IP address were the the first trusted
> > address or last external, but AWL uses the first routable address
> > which
> > is commonly dynamic.
>
> why is this in error ?

If you use /32 and the sender has a different IP address each time
there's no score averaging.

me at junc

Aug 1, 2013, 10:08 AM

Post #21 of 22 (130 views)
Permalink

Re: Creating new rules [In reply to]

RW skrev den 2013-08-01 18:00:

> If you use /32 and the sender has a different IP address each time
> there's no score averaging.

servers changeing sender ip daily ?, its not a real problem clients
does, there would be one static ip first

rwmaillists at googlemail

Aug 1, 2013, 11:52 AM

Post #22 of 22 (130 views)
Permalink

Re: Creating new rules [In reply to]

On Thu, 01 Aug 2013 19:08:12 +0200
Benny Pedersen wrote:

> RW skrev den 2013-08-01 18:00:
>
> > If you use /32 and the sender has a different IP address each time
> > there's no score averaging.
>
> servers changeing sender ip daily ?, its not a real problem clients
> does, there would be one static ip first

I think you have first and last the wrong way around. If the client has
a public IP address (as recorded by the submission server) then that's
the *first* routable address. With webmail the browser ip address is
used if it's recorded in a header.

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads