Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: SpamAssassin: users

Mailspike Performance

 

 

SpamAssassin users RSS feed   Index | Next | Previous | View Threaded


wtogami at gmail

Apr 12, 2011, 1:39 AM

Post #1 of 5 (2662 views)
Permalink
Mailspike Performance

We haven't had working statistics viewing for a few weeks, but now it is
fixed and I'm amazed by the performance of RCVD_IN_MSPIKE_BL.

http://ruleqa.spamassassin.org/20110409-r1090548-n/T_RCVD_IN_MSPIKE_BL/detail

RCVD_IN_MSPIKE_BL has nearly the highest spam detection ratio of all the
DNSBL's, second only to RCVD_IN_XBL. But our measurements also indicate
it is detecting this huge amount of spam with a very good ham safety rating.

* 84% overlap with RCVD_IN_XBL - redundancy isn't a huge problem here
because XBL is a tiny score. But 84% is surprisingly low overlap ratio
for such high spam detecting rule. This confirms that Mailspike is
doing an excellent job of building their IP reputation database in a
truly independent fashion.
* 67% overlap with RCVD_IN_PBL - overlap with PBL is concerning because
PBL is a high score. But 67% isn't too bad compared to other production
DNSBL's.
* 58% overlap with RCVD_IN_PSBL - pretty good

Given Mailspike's sustained decent performance since late 2009, it seems
clear that it is a great candidate for addition to spamassassin-3.4 by
default. It would be very interesting to see what it does to the scores
during an automatic rescoring of the network rules.

Thoughts about Future Rescoring
===============================
Before that rescoring, we may want to have a serious discussion about
reducing score pile-up in the case where multiple production DNSBL's all
hit at the same time. Adam Katz' approach is one possibility, albeit
confusing to users because users see subtractions in the score reports.
There may be other better approaches to this.


In related news...
==================
http://www.spamtips.org/2011/01/dnsbl-safety-report-1232011.html
The January DNSBL Safety report found RCVD_IN_SEMBLACK to be reasonably
safe, but at the time it overlapped with RCVD_IN_PBL 91% of the time
making it dangerously redundant due to PBL's high production score.

http://ruleqa.spamassassin.org/20110409-r1090548-n/T_RCVD_IN_SEMBLACK/detail
Our most recent measurements indicate that SEMBLACK is back to previous
behavior of extremely poor safety rating, with false positives on ~7% of
ham from recent weeks.

It was a bad idea to use SEMBLACK earlier this year due to the high
overlap with RCVD_IN_PBL, but this significant decline in safety rating
is a clear indication that you should not be using RCVD_IN_SEMBLACK.

http://ruleqa.spamassassin.org/20110409-r1090548-n/T_RCVD_IN_HOSTKARMA_BL/detail
HOSTKARMA_BL overlaps with MSPIKE_BL 88% of the time, but detects far
fewer spam and and with slightly more FP's. Compared to last year,
HOSTKARMA_BL's safety rating has improved considerably on a sustained
basis, and if we were evaluating it alone it wouldn't be too bad. But
now that we see the overlaps, HOSTKARMA_BL at this very moment is pretty
close to a redundant and slightly less safe subset of RCVD_IN_MSPIKE_BL.
Given these measurements, it probably isn't helpful to use HOSTKARMA_BL.

Warren Togami
warren [at] togami


mysqlstudent at gmail

Apr 12, 2011, 12:35 PM

Post #2 of 5 (2527 views)
Permalink
Re: Mailspike Performance [In reply to]

Hi,

> http://ruleqa.spamassassin.org/20110409-r1090548-n/T_RCVD_IN_HOSTKARMA_BL/detail
> HOSTKARMA_BL overlaps with MSPIKE_BL 88% of the time, but detects far fewer
> spam and and with slightly more FP's. Compared to last year, HOSTKARMA_BL's
> safety rating has improved considerably on a sustained basis, and if we were
> evaluating it alone it wouldn't be too bad. But now that we see the
> overlaps, HOSTKARMA_BL at this very moment is pretty close to a redundant
> and slightly less safe subset of RCVD_IN_MSPIKE_BL. Given these
> measurements, it probably isn't helpful to use HOSTKARMA_BL.

What is the recommended score for the MSPIKE rule(s)?

I currently have it set to 2.1. Should/can it be set higher?

Thanks for this update.

Alex


antispam at khopis

Apr 14, 2011, 2:51 PM

Post #3 of 5 (2609 views)
Permalink
Re: Mailspike Performance [In reply to]

On 04/12/2011 01:39 AM, Warren Togami Jr. wrote:
> We haven't had working statistics viewing for a few weeks, but now it
> is fixed and I'm amazed by the performance of RCVD_IN_MSPIKE_BL.
>
> http://ruleqa.spamassassin.org/20110409-r1090548-n/T_RCVD_IN_MSPIKE_BL/detail
>
>
> RCVD_IN_MSPIKE_BL has nearly the highest spam detection ratio of all
> the DNSBL's, second only to RCVD_IN_XBL. But our measurements also
> indicate it is detecting this huge amount of spam with a very good
> ham safety rating.
>
> * 84% overlap with RCVD_IN_XBL - redundancy isn't a huge problem
> here because XBL is a tiny score. But 84% is surprisingly low
> overlap ratio for such high spam detecting rule. This confirms that
> Mailspike is doing an excellent job of building their IP reputation
> database in a truly independent fashion.
> * 67% overlap with RCVD_IN_PBL - overlap with PBL is concerning
> because PBL is a high score. But 67% isn't too bad compared to other
> production DNSBL's.
> * 58% overlap with RCVD_IN_PSBL - pretty good

I created a meta for testing new DNSBLs a short while ago and didn't say
anything about it:

meta PUBLISHED_DNSBLS RCVD_IN_XBL || RCVD_IN_PBL || RCVD_IN_PSBL ||
RCVD_IN_SORBS_DUL || RCVD_IN_SORBS_WEB || RCVD_IN_BL_SPAMCOP_NET ||
RCVD_IN_RP_RNBL
tflags PUBLISHED_DNSBLS net nopublish # 20110127

meta PUBLISHED_DNSBLS_BRBL PUBLISHED_DNSBLS || RCVD_IN_BRBL_LASTEXT
tflags PUBLISHED_DNSBLS_BRBL net nopublish # 20110127

RCVD_IN_MSPIKE_BL has 99% overlap with the SA3.3 set and 98% with the
SA3.2 set. That leaves 0.6758% of spam uniquely hitting this DNSBL (1%
of its 67.5822%). RCVD_IN_SEMBLACK has the same story, resulting in
0.5138% unique spam from its 1% non-overlap (though note its lower s/o).

I'm guessing we have enough lists that they're all around this ballpark,
though we can't prove that without adding seven more meta rules (or
merely grepping the spam.log files).
Attachments: signature.asc (0.26 KB)


jm at jmason

Apr 15, 2011, 3:14 AM

Post #4 of 5 (2549 views)
Permalink
Re: Mailspike Performance [In reply to]

On Thu, Apr 14, 2011 at 22:51, Adam Katz <antispam [at] khopis> wrote:
> RCVD_IN_MSPIKE_BL has 99% overlap with the SA3.3 set and 98% with the
> SA3.2 set.  That leaves 0.6758% of spam uniquely hitting this DNSBL (1%
> of its 67.5822%).  RCVD_IN_SEMBLACK has the same story, resulting in
> 0.5138% unique spam from its 1% non-overlap (though note its lower s/o).

Good point. But what about the ham? if it hits the same spam, but
less ham, it's a better rule.

--j.


rwmaillists at googlemail

Apr 15, 2011, 7:48 AM

Post #5 of 5 (2521 views)
Permalink
Re: Mailspike Performance [In reply to]

On Mon, 11 Apr 2011 22:39:11 -1000
"Warren Togami Jr." <wtogami [at] gmail> wrote:

> We haven't had working statistics viewing for a few weeks, but now it
> is fixed and I'm amazed by the performance of RCVD_IN_MSPIKE_BL.
>
> http://ruleqa.spamassassin.org/20110409-r1090548-n/T_RCVD_IN_MSPIKE_BL/detail
>
> RCVD_IN_MSPIKE_BL has nearly the highest spam detection ratio of all
> the DNSBL's, second only to RCVD_IN_XBL. But our measurements also
> indicate it is detecting this huge amount of spam with a very good
> ham safety rating.
>
> * 84% overlap with RCVD_IN_XBL -
> ...
> * 58% overlap with RCVD_IN_PSBL - pretty good
>

IMO these overlaps aren't very important. Two lists could have a high
overlap because they both incorporate XBL, or they may just be
independently very good in the same niche.

What is important is the overlap in FPs. If a list has a weak FP
correlation with other lists, it's worthy of separate scoring even
if there's 99% overlap. If a list has strongly correlated FPs
against other lists it may be better to exclude it, or incorporate or
mitigate it through metarules.

Getting FP overlap figures is a little more difficult, but they are
much more objective.

SpamAssassin users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.