Mailing List Archive: SpamAssassin: devel

[Bug 6299] Update, enhance, and expand RCVD_ILLEGAL_IP

Index | Next | Previous | View Threaded

bugzilla-daemon at bugzilla

Jan 21, 2010, 4:04 PM

Post #1 of 12 (1484 views)
Permalink

[Bug 6299] Update, enhance, and expand RCVD_ILLEGAL_IP

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6299

Adam Katz <antispam [at] khopis> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |antispam [at] khopis

--- Comment #1 from Adam Katz <antispam [at] khopis> 2010-01-21 16:04:23 UTC ---
Checked into trunk/rulesrc/sandbox/khopesh/20_bug_6299.cf at r901932

Considering priority bump to address the pending FPs in the newly allocated
7.0.0.0/8 space (et al).

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

bugzilla-daemon at bugzilla

Jan 21, 2010, 4:05 PM

Post #2 of 12 (1428 views)
Permalink

[Bug 6299] Update, enhance, and expand RCVD_ILLEGAL_IP [In reply to]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6299

--- Comment #2 from Adam Katz <antispam [at] khopis> 2010-01-21 16:05:52 UTC ---
(In reply to comment #1)
> Checked into trunk/rulesrc/sandbox/khopesh/20_bug_6299.cf at r901932
>
> Considering priority bump to address the pending FPs in the newly allocated
> 7.0.0.0/8 space (et al).

uh. that's r901931

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

bugzilla-daemon at bugzilla

Jan 21, 2010, 4:35 PM

Post #3 of 12 (1423 views)
Permalink

[Bug 6299] Update, enhance, and expand RCVD_ILLEGAL_IP [In reply to]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6299

Justin Mason <jm [at] jmason> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |jm [at] jmason

--- Comment #3 from Justin Mason <jm [at] jmason> 2010-01-21 16:35:02 UTC ---
that fails lint:

lint: config: invalid regexp for rule T_RCVD_INVALID_IP: /
(?:by|ip)=(?!(?:(?:\.(?:1?\d?\d|2(?[0-4]\d|5[0-4])))\.(?:1?\d?\d|2(?[0-4]\d|5[0-4]))){3}
)/: Sequence (?[....) not recognized in regex; marked by <-- HERE in m/
(?:by|ip)=(?!(?:(?:\.(?:1?\d?\d|2(?[ <-- HERE
0-4]\d|5[0-4])))\.(?:1?\d?\d|2(?[0-4]\d|5[0-4]))){3} )/

commenting for now, feel free to uncomment when it's fixed ;)

: 32...; svn commit -m "comment out rule which fails lint"
rulesrc/sandbox/khopesh/20_bug_6299.cf
Sending rulesrc/sandbox/khopesh/20_bug_6299.cf
Transmitting file data .
Committed revision 901948.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

bugzilla-daemon at bugzilla

Jan 21, 2010, 4:38 PM

Post #4 of 12 (1424 views)
Permalink

[Bug 6299] Update, enhance, and expand RCVD_ILLEGAL_IP [In reply to]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6299

--- Comment #4 from Mark Martinec <Mark.Martinec [at] ijs> 2010-01-21 16:38:29 UTC ---
(In reply to comment #0)
> Then, I used its output with Mark's(?) version to hand-modify the rule:

Not mine, I just fixed it for Bug 6237 (removing 2.0.0.0/8 and 223.0.0.0/8).

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

bugzilla-daemon at bugzilla

Jan 21, 2010, 4:45 PM

Post #5 of 12 (1423 views)
Permalink

[Bug 6299] Update, enhance, and expand RCVD_ILLEGAL_IP [In reply to]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6299

--- Comment #5 from Mark Martinec <Mark.Martinec [at] ijs> 2010-01-21 16:45:01 UTC ---
Btw, the newly allocated 1.0.0.0/8 and 2.0.0.0/8 will start causing
false positives for SA 3.2.* as they come into more widespread use:
score RCVD_ILLEGAL_IP 3.199 3.196 2.902 1.908

Perhaps the new rule should be pushed into 3.2 updates.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

bugzilla-daemon at bugzilla

Jan 21, 2010, 8:13 PM

Post #6 of 12 (1417 views)
Permalink

[Bug 6299] Update, enhance, and expand RCVD_ILLEGAL_IP [In reply to]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6299

John Hardin <jhardin [at] impsec> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |jhardin [at] impsec

--- Comment #6 from John Hardin <jhardin [at] impsec> 2010-01-21 20:13:40 UTC ---
(In reply to comment #0)

> I know we want to avoid local and private net blocks,

We should also avoid the edge case where somebody is using a bogon subnet for
their private network, and only test the last external hop.

header RCVD_ILLEGAL_IP X-Spam-Relays-Untrusted =~ /^[^\]]+ (?:by|ip)=...etc/

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

bugzilla-daemon at bugzilla

Jan 21, 2010, 9:06 PM

Post #7 of 12 (1410 views)
Permalink

[Bug 6299] Update, enhance, and expand RCVD_ILLEGAL_IP [In reply to]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6299

--- Comment #7 from Adam Katz <antispam [at] khopis> 2010-01-21 21:06:15 UTC ---
(In reply to comment #3)
> that fails lint:

oops, missed the colons. fixed in r901999
any comment on how that rule can work with IPv6?

(In reply to comment #6)
> > I know we want to avoid local and private net blocks,
>
> We should also avoid the edge case where somebody is using a bogon subnet for
> their private network, and only test the last external hop.
>
> header RCVD_ILLEGAL_IP X-Spam-Relays-Untrusted =~ /^[^\]]+ (?:by|ip)=...etc/

I'm under the impression that spammers often forge extra headers to look more
authentic. Sometimes those are in private blocks, sometimes they are in
arbitrarily-chosen blocks.

We can test this theory, but my inclination is to leave it stand as is and
merely update the IP list.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

bugzilla-daemon at bugzilla

Jan 22, 2010, 4:09 AM

Post #8 of 12 (1393 views)
Permalink

[Bug 6299] Update, enhance, and expand RCVD_ILLEGAL_IP [In reply to]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6299

--- Comment #8 from Mark Martinec <Mark.Martinec [at] ijs> 2010-01-22 04:09:04 UTC ---
> I'm under the impression that spammers often forge extra headers to look more
> authentic. Sometimes those are in private blocks, sometimes they are in
> arbitrarily-chosen blocks.

Yes, this is my impression too, examining a bunch of messages during recent
days that hit the rule.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

bugzilla-daemon at bugzilla

Jan 26, 2010, 3:57 PM

Post #9 of 12 (1079 views)
Permalink

[Bug 6299] Update, enhance, and expand RCVD_ILLEGAL_IP [In reply to]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6299

--- Comment #9 from Adam Katz <antispam [at] khopis> 2010-01-26 15:57:08 UTC ---
(In reply to comment #5)
> Btw, the newly allocated 1.0.0.0/8 and 2.0.0.0/8 will start causing
> false positives for SA 3.2.* as they come into more widespread use:
> score RCVD_ILLEGAL_IP 3.199 3.196 2.902 1.908

It also FPs on 7.0.0.0/8, as does the replacement that went into 3.3..

Looks like it's already happening. Hege's two comparative rules (T_BUG_6295*)
show the newer version (_1) has far more FPs:
http://ruleqa.spamassassin.org/?rule=/RCVD_ILLEGAL_IP

Strangely enough, my version, which is more up-to-date, has slightly more FPs
than the version we're shipping. In triple-checking my regex, I actually found
100/8 was missing from my regex. The extremely mild increase in ham% was
offset by almost double the spam% and the highest S/O and GA rank of all the
variants.

> Perhaps the new rule should be pushed into 3.2 updates.

I agree.

(In reply to comment #6)
> We should also avoid the edge case where somebody is using a bogon subnet for
> their private network, and only test the last external hop.

I've added this to my sandbox to see what it does. As per my comment #7 I
suspect it reduces both ham and spam too much to be worthwhile.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

bugzilla-daemon at bugzilla

Jan 29, 2010, 11:39 AM

Post #10 of 12 (951 views)
Permalink

[Bug 6299] Update, enhance, and expand RCVD_ILLEGAL_IP [In reply to]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6299

Mark Martinec <Mark.Martinec [at] ijs> changed:

What |Removed |Added
----------------------------------------------------------------------------
Priority|P5 |P4
Target Milestone|Undefined |3.3.1

--- Comment #10 from Mark Martinec <Mark.Martinec [at] ijs> 2010-01-29 11:39:04 UTC ---
Reassigning to 3.3.1 before it is forgotten and gone for good.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

bugzilla-daemon at bugzilla

Jan 29, 2010, 12:42 PM

Post #11 of 12 (946 views)
Permalink

[Bug 6299] Update, enhance, and expand RCVD_ILLEGAL_IP [In reply to]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6299

--- Comment #11 from Adam Katz <antispam [at] khopis> 2010-01-29 12:42:40 UTC ---
(In reply to comment #8)
> > I'm under the impression that spammers often forge extra headers to look more
> > authentic. Sometimes those are in private blocks, sometimes they are in
> > arbitrarily-chosen blocks.
>
> Yes, this is my impression too, examining a bunch of messages during recent
> days that hit the rule.

Verified. Restricting this rule to last-external actually resulted in zero
hits in the masscheck, as evidenced by comparing T_KHOP_RCVD_ILLEGAL_IP_LE,
which is an exact copy of T_KHOP_RCVD_ILLEGAL_IP with the limitation suggested
by comment #6

Today's numbers are more of a mixed bag; the spam% of the updated rule is
almost exactly double the current rule, but the ham hits moved from 3 (0.0010%)
to 34 (0.0111%), as contrasted to the 20100126 results which showed only ONE
more ham hit on the updated rule.

Clearly, there are internal networks that allocate (ex-)bogon spaces rather
than using the reserved private network allocations of. Maybe hitting this
rule will help network administrators realize this.

I still think this is worth pushing forward, both on trunk and on each
supported branch.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

bugzilla-daemon at bugzilla

Feb 5, 2010, 5:48 PM

Post #12 of 12 (769 views)
Permalink

[Bug 6299] Update, enhance, and expand RCVD_ILLEGAL_IP [In reply to]

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6299

--- Comment #12 from Adam Katz <antispam [at] khopis> 2010-02-05 17:48:07 UTC ---
I am folding RCVD_TEST_NET and RCVD_LINK_LOCAL into the proposed
RCVD_ILLEGAL_IP despite their having no hits since they are fully reserved IPs;
they're only not on the IANA list as anything but footnotes because the list
contains only /8 networks.

I finally did the research on how SA handles IPv6 for this. No conflict and no
reason for checking things like 299.1.2.3 or 300.1.2.3 or 1234.5.6.7 as the
parser will refuse to put those into the pseudo-header. I removed those
aspects of the expression as well as the trailing octets (which made it far
simpler to add the non-/8 reserved networks).

In a few days, I'll push this bug to review. Just want to let that merging get
tested a little more and for more opportunity for people to voice opinions.

--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads