Mailing List Archive: SpamAssassin: users

JMF_W & URIBL_BLACK

Index | Next | Previous | View Threaded

mysqlstudent at gmail

Nov 9, 2009, 8:40 PM

Post #1 of 9 (889 views)
Permalink

JMF_W & URIBL_BLACK

Hi all,

I wrote a procmail script, and a few corresponding shell scripts to
put together a list of emails that were marked for both JMF_W
(HOSTKARMA_W) and URIBL_BLACK in the same message. While not
necessarily the complete conflict you might think (how can a mail
server be whitelisted while the message body contains a blacklisted
URL?), but I still thought it was worth investigating, and perhaps of
general interest to those that manage their respective lists.

So, I took about 6MB of mail, stripped the last "Received:" entry and
all of the URIs in the message and passed them through again to
multi.uribl.com, and produced a list of about 7k lines with the
Received line and corresponding blacklisted domains.

It was an interesting exercise, if nothing else. Hopefully someone can
make use of it. I'd like to hear if there are either some URIs or mail
servers that were listed wrong from this list. I know some of the
emails contained no blacklisted domains a few days after receiving the
email where they were originally marked (they had since been removed),
so it would be interesting if there were others like that on this
list.

For example, does salesforce.com know one of the domains included in
their emails is blacklisted?

Hopefully it's not too much text for the list...

Thanks,
Alex

Attachments:

uribl-list-2 (6.86 KB)

kdeugau at vianet

Nov 10, 2009, 8:05 AM

Post #2 of 9 (848 views)
Permalink

Re: JMF_W & URIBL_BLACK [In reply to]

Alex wrote:
> (how can a mail
> server be whitelisted while the message body contains a blacklisted
> URL?)

Pretty trivially; if spam with a blacklisted URI is forwarded from an
account handled by a trusted server, the final recipient will see both a
whitelisted/trusted relay and a blacklisted URI.

Of course, if mail is forwarded like that, it's not entirely
unreasonable to expect the forwarding system to do some spam filtering
of its own... in which case the spam wouldn't likely be forwarded in
the first place.

-kgd

mysqlstudent at gmail

Nov 10, 2009, 8:53 AM

Post #3 of 9 (843 views)
Permalink

Re: JMF_W & URIBL_BLACK [In reply to]

Hi,

>> (how can a mail
>> server be whitelisted while the message body contains a blacklisted
>> URL?)
>
> Pretty trivially; if spam with a blacklisted URI is forwarded from an
> account handled by a trusted server, the final recipient will see both a
> whitelisted/trusted relay and a blacklisted URI.

Another simple example is mail from Google, such as their "Google
Alert", which sends back links in an email every day based on your
query.

I was more looking for cases where that shouldn't be happening, and
one or the other (JMF_W or URIBL_BLACK) were incorrect. It seemed like
a good way to find anomalies.

When it does happen, what are you supposed to do if it is considered
severe enough? Block the server or whitelist the URL?

Thanks,
Alex

rob at invaluement

Nov 10, 2009, 10:26 AM

Post #4 of 9 (827 views)
Permalink

Re: JMF_W & URIBL_BLACK [In reply to]

Alex wrote:
> for both JMF_W
> (HOSTKARMA_W) and URIBL_BLACK in the same message.

I'm not involved in the management of either of these, but I have some
analysis which I think is accurate:

(1) Marc Perkel's domain whitelist is auto-generated. This has many
advantages... but one disadvantage is that some very dark-gray (or even
blackhat) are going to get on that whitelist more because they were very
shrewd and sneaky--but their messages were 99-100% UBE and NOT desired
in user's inboxes (but the messages *looked* legit enough to keep the
complaints down--a common situation)

(2) At the same time, uribl is (2.a) very good at listing just those
sort of sneaky ESPs who deserve to be blacklisted and are often missed
by other lists ...AND... (2.b) uribl also sometimes goes a bit too far
and lists some ESPs and hosters who have some legit uses, but send much spam

I think 2.a is happening more often here than 2.b

Regarding 2.b ...HEY... there are some really hard judgment calls here
that could go either way. For example, I'm starting to notice many
dark-gray ESPs who are sending 90% UBE... but the 10% legit mail they
are sending are *pure* *advertisements*... NOT things like order
confirmations, etc. There is then a strong argument that the collateral
damage of blocking that 10% pure ads really isn't that harmful in return
for the benefit of blocking the spams--since the end users are also
going to like such a tradeoff. But this still isn't easy because
sometimes that 10% involves large and famous companies (like AT&T recent
use of [withheld]'s ESP services)

And there are other examples which are a much harder to "call".

But i think this well explains the overlap between URIBL-black and
HostKarma's domain whitelist.

--
Rob McEwen
http://dnsbl.invaluement.com/
rob [at] invaluement
+1 (478) 475-9032

mysqlstudent at gmail

Nov 10, 2009, 7:59 PM

Post #5 of 9 (831 views)
Permalink

Re: JMF_W & URIBL_BLACK [In reply to]

Hi,

>> for both JMF_W
>> (HOSTKARMA_W) and URIBL_BLACK in the same message.
>
> I'm not involved in the management of either of these, but I have some
> analysis which I think is accurate:

Rob, thanks, I think you've hit the nail on the head on all counts.
That's what the spam race is all about -- the spammers trying to get
as much by as possible, while we try to definitively determine what
is, and isn't, junk or unsolicited as the line gets harder and harder
to distinguish.

This just becomes increasingly important when management drops an
email in the "Put Spam Here" folder for training that clearly isn't
spam, but something they've subscribed to, like a newsletter. For the
email that even I question sometimes, I'd like to be able to give them
a definitive answer as to why it is or isn't spam.

Thanks,
Alex

wtogami at redhat

Nov 10, 2009, 8:43 PM

Post #6 of 9 (830 views)
Permalink

Re: JMF_W & URIBL_BLACK [In reply to]

On 11/10/2009 10:59 PM, Alex wrote:
>
> This just becomes increasingly important when management drops an
> email in the "Put Spam Here" folder for training that clearly isn't
> spam, but something they've subscribed to, like a newsletter. For the
> email that even I question sometimes, I'd like to be able to give them
> a definitive answer as to why it is or isn't spam.
>
> Thanks,
> Alex

You know how it is unsafe to use unsubscribe on many spam because that
you are a live address?

What would be helpful is a list of "known mailing list service
providers" that we know are safe to use the unsubscribe procedure. Then
when our users classify such mail as spam, we could possibly
auto-unsubscribe with scripts and not blindly blacklist.

Does topica.com ever send any legitimate mail? I see nothing but spam
coming from there. I see many similar "providers" of unknown legitimacy
where I don't know if to report or unsubscribe. The business model of
these providers apparently do not reward them to be vigilant against
abusive customers, as we spam fighters seem unwilling to punish them for
repetitive abuse.

Warren

mysqlstudent at gmail

Nov 10, 2009, 9:38 PM

Post #7 of 9 (839 views)
Permalink

Re: JMF_W & URIBL_BLACK [In reply to]

Hi,

> You know how it is unsafe to use unsubscribe on many spam because that you
> are a live address?

Of course. If you read again, I'm talking about sites like eWeek.com,
where someone has intentionally subscribed, but can't figure out how
to unsubscribe, so they think if they just treat it as spam, it is
another way of making it go away.

> What would be helpful is a list of "known mailing list service providers"

Yeah, according to whose judgement? I think this is kind of what we
have with the JMF_W list; there shouldn't be anything on there that
permits the relaying of spam, else they should be delisted, if not
blacklisted.

I guess that also holds true for URIBL_BLACK? If the URL is
blacklisted, then ideally the user shouldn't see the unsubscribe link.

> Does topica.com ever send any legitimate mail? I see nothing but spam
> coming from there. I see many similar "providers" of unknown legitimacy

That was my question some time ago. They sure seem to be legitimate,
and I believe their relays are even whitelisted. They have several
domains, IIRC.

There's also us.edir1.com. I don't see anything legitimate from them,
but their JMF_W, IADB_GOODMAIL, DKIM_VERIFIED, and SENDERID, and
occasionally URIBL_BLACK, I believe. However, all that comes from them
is credit card offers. I'm now blocking them at the gateway, and no
complaints for about a week.

Then there's oriental-trading.com, which is JMF_BR and URIBL_BLACK,
yet they are a legitimate brick-and-mortar operation too, and people
actually like their stuff.

> where I don't know if to report or unsubscribe. The business model of these
> providers apparently do not reward them to be vigilant against abusive
> customers, as we spam fighters seem unwilling to punish them for repetitive
> abuse.

I would, if my users complained, but it must be so good that they
don't even know they're being spammed, and I don't want to be the one
who gets blamed for some executive not receiving their free coupons
from Oriental Trading. (hypothetically speaking, of course).

We're easily processing twice as much email as we were this time last
year. Still a billion rejects at the gateway for invalid helo.

Best,
Alex

csanterre at MerchantsOverseas

Nov 11, 2009, 6:34 AM

Post #8 of 9 (799 views)
Permalink

RE: JMF_W & URIBL_BLACK [In reply to]

> This just becomes increasingly important when management drops an
> email in the "Put Spam Here" folder for training that clearly isn't
> spam, but something they've subscribed to, like a newsletter. For the
> email that even I question sometimes, I'd like to be able to give them
> a definitive answer as to why it is or isn't spam.

This week, the president of the company told me to "Block all emails with
"deserve" in it!"

*sigh*

I had to school him on the fine art of antispam techniques with examples
like:

We deserve better servers..
Our health care coverage deserves to be fired...
The IT guys deserve a raise...

:)

--Chris

uhlar at fantomas

Nov 11, 2009, 6:44 AM

Post #9 of 9 (809 views)
Permalink

Re: JMF_W & URIBL_BLACK [In reply to]

> > This just becomes increasingly important when management drops an
> > email in the "Put Spam Here" folder for training that clearly isn't
> > spam, but something they've subscribed to, like a newsletter. For the
> > email that even I question sometimes, I'd like to be able to give them
> > a definitive answer as to why it is or isn't spam.

On 11.11.09 09:34, Chris Santerre wrote:
> This week, the president of the company told me to "Block all emails with
> "deserve" in it!"
>
> *sigh*
>
> I had to school him on the fine art of antispam techniques with examples
> like:
>
> We deserve better servers..
> Our health care coverage deserves to be fired...
> The IT guys deserve a raise...

I really wonder if THESE did convince him of NOT blocking such e-mail ;-)

--
Matus UHLAR - fantomas, uhlar [at] fantomas ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Spam is for losers who can't get business any other way.

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads