Mailing List Archive: SPF: Discuss

throwaway domains and whois

1 2

View All

Index | Next | Previous | View Threaded

stuart at bmsi

Oct 10, 2008, 10:44 AM

Post #1 of 36 (3770 views)
Permalink

throwaway domains and whois

My SPF based reputation system is working really well. I just added a
greylisting feature for domains with no reputation, and spam dropped so
much that several customers called complaining that something was wrong
with email since they weren't getting the steady trickle of spam they
were used to. The other filters were already cutting the spam from about
2000 per day per user to less than 20 per day per user. The greylisting
has dropped that to about 2 per day per user - but growing.

The surviving spammers have evolved. They register dozens
of new domains every day, with names like "strangecosmos.com" or
"lacerun.org" - random words glued together by a script. My system auto blocks
the domains after 20 spams - but those 20 annoy 20 users for each domain they
register. Each freshly registered domain has a valid and reasonable SPF
record. All identical - usually "v=spf1 a -all", and all such spam is SPF
pass. Their spam software has a state machine which retries after greylisting,
just like a real sender. There is a valid DomainKeys header for the 2822 From
field. The IP address is from a different part of the world for each freshly
registered domain. The message is presented as an image - making bayesian
filters rely on headers and meta-info for recognition. As this new breed of
spam software gets adopted, the spam that makes it through grows.

If you do a whois on these throwaway domains, the registrant is always
a front company, like "Protected Domain Services" or "Domains by Proxy".
My idea is to start tracking reputation by domain registrant.
I would like to reject all mail from the above two registrants, for instance,
regardless of domain name du jour. What are the restrictions on using
whois? Can I simply script running it for every domain, with a cache
to remember results? Or will reigstrars start blocking me for abuse?
What is the most efficient way to obtain whois info on a domain?

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

greg at hewgill

Oct 10, 2008, 11:33 AM

Post #2 of 36 (3654 views)
Permalink

Re: throwaway domains and whois [In reply to]

On Fri, Oct 10, 2008 at 01:44:40PM -0400, Stuart D. Gathman wrote:
> I would like to reject all mail from the above two registrants, for instance,
> regardless of domain name du jour. What are the restrictions on using
> whois? Can I simply script running it for every domain, with a cache
> to remember results? Or will reigstrars start blocking me for abuse?
> What is the most efficient way to obtain whois info on a domain?

If you run automated whois queries, I believe you will quickly get
blocked. I ran into one registrar who didn't allow me to do more than
something like three whois queries per day per IP. I hit that limit
just doing manual queries!

I don't know of any registrars that offer their whois database for bulk
download.

Have you considered assigning reputation to nameservers? I've thought
about that idea before but haven't yet built anything to try it out.
Some time ago I noticed that a particular breed of spam was coming from
a different domain every time, but the nameserver address was always
the same. Dodgy domains registered with a dodgy registrar would likely
tend to have different nameserver addresses than stable, legitimate
domains. Querying for NS records would not be subject to rate limiting.

Greg Hewgill
http://hewgill.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

stuart at bmsi

Oct 10, 2008, 12:43 PM

Post #3 of 36 (3655 views)
Permalink

Re: throwaway domains and whois [In reply to]

On Fri, 10 Oct 2008, Greg Hewgill wrote:

> Have you considered assigning reputation to nameservers? I've thought

I think that simply inserting pseudo headers like:

X-NameServer: ns39.domaincontrol.com
X-NameServer: ns40.domaincontrol.com

Into the message fed to bayesian content filtering would do the trick. Mail
from spammer nameservers will make it past SMTP envelope, but should
end up in quarantine - and each days batch of domains should get blocked
in SMTP envelope after 20 quarantined spams each.

Looking up multiple reputations in MFROM could be a problem - some of
my systems get 400000+ connections a day, and I depend on rejecting
efficiently in envelope.

If I was going to do complex processing at envelope, I would feed everything
known so far about the sender: MXs, NSs, As, PTRs, HELO, MFROM, RCPT, even hour
of day (I've noticed spam tends to prefers certain time slots) to a bayesian
filter.

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

scott at kitterman

Oct 10, 2008, 12:53 PM

Post #4 of 36 (3656 views)
Permalink

Re: throwaway domains and whois [In reply to]

This one isn't free, but maybe something like this would help:

http://www.support-intelligence.com/dob/

Scott K

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

dmquigg-spf at yahoo

Oct 10, 2008, 7:16 PM

Post #5 of 36 (3650 views)
Permalink

Re: throwaway domains and whois [In reply to]

At 01:44 PM 10/10/2008 -0400, Stuart D. Gathman wrote:

>My SPF based reputation system is working really well.
>... The greylisting
>has dropped that to about 2 per day per user - but growing.
>
>The surviving spammers have evolved. ...
>Their spam software has a state machine which retries after greylisting,
>just like a real sender. ... As this new breed of
>spam software gets adopted, the spam that makes it through grows.

Greylisting is like a weak antibiotic. The survivors become the dominant strain and now the whole population is resistant to the treatment.

>If you do a whois on these throwaway domains, the registrant is always
>a front company, like "Protected Domain Services" or "Domains by Proxy".
>My idea is to start tracking reputation by domain registrant.
>I would like to reject all mail from the above two registrants, for instance,
>regardless of domain name du jour. What are the restrictions on using
>whois? Can I simply script running it for every domain, with a cache
>to remember results? Or will reigstrars start blocking me for abuse?
>What is the most efficient way to obtain whois info on a domain?

You might want to check with the folks at dnsstuff.com. I use their service for manual whois queries, but I assume they do the real query in an automated fashion. They probably have an agreement with some registrar. Instead of trying to get an agreement with a registrar, you might work out something with dnsstuff.com to relay your queries.

My worry about your proposal is that you might lose mail from legitimate senders who happen to use the same "front company". I think at one time I was considering Domains by Proxy, but then my own registrar Domains Made Easy offered the privacy service, and I used it.

How about sending a challenge (via SMTP reject) on mail that would otherwise be greylisted? The folks at Stanford have an excellent discussion of the topic in their paper on whitelisting. See the paper by Erikson, et.al. at http://www.ceas.cc/

-- Dave

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

michael at talamasca

Oct 13, 2008, 5:43 AM

Post #6 of 36 (3635 views)
Permalink

Re: throwaway domains and whois [In reply to]

On Fri, 10 Oct 2008, David MacQuigg wrote:
> Greylisting is like a weak antibiotic. The survivors become the
> dominant strain and now the whole population is resistant to the
> treatment.

You could say that about most anti-spam methods. Many people do about
SPF itself. And the botnet epidemic can be considered the spammer attempt
at "immunity" to IP blacklists.

Greylisting has two effects:

1. Sloppy spamware that doesn't re-queue deferred messages is completely
blocked.

2. To get through, the spammer needs to use the same IP address after the
timeout. He cannot jump IPs to avoid blacklists during the timeout
period, except by starting over with a new timeout.

We knew all along that #1 wouldn't last. The original Greylisting
whitepaper states that the data suggested a mere 1 minute lockdown would
be nearly as effective as longer ones. But because they expected spamware
to improve, they recommended a 1 hour timeout.

#2 still has potential. By combining sufficiently slow greylisting with
a sufficiently fast IP reputation service, spammers can still be
significantly hampered no matter how clever they are.

---- Michael Deutschmann <michael [at] talamasca>

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

dmquigg-spf at yahoo

Oct 13, 2008, 12:59 PM

Post #7 of 36 (3643 views)
Permalink

Re: throwaway domains and whois [In reply to]

At 05:43 AM 10/13/2008 -0700, you wrote:

>On Fri, 10 Oct 2008, David MacQuigg wrote:
>> Greylisting is like a weak antibiotic. The survivors become the
>> dominant strain and now the whole population is resistant to the
>> treatment.
>
>You could say that about most anti-spam methods. Many people do about
>SPF itself. And the botnet epidemic can be considered the spammer attempt
>at "immunity" to IP blacklists.
>
>Greylisting has two effects:
>
>1. Sloppy spamware that doesn't re-queue deferred messages is completely
>blocked.
>
>2. To get through, the spammer needs to use the same IP address after the
>timeout. He cannot jump IPs to avoid blacklists during the timeout
>period, except by starting over with a new timeout.
>
>We knew all along that #1 wouldn't last. The original Greylisting
>whitepaper states that the data suggested a mere 1 minute lockdown would
>be nearly as effective as longer ones. But because they expected spamware
>to improve, they recommended a 1 hour timeout.
>
>#2 still has potential. By combining sufficiently slow greylisting with
>a sufficiently fast IP reputation service, spammers can still be
>significantly hampered no matter how clever they are.

Good points all. I hadn't considered the possibility of making the timeout so long that the IP blacklists have time to respond. This could make greylisting a viable long-term strategy. Maybe Stuart could try this with his greylisting setup, and let us know if it works.

I have heard, but I don't have direct experience, that large senders sometimes resend from a different transmitter. Maybe after a number of retries, they re-use an address already on the pending list. The message then gets through, but it could be delayed by days.

I didn't see any response to my suggestion of sending a challenge to the author via SMTP reject. That still seem like the better alternative, and one that doesn't have the "weak antibiotic" problem. Also the author of the message has a guaranteed way to get his message through. The Stanford study showed that many challenges are never answered, but at least then we know that the message was not very important to the author.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

vesely at tana

Oct 13, 2008, 11:39 PM

Post #8 of 36 (3631 views)
Permalink

Re: throwaway domains and whois [In reply to]

Scott Kitterman wrote:
> This one isn't free, but maybe something like this would help:
>
> http://www.support-intelligence.com/dob/

Given that Stuart wrote:
>> Each freshly registered domain has a valid and reasonable SPF record.
>> All identical - usually "v=spf1 a -all", and all such spam is SPF pass.

SPF-wise, to block domains that are newer than 5 days looks like the
perfect answer. However, I still don't know why spammers stop using
domains after a few days. I recall finding that spammers domain had
been shut down within a few days; however, that was from manual
lookups so I have no statistically relevant data. Is that due to
individual providers' initiative, possibly driven by users'
complaints, or are there some more structured mechanisms that operate
within those 5 days?

--
TIA
Ale

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

scott at kitterman

Oct 14, 2008, 3:00 AM

Post #9 of 36 (3640 views)
Permalink

Re: throwaway domains and whois [In reply to]

On Tue, 14 Oct 2008 08:39:40 +0200 Alessandro Vesely <vesely [at] tana> wrote:
>Scott Kitterman wrote:
>> This one isn't free, but maybe something like this would help:
>>
>> http://www.support-intelligence.com/dob/
>
>Given that Stuart wrote:
>>> Each freshly registered domain has a valid and reasonable SPF record.
>>> All identical - usually "v=spf1 a -all", and all such spam is SPF pass.
>
>SPF-wise, to block domains that are newer than 5 days looks like the
>perfect answer. However, I still don't know why spammers stop using
>domains after a few days. I recall finding that spammers domain had
>been shut down within a few days; however, that was from manual
>lookups so I have no statistically relevant data. Is that due to
>individual providers' initiative, possibly driven by users'
>complaints, or are there some more structured mechanisms that operate
>within those 5 days?

IIRC it's because of payment timing. You can use a domain somehow for up to
4 days before you pay for it.

Scott K

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

iane at sussex

Oct 14, 2008, 3:30 AM

Post #10 of 36 (3631 views)
Permalink

Re: throwaway domains and whois [In reply to]

--On 13 October 2008 12:59:52 -0700 David MacQuigg <dmquigg-spf [at] yahoo>
wrote:

>
> I didn't see any response to my suggestion of sending a challenge to the
> author via SMTP reject. That still seem like the better alternative, and
> one that doesn't have the "weak antibiotic" problem. Also the author of
> the message has a guaranteed way to get his message through. The
> Stanford study showed that many challenges are never answered, but at
> least then we know that the message was not very important to the author.
>

That would only be sensible if you could guarantee that the sender address
didn't belong to a real person who isn't the sender of the email.

--
Ian Eiloart
IT Services, University of Sussex
x3148

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

dmquigg-spf at yahoo

Oct 14, 2008, 7:55 AM

Post #11 of 36 (3633 views)
Permalink

Re: throwaway domains and whois [In reply to]

At 11:30 AM 10/14/2008 +0100, Ian Eiloart wrote:
>--On 13 October 2008 12:59:52 -0700 David MacQuigg <dmquigg-spf [at] yahoo> wrote:
>>
>>I didn't see any response to my suggestion of sending a challenge to the
>>author via SMTP reject. That still seem like the better alternative, and
>>one that doesn't have the "weak antibiotic" problem. Also the author of
>>the message has a guaranteed way to get his message through. The
>>Stanford study showed that many challenges are never answered, but at
>>least then we know that the message was not very important to the author.
>
>That would only be sensible if you could guarantee that the sender address didn't belong to a real person who isn't the sender of the email.

That is guaranteed by the use of an SMTP reject, instead of a bounce message to the alleged sender.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

michael at talamasca

Oct 14, 2008, 10:39 AM

Post #12 of 36 (3628 views)
Permalink

Re: throwaway domains and whois [In reply to]

On Mon, 13 Oct 2008, David MacQuigg wrote:
> I didn't see any response to my suggestion of sending a challenge to
> the author via SMTP reject. That still seem like the better alternative,
> and one that doesn't have the "weak antibiotic" problem.

Uh, of all the anti-spam methods, C/R is probably the poster child for
the antibiotic problem.

When a single person uses a homebuilt C/R, it's extremely effective, with
the only false negatives being when the backscattering variant is used and
angry backscatter victims deliberately confirm forged e-mail to punish the
C/R user.

But if it's widely deployed, many people will be using identical C/R
software, and spamware will be extended to handshake with it. Captcha is
not an impregnable obstacle.

We just haven't seen this because most people competent enough to
implement C/R reject it out of hand due to the backscatter problem, which
has clamped usage to the point that spammers haven't had to adapt.

C/R in the 550 message, as you propose, avoids the backscatter problem
but not the antibiotic problem. It also has the issue that some senders
may not ever see the 550 text.

---- Michael Deutschmann <michael [at] talamasca>

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

sandy at cypressintegrated

Oct 14, 2008, 11:27 AM

Post #13 of 36 (3638 views)
Permalink

Re[2]: throwaway domains and whois [In reply to]

> It also has the issue that some senders may not ever see the 550
> text.

Ya think?

Put it this way: you might have a FUSSP if you...

...conclude that unresponded challenges mean "the message was not very
important to the author."

--Sandy

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

stuart at bmsi

Oct 14, 2008, 11:31 AM

Post #14 of 36 (3625 views)
Permalink

Re: throwaway domains and whois [In reply to]

On Tue, 14 Oct 2008, Michael Deutschmann wrote:

> C/R in the 550 message, as you propose, avoids the backscatter problem
> but not the antibiotic problem. It also has the issue that some senders
> may not ever see the 550 text.

I have extensive experience putting text in the 4xx and 5xx responses. Almost
all users do see the text - only once was there a mail system so braindead
as to remove it. The problem is, almost nobody reads it. They see something
about "failed delivery", and their eyes glaze over and they never actually read
the details, and then they delete it before calling support so that there is no
evidence left of what really happened other than "my mail isn't getting
through".

I also log the 4xx and 5xx responses, of course, so if I can only extract
an approximate time or email address, I can find it.

Occassionally, maybe a dozen times in last few years out of millions of
messages, someone with half a brain will read the text, act on the
advice, and thank me via email to postmaster for the information.

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

sandy at cypressintegrated

Oct 14, 2008, 11:44 AM

Post #15 of 36 (3626 views)
Permalink

Re[2]: throwaway domains and whois [In reply to]

> The problem is, almost nobody reads it.

I agree and expect that that's what Michael actually meant by "see".

--Sandy

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

dmquigg-spf at yahoo

Oct 14, 2008, 4:12 PM

Post #16 of 36 (3623 views)
Permalink

Re: throwaway domains and whois [In reply to]

At 01:44 PM 10/10/2008 -0400, Stuart D. Gathman wrote:

>The other filters were already cutting the spam from about
>2000 per day per user to less than 20 per day per user. The greylisting
>has dropped that to about 2 per day per user - but growing.

What is your timeout window for the sender to try again (min and max)?

What do you think of Michael's suggestion to make the minimum really long? Will that avoid the "weak antibiotic" problem?

-- Dave

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

stuart at bmsi

Oct 14, 2008, 5:00 PM

Post #17 of 36 (3623 views)
Permalink

Re: throwaway domains and whois [In reply to]

On Tue, 14 Oct 2008, David MacQuigg wrote:

> >The other filters were already cutting the spam from about
> >2000 per day per user to less than 20 per day per user. The greylisting
> >has dropped that to about 2 per day per user - but growing.
>
> What is your timeout window for the sender to try again (min and max)?

5 minutes min - because a lot of legit senders retry in 5 min
6 hours max - because several legit senders were waiting 6 hours to retry

> What do you think of Michael's suggestion to make the minimum really long?

Not feasible. Users have enough trouble accepting a 5 minute delay.
"Ok - I sent you an email. Did you get it? No. Hmmm. Let me try again.
Did you get that one. No? Is your email working? ... "

> Will that avoid the "weak antibiotic" problem?

Only if the delay is 4 days - to make the spammers actually pay for
their throwaway domains. That certainly won't fly.

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

michael at talamasca

Oct 15, 2008, 12:20 AM

Post #18 of 36 (3620 views)
Permalink

Re: throwaway domains and whois [In reply to]

On Tue, 14 Oct 2008, Stuart D. Gathman wrote:
> On Tue, 14 Oct 2008, David MacQuigg wrote:
> > What do you think of Michael's suggestion to make the minimum really long?
>
> Not feasible. Users have enough trouble accepting a 5 minute delay.
> "Ok - I sent you an email. Did you get it? No. Hmmm. Let me try again.
> Did you get that one. No? Is your email working? ... "

Five minutes is way too short. If you really can't increase the minimum
beyond that, you'll have to resign yourself to the eventual uselessness of
your greylisting implementation. All you'll catch are the spammers with
old software, and perhaps ones that apply a one-size-fits-all retry
schedule that does not exploit your wide window.

> > Will that avoid the "weak antibiotic" problem?
>
> Only if the delay is 4 days - to make the spammers actually pay for
> their throwaway domains. That certainly won't fly.

Four days is obviously unreasonable. But a shorter time, on the order of
hours, should be enough for automated blacklists (eg: Spamcop, XBL,
UCEPROTECT) to react to the spammer's initial run, spiking his attempt at
a follow-up run to hit the greylist-protected addresses. Remember, he has
to use the *same zombies* for the follow-up run in order to pass
greylisting.

The optimal lockdown time would depend on the reaction time of the
blacklists you use. I'm not sure what that is, but it would be
straightforward enough to research.

---- Michael Deutschmann <michael [at] talamasca>

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

stuart at bmsi

Oct 15, 2008, 7:37 AM

Post #19 of 36 (3622 views)
Permalink

Re: throwaway domains and whois [In reply to]

On Wed, 15 Oct 2008, Michael Deutschmann wrote:

> The optimal lockdown time would depend on the reaction time of the
> blacklists you use. I'm not sure what that is, but it would be
> straightforward enough to research.

In our case, we use our own reputation based blacklist. There are
too many false positives with Spam Haus and the like. The greylisting
is serving its intended purpose of blocking the spammers with old
software.

I am working on tracking reputation of NS servers. Unlike whois, that info
is readily available and might help with throwaway domains.

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

michael at talamasca

Oct 15, 2008, 4:40 PM

Post #20 of 36 (3616 views)
Permalink

Re: throwaway domains and whois [In reply to]

On Wed, 15 Oct 2008, Stuart D. Gathman wrote:
> In our case, we use our own reputation based blacklist. There are
> too many false positives with Spam Haus and the like.

Really? I'd have thought using Spamhaus ZEN would be quite safe, since
even if there is an FP, you'd be among thousands of domains the sender
can't reach. You can expect the sender to realize he has a *major
problem* before he even composes a mail to one of your users....

Anyhow, it seems you need the social aspect of a public blacklist. A
private blacklist is vulnerable to "spread spectrum" use of zombies -- the
spammer could use a different, fresh, zombie for each attempt at one of
your mailboxes.

But it's not possible to spam the whole world without re-using IP
addresses, so there is a good chance that any IP used to spam you has
already hit a public blacklist's spamtrap (or will hit it soon, hence the
value of a long greylist lockdown).

---- Michael Deutschmann <michael [at] talamasca>

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

stuart at bmsi

Oct 15, 2008, 5:48 PM

Post #21 of 36 (3618 views)
Permalink

Re: throwaway domains and whois [In reply to]

On Wed, 15 Oct 2008, Michael Deutschmann wrote:

> On Wed, 15 Oct 2008, Stuart D. Gathman wrote:
> > In our case, we use our own reputation based blacklist. There are
> > too many false positives with Spam Haus and the like.
>
> Really? I'd have thought using Spamhaus ZEN would be quite safe, since
> even if there is an FP, you'd be among thousands of domains the sender
> can't reach. You can expect the sender to realize he has a *major
> problem* before he even composes a mail to one of your users....

Right there is the problem. The sender is clueless and incompetent,
and has no idea how to solve the problem. They can't even configure
a valid HELO. But they are the overseas partner of my customer, and their mail
has to go through - but I still have to block all the other garbage. This
level of filtering is inherently custom.

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

michael at talamasca

Oct 15, 2008, 6:46 PM

Post #22 of 36 (3616 views)
Permalink

Re: throwaway domains and whois [In reply to]

On Wed, 15 Oct 2008, Stuart D. Gathman wrote:
> But they are the overseas partner of my customer, and their mail
> has to go through - but I still have to block all the other garbage. This
> level of filtering is inherently custom.

Still, replacing public blacklists with custom ones isn't necessarily an
improvement. A public blacklist can be effective with a shorter expiry
time than a private one, which means it is less likely to FP when IP
addresses change ownership.

It's better to use public blacklists, but maintain a private whitelist to
override if necessary.

You could set your system up to automatically whitelist any address your
customer mails to. This will protect you if someone your customer is
already talking to suddenly gets listed.

One other thing -- you mentioned the "evolved" spammers are evading your
content-filtering by sending images. A simple no-binary-attachments
content filter would stop those cold. That's too harsh for most sites,
let alone your FP-phobic environment, but if it was only applied to
never-seen-before domains, it might work.

---- Michael Deutschmann <michael [at] talamasca>

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

dmquigg-spf at yahoo

Oct 16, 2008, 9:20 AM

Post #23 of 36 (3610 views)
Permalink

Re: clueless senders [In reply to]

At 08:48 PM 10/15/2008 -0400, Stuart D. Gathman wrote:

>On Wed, 15 Oct 2008, Michael Deutschmann wrote:
>
>> On Wed, 15 Oct 2008, Stuart D. Gathman wrote:
>> > In our case, we use our own reputation based blacklist. There are
>> > too many false positives with Spam Haus and the like.
>>
>> Really? I'd have thought using Spamhaus ZEN would be quite safe, since
>> even if there is an FP, you'd be among thousands of domains the sender
>> can't reach. You can expect the sender to realize he has a *major
>> problem* before he even composes a mail to one of your users....

I use the most conservative IP blacklists (CBL and Spamhaus) as my initial filter, basically blocking addresses that are currently and heavily being abused. I call this my "Block DoS" stage, and it drops about 2/3 of the attempted connections, sending an SMTP reject with a link, so the sender can check the blacklist.

My mailflow is currently too small to get any good stats on false rejects from the Block DoS stage, but I'm not worried about it. As Michael says, if a sender is on one of these lists, he will know he has a problem, even if he doesn't know how to fix it.

>Right there is the problem. The sender is clueless and incompetent,
>and has no idea how to solve the problem. They can't even configure
>a valid HELO.

I would change the wording a little. The sender is not an email admin, and is not going to get involved in even the simplest technical problem, like setting a valid HELO.

I think we can expect the sender to solve the problem, however. There are plenty of services, even free services, that will relay your outgoing mail, keep their transmitters clean, never get on an IP blacklist, and give you the best possible assurance that your mail will be delivered. I use yahoo.com and controlledmail.com.

I am willing and able to fix any problems with my own transmitter, but I would rather spend my time on other projects. I use my own transmitter for occasional tests, never for mail that needs reliable delivery. Even with *no known problems*, now or in the past, I still have difficulty getting delivery to services that treat mail from unknown transmitters as potential spam.

I think the real challenge is how do we motivate senders to fix a problem, when our only communication with them is an SMTP reject. Maybe we need a webpage with a list of services they can call for help.

-- Dave

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

stuart at bmsi

Oct 16, 2008, 4:02 PM

Post #24 of 36 (3610 views)
Permalink

Re: throwaway domains and whois [In reply to]

On Wed, 15 Oct 2008, Michael Deutschmann wrote:

> You could set your system up to automatically whitelist any address your
> customer mails to. This will protect you if someone your customer is
> already talking to suddenly gets listed.

Already do that. And use honeypots for blacklisting (and content filter
training) - as well as behavioural triggers, like 4 forged MFROMs in a row from
an IP gets it blacklisted, and 4 nonexistent RCPTs from a MFROM gets the IP
blacklisted (and the MFROM blacklisted if SPF Pass). The blacklist (and
whitelist) is self-maintaining. Paying for a public blacklist, which I then
have to make exceptions for, would not save me or the MTA any work.

One customer was already using 4 public blacklists - and still getting hundreds
of thousands of spams per day to the inbox (2 million connections per day). We
purchased a commercial spam filtering contract. As soon as I switched the
MX records, the company (who shall remain nameless) deleted our account
and threatened to sue us for crashing their mail server and denying
service to all their other customers (Duh - that's why we needed your
spam service people). We then switched to spamsoap, which had no problem with
the volume, but still let thousands of spams per day through. So I upgraded
the MTA to a low end 2.8Ghz Pentium D with 1G ram on a 5mbit cable account, and
handled it myself. The python script quickly built a blacklist and handles the
2 million connections while serving PHP pages without breaking a sweat (95%
idle). The first company must have been doing something lame like trying
to run spamassassin on each and every message. Spamsoap could be worthwile
just to cut the connection rate (they use public blacklists) - but then SPF has
to use the Received header field.

I think the next feature will be NS reputation. Unfortunately, the spammers
will simply start using throwaway domains for NS servers. But they
aren't yet.

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

vesely at tana

Oct 17, 2008, 1:47 AM

Post #25 of 36 (3609 views)
Permalink

Re: throwaway domains and whois [In reply to]

Stuart D. Gathman wrote:
> On Wed, 15 Oct 2008, Michael Deutschmann wrote:
>
>> You could set your system up to automatically whitelist any address your
>> customer mails to. This will protect you if someone your customer is
>> already talking to suddenly gets listed.

Doesn't that assume "+mx -all"? One should whitelist the IPs where
replied messages actually came from... Rather, I would whitelist the
sender and just flag the DNSBL lookup response, delaying any reaction
until responding to the RCPT command. That way it is also possible to
forget DNSBL altogether for specific recipients who don't want that
filtering.

> Already do that. And use honeypots for blacklisting (and content filter
> training) - as well as behavioural triggers, like 4 forged MFROMs in a row from
> an IP gets it blacklisted, and 4 nonexistent RCPTs from a MFROM gets the IP
> blacklisted (and the MFROM blacklisted if SPF Pass). The blacklist (and
> whitelist) is self-maintaining.

May I ask how do you manage black list entries rehabilitation? I'm
planning to do something similar to Stockade(*), i.e. to have a decay
rate so that the probability that a listed IP gets blocked is
automatically halved every that many seconds, until it eventually
vanishes. That implies fuzzy blocking, though.

[*] http://caia.swin.edu.au/stockade/
They do it at the IP level.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads