Mailing List Archive: SPF: Help

SPF prevents mail from being delivered to Hotmail

Index | Next | Previous | View Threaded

markwolk at gmail

Oct 26, 2006, 8:03 PM

Post #1 of 8 (16016 views)
Permalink

SPF prevents mail from being delivered to Hotmail

In the Hotmail saga, here is my latest discovery.

As we all know, Hotmail receives email very unreliably. Many
legitimate mails end up in the junk mail folder, many others simply
disappear.

Mail to Hotmail seemed to work better depending on providers used.
I.e. mails from my domains sent though ControlledMail would arrive
into Hotmail's inbox perfectly; whereas the same mails sent through
Tuffmail would end up in Hotmail's junk mail folder.

As an experiment, I have totally removed the SPF record of one of my
domains, guidedvacation.com. I waited ca 2 weeks (knowing that
Hotmails caches SPF records). And here is my experiment of today:

- mail from guidedvacation.com sent through Tuffmail arrives perfectly
in Hotmail's inbox

- mail from any other of my domains, that still have an SPF record,
sent through Tuffmail arrives in Hotmail's junk mail folder

In other terms having an SPF record is harmful if you want to send
mail to Hotmail. Removing the SPF record will improve reliability of
email delivery to Hotmail.

SPF also seems to be useless with joejob prevention. Spammers use my
SPF-protected domains more and more to send their spam. I receive
thousands of returned mails which I have never sent before, and I have
constantly to increase my filters so they don't flood my inbox. So SPF
does not seem to do the job it was designed to, and I wonder, at the
end, what is its use now?

-------
Archives at http://archives.listbox.com/spf-help/current/ or
http://www.gossamer-threads.com/lists/spf/help/ (easier to search)
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?member_id=1311530&user_secret=cbdbbc81

steve at teamITS

Oct 26, 2006, 10:48 PM

Post #2 of 8 (15200 views)
Permalink

RE: SPF prevents mail from being delivered to Hotmail [In reply to]

Mark Wolk <mailto:markwolk[at]gmail.com> wrote on Thursday, October 26,
2006 10:04 PM:

> In other terms having an SPF record is harmful if you want to send
> mail to Hotmail. Removing the SPF record will improve reliability of
> email delivery to Hotmail.

I can't speak directly to the accuracy of your claim since I
haven't tried it, however, 1) I have a client whose domain doesn't use
SPF and has trouble sending to HotMail, and 2) HotMail, being a
Microsoft property, likely doesn't use SPF, it uses SPF records to
perform Sender ID calculations.

Did you also set up a Sender ID record for this domain and test
with that? Is there an SPF record set up for the HELO greeting of the
outgoing (Tuffmail ?) mail server?

> SPF also seems to be useless with joejob prevention. Spammers use my
> SPF-protected domains more and more to send their spam. I receive
> thousands of returned mails which I have never sent before, and I have
> constantly to increase my filters so they don't flood my inbox. So SPF
> does not seem to do the job it was designed to, and I wonder, at the
> end, what is its use now?

This is sort of a common misconception I think. SPF will only
work well to block forged e-mails when every or most every mail server
checks for it. If the receiving mail server doesn't check for SPF, then
SPF has no effect. The whole
SPF-is-merging-with-Sender-ID-oops-no-it's-not thing slowed things down
IMHO, and even now I don't think many mail servers natively support SPF
except via add-ons. So I don't think it's caught on as fast as everyone
would like. I think a push to get mail servers to check for SPF would
be a really good idea.

Being on this list for a while now it is apparent to me that a
"single mail server" setup is fairly easy to get right with SPF but more
complicated setups are often set up incorrectly by SPF newbies. Having
an "include" that doesn't resolve will render an SPF record useless, for
example.

- Steve Yates
- ITS, Inc.
- Computers will help us to solve problems we wouldn't have without
them.

~ Taglines by Taglinator - www.srtware.com ~

-------
Archives at http://archives.listbox.com/spf-help/current/ or
http://www.gossamer-threads.com/lists/spf/help/ (easier to search)
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?member_id=1311530&user_secret=cbdbbc81

wmark at markwolk

Oct 27, 2006, 12:07 AM

Post #3 of 8 (15231 views)
Permalink

Re: SPF prevents mail from being delivered to Hotmail [In reply to]

On 10/27/06, Steve Yates <steve[at]teamits.com> wrote:
> Mark Wolk <mailto:markwolk[at]gmail.com> wrote on Thursday, October 26,
> 2006 10:04 PM:
>
> > In other terms having an SPF record is harmful if you want to send
> > mail to Hotmail. Removing the SPF record will improve reliability of
> > email delivery to Hotmail.
>
> I can't speak directly to the accuracy of your claim since I
> haven't tried it, however, 1) I have a client whose domain doesn't use
> SPF and has trouble sending to HotMail, and 2) HotMail, being a
> Microsoft property, likely doesn't use SPF, it uses SPF records to
> perform Sender ID calculations.
>
> Did you also set up a Sender ID record for this domain and test
> with that? Is there an SPF record set up for the HELO greeting of the
> outgoing (Tuffmail ?) mail server?
>
> > SPF also seems to be useless with joejob prevention. Spammers use my
> > SPF-protected domains more and more to send their spam. I receive
> > thousands of returned mails which I have never sent before, and I have
> > constantly to increase my filters so they don't flood my inbox. So SPF
> > does not seem to do the job it was designed to, and I wonder, at the
> > end, what is its use now?
>
> This is sort of a common misconception I think. SPF will only
> work well to block forged e-mails when every or most every mail server
> checks for it. If the receiving mail server doesn't check for SPF, then
> SPF has no effect. The whole
> SPF-is-merging-with-Sender-ID-oops-no-it's-not thing slowed things down
> IMHO, and even now I don't think many mail servers natively support SPF
> except via add-ons. So I don't think it's caught on as fast as everyone
> would like. I think a push to get mail servers to check for SPF would
> be a really good idea.
>
> Being on this list for a while now it is apparent to me that a
> "single mail server" setup is fairly easy to get right with SPF but more
> complicated setups are often set up incorrectly by SPF newbies. Having
> an "include" that doesn't resolve will render an SPF record useless, for
> example.
>
> - Steve Yates
> - ITS, Inc.
> - Computers will help us to solve problems we wouldn't have without
> them.
>
> ~ Taglines by Taglinator - www.srtware.com ~
>
> -------
> Archives at http://archives.listbox.com/spf-help/current/ or
> http://www.gossamer-threads.com/lists/spf/help/ (easier to search)
> To unsubscribe, change your address, or temporarily deactivate your subscription,
> please go to http://v2.listbox.com/member/?&

Steve,

You wrote: *****Did you also set up a Sender ID record for this domain
and test with that? Is there an SPF record set up for the HELO
greeting of the outgoing (Tuffmail ?) mail server?*****

No, I have no sender ID set up. For all my domains that use SPF, I
have an include to Tuffmail and I am sure the SPF record is correct.
Tuffmail does have an SPF record; not sure if they have Sender ID.

You are correct; NOT having an SPF record is probably not the only
secret behind being able to successfully send mail to Hotmail and,
conversely, the existence of an SPF record does not mean that email
from a given domain will not be delivered to Hotmail. In my case, and
in combination with Tuffmail, it worked. I thought it was interesting
to point out that, in this particular example, the removal of my SPF
record had a very obvious link with the ability to successfully send
mail to Hotmail.

As for the other considerations regarding general SPF usefulness, I am
personally not technical enough to understand all of it. All what I
see is that SPF comes from a good idea and a good cause, but it's not
good enough. Hopefully some minds better than mine are working on
improving it.

Mark Wolk

-------
Archives at http://archives.listbox.com/spf-help/current/ or
http://www.gossamer-threads.com/lists/spf/help/ (easier to search)
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?member_id=1311530&user_secret=cbdbbc81

nobody at xyzzy

Oct 27, 2006, 1:14 AM

Post #4 of 8 (15186 views)
Permalink

Re: SPF prevents mail from being delivered to Hotmail [In reply to]

Mark Wolk wrote:

> Tuffmail does have an SPF record; not sure if they have Sender ID.

I've found http://www.tuffmail.com/faq.php#spf - they offer a policy
for inclusion at customer-spf.mxes.net

With `nslookup -q=txt customer-spf.mxes.net` you can see what they
have: text = "v=spf1 ip4:216.86.168.0/24 ip4:205.237.194.0/26
ip4:205.237.194.64/27 ip4:216.86.160.224/27 ip4:216.86.167.192/26"

No "spf2.0/pra" or anything else mentioning PRA (= Sender ID). If
their list of sending IPs would be incomplete it would be bad, but
you've probably tested that it's okay.

> in combination with Tuffmail, it worked. I thought it was interesting
> to point out that, in this particular example, the removal of my SPF
> record had a very obvious link with the ability to successfully send
> mail to Hotmail.

Yes, but we're far from understanding _why_ :-)

> As for the other considerations regarding general SPF usefulness,
> I am personally not technical enough to understand all of it.

If you followed the tuffmail advice with "~all" (SOFTFAIL) at the end
of your policies don't expect too much, that's mainly for testing.

For an effect wrt forgeries you'd need "-all" (FAIL). And then it
can take some time until the spammers note that abusing your addresses
is a stupid plan. Two years ago "my" spammer needed four months to
figure this out. Last year it took him or her two weeks to verify
that abusing my addresses isn't what he or she wants. This year the
misdirected bounces I get are minimal (about as much as 2003).

Frank

-------
Archives at http://archives.listbox.com/spf-help/current/ or
http://www.gossamer-threads.com/lists/spf/help/ (easier to search)
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?member_id=1311530&user_secret=cbdbbc81

markwolk at gmail

Oct 27, 2006, 2:51 AM

Post #5 of 8 (15174 views)
Permalink

Re: Re: SPF prevents mail from being delivered to Hotmail [In reply to]

On 10/27/06, Frank Ellermann <nobody[at]xyzzy.claranet.de> wrote:
> Mark Wolk wrote:
>
> > Tuffmail does have an SPF record; not sure if they have Sender ID.
>
> I've found http://www.tuffmail.com/faq.php#spf - they offer a policy
> for inclusion at customer-spf.mxes.net
>
> With `nslookup -q=txt customer-spf.mxes.net` you can see what they
> have: text = "v=spf1 ip4:216.86.168.0/24 ip4:205.237.194.0/26
> ip4:205.237.194.64/27 ip4:216.86.160.224/27 ip4:216.86.167.192/26"
>
> No "spf2.0/pra" or anything else mentioning PRA (= Sender ID). If
> their list of sending IPs would be incomplete it would be bad, but
> you've probably tested that it's okay.
>
> > in combination with Tuffmail, it worked. I thought it was interesting
> > to point out that, in this particular example, the removal of my SPF
> > record had a very obvious link with the ability to successfully send
> > mail to Hotmail.
>
> Yes, but we're far from understanding _why_ :-)
>
> > As for the other considerations regarding general SPF usefulness,
> > I am personally not technical enough to understand all of it.
>
> If you followed the tuffmail advice with "~all" (SOFTFAIL) at the end
> of your policies don't expect too much, that's mainly for testing.
>
> For an effect wrt forgeries you'd need "-all" (FAIL). And then it
> can take some time until the spammers note that abusing your addresses
> is a stupid plan. Two years ago "my" spammer needed four months to
> figure this out. Last year it took him or her two weeks to verify
> that abusing my addresses isn't what he or she wants. This year the
> misdirected bounces I get are minimal (about as much as 2003).
>
> Frank
>
>
> -------
> Archives at http://archives.listbox.com/spf-help/current/ or
> http://www.gossamer-threads.com/lists/spf/help/ (easier to search)
> To unsubscribe, change your address, or temporarily deactivate your subscription,
> please go to http://v2.listbox.com/member/?&
>

Frank,

You wrote: *****If you followed the tuffmail advice with "~all"
(SOFTFAIL) at the end of your policies don't expect too much, that's
mainly for testing.*****

The thousands of returned junk mails I mentioned had been sent by
spammers from domains of mine that have "v=spf1 -all", i.e. which
should never send any email at all.

I am still reluctant in using "-all" on my domains which are
authorised to send mail, as some incorrectly configured recipients
reject them.

Mark Wolk

-------
Archives at http://archives.listbox.com/spf-help/current/ or
http://www.gossamer-threads.com/lists/spf/help/ (easier to search)
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?member_id=1311530&user_secret=cbdbbc81

nobody at xyzzy

Oct 27, 2006, 4:32 AM

Post #6 of 8 (15239 views)
Permalink

Re: SPF prevents mail from being delivered to Hotmail [In reply to]

Mark Wolk wrote:

> The thousands of returned junk mails I mentioned had been sent by
> spammers from domains of mine that have "v=spf1 -all", i.e. which
> should never send any email at all.

Then that spammer and the receivers bouncing this crap instead of
rejecting it as proposed by your "v=spf1 -all" need a clue. What I
did last year was to report the spam with spamcop. For that you'd
need the "quick report" feature (it's free, but only available for
reporters with a minimal rate of false positives)

And you can't report "thousands" per day, there's a limit as I've
found out two weeks ago, something like say 3000/day should work.

Probably no recipe for you, but it did the trick for me (as far as
I can judge it - of course "my" spammer fails to inform me about
s/h/it's "business decisions" wrt abusing my @xyzzy vanity domain)

Another plan could be to block abusive bouncers. Or add the SCBL
to your mix of blocklists (scoring, not exclusively, there are too
many dubious entries). Or rearrange your mail infrastructure in a
way allowing to use some kind of BATV.

For domains never sending mail as in "v=spf1 -all" that's kind of
simple, reject anything sent to these domains, and while you're at
it note such senders as hopeless rejecting also all their other
mails, until they cleaned up their act and reject SPF FAIL instead
of forwarding the spam (disguised as wannabe-bounces) to you.

There's absolutely no way which won't hurt for some corner cases
like "oops, I got my SPF policy wrong", or whatever else you try.

> I am still reluctant in using "-all" on my domains which are
> authorised to send mail, as some incorrectly configured recipients
> reject them.

Actually if they reject them it's working as designed. I had this
once in now 30 months: The Tech-C address in a whois database was
"protected" by a weird crypto-local-part[at]registry.example address.
And forwarded to the real Tech-C elsewhere, supporting SPF. And of
course the IP of this registry isn't permitted in my sender policy.

Therefore this hop supporting SPF rejected the mail, the registry
sent the error text as bounce to me, it contained the real address.
I sent my mail again to the real address, that worked (as expected).

Apparently it's a very rare case where "receivers" get this wrong,
here an unlucky combination of registry + next hop + user (Tech-C).

Hard to judge who got it wrong, the registry is free to ignore SPF,
the next hop is free to respect it, and probably the Tech-C had no
clue what that's all about. But no real problem in this case.

It could get ugly if the next hop does NOT reject an SPF FAIL, but
only marks it as potential spam. The clueless user would probably
delete all mails tagged as potential spam without checking if that
is okay, and then the mail is lost.

SPF is based on "reject FAIL", other uses are extremely dangerous,
minus the obvious "PASS from a white-listed sender" scenarios.

Frank

-------
Archives at http://archives.listbox.com/spf-help/current/ or
http://www.gossamer-threads.com/lists/spf/help/ (easier to search)
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?member_id=1311530&user_secret=cbdbbc81

alex at ergens

Oct 27, 2006, 7:12 AM

Post #7 of 8 (15179 views)
Permalink

Re: SPF prevents mail from being delivered to Hotmail [In reply to]

On Fri, Oct 27, 2006 at 04:03:34PM +1300, Mark Wolk wrote:

> As we all know, Hotmail receives email very unreliably. Many
> legitimate mails end up in the junk mail folder, many others simply
> disappear.

A hunch: they guess, and sometimes they guess wrong or very wrong.
(I mean something different from, but similar to, spamassassin)

> In other terms having an SPF record is harmful if you want to send
> mail to Hotmail. Removing the SPF record will improve reliability of
> email delivery to Hotmail.

Put an SPF record on the domain, and they don't have to guess anymore.

Of course, if you send from a host you did not authorize, or if there's
an error in your record, things are to be expected to fail.

It's a shame they don't reject at the border, it's fortunate that they
don't bounce all that junk anymore. What's left is to put it in the
spamfolder, or even delete it. Isn't this a user setting by the way?

Then again, hotmail doesn't do SPF, they do senderID. That's not
"SPF and a bit more", that's "almost-but-not-quite SPF and some else".
This could also be a source of problems.

- ptr is reported to be problematic
- DNS caching seems, er..., odd

and last but certainly not least: checking of the wrong property,
RFC822 stuff, for which SPF records are not designed.

alex

-------
Archives at http://archives.listbox.com/spf-help/current/ or
http://www.gossamer-threads.com/lists/spf/help/ (easier to search)
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?member_id=1311530&user_secret=cbdbbc81

scott at kitterman

Oct 27, 2006, 8:32 AM

Post #8 of 8 (15208 views)
Permalink

Re: SPF prevents mail from being delivered to Hotmail [In reply to]

On Thursday 26 October 2006 23:03, Mark Wolk wrote:
> In the Hotmail saga, here is my latest discovery.
>
> As we all know, Hotmail receives email very unreliably. Many
> legitimate mails end up in the junk mail folder, many others simply
> disappear.
>
> Mail to Hotmail seemed to work better depending on providers used.
> I.e. mails from my domains sent though ControlledMail would arrive
> into Hotmail's inbox perfectly; whereas the same mails sent through
> Tuffmail would end up in Hotmail's junk mail folder.

I don't know anything about the other providers that you use, so I can't
directly comment, but I can add some background here...

With controlledmail.com, I produce RFC compliant mail with good
forward/reverse DNS, etc. Additionally, the service is a boutique service.
I screen customers more closely that most anyone else does and am careful
about who I am willing to let send the my MTAs. This keeps the revenue curve
rather flatter than I would prefer in the short term, but will eventually, I
think, pay off. As a result, I am virtually certain the the number of
messages that could reasonably be classified as spam coming out of my MTAs is
zero (unreasonable classification I can't prevent).

So, I thinking that Hotmail may be applying some sort of additional reputation
checks to SID compliant mail that is causing less careful provider's mail to
get dumped (I'm basically agreeing with Alex on this). They do have a
service that your provider can use to see what 'spam' Hotmail have gotten
from the provider's IP range. I've signed up for it, but it has yet to show
me any data. Perhaps the other provider can find out some specifics that way
and help clear this up.

> As an experiment, I have totally removed the SPF record of one of my
> domains, guidedvacation.com. I waited ca 2 weeks (knowing that
> Hotmails caches SPF records). And here is my experiment of today:
>
> - mail from guidedvacation.com sent through Tuffmail arrives perfectly
> in Hotmail's inbox
>
> - mail from any other of my domains, that still have an SPF record,
> sent through Tuffmail arrives in Hotmail's junk mail folder

I think this is consistent with the extra reputation checks theory I suggested
above.

> In other terms having an SPF record is harmful if you want to send
> mail to Hotmail. Removing the SPF record will improve reliability of
> email delivery to Hotmail.

I'd put it slightly differently, it seems to me that this is true only for
some senders, otherwise the traffic you send through my service would be
equally unreliable. The risk is SPF plus some other unknown factor.

> SPF also seems to be useless with joejob prevention. Spammers use my
> SPF-protected domains more and more to send their spam. I receive
> thousands of returned mails which I have never sent before, and I have
> constantly to increase my filters so they don't flood my inbox. So SPF
> does not seem to do the job it was designed to, and I wonder, at the
> end, what is its use now?

One of the things with spam is that it seems to me that there are a relatively
few number of spammers doing most of this kind of stuff. At this point, SPF
is still a young technology and the preventative effect is mostly as a
deterrent. Frank and I (and others) have had the 'fortune' to have been
plagued by spammers that were deterred from forging our domains at the
current level of deployment. Clearly your spammer is less concerned.

I have seen, in the last 6 - 9 months, a significant upswing in the number of
questions we get about how to deploy SPF checking. I believe that as the
rejection rate due to SPF increases and forging SPF domains begins to have a
larger effect on deliverability, more spammers will be deterred, so I would
council patience wrt SPF. I you want, contact me offlist and we can see if I
can do anything to help you through other means in the meantime.

On Friday 27 October 2006 01:48, Steve Yates <steve[at]teamits.com> wrote:

> Mark Wolk <mailto:markwolk[at]gmail.com> wrote on Thursday, October 26,

> Did you also set up a Sender ID record for this domain and test
> with that? Is there an SPF record set up for the HELO greeting of the
> outgoing (Tuffmail ?) mail server?

Since Mark is a customer of mine, I've reviewed his SPF records in detail and
they do not have any of the usual problems that would trip up Hotmail or SID.
For his use case the v=spf1 records should be adequate.

I checked one of the Tuffmail outbound servers and it does have an SPF record
for EHLO.

On Friday 27 October 2006 05:51, Mark Wolk wrote:

> You wrote: *****If you followed the tuffmail advice with "~all"
> (SOFTFAIL) at the end of your policies don't expect too much, that's
> mainly for testing.*****
>
> The thousands of returned junk mails I mentioned had been sent by
> spammers from domains of mine that have "v=spf1 -all", i.e. which
> should never send any email at all.

Which is confirmation that your spammer is not yet intimidated by SPF.

> I am still reluctant in using "-all" on my domains which are
> authorised to send mail, as some incorrectly configured recipients
> reject them.

That is not an unreasonable concern, but my experience has been that this is a
noise level issue. I've had -all records for over two years and have had a
total of 4 e-mails rejected in that time. Here are the root causes:

1 - My ISP at the time was adding an SPF-Received header to outgoing mail and
one receiver incorrectly rejected on that (the mails were all going out with
Fail in the header because of course my client IP address wasn't in my SPF
record). So that one took two errors.

1 - One receiver had incorrectly enabled SPF checking on an internal server.
What happened was that the server used to be a border MTA, but then due to a
merger was no longer on the border for the larger company and it wasn't
turned off.

2 - Actualy forwarding related rejections. One of my consulting customers got
spun off to a new company. The old company set up mail forwarding from the
old addresses to the new addresses and the new company checked SPF. I had
two messages rejected and since the new e-mail address was in the rejection
message it took all of a minute to resend the messages to the correct
addresses.

So, in my experience the risk is pretty low. Most people I've heard from have
had similar experiences. Since you have multiple domains, you might try it
with one or two and see how it goes.

Scott K

-------
Archives at http://archives.listbox.com/spf-help/current/ or
http://www.gossamer-threads.com/lists/spf/help/ (easier to search)
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?member_id=1311530&user_secret=cbdbbc81

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com