Mailing List Archive: SPF: Discuss

Yahoo mail and Gmail policy explanation?

Index | Next | Previous | View Threaded

openspf at mart

Feb 9, 2011, 12:36 PM

Post #1 of 7 (1096 views)
Permalink

Yahoo mail and Gmail policy explanation?

Hi guys, hope you can explain something for me!

Neither yahoo mail nor gmail seem to reject incoming mail if there is no DNS
record at all for the envelope sender address, allowing spammers to just use
completely fake domains.

Does anyone know why they might choose to have that policy? Is there any
legitimate reason for allowing incoming mail from a fake domain? It doesn't
even get blocked when you turn on the spam filter!

I understand the argument that if all servers blocked mail from fake
domains, and every legitimate domain used SPF, then spammers would have a
harder time finding sender addresses to use, but doesn't this policy from
yahoo and gmail compromise the ability of SPF to contribute to the overall
reduction of spam?

Thanks in advance,
Martin

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/1311532-17d8a1ba
Modify Your Subscription: https://www.listbox.com/member/?member_id=1311532&id_secret=1311532-f2ea6ed9
Unsubscribe Now: https://www.listbox.com/unsubscribe/?member_id=1311532&id_secret=1311532-bdbb122a&post_id=20110209153649:5064C79A-348C-11E0-95B5-8496CA854E3C
Powered by Listbox: http://www.listbox.com

stuart at bmsi

Feb 9, 2011, 5:31 PM

Post #2 of 7 (1062 views)
Permalink

Re: Yahoo mail and Gmail policy explanation? [In reply to]

On Thu, 10 Feb 2011, Martin Jericho wrote:

> Neither yahoo mail nor gmail seem to reject incoming mail if there is no DNS
> record at all for the envelope sender address, allowing spammers to just use
> completely fake domains.
>
> Does anyone know why they might choose to have that policy? Is there any
> legitimate reason for allowing incoming mail from a fake domain? It doesn't
> even get blocked when you turn on the spam filter!

Believe it or not, there are countless clueless, but otherwise "legitimate"
senders who can't get basic things like HELO or MAIL FROM right (much
less SPF). Our customers get such mail rejected from their customers every
month or so. We immediately search the logs, find what brainless thing their
customer is doing, attempt to send mail to postmaster (which usually fails,
because they are after all clueless), and add a special "whitelist" (like
"accept mail from invalid domain email-clueless.com" and hope spammers don't
use it).

Free email outfits like yahoo or gmail simply can't afford to offer this
kind of email tech support. Their system has to be entirely self-serve.
Statistically routing mail to a "spam" folder is something end users
can handle on their own when it doesn't do what they want. Diagnosing
what what idiotic thing this particular sender did, and constructing
a complex whitelist to work around it is not something end users can handle.
It is *not* as simple as "whitelist this email" when the domain is invalid
or forged.

If there was a particular email, then a "Whitelist" button could run
heuristics to identify common sender problems and apply standard workarounds.
But if there was an email, then our customer would not be complaining!
The first problem is *finding* what random invalid domain the stupid
sender is trying to use in the log. In the case of my church, for instance,
they simply had a typo in their MTA config for the MAIL FROM (and
they rewrote the MAIL FROM of all client submissions with the wrong domain).

If people would just send a test email to something like spf-test [at] openspf
after configuring their server, things would be so much easier.
It would also be nice if they tested their SPF record on openspf.org
before publishing it.

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/1311532-17d8a1ba
Modify Your Subscription: https://www.listbox.com/member/?member_id=1311532&id_secret=1311532-f2ea6ed9
Unsubscribe Now: https://www.listbox.com/unsubscribe/?member_id=1311532&id_secret=1311532-bdbb122a&post_id=20110209203212:93B6D168-34B5-11E0-BF88-9E0B634668CC
Powered by Listbox: http://www.listbox.com

openspf at mart

Feb 9, 2011, 9:33 PM

Post #3 of 7 (1062 views)
Permalink

Re: Yahoo mail and Gmail policy explanation? [In reply to]

Hi Stuart,

Thanks for your explanation - although I'm not convinced about the
reasoning.

If Google and Yahoo just allow email from invalid domains because there are
too many incorrectly configured sender mail servers, wouldn't the same
argument mean they also just ignore SPF because there are so many
incorrectly configured SPF records? From the evidence I've seen they do
check SPF records for real domains but just let fake domains through without
even marking them as spam.

If all mail servers did the sensible thing as mentioned here:
http://www.openspf.org/FAQ/Blocking_spam, and all domains had SPF records,
spammers would indeed have a hard time. The fact that Google and Yahoo allow
fake domains through really makes SPF completely impotent as a weapon to
fight spam. In the cartoon guide (http://old.openspf.org/aspen.html) this
would be represented by a huge bell curve called "fake domains" that dwarfs
the other two and is a free ticket to spammers.

The fundamental question I'm trying to get at is:
Should I bother setting up and maintaining SPF records if my domains are not
currently suffering from any forged identity problems? I would do it happily
if it contributed to the internet community's fight against spam, but unless
major email service providers close the fake domain loophole, there doesn't
seem to be any point.

Cheers
Martin

----- Original Message -----
From: "Stuart D. Gathman" <stuart [at] bmsi>
To: <spf-discuss [at] listbox>
Sent: Thursday, February 10, 2011 12:31 PM
Subject: Re: [spf-discuss] Yahoo mail and Gmail policy explanation?

> On Thu, 10 Feb 2011, Martin Jericho wrote:
>
>> Neither yahoo mail nor gmail seem to reject incoming mail if there is no
>> DNS
>> record at all for the envelope sender address, allowing spammers to just
>> use
>> completely fake domains.
>>
>> Does anyone know why they might choose to have that policy? Is there any
>> legitimate reason for allowing incoming mail from a fake domain? It
>> doesn't
>> even get blocked when you turn on the spam filter!
>
> Believe it or not, there are countless clueless, but otherwise
> "legitimate"
> senders who can't get basic things like HELO or MAIL FROM right (much
> less SPF). Our customers get such mail rejected from their customers
> every
> month or so. We immediately search the logs, find what brainless thing
> their
> customer is doing, attempt to send mail to postmaster (which usually
> fails,
> because they are after all clueless), and add a special "whitelist" (like
> "accept mail from invalid domain email-clueless.com" and hope spammers
> don't
> use it).
>
> Free email outfits like yahoo or gmail simply can't afford to offer this
> kind of email tech support. Their system has to be entirely self-serve.
> Statistically routing mail to a "spam" folder is something end users
> can handle on their own when it doesn't do what they want. Diagnosing
> what what idiotic thing this particular sender did, and constructing
> a complex whitelist to work around it is not something end users can
> handle.
> It is *not* as simple as "whitelist this email" when the domain is invalid
> or forged.
>
> If there was a particular email, then a "Whitelist" button could run
> heuristics to identify common sender problems and apply standard
> workarounds.
> But if there was an email, then our customer would not be complaining!
> The first problem is *finding* what random invalid domain the stupid
> sender is trying to use in the log. In the case of my church, for
> instance,
> they simply had a typo in their MTA config for the MAIL FROM (and
> they rewrote the MAIL FROM of all client submissions with the wrong
> domain).
>
> If people would just send a test email to something like
> spf-test [at] openspf
> after configuring their server, things would be so much easier.
> It would also be nice if they tested their SPF record on openspf.org
> before publishing it.
>
> --
> Stuart D. Gathman <stuart [at] bmsi>
> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
> "Confutatis maledictis, flammis acribus addictis" - background song for
> a Microsoft sponsored "Where do you want to go from here?" commercial.
>
>
> -------------------------------------------
> Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
> Modify Your Subscription: http://www.listbox.com/member/
> [http://www.listbox.com/member/]
>
> Archives: https://www.listbox.com/member/archive/735/=now
> RSS Feed: https://www.listbox.com/member/archive/rss/735/20472388-958fad67
> Modify Your Subscription:
> https://www.listbox.com/member/?&
> Unsubscribe Now:
> https://www.listbox.com/unsubscribe/?&&post_id=20110209203212:93B6D168-34B5-11E0-BF88-9E0B634668CC
> Powered by Listbox: http://www.listbox.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/1311532-17d8a1ba
Modify Your Subscription: https://www.listbox.com/member/?member_id=1311532&id_secret=1311532-f2ea6ed9
Unsubscribe Now: https://www.listbox.com/unsubscribe/?member_id=1311532&id_secret=1311532-bdbb122a&post_id=20110210003411:626FF0A4-34D7-11E0-8032-CB15B0C564E0
Powered by Listbox: http://www.listbox.com

shane at red

Feb 10, 2011, 6:56 AM

Post #4 of 7 (1076 views)
Permalink

Re: Yahoo mail and Gmail policy explanation? [In reply to]

Hello Martin,

MJ> The fundamental question I'm trying to get at is:
MJ> Should I bother setting up and maintaining SPF records if my domains are not
MJ> currently suffering from any forged identity problems?

Before answering your question it's important to make clear is that
SPF is all about protecting a Domain's email Sending reputation &
thereby reducing forging of a Domain's email. A Domain owner is able
to state with a choice of granularity which mail servers are
permitted to send authorised mail.

A Receiver can read a Sending Domain SPF records in DNS to aid
rejection of non-authorised mail servers which likely will be spam.
If valid mail is sent via non-authorised mail servers a tight Sender
SPF record statement will say to a Receiver "Please feel free
to reject this email without further checks."

It follows that any reduction of spam is a side effect of using
SPF by making it easier to identify forged identity email.
Spammers using slack SPF statements for their own domains will
have a slack reputation for those domains & have spam detected
as usual by other methods. A spammer using a tight SPF record
statement makes it easier for a Receiver to reject spammer Domain
email via blocklists.

Answering your question: Yes you should protect your domains
against forged identity problems before they occur.

This is about saving damage to your Domain's reputation &
mailserver downtime particularly in a commercial context.
You do not want to be tagged as sending spam!

Forged identity can create massive load on servers from Receiver
rejection emails interfering with legitimate email.

Setting SPF records after identity spoofing starts is too late, a
Domain will be quickly widely blocklisted by both specialist and
local Receiver blocking filters. Once blocklisted it may take
weeks to restore Domain reputation and get unblocked.

Many well known specialist centralised blocklists require
manual requests via webpage or other hoops including enforced
delay of 7 days or more before a blocklisted server retest or
blocklist reset.

MJ> I would do it happily
MJ> if it contributed to the internet community's fight against spam, but unless
MJ> major email service providers close the fake domain loophole, there doesn't
MJ> seem to be any point.

I don't see a problem choosing to use SPF with your Domains.

As above SPF helps fight Domain forged identity, any spam
reduction is just a side effect. The more Domain Senders &
Receivers use SPF, the more spammers will find Domain forged
identity less useful and use their own domains which may be
spam blocked by other methods such as automated centralised
blocklists.

Re loophole: Are the email service providers filtering fake domain
emails identified as spam ?

If Domain forged identity emails are being observed not being
rejected causing agro perhaps you should consider POP3 collection
& a separate filter, http://keir.net/k9.html works fine and uses
external IP blocklists.

I feel it would be wrong to assume SPF & other reputation methods
are not being used to detect forged identity & non-authorised
Domain servers when scoring for email filtering.

It's not clear to me if scoring is used for statistics alone or
if a level of scoring is set that triggers a level of forged
identity email rejection.

I'd be surprised if forged domain email identified as spam is
getting past the filters. Forged domain emails rejected as spam
by the email provider should be passed to IP blocklists but not
Domain blocklists.

Likewise the service provider should be using advanced outbound
Sender email filters in order to try and identify outgoing
spam from their own servers.

--
Best regards,
Shane mailto:shane [at] red

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/1311532-17d8a1ba
Modify Your Subscription: https://www.listbox.com/member/?member_id=1311532&id_secret=1311532-f2ea6ed9
Unsubscribe Now: https://www.listbox.com/unsubscribe/?member_id=1311532&id_secret=1311532-bdbb122a&post_id=20110210095708:06D076B6-3526-11E0-BCD0-8FD735712E33
Powered by Listbox: http://www.listbox.com

stuart at bmsi

Feb 10, 2011, 11:27 AM

Post #5 of 7 (1056 views)
Permalink

Re: Yahoo mail and Gmail policy explanation? [In reply to]

On Thu, 10 Feb 2011, Martin Jericho wrote:

> If Google and Yahoo just allow email from invalid domains because there are
> too many incorrectly configured sender mail servers, wouldn't the same
> argument mean they also just ignore SPF because there are so many incorrectly
> configured SPF records? From the evidence I've seen they do check SPF
> records for real domains but just let fake domains through without even
> marking them as spam.

I agree with you - they really should at least require an A record. I
was just trying to explain their motivation - which is to avoid tech
support at all costs.

> The fundamental question I'm trying to get at is:
> Should I bother setting up and maintaining SPF records if my domains are not
> currently suffering from any forged identity problems? I would do it happily
> if it contributed to the internet community's fight against spam, but unless
> major email service providers close the fake domain loophole, there doesn't
> seem to be any point.

*I* certain appreciate having valid SPF records, and it is an integral
part of blocking email forgery (which is a different issue than "spam"
as you seem to realize) for our customer. Even SPF from spammers allows
me to ban the domain with confidence:-) We are not as big as gmail,
but we reject about 200000 forged emails every day. (With one
occasionally being a clueless sender such as I described yesterday.)

On the positive side, your recipients can be more confident that emails
are truly from your domain when their email system gets an SPF pass. US
Banks find this valuable enough to make it recommended industry
practice.

SMTP TLS has been available in robust form for 15 years - but very few
people use it. Does that mean you shouldn't use it?

SPF and other authentication protocols (like DKIM and S/MIME)
distinguish responsible email admist a sea of twitter twaddle.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/1311532-17d8a1ba
Modify Your Subscription: https://www.listbox.com/member/?member_id=1311532&id_secret=1311532-f2ea6ed9
Unsubscribe Now: https://www.listbox.com/unsubscribe/?member_id=1311532&id_secret=1311532-bdbb122a&post_id=20110210142708:BEF42B14-354B-11E0-8889-BB837BF0678F
Powered by Listbox: http://www.listbox.com

terry at ashtonwoodshomes

Feb 10, 2011, 12:15 PM

Post #6 of 7 (1115 views)
Permalink

Re: Yahoo mail and Gmail policy explanation? [In reply to]

There is still a point, because although some ESP's may not block
forged/non-existent domains, many mail servers do, more and more.
And more and more use SPF as an indicator toward spam (which is often
forgery) or not.
So having an SPF record can help your domain get successful delivery.
And can help you from some forgery should it ever sneak up on you.

I am all for "do not fix something that is not broken",
but remember also that "an ounce of prevention is worth a pound of cure".
SPF may be that ounce of prevention.

Terry

Terry Fielder
terry [at] greatgulfhomes
Associate Director Software Development and Deployment
Great Gulf Homes / Ashton Woods Homes
Fax: (416) 441-9085

On 2/10/2011 12:33 AM, Martin Jericho wrote:
> Hi Stuart,
>
> Thanks for your explanation - although I'm not convinced about the
> reasoning.
>
> If Google and Yahoo just allow email from invalid domains because
> there are too many incorrectly configured sender mail servers,
> wouldn't the same argument mean they also just ignore SPF because
> there are so many incorrectly configured SPF records? From the
> evidence I've seen they do check SPF records for real domains but just
> let fake domains through without even marking them as spam.
>
> If all mail servers did the sensible thing as mentioned here:
> http://www.openspf.org/FAQ/Blocking_spam, and all domains had SPF
> records, spammers would indeed have a hard time. The fact that Google
> and Yahoo allow fake domains through really makes SPF completely
> impotent as a weapon to fight spam. In the cartoon guide
> (http://old.openspf.org/aspen.html) this would be represented by a
> huge bell curve called "fake domains" that dwarfs the other two and is
> a free ticket to spammers.
>
> The fundamental question I'm trying to get at is:
> Should I bother setting up and maintaining SPF records if my domains
> are not currently suffering from any forged identity problems? I would
> do it happily if it contributed to the internet community's fight
> against spam, but unless major email service providers close the fake
> domain loophole, there doesn't seem to be any point.
>
> Cheers
> Martin
>
>
>
> ----- Original Message ----- From: "Stuart D. Gathman" <stuart [at] bmsi>
> To: <spf-discuss [at] listbox>
> Sent: Thursday, February 10, 2011 12:31 PM
> Subject: Re: [spf-discuss] Yahoo mail and Gmail policy explanation?
>
>
>> On Thu, 10 Feb 2011, Martin Jericho wrote:
>>
>>> Neither yahoo mail nor gmail seem to reject incoming mail if there
>>> is no DNS
>>> record at all for the envelope sender address, allowing spammers to
>>> just use
>>> completely fake domains.
>>>
>>> Does anyone know why they might choose to have that policy? Is there
>>> any
>>> legitimate reason for allowing incoming mail from a fake domain? It
>>> doesn't
>>> even get blocked when you turn on the spam filter!
>>
>> Believe it or not, there are countless clueless, but otherwise
>> "legitimate"
>> senders who can't get basic things like HELO or MAIL FROM right (much
>> less SPF). Our customers get such mail rejected from their customers
>> every
>> month or so. We immediately search the logs, find what brainless
>> thing their
>> customer is doing, attempt to send mail to postmaster (which usually
>> fails,
>> because they are after all clueless), and add a special "whitelist"
>> (like
>> "accept mail from invalid domain email-clueless.com" and hope
>> spammers don't
>> use it).
>>
>> Free email outfits like yahoo or gmail simply can't afford to offer this
>> kind of email tech support. Their system has to be entirely self-serve.
>> Statistically routing mail to a "spam" folder is something end users
>> can handle on their own when it doesn't do what they want. Diagnosing
>> what what idiotic thing this particular sender did, and constructing
>> a complex whitelist to work around it is not something end users can
>> handle.
>> It is *not* as simple as "whitelist this email" when the domain is
>> invalid
>> or forged.
>>
>> If there was a particular email, then a "Whitelist" button could run
>> heuristics to identify common sender problems and apply standard
>> workarounds.
>> But if there was an email, then our customer would not be complaining!
>> The first problem is *finding* what random invalid domain the stupid
>> sender is trying to use in the log. In the case of my church, for
>> instance,
>> they simply had a typo in their MTA config for the MAIL FROM (and
>> they rewrote the MAIL FROM of all client submissions with the wrong
>> domain).
>>
>> If people would just send a test email to something like
>> spf-test [at] openspf
>> after configuring their server, things would be so much easier.
>> It would also be nice if they tested their SPF record on openspf.org
>> before publishing it.
>>
>> --
>> Stuart D. Gathman <stuart [at] bmsi>
>> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703
>> 591-6154
>> "Confutatis maledictis, flammis acribus addictis" - background song for
>> a Microsoft sponsored "Where do you want to go from here?" commercial.
>>
>>
>> -------------------------------------------
>> Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
>> Modify Your Subscription: http://www.listbox.com/member/
>> [http://www.listbox.com/member/]
>>
>> Archives: https://www.listbox.com/member/archive/735/=now
>> RSS Feed:
>> https://www.listbox.com/member/archive/rss/735/20472388-958fad67
>> Modify Your Subscription: https://www.listbox.com/member/?&
>> Unsubscribe Now:
>> https://www.listbox.com/unsubscribe/?&&post_id=20110209203212:93B6D168-34B5-11E0-BF88-9E0B634668CC
>> Powered by Listbox: http://www.listbox.com
>
>
>
> -------------------------------------------
> Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
> Modify Your Subscription: http://www.listbox.com/member/
> [http://www.listbox.com/member/]
>
> Archives: https://www.listbox.com/member/archive/735/=now
> RSS Feed: https://www.listbox.com/member/archive/rss/735/1068817-8ce620fc
> Modify Your Subscription:
> https://www.listbox.com/member/?&
> Unsubscribe Now:
> https://www.listbox.com/unsubscribe/?&&post_id=20110210003411:626FF0A4-34D7-11E0-8032-CB15B0C564E0
> Powered by Listbox: http://www.listbox.com
>

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/1311532-17d8a1ba
Modify Your Subscription: https://www.listbox.com/member/?member_id=1311532&id_secret=1311532-f2ea6ed9
Unsubscribe Now: https://www.listbox.com/unsubscribe/?member_id=1311532&id_secret=1311532-bdbb122a&post_id=20110210151545:A2B08054-3552-11E0-8456-1A52F559ED1D
Powered by Listbox: http://www.listbox.com

openspf at mart

Feb 10, 2011, 1:59 PM

Post #7 of 7 (1057 views)
Permalink

Re: Yahoo mail and Gmail policy explanation? [In reply to]

Thanks for all the responses.

The description of the what I would have to go through if my domain ever
gets onto a blacklist has convinced me. I naively thought it would be a
simple process!

Regards,
Martin

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/1311532-17d8a1ba
Modify Your Subscription: https://www.listbox.com/member/?member_id=1311532&id_secret=1311532-f2ea6ed9
Unsubscribe Now: https://www.listbox.com/unsubscribe/?member_id=1311532&id_secret=1311532-bdbb122a&post_id=20110210165931:2CAA82EC-3561-11E0-AA11-B6A71B2BADF7
Powered by Listbox: http://www.listbox.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads