Mailing List Archive: SPF: Discuss

Re: [spf-help] How reliable is it to block/reject on SPF fail?

1 2

View All

Index | Next | Previous | View Threaded

spf at jubileegroup

Nov 21, 2009, 7:10 AM

Post #1 of 26 (4626 views)
Permalink

Re: [spf-help] How reliable is it to block/reject on SPF fail?

Hi there,

On Sat, 21 Nov 2009, Thomas Harold wrote:

> What is the current thinking on rejecting at the SMTP transaction if you
> encounter an SPF fail? Are there a lot of false positives?

This implies an accepted meaning for the expression "false positive"
and I'm not sure that there is one. Spammers are as easily able to
set up DNS records as anyone else. Sometimes it seems that spammers
are implementing SPF faster than anyone else, and at one time people
were (only half-jokingly:) suggesting that mail should be rejected on
SPF pass! But that misses the point. The question is one of forgery.
In my book nobody has any business forging anything, so when I get an
SPF fail I reject, usually without even asking for the DATA. If I do
ask for the data it's probably to send it to a honey trap, or abuse@.
If I get a softfail I take a view. Usually a dim one.

Does this help a lot in fighting spam? Well it does, but very little.
So far this month out of about half a million attempts to send spam to
the dozen or so domains for which I handle mail, two were rejected on
SPF failure. Typical numbers, but bear in mind that the vast majority
of attempts didn't even get as far as the SPF check so it isn't as bad
as those numbers might suggest.

To put it another way:

If any given domain publishes SPF records, then this month all the
attempts to send mail forged to appear to be from that domain were
rejected. And the previous month. And the one before that, and...

Sounds better that way. :)

100% of two forgery attemps still might not sound great, but there's
another little-mentioned benefit. My servers suffered no DOS attacks
from _legitimate_ mailservers which (a) were the recipients of botnet
generated mail forged to appear to be from one or more of the domains
I manage and (b) have implemented SPF checking. Admittedly I have but
a vague idea of how many attacks that might be, but I did experience a
couple two or three years back and that was hell. Mail volumes rose
by more than an order of magnitude in a few minutes, and stayed that
way for weeks. The first attack brought a server down for a few days.
The second one didn't. One reason that I don't have any real numbers
for this important metric is that if the legitimate servers see mail
forged to come from my domains, and if they also check my SPF records,
then I don't see any traffic except mail to my own abuse@ addresses -
and I can't remember the last time I saw any. Which is a Good Thing.

So your question will probably boil down to "how much do I trust the
domain's DNS records?". Leaving aside misconfigured records, my view
is that you have no choice but to trust them, just as you must trust
them when you ask for the A or MX records. That's how things are.

> I'm trying to decide whether to to remove the "warn_on_reject" in
> Postfix's main.cf to go ahead and return a 550 5.7.1 error code.

I'm slightly to the right of Attila the Hun on mail rejection, so as
a non-representative sample I'll leave others to help you with that.

--

73,
Ged.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

iane at sussex

Nov 25, 2009, 7:58 AM

Post #2 of 26 (4510 views)
Permalink

Re: Re: [spf-help] How reliable is it to block/reject on SPF fail? [In reply to]

--On 21 November 2009 15:10:04 +0000 "G.W. Haywood"
<spf [at] jubileegroup> wrote:

> Hi there,
>
> On Sat, 21 Nov 2009, Thomas Harold wrote:
>
>> What is the current thinking on rejecting at the SMTP transaction if you
>> encounter an SPF fail? Are there a lot of false positives?
>

As I understand it, most "-all" records are for domains that don't emit
email at all. But, they're not widely published.

That's one of the clearest use cases for SPF though. You can publish an A
record without establishing an email domain. You should reject an SPF fail.
Any "false positives" can squarely be blamed on the publisher of the record
or the forwarder of the email.

--
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

spfdiscuss at alandoherty

Nov 25, 2009, 9:35 AM

Post #3 of 26 (4494 views)
Permalink

Re: Re: [spf-help] How reliable is it to block/reject on SPF fail? [In reply to]

At 15:58 25/11/2009 Wednesday, Ian Eiloart wrote:

>--On 21 November 2009 15:10:04 +0000 "G.W. Haywood" <spf [at] jubileegroup> wrote:
>
>>Hi there,
>>
>>On Sat, 21 Nov 2009, Thomas Harold wrote:
>>
>>>What is the current thinking on rejecting at the SMTP transaction if you
>>>encounter an SPF fail? Are there a lot of false positives?
>
>As I understand it, most "-all" records are for domains that don't emit email at all. But, they're not widely published.

if you mean "v=spf1 -all"
ie a record with no passing syntax before the -all
but if you mean records ending with -all you would be incorrect most people with established spf have there records terminated with a -all
?all being really equivalent to not having any restriction on forgery/spf-non-believer
~all being a little better but still effectively saying you want the receiver to consider accepting forgeries also {but possibly via spam-folder}

>That's one of the clearest use cases for SPF though. You can publish an A record without establishing an email domain.

yes like the common www.example.com

>You should reject an SPF fail.

i think you mean publish an spf fail ie "v=spf1 -all"
{rejection being down to the recipient}
and i wholeheartedly agree and recommend this approach
{i also reccomend publishing a null MX record so mta's not checking spf
can see they cannot send mail to the domain and should not attempt "fallback to A" {broken} standard behaviour
{thus any checking the validity of incoming adress's in DNS get an explicit fail too}
this is done via
xxx.example.com IN MX 10 .

> Any "false positives" can squarely be blamed on the publisher of the record or the forwarder of the email.

true, {as to the publisher}

the non-srs-forwarder is blameless though as well his-server his-rules as far as how he checks incomming mail for validity
and non-srs-forwarding is done on behalf of the shared-user of both systems
{ie user who has account with non-srs-forwarder and the final reciever}

so in that case its a failure of the user to {select a non-srs-forwarder with adequate inbound filtering}
and to inform the end-recieving system to not perform spf filtering on mail non-srs-forwarded to him from non-srs-forwarders-ip

if user has no method of disabling spf-checking on mail to him from non-srs-forwarders-ip then he should be selecting a different end-reception supplier
as the current one is ill equipped to accept non-srs-forwards {and thus should either ban non-srs-forwarding or not reject on spf-fail}

{srs-based-forwarders re-write the envelope sender so couldn't fail SPF}

*yes anyone rejecting on spf fail needs to have a system for allowing users to whitelist their non-srs-forwarders, the software capable of doing this has been around for long enough, and unfortunately srs is still not common enough in most low end MTA's {like exchange}

*we see a non-insignificant demand for our offering to corporate customers of forward re-handling ie they non-srs-forward to an alias here, we srs-forward on to the actual destination blackberry/gmail/hotmail/whatever as their is a legit need for forwarding, but too many end-receivers give no method for users to whitelist their non-srs-forwarders-ips and also reject on SPF fail

>--
>Ian Eiloart
>IT Services, University of Sussex
>01273-873148 x3148
>For new support requests, see http://www.sussex.ac.uk/its/help/
>
>
>-------------------------------------------
>Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
>Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]
>
>Archives: https://www.listbox.com/member/archive/735/=now
>RSS Feed: https://www.listbox.com/member/archive/rss/735/
>Powered by Listbox: http://www.listbox.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

iane at sussex

Nov 26, 2009, 3:23 AM

Post #4 of 26 (4500 views)
Permalink

Re: Re: [spf-help] How reliable is it to block/reject on SPF fail? [In reply to]

--On 25 November 2009 17:35:29 +0000 alan <spfdiscuss [at] alandoherty>
wrote:

> At 15:58 25/11/2009 Wednesday, Ian Eiloart wrote:
>
>
>> --On 21 November 2009 15:10:04 +0000 "G.W. Haywood"
>> <spf [at] jubileegroup> wrote:
>>
>>> Hi there,
>>>
>>> On Sat, 21 Nov 2009, Thomas Harold wrote:
>>>
>>>> What is the current thinking on rejecting at the SMTP transaction if
>>>> you encounter an SPF fail? Are there a lot of false positives?
>>
>> As I understand it, most "-all" records are for domains that don't emit
>> email at all. But, they're not widely published.
>
> if you mean "v=spf1 -all"
> ie a record with no passing syntax before the -all
> but if you mean records ending with -all you would be incorrect most
> people with established spf have there records terminated with a -all
> ?all being really equivalent to not having any restriction on
> forgery/spf-non-believer ~all being a little better but still effectively
> saying you want the receiver to consider accepting forgeries also {but
> possibly via spam-folder}

Not according to the only stats that I've seen on the matter. They say 9.9%
of domains publish SPF, 46.8% of those publish "-all" records, and 70.2% of
those are "v=spf1 -all".

For those with "v=spf1 -all" everyone would agree that it can never be
wrong to reject the email, surely? For the other 29.8%, you should still
respect the publisher's policy, but that might result in rejecting wanted
email. Minimise those false positives by allowing your recipient users to
whitelist their forwarders.

>> That's one of the clearest use cases for SPF though. You can publish an
>> A record without establishing an email domain.
>
> yes like the common www.example.com

Yes, but more importantly, we have hundreds of machines registered in the
sussex.ac.uk domain, but none of them are email domains.

>> You should reject an SPF fail.
>
> i think you mean publish an spf fail ie "v=spf1 -all"
> {rejection being down to the recipient}

Actually, I meant what I said. If you see an SPF fail, you should reject
the email. This comment wasn't related to machines that aren't email
domains. Your next comment is useful though, I wasn't aware of this
possibility. I wonder how many MTAs handle this correctly? I guess they
can't do worse than ignore the record.

> and i wholeheartedly agree and recommend this approach
> {i also reccomend publishing a null MX record so mta's not checking spf
> can see they cannot send mail to the domain and should not attempt
> "fallback to A" {broken} standard behaviour {thus any checking the
> validity of incoming adress's in DNS get an explicit fail too} this is
> done via
> xxx.example.com IN MX 10 .
>
>> Any "false positives" can squarely be blamed on the publisher of the
>> record or the forwarder of the email.
>
> true, {as to the publisher}
>
> the non-srs-forwarder is blameless though as well his-server his-rules as
> far as how he checks incomming mail for validity and non-srs-forwarding
> is done on behalf of the shared-user of both systems {ie user who has
> account with non-srs-forwarder and the final reciever}

His server, his rules. Agreed. However, if he knowingly forwards mail with
broken SPF, then that's nobody else's fault.

> so in that case its a failure of the user to {select a non-srs-forwarder
> with adequate inbound filtering} and to inform the end-recieving system
> to not perform spf filtering on mail non-srs-forwarded to him from
> non-srs-forwarders-ip

But many people don't "select" their forwarders. For example, our students
can opt to have their "sussex.ac.uk" email forwarded to a third party
account. They can't choose who does the forwarding, though!

>
> if user has no method of disabling spf-checking on mail to him from
> non-srs-forwarders-ip then he should be selecting a different
> end-reception supplier as the current one is ill equipped to accept
> non-srs-forwards {and thus should either ban non-srs-forwarding or not
> reject on spf-fail}

Perhaps they'd be well advised to. Unfortunately, most end users don't know
about SPF. And, they don't know that they don't know!

> {srs-based-forwarders re-write the envelope sender so couldn't fail SPF}
>
> *yes anyone rejecting on spf fail needs to have a system for allowing
> users to whitelist their non-srs-forwarders, the software capable of
> doing this has been around for long enough, and unfortunately srs is
> still not common enough in most low end MTA's {like exchange}

True. Though I believe that the latest version of Exchange forwards email
with the return-path set to the original recipient address. Not good, but
it doesn't break SPF!

> *we see a non-insignificant demand for our offering to corporate
> customers of forward re-handling ie they non-srs-forward to an alias
> here, we srs-forward on to the actual destination
> blackberry/gmail/hotmail/whatever as their is a legit need for
> forwarding, but too many end-receivers give no method for users to
> whitelist their non-srs-forwarders-ips and also reject on SPF fail
>
>
>> --
>> Ian Eiloart
>> IT Services, University of Sussex
>> 01273-873148 x3148
>> For new support requests, see http://www.sussex.ac.uk/its/help/
>>
>>
>> -------------------------------------------
>> Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
>> Modify Your Subscription: http://www.listbox.com/member/
>> [http://www.listbox.com/member/]
>>
>> Archives: https://www.listbox.com/member/archive/735/=now
>> RSS Feed: https://www.listbox.com/member/archive/rss/735/
>> Powered by Listbox: http://www.listbox.com
>
>
>
> -------------------------------------------
> Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
> Modify Your Subscription: http://www.listbox.com/member/
> [http://www.listbox.com/member/]
>
> Archives: https://www.listbox.com/member/archive/735/=now
> RSS Feed: https://www.listbox.com/member/archive/rss/735/
> Powered by Listbox: http://www.listbox.com

--
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

spfdiscuss at alandoherty

Nov 26, 2009, 11:42 AM

Post #5 of 26 (4503 views)
Permalink

Re: Re: [spf-help] How reliable is it to block/reject on SPF fail? [In reply to]

At 11:23 26/11/2009 Thursday, Ian Eiloart wrote:

>--On 25 November 2009 17:35:29 +0000 alan <spfdiscuss [at] alandoherty> wrote:
>
>>At 15:58 25/11/2009 Wednesday, Ian Eiloart wrote:
>>
>>
>>>--On 21 November 2009 15:10:04 +0000 "G.W. Haywood"
>>><spf [at] jubileegroup> wrote:
>>>
>>>>Hi there,
>>>>
>>>>On Sat, 21 Nov 2009, Thomas Harold wrote:
>>>>
>>>>>What is the current thinking on rejecting at the SMTP transaction if
>>>>>you encounter an SPF fail? Are there a lot of false positives?
>>>
>>>As I understand it, most "-all" records are for domains that don't emit
>>>email at all. But, they're not widely published.
>>
>>if you mean "v=spf1 -all"
>>ie a record with no passing syntax before the -all
>>but if you mean records ending with -all you would be incorrect most
>>people with established spf have there records terminated with a -all
>>?all being really equivalent to not having any restriction on
>>forgery/spf-non-believer ~all being a little better but still effectively
>>saying you want the receiver to consider accepting forgeries also {but
>>possibly via spam-folder}
>
>Not according to the only stats that I've seen on the matter. They say 9.9% of domains publish SPF, 46.8% of those publish "-all" records, and 70.2% of those are "v=spf1 -all".
>
>For those with "v=spf1 -all" everyone would agree that it can never be wrong to reject the email, surely? For the other 29.8%, you should still respect the publisher's policy, but that might result in rejecting wanted email. Minimise those false positives by allowing your recipient users to whitelist their forwarders.

I'm not seeing where you disagree with my point here {that -all dosnt equate to dosn't emit email, only v=spf1 -all does that
but i agree that few are using -all and ~all records and thus most publishing spf, are not actually using it for its designed purpose of limiting/stopping forgery {they are using it to say mail that passes IS good, mail that fails... treat it like it has no SPF {ie take it too}

>>>That's one of the clearest use cases for SPF though. You can publish an
>>>A record without establishing an email domain.
>>
>>yes like the common www.example.com
>
>Yes, but more importantly, we have hundreds of machines registered in the sussex.ac.uk domain, but none of them are email domains.

well yes domains like that should also block forgeries of their hosts/subdomains i was just citing the most common example in pretty much every domain

>>>You should reject an SPF fail.
>>
>>i think you mean publish an spf fail ie "v=spf1 -all"
>>{rejection being down to the recipient}
>
>Actually, I meant what I said. If you see an SPF fail, you should reject the email. This comment wasn't related to machines that aren't email domains. Your next comment is useful though, I wasn't aware of this possibility. I wonder how many MTAs handle this correctly? I guess they can't do worse than ignore the record.

I just {understandably} didn't see the move from Publisher of SPF role to receiver of mail role

>>and i wholeheartedly agree and recommend this approach
>>{i also reccomend publishing a null MX record so mta's not checking spf
>>can see they cannot send mail to the domain and should not attempt
>>"fallback to A" {broken} standard behaviour {thus any checking the
>>validity of incoming adress's in DNS get an explicit fail too} this is
>>done via
>>xxx.example.com IN MX 10 .
>>
>>>Any "false positives" can squarely be blamed on the publisher of the
>>>record or the forwarder of the email.
>>
>>true, {as to the publisher}
>>
>>the non-srs-forwarder is blameless though as well his-server his-rules as
>>far as how he checks incomming mail for validity and non-srs-forwarding
>>is done on behalf of the shared-user of both systems {ie user who has
>>account with non-srs-forwarder and the final reciever}
>
>His server, his rules. Agreed. However, if he knowingly forwards mail with broken SPF, then that's nobody else's fault.
{only if his server is capable of SRS few are} 90% of servers i see day to day are just plain incapable M$ exchange being the most common
and few admins decide to feorward mails, this is usually the choice of the end user {management with blackbery or road-warrior wanting to use gmail}

>>so in that case its a failure of the user to {select a non-srs-forwarder
>>with adequate inbound filtering} and to inform the end-recieving system
>>to not perform spf filtering on mail non-srs-forwarded to him from
>>non-srs-forwarders-ip
>
>But many people don't "select" their forwarders. For example, our students can opt to have their "sussex.ac.uk" email forwarded to a third party account. They can't choose who does the forwarding, though!

well actually they could {if you didn't already do srs} they could get a provider like ourselves to middle-man the mail
{by setting yours to forward to us}
so we make it SRS compliant and before it reaches the whitelisting incapable end destination {like a lot of businesses with exchange have to}

>>if user has no method of disabling spf-checking on mail to him from
>>non-srs-forwarders-ip then he should be selecting a different
>>end-reception supplier as the current one is ill equipped to accept
>>non-srs-forwards {and thus should either ban non-srs-forwarding or not
>>reject on spf-fail}
>
>Perhaps they'd be well advised to. Unfortunately, most end users don't know about SPF. And, they don't know that they don't know!

they shouldn't have to but you just {like us} have a big button allowing them to whitlist the sender {in webmail} {only shown if the mail appeared to be forwarded} that brings them to the whitlisting of forwarders section of the api
{or in our case a link in our web based "your mail that was rejected log" beside any rejected due to SPF fail
but the means are many, hotmail do it by encouraging users to give their forwarding-here address and i suspect then probing them to see if they need whitlisting or not
{i must play with this option for our own few users who arn't techies, luckily very few atm}

>>{srs-based-forwarders re-write the envelope sender so couldn't fail SPF}
>>
>>*yes anyone rejecting on spf fail needs to have a system for allowing
>>users to whitelist their non-srs-forwarders, the software capable of
>>doing this has been around for long enough, and unfortunately srs is
>>still not common enough in most low end MTA's {like exchange}
>
>True. Though I believe that the latest version of Exchange forwards email with the return-path set to the original recipient address. Not good, but it doesn't break SPF!

if it does is very new
its long been my argument against sender-id that if it was any use M$ would at least make its own exchange capable of forwarding in a sender-id compliant way
{which of course is even more complex than SRS as it involves body re-writing too, and it wasn't even capable of the SRS based bit}
{they of course allow you to block on sender-id fail, making all forwarding within an exchange only group impossible, DOH!}

>>*we see a non-insignificant demand for our offering to corporate
>>customers of forward re-handling ie they non-srs-forward to an alias
>>here, we srs-forward on to the actual destination
>>blackberry/gmail/hotmail/whatever as their is a legit need for
>>forwarding, but too many end-receivers give no method for users to
>>whitelist their non-srs-forwarders-ips and also reject on SPF fail
>>
>>
>>>--
>>>Ian Eiloart
>>>IT Services, University of Sussex
>>>01273-873148 x3148
>>>For new support requests, see http://www.sussex.ac.uk/its/help/
>>>
>>>
>>>-------------------------------------------
>>>Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
>>>Modify Your Subscription: http://www.listbox.com/member/
>>>[http://www.listbox.com/member/]
>>>
>>>Archives: https://www.listbox.com/member/archive/735/=now
>>>RSS Feed: https://www.listbox.com/member/archive/rss/735/
>>>Powered by Listbox: http://www.listbox.com
>>
>>
>>
>>-------------------------------------------
>>Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
>>Modify Your Subscription: http://www.listbox.com/member/
>>[http://www.listbox.com/member/]
>>
>>Archives: https://www.listbox.com/member/archive/735/=now
>>RSS Feed: https://www.listbox.com/member/archive/rss/735/
>>Powered by Listbox: http://www.listbox.com
>
>
>
>--
>Ian Eiloart
>IT Services, University of Sussex
>01273-873148 x3148
>For new support requests, see http://www.sussex.ac.uk/its/help/
>
>
>-------------------------------------------
>Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
>Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]
>
>Archives: https://www.listbox.com/member/archive/735/=now
>RSS Feed: https://www.listbox.com/member/archive/rss/735/
>Powered by Listbox: http://www.listbox.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

vesely at tana

Nov 26, 2009, 11:48 PM

Post #6 of 26 (4509 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

alan wrote:
> At 11:23 26/11/2009 Thursday, Ian Eiloart wrote:
>>>> Any "false positives" can squarely be blamed on the publisher of the
>>>> record or the forwarder of the email.
>>> true, {as to the publisher}
>>> the non-srs-forwarder is blameless though as well his-server his-rules as
>>> far as how he checks incomming mail for validity and non-srs-forwarding
>>> is done on behalf of the shared-user of both systems {ie user who has
>>> account with non-srs-forwarder and the final reciever}
>> His server, his rules. Agreed. However, if he knowingly forwards mail with broken SPF, then that's nobody else's fault.
> {only if his server is capable of SRS few are} 90% of servers i see day to day are just plain incapable M$ exchange being the most common
> and few admins decide to feorward mails, this is usually the choice of the end user {management with blackbery or road-warrior wanting to use gmail}

The decision that is up to users is to enable either forwarding by
one server or fetching by the other one. Gmail and many other
servers offer fetching: it is the solution with fewer problems.

>>> so in that case its a failure of the user to {select a non-srs-forwarder
>>> with adequate inbound filtering} and to inform the end-recieving system
>>> to not perform spf filtering on mail non-srs-forwarded to him from
>>> non-srs-forwarders-ip
>> But many people don't "select" their forwarders. For example, our students can opt to have their "sussex.ac.uk" email forwarded to a third party account. They can't choose who does the forwarding, though!
>
> well actually they could {if you didn't already do srs} they could get a provider like ourselves to middle-man the mail
> {by setting yours to forward to us}
> so we make it SRS compliant and before it reaches the whitelisting incapable end destination {like a lot of businesses with exchange have to}

However, the middle address will be visible in case the final server
rejects a message because the mailbox is full. That may be
unacceptable in case the first (vanity) address bears utter
significance.

BTW, what you call "SRS compliant" should actually be called "SPF
compliant". SRS is not the only technique to change envelope senders
and thus avoid breaking SPF.

>>> if user has no method of disabling spf-checking on mail to him from
>>> non-srs-forwarders-ip then he should be selecting a different
>>> end-reception supplier as the current one is ill equipped to accept
>>> non-srs-forwards {and thus should either ban non-srs-forwarding or not
>>> reject on spf-fail}
>> Perhaps they'd be well advised to. Unfortunately, most end users don't know about SPF. And, they don't know that they don't know!
>
> they shouldn't have to but you just {like us} have a big button allowing them to whitlist the sender {in webmail} {only shown if the mail appeared to be forwarded} that brings them to the whitlisting of forwarders section of the api
> {or in our case a link in our web based "your mail that was rejected log" beside any rejected due to SPF fail
> but the means are many, hotmail do it by encouraging users to give their forwarding-here address and i suspect then probing them to see if they need whitlisting or not
> {i must play with this option for our own few users who arn't techies, luckily very few atm}

I cannot understand this. Webmail servers only have the chance to
determine that a message appears to be forwarded, and hence present
the big button, _after_ they have already accepted the message. What
do they do if the user does _not_ whitelist the sender at that stage?

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

iane at sussex

Nov 27, 2009, 3:05 AM

Post #7 of 26 (4503 views)
Permalink

Re: Re: [spf-help] How reliable is it to block/reject on SPF fail? [In reply to]

--On 26 November 2009 19:42:05 +0000 alan <spfdiscuss [at] alandoherty>
wrote:

> At 11:23 26/11/2009 Thursday, Ian Eiloart wrote:
>
>
>> --On 25 November 2009 17:35:29 +0000 alan <spfdiscuss [at] alandoherty>
>> wrote:
>>
>>> At 15:58 25/11/2009 Wednesday, Ian Eiloart wrote:
>>>
>>>
>>>> --On 21 November 2009 15:10:04 +0000 "G.W. Haywood"
>>>> <spf [at] jubileegroup> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> On Sat, 21 Nov 2009, Thomas Harold wrote:
>>>>>
>>>>>> What is the current thinking on rejecting at the SMTP transaction if
>>>>>> you encounter an SPF fail? Are there a lot of false positives?
>>>>
>>>> As I understand it, most "-all" records are for domains that don't emit
>>>> email at all. But, they're not widely published.
>>>
>>> if you mean "v=spf1 -all"
>>> ie a record with no passing syntax before the -all
>>> but if you mean records ending with -all you would be incorrect most
>>> people with established spf have there records terminated with a -all
>>> ?all being really equivalent to not having any restriction on
>>> forgery/spf-non-believer ~all being a little better but still
>>> effectively saying you want the receiver to consider accepting
>>> forgeries also {but possibly via spam-folder}
>>
>> Not according to the only stats that I've seen on the matter. They say
>> 9.9% of domains publish SPF, 46.8% of those publish "-all" records, and
>> 70.2% of those are "v=spf1 -all".
>>
>> For those with "v=spf1 -all" everyone would agree that it can never be
>> wrong to reject the email, surely? For the other 29.8%, you should still
>> respect the publisher's policy, but that might result in rejecting
>> wanted email. Minimise those false positives by allowing your recipient
>> users to whitelist their forwarders.
>
> I'm not seeing where you disagree with my point here {that -all dosnt
> equate to doesn't emit email, only v=spf1 -all does that but i agree that
> few are using -all and ~all records and thus most publishing spf, are not
> actually using it for its designed purpose of limiting/stopping forgery
> {they are using it to say mail that passes IS good, mail that fails...
> treat it like it has no SPF {ie take it too}

Perhaps I misunderstood you. I thought you were reading me as saying "most
use of 'v=spf1 -all' is for domains that don't emit mail. Perhaps I should
have said "most domains with '-all' use 'v=spf1 -all' records, to indicate
that they don't emit mail.

But, I think you're claiming that most domains use "-all", whether they
emit mail or not. That's not true according to spf-all.com. They say that
most domains don't use spf. Among domains that use SPF, 53.2% don't use
"-all" at all, 32.9% use "v=spf1 -all", so only 14% use "-all" AND emit
email. Alternatively, of email emitting domains that use SPF, only about
20% use "-all" - assuming that all spf users that aren't using "-all" do
actually emit email.

It would be easier to follow what you are saying if you used some
punctuation. At least so we know where one sentence begins, and the next
ends. However, I think maybe you have a typo in "most people with
established spf have there [sic] records terminated with a -all [sic]". I
think you meant "their", and "~all", perhaps?

--
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

iane at sussex

Nov 27, 2009, 3:13 AM

Post #8 of 26 (4507 views)
Permalink

Re: Re: [spf-help] How reliable is it to block/reject on SPF fail? [In reply to]

--On 26 November 2009 19:42:05 +0000 alan <spfdiscuss [at] alandoherty>
wrote:

>
>> But many people don't "select" their forwarders. For example, our
>> students can opt to have their "sussex.ac.uk" email forwarded to a third
>> party account. They can't choose who does the forwarding, though!
>
> well actually they could {if you didn't already do srs} they could get a
> provider like ourselves to middle-man the mail {by setting yours to
> forward to us}
> so we make it SRS compliant and before it reaches the whitelisting
> incapable end destination {like a lot of businesses with exchange have
> to}

Oh, OK. I didn't know that service existed. What's your web site?

>>> if user has no method of disabling spf-checking on mail to him from
>>> non-srs-forwarders-ip then he should be selecting a different
>>> end-reception supplier as the current one is ill equipped to accept
>>> non-srs-forwards {and thus should either ban non-srs-forwarding or not
>>> reject on spf-fail}
>>
>> Perhaps they'd be well advised to. Unfortunately, most end users don't
>> know about SPF. And, they don't know that they don't know!

> they shouldn't have to but you just {like us} have a big button allowing
> them to whitlist the sender {in webmail} {only shown if the mail appeared
> to be forwarded} that brings them to the whitlisting of forwarders
> section of the api {or in our case a link in our web based "your mail
> that was rejected log" beside any rejected due to SPF fail but the means
> are many, hotmail do it by encouraging users to give their
> forwarding-here address and i suspect then probing them to see if they
> need whitlisting or not {i must play with this option for our own few
> users who arn't techies, luckily very few atm}

Yes, it would be helpful if all mail providers did that. Would be nice to
have a way that mail clients like Thunderbird could set such whitelisting
options. Unfortunately, they don't always interact with the inbound MTA at
all, so some new standard might be needed for this.

>>> {srs-based-forwarders re-write the envelope sender so couldn't fail SPF}
>>>
>>> *yes anyone rejecting on spf fail needs to have a system for allowing
>>> users to whitelist their non-srs-forwarders, the software capable of
>>> doing this has been around for long enough, and unfortunately srs is
>>> still not common enough in most low end MTA's {like exchange}
>>
>> True. Though I believe that the latest version of Exchange forwards
>> email with the return-path set to the original recipient address. Not
>> good, but it doesn't break SPF!
>
> if it does is very new

I read some complaints about the facility. I don't have any direct
experience, though.

> its long been my argument against sender-id that if it was any use M$
> would at least make its own exchange capable of forwarding in a sender-id
> compliant way {which of course is even more complex than SRS as it
> involves body re-writing too, and it wasn't even capable of the SRS based
> bit} {they of course allow you to block on sender-id fail, making all
> forwarding within an exchange only group impossible, DOH!}

--
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

iane at sussex

Nov 27, 2009, 3:20 AM

Post #9 of 26 (4508 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

--On 27 November 2009 08:48:44 +0100 Alessandro Vesely <vesely [at] tana>
wrote:

>
> The decision that is up to users is to enable either forwarding by one
> server or fetching by the other one. Gmail and many other servers offer
> fetching: it is the solution with fewer problems.
>

Except that it requires you to Gmail to store your password for the remote
account in a recoverable format. For our users, that means violating our
terms and conditions of use. In fact, we'd not permit them to share our
passwords with Gmail even if Gmail were storing them securely. We have a
common authentication mechanism that means you can do more with the
password than just read email.

We'd need something like OAuth to make it work securely.

As an intermediate effort, we could provide a second account with distinct
credentials that was used ONLY for IMAP, and had read permissions on the
mailbox of the primary account, though. It's probably easier to implement
SRS, but we'd still like the receiving account to whitlist our forwarding,
so that we don't pollute our reputation.

--
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

spfdiscuss at alandoherty

Nov 27, 2009, 6:00 AM

Post #10 of 26 (4499 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

At 07:48 27/11/2009 Friday, Alessandro Vesely wrote:
>alan wrote:
>>At 11:23 26/11/2009 Thursday, Ian Eiloart wrote:
>>>>>Any "false positives" can squarely be blamed on the publisher of the
>>>>>record or the forwarder of the email.
>>>>true, {as to the publisher}
>>>>the non-srs-forwarder is blameless though as well his-server his-rules as
>>>>far as how he checks incomming mail for validity and non-srs-forwarding
>>>>is done on behalf of the shared-user of both systems {ie user who has
>>>>account with non-srs-forwarder and the final reciever}
>>>His server, his rules. Agreed. However, if he knowingly forwards mail with broken SPF, then that's nobody else's fault.
>>{only if his server is capable of SRS few are} 90% of servers i see day to day are just plain incapable M$ exchange being the most common
>>and few admins decide to feorward mails, this is usually the choice of the end user {management with blackbery or road-warrior wanting to use *webmail-provider*}
>
>The decision that is up to users is to enable either forwarding by one server or fetching by the other one. Gmail and many other servers offer fetching: it is the solution with fewer problems.

true this is what we suggest for all gmail users {i should stop using gmail as default webmail service in statements like above now ammended}
we even offer them a how-to for setting up gmail to pop/submit
so they can do both incoming/outgoing via gmail-interface without even altering their SPF as both flows actually go via us
{its a pity more webmail systems don't offer the utility}

>>>>so in that case its a failure of the user to {select a non-srs-forwarder
>>>>with adequate inbound filtering} and to inform the end-recieving system
>>>>to not perform spf filtering on mail non-srs-forwarded to him from
>>>>non-srs-forwarders-ip
>>>But many people don't "select" their forwarders. For example, our students can opt to have their "sussex.ac.uk" email forwarded to a third party account. They can't choose who does the forwarding, though!
>>well actually they could {if you didn't already do srs} they could get a provider like ourselves to middle-man the mail {by setting yours to forward to us}
>>so we make it SRS compliant and before it reaches the whitelisting incapable end destination {like a lot of businesses with exchange have to}
>
>However, the middle address will be visible in case the final server rejects a message because the mailbox is full. That may be unacceptable in case the first (vanity) address bears utter significance.

true but few business-related forwarders have the mailbox full scenario
but yes this would be an issue if it happened {but less of one than their server returning DSN's to ever SPF utilising sender}

>BTW, what you call "SRS compliant" should actually be called "SPF compliant". SRS is not the only technique to change envelope senders and thus avoid breaking SPF.

true my bad ;)

>>>>if user has no method of disabling spf-checking on mail to him from
>>>>non-srs-forwarders-ip then he should be selecting a different
>>>>end-reception supplier as the current one is ill equipped to accept
>>>>non-srs-forwards {and thus should either ban non-srs-forwarding or not
>>>>reject on spf-fail}
>>>Perhaps they'd be well advised to. Unfortunately, most end users don't know about SPF. And, they don't know that they don't know!
>>they shouldn't have to but you just {like us} have a big button allowing them to whitlist the sender {in webmail} {only shown if the mail appeared to be forwarded} that brings them to the whitlisting of forwarders section of the api
>>{or in our case a link in our web based "your mail that was rejected log" beside any rejected due to SPF fail
>>but the means are many, hotmail do it by encouraging users to give their forwarding-here address and i suspect then probing them to see if they need whitlisting or not
>>{i must play with this option for our own few users who arn't techies, luckily very few atm}
>
>I cannot understand this. Webmail servers only have the chance to determine that a message appears to be forwarded, and hence present the big button, _after_ they have already accepted the message. What do they do if the user does _not_ whitelist the sender at that stage?

yes this only happens on the 90% of non SPF using mail that gets through {but appears non-srs-forwarded}
the link is presented beside the rejected mail {due to SPF failure} in their web based view of the rejected mail log
{we provide a rejected mail log to the user so they can tweak which DNSBL's,HELO-checks,PTR-checks,manual whitelists/blacklists etc are working/not-working for them, and ammend their filtering config appropriatly}
we have found it works well with over time the user turning on more and more aggressive checks and whitelisting senders when needed, {often pruning their whitelists after convincing sender to stop doing whatever dumb thing caused the test to fail, like helo-as blah.blah.local}
most of our users are geeks thats why they come to us when they want more control for their spamfiltering.

most have been with us since the past when their api to altering their rejection criteria was a phonecall to me, but it was still a better api than elsewhere

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

spfdiscuss at alandoherty

Nov 27, 2009, 6:09 AM

Post #11 of 26 (4510 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

At 11:20 27/11/2009 Friday, you wrote:

>--On 27 November 2009 08:48:44 +0100 Alessandro Vesely <vesely [at] tana> wrote:
>
>>
>>The decision that is up to users is to enable either forwarding by one
>>server or fetching by the other one. Gmail and many other servers offer
>>fetching: it is the solution with fewer problems.
>
>Except that it requires you to Gmail to store your password for the remote account in a recoverable format. For our users, that means violating our terms and conditions of use. In fact, we'd not permit them to share our passwords with Gmail even if Gmail were storing them securely. We have a common authentication mechanism that means you can do more with the password than just read email.
>
>We'd need something like OAuth to make it work securely.
>
>As an intermediate effort, we could provide a second account with distinct credentials that was used ONLY for IMAP, and had read permissions on the mailbox of the primary account, though. It's probably easier to implement SRS, but we'd still like the receiving account to whitlist our forwarding, so that we don't pollute our reputation.

I'd suggest that SRS-forwarding to gmail would be less benefit
than say offering a dedicated second pop3/smtp-submission server with dedicated credentials

just for the reason of ip-reputation {which srs-forwarding unfortunately does tend to muddy}
and additionally it allows users to webmail reply from their work address via your servers {thus without having to allow all gmail users the potential for forging mail and passing SPF}

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

iane at sussex

Nov 27, 2009, 6:51 AM

Post #12 of 26 (4511 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

>
> I'd suggest that SRS-forwarding to gmail would be less benefit
> than say offering a dedicated second pop3/smtp-submission server with
> dedicated credentials
>
> just for the reason of ip-reputation {which srs-forwarding unfortunately
> does tend to muddy} and additionally it allows users to webmail reply
> from their work address via your servers {thus without having to allow
> all gmail users the potential for forging mail and passing SPF}
>

I just don't like the security risk associated with Google holding a load
of our passwords in plain text.

--
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

spfdiscuss at alandoherty

Nov 27, 2009, 6:54 AM

Post #13 of 26 (4505 views)
Permalink

Re: Re: [spf-help] How reliable is it to block/reject on SPF fail? [In reply to]

At 11:05 27/11/2009 Friday, Ian Eiloart wrote:

>--On 26 November 2009 19:42:05 +0000 alan <spfdiscuss [at] alandoherty> wrote:
>
>>At 11:23 26/11/2009 Thursday, Ian Eiloart wrote:
>>
>>
>>>--On 25 November 2009 17:35:29 +0000 alan <spfdiscuss [at] alandoherty>
>>>wrote:
>>>
>>>>At 15:58 25/11/2009 Wednesday, Ian Eiloart wrote:
>>>>
>>>>
>>>>>--On 21 November 2009 15:10:04 +0000 "G.W. Haywood"
>>>>><spf [at] jubileegroup> wrote:
>>>>>
>>>>>>Hi there,
>>>>>>
>>>>>>On Sat, 21 Nov 2009, Thomas Harold wrote:
>>>>>>
>>>>>>>What is the current thinking on rejecting at the SMTP transaction if
>>>>>>>you encounter an SPF fail? Are there a lot of false positives?
>>>>>
>>>>>As I understand it, most "-all" records are for domains that don't emit
>>>>>email at all. But, they're not widely published.
>>>>
>>>>if you mean "v=spf1 -all"
>>>>ie a record with no passing syntax before the -all
>>>>but if you mean records ending with -all you would be incorrect most
>>>>people with established spf have there records terminated with a -all
>>>>?all being really equivalent to not having any restriction on
>>>>forgery/spf-non-believer ~all being a little better but still
>>>>effectively saying you want the receiver to consider accepting
>>>>forgeries also {but possibly via spam-folder}
>>>
>>>Not according to the only stats that I've seen on the matter. They say
>>>9.9% of domains publish SPF, 46.8% of those publish "-all" records, and
>>>70.2% of those are "v=spf1 -all".
>>>
>>>For those with "v=spf1 -all" everyone would agree that it can never be
>>>wrong to reject the email, surely? For the other 29.8%, you should still
>>>respect the publisher's policy, but that might result in rejecting
>>>wanted email. Minimise those false positives by allowing your recipient
>>>users to whitelist their forwarders.
>>
>>I'm not seeing where you disagree with my point here {that -all dosnt
>>equate to doesn't emit email, only v=spf1 -all does that but i agree that
>>few are using -all and ~all records and thus most publishing spf, are not
>>actually using it for its designed purpose of limiting/stopping forgery
>>{they are using it to say mail that passes IS good, mail that fails...
>>treat it like it has no SPF {ie take it too}
>
>Perhaps I misunderstood you. I thought you were reading me as saying "most use of 'v=spf1 -all' is for domains that don't emit mail. Perhaps I should have said "most domains with '-all' use 'v=spf1 -all' records, to indicate that they don't emit mail.

i still think we have wires crossed but its pointless i think

>But, I think you're claiming that most domains use "-all", whether they emit mail or not.

no i will clarify all in one moment about my claims {i won't further try and analise yours}

> That's not true according to spf-all.com. They say that most domains don't use spf. Among domains that use SPF, 53.2% don't use "-all" at all, 32.9% use "v=spf1 -all", so only 14% use "-all" AND emit email. Alternatively, of email emitting domains that use SPF, only about 20% use "-all" - assuming that all spf users that aren't using "-all" do actually emit email.

i was saying {and assuming as understood anything with *}
most domains publish no SPF*

those few that do publish SPF do so ineffectively/uselessly {ie using ?all at end}*

most domains "effectivly" utilising SPF that emit email use -all at the end {with a few doing ~all}

most domains "seen" using -all at the end are emitters of email

it is possible that vastly more domains have "v=spf -all" to block potential-forgeries but these are hardly ever seen and impossible to enumerate {in my own zone files yes they would be 15 to 1} but few people would ever query/guess these host/domain-names or know their existence}

I do encourage any/every hostmaster to publish "v=spf -all" on all domains not used in mfrom or helo
It would be nice if this number is growing

>It would be easier to follow what you are saying if you used some punctuation. At least so we know where one sentence begins, and the next ends. However, I think maybe you have a typo in "most people with established spf have there [sic] records terminated with a -all [sic]". I think you meant "their", and "~all", perhaps?

yes i admit i have no ability to spell or punctuate

yes i meant their
no i meant -all

but should have used a sentence instead of "established"
read established as:
tested working and free from false positives by period of terminating with ?all
then made useful by clue-full admin by switching to -all

as opposed to the majority of SPF publishers who do not "use/utilise" SPF and stay in the ?all phase permanently so their passing mail gets a green light from SPF, while their non-passing mail {the spammers} also gets delivered unhindered.

if -all became more common non-srs-forwarder implementations would lessen
receivers would be pressured to allow white listing of non-srs-forwarders
{actually whitelisting of SRS-forwarders would be necessary too for people offering a this-is-spam type button to users scoring sending ip reputation on hits}
or alternately relievers would start offering pop3 pickup from remote as a better alternative to forwarding {like gmail}

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

spfdiscuss at alandoherty

Nov 27, 2009, 7:07 AM

Post #14 of 26 (4508 views)
Permalink

Re: Re: [spf-help] How reliable is it to block/reject on SPF fail? [In reply to]

At 11:13 27/11/2009 Friday, Ian Eiloart wrote:

>--On 26 November 2009 19:42:05 +0000 alan <spfdiscuss [at] alandoherty> wrote:
>
>>
>>>But many people don't "select" their forwarders. For example, our
>>>students can opt to have their "sussex.ac.uk" email forwarded to a third
>>>party account. They can't choose who does the forwarding, though!
>>
>>well actually they could {if you didn't already do srs} they could get a
>>provider like ourselves to middle-man the mail {by setting yours to
>>forward to us}
>>so we make it SRS compliant and before it reaches the whitelisting
>>incapable end destination {like a lot of businesses with exchange have
>>to}
>
>Oh, OK. I didn't know that service existed. What's your web site?

none atm {there is just an internal how-to for users/dsl subscribers}
yeah they are small startup and not effectivly marketing beyond the geeks{and the companies they work for}/word of mouth atm

>>>>if user has no method of disabling spf-checking on mail to him from
>>>>non-srs-forwarders-ip then he should be selecting a different
>>>>end-reception supplier as the current one is ill equipped to accept
>>>>non-srs-forwards {and thus should either ban non-srs-forwarding or not
>>>>reject on spf-fail}
>>>
>>>Perhaps they'd be well advised to. Unfortunately, most end users don't
>>>know about SPF. And, they don't know that they don't know!
>
>>they shouldn't have to but you just {like us} have a big button allowing
>>them to whitlist the sender {in webmail} {only shown if the mail appeared
>>to be forwarded} that brings them to the whitlisting of forwarders
>>section of the api {or in our case a link in our web based "your mail
>>that was rejected log" beside any rejected due to SPF fail but the means
>>are many, hotmail do it by encouraging users to give their
>>forwarding-here address and i suspect then probing them to see if they
>>need whitlisting or not {i must play with this option for our own few
>>users who arn't techies, luckily very few atm}
>
>Yes, it would be helpful if all mail providers did that. Would be nice to have a way that mail clients like Thunderbird could set such whitelisting options. Unfortunately, they don't always interact with the inbound MTA at all, so some new standard might be needed for this.

we provide links in the headers for pop3/imap users to the section of the received mail log for that mail {on the internal web-site}
which in turn has inline linked options to turn on blocking for each smtp time test the received mail failed {that they were obviously currently not blocking on}
and to whitelist/blacklist the sender ip {and the whitelist page offers whitelist for all-failures/spf-failures/pbl-failures......the list goes on}
its all quite user tweakable
from draconian
accept no mail that isn't whitelisted
to accept all mail even the infected crap

someday I'll make the web based admin stuff look nice and fully functional {and secure}
{features are added to exim about 3 months before the user interface to feature is built}
{so during the cross over the api is phone me}

then open source release the lot
{any c coders wanting to help with a related sub project are welcome to volunteer time}

>>>>{srs-based-forwarders re-write the envelope sender so couldn't fail SPF}
>>>>
>>>>*yes anyone rejecting on spf fail needs to have a system for allowing
>>>>users to whitelist their non-srs-forwarders, the software capable of
>>>>doing this has been around for long enough, and unfortunately srs is
>>>>still not common enough in most low end MTA's {like exchange}
>>>
>>>True. Though I believe that the latest version of Exchange forwards
>>>email with the return-path set to the original recipient address. Not
>>>good, but it doesn't break SPF!
>>
>>if it does is very new
>
>I read some complaints about the facility. I don't have any direct experience, though.
>
>>its long been my argument against sender-id that if it was any use M$
>>would at least make its own exchange capable of forwarding in a sender-id
>>compliant way {which of course is even more complex than SRS as it
>>involves body re-writing too, and it wasn't even capable of the SRS based
>>bit} {they of course allow you to block on sender-id fail, making all
>>forwarding within an exchange only group impossible, DOH!}
>
>
>
>--
>Ian Eiloart
>IT Services, University of Sussex
>01273-873148 x3148
>For new support requests, see http://www.sussex.ac.uk/its/help/
>
>
>-------------------------------------------
>Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
>Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]
>
>Archives: https://www.listbox.com/member/archive/735/=now
>RSS Feed: https://www.listbox.com/member/archive/rss/735/
>Powered by Listbox: http://www.listbox.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

vesely at tana

Nov 28, 2009, 7:14 AM

Post #15 of 26 (4477 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

alan wrote:
> At 07:48 27/11/2009 Friday, Alessandro Vesely wrote:
>> I cannot understand this. Webmail servers only have the chance to determine that a message appears to be forwarded, and hence present the big button, _after_ they have already accepted the message. What do they do if the user does _not_ whitelist the sender at that stage?
>
> yes this only happens on the 90% of non SPF using mail that gets through {but appears non-srs-forwarded}
> the link is presented beside the rejected mail {due to SPF failure} in their web based view of the rejected mail log
> {we provide a rejected mail log to the user so they can tweak which DNSBL's,HELO-checks,PTR-checks,manual whitelists/blacklists etc are working/not-working for them, and ammend their filtering config appropriatly}
> we have found it works well with over time the user turning on more and more aggressive checks and whitelisting senders when needed, {often pruning their whitelists after convincing sender to stop doing whatever dumb thing caused the test to fail, like helo-as blah.blah.local}

Ah, now I got it. It seems clever enough to me, even if users have
to miss the very first messages --probably tests in most cases.

I agree with Ian that it would be nice to have something like this
as a standard. That might provide for the sender setting up any
relevant detail, and (non-geek) users just having to agree, either
manually or automatically. However, the obvious advantage of your
approach is that it requires minimal or no compliance from senders.

One added benefit is that users have a means to maintain a list of
their whitelistings, subscriptions, etcetera.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

spfdiscuss at alandoherty

Nov 28, 2009, 8:16 AM

Post #16 of 26 (4482 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

At 15:14 28/11/2009 Saturday, Alessandro Vesely wrote:
>alan wrote:
>>At 07:48 27/11/2009 Friday, Alessandro Vesely wrote:
>>>I cannot understand this. Webmail servers only have the chance to determine that a message appears to be forwarded, and hence present the big button, _after_ they have already accepted the message. What do they do if the user does _not_ whitelist the sender at that stage?
>>yes this only happens on the 90% of non SPF using mail that gets through {but appears non-srs-forwarded}
>>the link is presented beside the rejected mail {due to SPF failure} in their web based view of the rejected mail log
>>{we provide a rejected mail log to the user so they can tweak which DNSBL's,HELO-checks,PTR-checks,manual whitelists/blacklists etc are working/not-working for them, and ammend their filtering config appropriatly}
>>we have found it works well with over time the user turning on more and more aggressive checks and whitelisting senders when needed, {often pruning their whitelists after convincing sender to stop doing whatever dumb thing caused the test to fail, like helo-as blah.blah.local}
>
>Ah, now I got it. It seems clever enough to me, even if users have to miss the very first messages --probably tests in most cases.
>
>I agree with Ian that it would be nice to have something like this as a standard. That might provide for the sender setting up any relevant detail, and (non-geek) users just having to agree, either manually or automatically. However, the obvious advantage of your approach is that it requires minimal or no compliance from senders.

well the preferences are currently inherited from owner thus

root can change global defaults for all domains {and add postmasters and domains} {also can access per postmaster defaults, per domain defaults and per user defaults and per address settings}
a-postmaster can change defaults for all domains he controls {and add users and address} {also can set per-domain defaults, and per user defaults and per address settings}
a-user can change defaults for all address allocated to them by the postmaster and settings for each address only

root and postmaster have the options hard-on, hard-off, on and off
postmaster and user have the extra option of inherit
users only have on off or inherit

hard-on forces lower {postmasters/users} to only have inherit selectable {with an idicator showing value is on}
hard-off forces lower {postmasters/users} to only have inherit selectable {with an idicator showing value is off}
on means option is on for all below with inherit selected
off means option is off for all below with inherit selected

we give a postmaster id-password to the it/mail guy at receiving site we do inbound mail for and some just never create user accounts they just keep all address' under postmaster control and handle the calls/changes tweaks for their users

but some do force all users to handle their own, and some do the most common of only creating user accounts for those competent enough to run them themselves

>One added benefit is that users have a means to maintain a list of their whitelistings, subscriptions, etcetera.

that was my main aim in my building of this
{ok first motive was so i could run draconian anti-spammer and anti-idioticly-setup-mta filtering on my own address' without affecting other users and leaving abuse and postmaster open for all spam*}
{but later i realised that others might appreciate having the utility to alter their own filtering per-address also}
{i'm hoping it will lead to users pressuring idiotic-mta owners to clean-up, due to them seeing that 90% of spam would be blocked by the same test-fail that idiot-legit-sender keeps hitting so by convincing legit-sender to fix his mta they can then turn on reject-on-fail for that test}
{also we add headers for each test-fail on accepted mail so user can see the idiot-legit-senders has issues, hoping that embarrassment will also motivate senders to send mail so it dosn't gain an X-DUMB-xxx header}

*anyone ever trying to report spam/piracy to hotmail or piracy@M$ knows why spamfiltering postmaster and abuse is counterproductive {you don't get the complaints so you never kill the customer so your rep rapidly hits floor}

>-------------------------------------------
>Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
>Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]
>
>Archives: https://www.listbox.com/member/archive/735/=now
>RSS Feed: https://www.listbox.com/member/archive/rss/735/
>Powered by Listbox: http://www.listbox.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

spfdiscuss at alandoherty

Nov 28, 2009, 8:29 AM

Post #17 of 26 (4470 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

At 15:14 28/11/2009 Saturday, Alessandro Vesely wrote:
>alan wrote:
>>At 07:48 27/11/2009 Friday, Alessandro Vesely wrote:
>>>I cannot understand this. Webmail servers only have the chance to determine that a message appears to be forwarded, and hence present the big button, _after_ they have already accepted the message. What do they do if the user does _not_ whitelist the sender at that stage?
>>yes this only happens on the 90% of non SPF using mail that gets through {but appears non-srs-forwarded}
>>the link is presented beside the rejected mail {due to SPF failure} in their web based view of the rejected mail log
>>{we provide a rejected mail log to the user so they can tweak which DNSBL's,HELO-checks,PTR-checks,manual whitelists/blacklists etc are working/not-working for them, and ammend their filtering config appropriatly}
>>we have found it works well with over time the user turning on more and more aggressive checks and whitelisting senders when needed, {often pruning their whitelists after convincing sender to stop doing whatever dumb thing caused the test to fail, like helo-as blah.blah.local}
>
>Ah, now I got it. It seems clever enough to me, even if users have to miss the very first messages --probably tests in most cases.
>
>I agree with Ian that it would be nice to have something like this as a standard. That might provide for the sender setting up any relevant detail, and (non-geek) users just having to agree, either manually or automatically. However, the obvious advantage of your approach is that it requires minimal or no compliance from senders.

as far as standards yup hopefully in the end {whenever that is} the websites api will be documented well enough and structured in a way
that 3rd party plugins could talk direct to it so all the user has to give the plugin is the base url of the admin site and their admin id/password
and then thus plugin could present whatever options it wished from the mail client
{and similarly webmail clients could be tweaked to present their own api to the filtering}
but currently even the filtering site api dosn't present all available options to the user, many still require calling me and having me alter their preferences

as the system as-is is hoped to be adopted by providers or admins wanting to stick a filter in front of exchange/in-house-system
{as its designed to be a front-end only and validates attempted address' by calling forward to backend mailbox server {split-dns so it only sees the mx for backend server}
thus if a users address isn't previously defined it uses the per domain defaults
{and thus the postmaster only has to add users/address' to the filtering api when exceptions to the defaults are needed}

>One added benefit is that users have a means to maintain a list of their whitelistings, subscriptions, etcetera.
>
>
>
>-------------------------------------------
>Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
>Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]
>
>Archives: https://www.listbox.com/member/archive/735/=now
>RSS Feed: https://www.listbox.com/member/archive/rss/735/
>Powered by Listbox: http://www.listbox.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

stuart at bmsi

Nov 28, 2009, 8:41 AM

Post #18 of 26 (4489 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

On Fri, 27 Nov 2009, Ian Eiloart wrote:

> > server or fetching by the other one. Gmail and many other servers offer
> > fetching: it is the solution with fewer problems.
> >
>
> Except that it requires you to Gmail to store your password for the remote
> account in a recoverable format. For our users, that means violating our terms
> and conditions of use. In fact, we'd not permit them to share our passwords
> with Gmail even if Gmail were storing them securely. We have a common
> authentication mechanism that means you can do more with the password than
> just read email.

I've always wondered why more users don't just run their own mail domain,
and either buy Exchange for Windows or run an open source OS.

But then I remember how frustrated I get with all the supposedly professional
email providers that don't know diddly squat about RFCs and cause all
kinds of trouble.

And while a 3rd party service could give you your own domain for a
few bucks a month (for small volumes), it is hard to compete with "free"
(as in beer).

Is it even possible to have an open source (or even proprietary) email server
that even the most rfc ignorant can successfully configure out of the box?
Perhaps it would include a system where every installation would be
automatically tested (remotely using only SMTP related ports and by having the
new install send test emails to designated servers) by other accepted
installations, and only activated when it passes.

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

vesely at tana

Nov 29, 2009, 1:30 AM

Post #19 of 26 (4473 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

alan wrote:
>> I agree with Ian that it would be nice to have something like this as a standard. That might provide for the sender setting up any relevant detail, and (non-geek) users just having to agree, either manually or automatically. However, the obvious advantage of your approach is that it requires minimal or no compliance from senders.
>
> as far as standards yup hopefully in the end {whenever that is} the websites api will be documented well enough and structured in a way
> that 3rd party plugins could talk direct to it so all the user has to give the plugin is the base url of the admin site and their admin id/password
> and then thus plugin could present whatever options it wished from the mail client
> {and similarly webmail clients could be tweaked to present their own api to the filtering}
> but currently even the filtering site api dosn't present all available options to the user, many still require calling me and having me alter their preferences

It probably makes sense to standardize those filtering options that
are based on envelope data. It is easier, because there are not many
of those. For the content, I'd only bother a few relevant header
fields, e.g. Authentication-Results, X-Spam-Status, X-Spam-Level,
SPF-Received, if at all. It is not practical to require expansive
filtering.

Obviously, any mail site may add whatever (optional) filtering
rules. The purpose of _requiring_ some of them is in case a message
is forwarded multiple times. Consider this scenario:

sender-> user [at] sit -> alias [at] intermediat -> other [at] fina

In such cases, the rules at the final site must not be stricter than
those at the intermediate one. Thus, there should be no need to
fine-tune downstream sites. It would then be sufficient to configure
whitelistings at one site only, as standardized settings can easily
be exchanged between compliant sites, if the given standardization
provides a way for doing that.

>> One added benefit is that users have a means to maintain a list of their whitelistings, subscriptions, etcetera.

In the above scenario, subscriptions might be retrieved recursively
and result in a single web page.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

spfdiscuss at alandoherty

Nov 29, 2009, 7:00 AM

Post #20 of 26 (4461 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

At 09:30 29/11/2009 Sunday, Alessandro Vesely wrote:
>alan wrote:
>>>I agree with Ian that it would be nice to have something like this as a standard. That might provide for the sender setting up any relevant detail, and (non-geek) users just having to agree, either manually or automatically. However, the obvious advantage of your approach is that it requires minimal or no compliance from senders.
>>as far as standards yup hopefully in the end {whenever that is} the websites api will be documented well enough and structured in a way
>>that 3rd party plugins could talk direct to it so all the user has to give the plugin is the base url of the admin site and their admin id/password
>>and then thus plugin could present whatever options it wished from the mail client
>>{and similarly webmail clients could be tweaked to present their own api to the filtering}
>>but currently even the filtering site api dosn't present all available options to the user, many still require calling me and having me alter their preferences
>
>It probably makes sense to standardize those filtering options that are based on envelope data. It is easier, because there are not many of those.

not many???
sending ip > countless DNSBL and several DNSWL {user selectable} {most locally served from mirrors as aggregate zone files ie one lookup serves 32 bit encoded DNSBL results}
then PTR tests {ie does it have ptr}
does it have FQRDNS
then enumerating it to a country {for users blocking based on country code}
and enumerating to an esp/isp {for users whitelisting/blacklisting by isp {hotmail/gmail etc}
sending HELO 60ish tests
then mail-from several DNSRL {dns reputation lists like rfc ignorant etc}
then rcpt to {take results of above 32 at a time and logical-and them with recipients options then reject on any non zero result}

currently most of our filtering options to the user are pre-data

> For the content, I'd only bother a few relevant header fields, e.g. Authentication-Results, X-Spam-Status, X-Spam-Level, SPF-Received, if at all. It is not practical to require expansive filtering.

the content filtering atm is clamav + spamassasin so we have a per user SA reject score a
and a binary reject viri and binary reject on {phish/other non-viri clamav result}
someday we hope to have more on the content filtering, like i'm building dkim in soon

but soon want to investigate separate classes of SA rulset and depending on class of adress(s) accepting the email at this pass, run the tweaked SA over the email

{by class of user i mean currently the first address to accept at RCPT time decides the class {8 digit binary}
any further recipients rejecting on envelope details rejects as usual
any further recipients not-rejecting on envelope and being in the same content-class accept the mail {thus far}
any further recipients not-rejecting on envelope but in a different content-class defer the mail

thus mail to 6 users in six different content classes {with say only one rejecting on envelope}
will try 5 times and each will get 1 user

but a mail to 6 users in two content classes {with say the same one rejecting on envelope}
will try twice to get to all

{as most users fall into the default content class of reject on all clamav hits reject on SA score 10+
11101010
postmaster and abuse though are
10000000
ie reject only viri

viri phish use-SA {SA-Score}

>Obviously, any mail site may add whatever (optional) filtering rules. The purpose of _requiring_ some of them is in case a message is forwarded multiple times. Consider this scenario:
>
>sender-> user [at] sit -> alias [at] intermediat -> other [at] fina
>
>In such cases, the rules at the final site must not be stricter than those at the intermediate one.

or more simply what we do is mail to a forwarding recipient, from a forwarders ip, is just accepted regardless, {so we aren't creating backscatter due to the forwarders crappy filters}
{it is not our job to save them from spam that their other receiver didn't block! as if we do there is no pressure on the forwarder to institute decent anti-spam solutions}
{its an idealogical thing ie filtering should all be at the point of initial reception, as once past this point the spammer has been paid and any further rejection will only result in
A DSN's bothering the forged-from
B DSN repression causing the non-forged-from {but sending suspicious mail} not being notified that the user didn't get it

both are worse than user getting the spam {suitably marked as such} in my book
I think we should all try to ensure DSN's are only normally generated by originating-senders where possible {to force them like us to police their outflow and ensure their users cannot generate forgeries}

in our case every authenticated submission can only send envelope-from that users list of verified from address's, and mail servers can only send envelope-from their list of allowed domains {or from-any to their list of outbound-forwarding destinations}

so if a user here generates spam {uce or ube} it is entirely traceable and the NDR's are local and logged by us for monitoring so we can TOS them

> Thus, there should be no need to fine-tune downstream sites. It would then be sufficient to configure whitelistings at one site only, as standardized settings can easily be exchanged between compliant sites, if the given standardization provides a way for doing that.

not really a good idea in my experience most users with more than one address want wildly different filtering on each
so it would be bad to assume a forwarded address would want the same filtering as a receiving address
{as for example some of our users have the forwarders receiving all kinds of crap but have the end receivers {usually dedicated address' for forward reception} simply setup to deny 0.0.0.0/0 except forwarders-ip's
this obviously wouldn't want to be auto-propagated

>>>One added benefit is that users have a means to maintain a list of their whitelistings, subscriptions, etcetera.
>
>In the above scenario, subscriptions might be retrieved recursively and result in a single web page.
>
>
>
>
>-------------------------------------------
>Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
>Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]
>
>Archives: https://www.listbox.com/member/archive/735/=now
>RSS Feed: https://www.listbox.com/member/archive/rss/735/
>Powered by Listbox: http://www.listbox.com

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

iane at sussex

Nov 30, 2009, 3:07 AM

Post #21 of 26 (4461 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

--On 28 November 2009 11:41:55 -0500 "Stuart D. Gathman" <stuart [at] bmsi>
wrote:

> On Fri, 27 Nov 2009, Ian Eiloart wrote:
>
>> > server or fetching by the other one. Gmail and many other servers offer
>> > fetching: it is the solution with fewer problems.
>> >
>>
>> Except that it requires you to Gmail to store your password for the
>> remote account in a recoverable format. For our users, that means
>> violating our terms and conditions of use. In fact, we'd not permit them
>> to share our passwords with Gmail even if Gmail were storing them
>> securely. We have a common authentication mechanism that means you can
>> do more with the password than just read email.
>
> I've always wondered why more users don't just run their own mail domain,
> and either buy Exchange for Windows or run an open source OS.
>
> But then I remember how frustrated I get with all the supposedly
> professional email providers that don't know diddly squat about RFCs and
> cause all kinds of trouble.
>
> And while a 3rd party service could give you your own domain for a
> few bucks a month (for small volumes), it is hard to compete with "free"
> (as in beer).
>
> Is it even possible to have an open source (or even proprietary) email
> server that even the most rfc ignorant can successfully configure out of
> the box? Perhaps it would include a system where every installation would
> be automatically tested (remotely using only SMTP related ports and by
> having the new install send test emails to designated servers) by other
> accepted installations, and only activated when it passes.

The problems? I always look to the analogy of the car. When they first came
out, you needed a chauffeur with technical skills to repair the vehicle
they were driving. Now, almost anyone can buy a reliable car and learn to
use it with a few hours lessons. Those lessons, though, are basically about
learning not to drive your car without killing yourself or anyone else.
Very few people though have the skills and the inclination to build even
the simplest kit car.

We're at that point with the Internet. The biggest stumbling block with
roll-your-own mail server is probably the question of where you get an IP
address that isn't on an RBL. Frankly, I'm very pleased that a large
proportion of the IP space is protected by policy block lists, and would
not want millions more home PCs permitted to emit unauthenticated mail on
port 25 (especially Windows, but other OS too).

So, you'd probably want these domains hosted on some other mail server.
Google hosts my domain for free - I just pay the registration cost. I have
a virtual host, but can't be bothered to run my own mail server on it.

--
Ian Eiloart
IT Services, University of Sussex
01273-873148 x3148
For new support requests, see http://www.sussex.ac.uk/its/help/

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

stuart at bmsi

Nov 30, 2009, 11:52 AM

Post #22 of 26 (4458 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

On Mon, 30 Nov 2009, Ian Eiloart wrote:

> So, you'd probably want these domains hosted on some other mail server. Google
> hosts my domain for free - I just pay the registration cost. I have a virtual
> host, but can't be bothered to run my own mail server on it.

While this might be ok for a casual user, I don't want my mail traversing
google or any other provider in plaintext. I suppose I could use S/MIME
or GPG, but TLS is no effort privacy measure that doesn't involve trying to
tell non-geek correspondents how to create a key pair. When a
correspondent has a small office mail server supporting TLS, adequate privacy
is automatic.

I'm not thinking about casual "home" users, but about "small business" users
with a static IP. This includes "home office" users with a static IP. My
biggest email client has a /27 block and servers in his basement (blocks
400,000 forged emails a day to himself and his customers as well as serving
web pages).

There are commercial products that are "plug and play" email appliances.
Unfortunately, the most popular ones have serious RFC and general
stupidity issues (e.g. replying to DSNs, sending "you have a virus"
replies to emails they have already identified as sent by a virus, invalid
HELO name, etc, etc).

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

spfdiscuss at alandoherty

Nov 30, 2009, 1:10 PM

Post #23 of 26 (4457 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

At 19:52 30/11/2009 Monday, Stuart D. Gathman wrote:
>On Mon, 30 Nov 2009, Ian Eiloart wrote:
>
>> So, you'd probably want these domains hosted on some other mail server. Google
>> hosts my domain for free - I just pay the registration cost. I have a virtual
>> host, but can't be bothered to run my own mail server on it.
>
>While this might be ok for a casual user, I don't want my mail traversing
>google or any other provider in plaintext. I suppose I could use S/MIME
>or GPG, but TLS is no effort privacy measure that doesn't involve trying to
>tell non-geek correspondents how to create a key pair. When a
>correspondent has a small office mail server supporting TLS, adequate privacy
>is automatic.
>
>I'm not thinking about casual "home" users, but about "small business" users
>with a static IP. This includes "home office" users with a static IP. My
>biggest email client has a /27 block and servers in his basement (blocks
>400,000 forged emails a day to himself and his customers as well as serving
>web pages).
>
>There are commercial products that are "plug and play" email appliances.
>Unfortunately, the most popular ones have serious RFC and general
>stupidity issues (e.g. replying to DSNs, sending "you have a virus"
>replies to emails they have already identified as sent by a virus, invalid
>HELO name, etc, etc).

i agree with all of the above

my modis operandi is to not default block {some} common idiotic behaviour
but to highlight any/all of it to the receiver
thus they see the sender has issues and either complains to them
"i cannot turn on spam/ratware filter X as your idiot server keeps failing the legitimacy tests"

or chooses to ignore

given enough time and enough spam {that would otherwise have been blocked} some users will complain
and those some senders will clean up {and often realize this improves delivery to others}
and some won't and will find themselves blocked when/if we reach a stage when everyone blocks on idiot test X, as it is no longer a common mistake

I think one of the issues currently is too many badly setup senders and to few recipients knowing/complaining
{and many admins unwilling to block due to users complaining to them}
so i just give the users the decision of what to block or not, and let them cleanup the senders they deal with
{which works pretty well on small business to small business sender/reciever}

and sometimes generates a little work for me if they need consultancy to fix the issue
{but most just avail of the "I'll walk you through it for free on the phone if your doing it to improve your sending of mail to one of our customers" offer, as phonecalls that cleanup any portion of the net i can do unbilled, [why i'm broke but my karma is good]}

i regard it all like open relays
at first it was common,
then became abused
then then some receivers started blocking open-relays to limit the abuse
the users complained the open-relays complained
nearly all non-pig-headed open-relays got fixed
eventually nearly everyone started blocking open-relays to limit the abuse
nowdays there are pretty much no open relays

all issues like this will be tackled by a few voting with their mailboxes to not accept "shady" senders
eventually all conscientious/legit senders will follow best practices to get any useful number of deliveries
after a time even the cautious receivers will be able to block "shady" senders
as the false positive rate will diminish
then we reach a stage where "is it a legit MTA or ratware" becomes black n white {long way off}

and we move the fight to
legit MTA's that have poor defences against spammer abuse/vs/ legit MTA's that catch/block attempted abuse quickly and effectively

which we are also working on but i think few are yet {because ratware is still a common source of spam}

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

vesely at tana

Dec 1, 2009, 4:16 AM

Post #24 of 26 (4453 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

alan wrote:
>>It probably makes sense to standardize those filtering options that are based on envelope data. It is easier, because there are not many of those.
>
> not many???

Well, some...

> sending ip > countless DNSBL and several DNSWL {user selectable} {most locally served from mirrors as aggregate zone files ie one lookup serves 32 bit encoded DNSBL results}

http://www.mxtoolbox.com/blacklists.aspx lists just about one hundred.
However, only a couple of them have widespread significance. And I
know of only one dnswl.org.

> then PTR tests {ie does it have ptr}
> does it have FQRDNS

These are standard checks.

> then enumerating it to a country {for users blocking based on country code}

(nonsensical, IMHO: only reflects whether an operator's customers have
any correspondent in those countries.)

> and enumerating to an esp/isp {for users whitelisting/blacklisting by isp {hotmail/gmail etc}

By mail domain may make more sense than by ISP. But then it is not
clearly stated what should the helo name reflect. That's why SPF has
to wait for the MAIL FROM command.

> sending HELO 60ish tests
> then mail-from several DNSRL {dns reputation lists like rfc ignorant etc}
> then rcpt to {take results of above 32 at a time and logical-and them with recipients options then reject on any non zero result}
>
> currently most of our filtering options to the user are pre-data

It's still a pretty manageable set, specially if compared to SA rules.

>> For the content, I'd only bother a few relevant header fields, e.g. Authentication-Results, X-Spam-Status, X-Spam-Level, SPF-Received, if at all. It is not practical to require expansive filtering.
>
> the content filtering atm is clamav + spamassasin so we have a per user SA reject score a
> and a binary reject viri and binary reject on {phish/other non-viri clamav result}
> someday we hope to have more on the content filtering, like i'm building dkim in soon
>
> but soon want to investigate separate classes of SA rulset and depending on class of adress(s) accepting the email at this pass, run the tweaked SA over the email

This varies a lot more, among mail sites.

> {by class of user i mean currently the first address to accept at RCPT time decides the class {8 digit binary}
> [snip descr of recipient partitioning]
(I think this is standard, although not currently well worded)

>>Obviously, any mail site may add whatever (optional) filtering rules. The purpose of _requiring_ some of them is in case a message is forwarded multiple times. Consider this scenario:
>>
>>sender-> user [at] sit -> alias [at] intermediat -> other [at] fina
>>
>>In such cases, the rules at the final site must not be stricter than those at the intermediate one.
>
> or more simply what we do is mail to a forwarding recipient, from a forwarders ip, is just accepted regardless, {so we aren't creating backscatter due to the forwarders crappy filters}

Yup. However, it would be preferable to check that the forwarder
actually performed the checks that it is supposed to perform, on each
message. You can only double-check those checks that your server is in
turn configured for.

It could be done this way: Alice goes to the "final" webmail site; it
downloads an html form from the "intermediate" site; the latter, in
turn downloads a form from the "site". Although Alice is working on
the "final" webmail page, she uses that composite form for configuring
her "user [at] sit" upstream address. Alice sets any available option and
submits the form to "final". The "final" site looks at the options it
understands and saves them for double-checks, then submits the form to
"intermediate". The "intermediate" site does the same, so the form is
eventually submitted to "site".

When a message is forwarded, both "intermediate" and "final" can
double-check some of the options. Is that too cumbersome?

> {it is not our job to save them from spam that their other receiver didn't block! as if we do there is no pressure on the forwarder to institute decent anti-spam solutions}

Agreed, but we only marginally trust the other receivers.
Double-checks allow to verify their work.

> {its an idealogical thing ie filtering should all be at the point of initial reception, as once past this point the spammer has been paid and any further rejection will only result in
> A DSN's bothering the forged-from
> B DSN repression causing the non-forged-from {but sending suspicious mail} not being notified that the user didn't get it

Agreed.

> both are worse than user getting the spam {suitably marked as such} in my book
> I think we should all try to ensure DSN's are only normally generated by originating-senders where possible {to force them like us to police their outflow and ensure their users cannot generate forgeries}

Policing issues may be better handled via abuse reports, when it will
be clear where they should be addressed.

> in our case every authenticated submission can only send envelope-from that users list of verified from address's, and mail servers can only send envelope-from their list of allowed domains {or from-any to their list of outbound-forwarding destinations}

Fine. However, if the "cumbersome" behavior described above has been
agreed between a forwarder and its target, there may be better options
than checking the envelope-from.

> so if a user here generates spam {uce or ube} it is entirely traceable and the NDR's are local and logged by us for monitoring so we can TOS them

Abuse reports would also allow to trace whether the upstream
transmitter blocks its spammy users after spam has been notified.

>> Thus, there should be no need to fine-tune downstream sites. It would then be sufficient to configure whitelistings at one site only, as standardized settings can easily be exchanged between compliant sites, if the given standardization provides a way for doing that.
>
> not really a good idea in my experience most users with more than one address want wildly different filtering on each
> so it would be bad to assume a forwarded address would want the same filtering as a receiving address
> {as for example some of our users have the forwarders receiving all kinds of crap but have the end receivers {usually dedicated address' for forward reception} simply setup to deny 0.0.0.0/0 except forwarders-ip's
> this obviously wouldn't want to be auto-propagated

Yes, we are saying the same thing. I just wrote it loosely: when I wrote

>> Thus, there should be no need to fine-tune downstream sites.

I meant

>> Thus, there should be no need to fine-tune downstream sites as far as the forwarded stuff is concerned.

Alice would configure her "other [at] fina" options using a different
form, still at the "final" webmail site.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

vesely at tana

Dec 1, 2009, 4:27 AM

Post #25 of 26 (4452 views)
Permalink

Re: How reliable is it to block/reject on SPF fail? [In reply to]

Stuart D. Gathman wrote:
> On Mon, 30 Nov 2009, Ian Eiloart wrote:
>
>> So, you'd probably want these domains hosted on some other mail server. Google
>> hosts my domain for free - I just pay the registration cost. I have a virtual
>> host, but can't be bothered to run my own mail server on it.
>
> While this might be ok for a casual user, I don't want my mail traversing
> google or any other provider in plaintext. I suppose I could use S/MIME
> or GPG, but TLS is no effort privacy measure that doesn't involve trying to
> tell non-geek correspondents how to create a key pair. When a
> correspondent has a small office mail server supporting TLS, adequate privacy
> is automatic.

+1: Trusting one's mailbox provider is fundamental.

> There are commercial products that are "plug and play" email appliances.
> Unfortunately, the most popular ones have serious RFC and general
> stupidity issues (e.g. replying to DSNs, sending "you have a virus"
> replies to emails they have already identified as sent by a virus, invalid
> HELO name, etc, etc).

Back to Ian's car metaphor, these issues would compare to knowing the
terrain, the roads, and the streets. Those appliances are buggy tom
toms that users without further knowledge shouldn't rely upon.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org [http://www.openspf.org]
Modify Your Subscription: http://www.listbox.com/member/ [http://www.listbox.com/member/]

Archives: https://www.listbox.com/member/archive/735/=now
RSS Feed: https://www.listbox.com/member/archive/rss/735/
Powered by Listbox: http://www.listbox.com

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads