Mailing List Archive: SPF: Devel

Test Suite Issue

Index | Next | Previous | View Threaded

scott at kitterman

Sep 24, 2006, 7:27 PM

Post #1 of 9 (2274 views)
Permalink

Test Suite Issue

I just got done fixing pySPF so that it does what RFC 4408 says for test:

redirect-cancels-exp:
description: >-
when executing "redirect", exp= from the original domain MUST NOT be
used.
spec: 6.2/13
helo: mail.example.com
host: 1.2.3.4
mailfrom: foo [at] e1
result: fail
explanation: DEFAULT

The problem is that since 'DEFAULT' is not the default fail explanation in
pySPF, the test still fails. It seems to me this is a test suite issue.
Ideally one would specify the result of this test as NOT 'No-see-um'.

Is there a way we can do that?

Scott K

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel [at] v2

julian at mehnle

Sep 25, 2006, 2:32 AM

Post #2 of 9 (2201 views)
Permalink

Re: Test Suite Issue [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Scott Kitterman wrote:
> I just got done fixing pySPF so that it does what RFC 4408 says for
> test:
>
> redirect-cancels-exp:
> description: >-
> when executing "redirect", exp= from the original domain MUST NOT
> be used.
> spec: 6.2/13
> helo: mail.example.com
> host: 1.2.3.4
> mailfrom: foo [at] e1
> result: fail
> explanation: DEFAULT
>
> The problem is that since 'DEFAULT' is not the default fail explanation
> in pySPF, the test still fails. It seems to me this is a test suite
> issue.

Not really. The test suite schema documentation[1] says:

| If the explanation attribute is the magic string DEFAULT, then the test
| driver should verify that the default explanation was returned. If the
| implementation supports setting the default explanation, this is easily
| accomplished by setting the default explanation to "DEFAULT".

The explanation string "DEFAULT" should not be taken literally.

References:
1. http://new.openspf.org/Test_Suite/Schema

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFF6IvwL7PKlBZWjsRAg7DAKDtEe6814iwvXVxXBTFpdx4y55K7gCfYowM
WXYK84EQ+rbbVBQ46h8+7G0=
=ZECU
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel [at] v2

scott at kitterman

Sep 25, 2006, 8:21 AM

Post #3 of 9 (2196 views)
Permalink

Re: Re: Test Suite Issue [In reply to]

On Mon, 25 Sep 2006 09:32:23 +0000 Julian Mehnle <julian [at] mehnle> wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Scott Kitterman wrote:
>> I just got done fixing pySPF so that it does what RFC 4408 says for
>> test:
>>
>> redirect-cancels-exp:
>> description: >-
>> when executing "redirect", exp= from the original domain MUST NOT
>> be used.
>> spec: 6.2/13
>> helo: mail.example.com
>> host: 1.2.3.4
>> mailfrom: foo [at] e1
>> result: fail
>> explanation: DEFAULT
>>
>> The problem is that since 'DEFAULT' is not the default fail explanation
>> in pySPF, the test still fails. It seems to me this is a test suite
>> issue.
>
>Not really. The test suite schema documentation[1] says:
>
>| If the explanation attribute is the magic string DEFAULT, then the test
>| driver should verify that the default explanation was returned. If the
>| implementation supports setting the default explanation, this is easily
>| accomplished by setting the default explanation to "DEFAULT".
>
>The explanation string "DEFAULT" should not be taken literally.
>
Oh, it seems like you actually expect people to read documentation? ;)

I'll take a look and see if I can deal with it in the pySPF specific test
driver.

Thanks,

Scott K

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel [at] v2

stuart at bmsi

Sep 25, 2006, 9:45 AM

Post #4 of 9 (2200 views)
Permalink

Re: Test Suite Issue [In reply to]

On Sun, 24 Sep 2006, Scott Kitterman wrote:

> I just got done fixing pySPF so that it does what RFC 4408 says for test:

> redirect-cancels-exp:
...

> The problem is that since 'DEFAULT' is not the default fail explanation in
> pySPF, the test still fails. It seems to me this is a test suite issue.
> Ideally one would specify the result of this test as NOT 'No-see-um'.
>
> Is there a way we can do that?

Your fix for redirect-cancels-exp broke (or maybe my preliminary attempts
to fix exp bugs broke):

q.set_default_explanation('DEFAULT')

You can see details of failing tests with -v:

$ python testspf.py -v
...
DEFAULT != SPF fail - not authorized
redirect-cancels-exp in rfc4408-tests.yml failed, 6.2/13
...

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel [at] v2

scott at kitterman

Sep 25, 2006, 9:59 AM

Post #5 of 9 (2187 views)
Permalink

Re: Test Suite Issue [In reply to]

On Monday 25 September 2006 12:45, Stuart D. Gathman wrote:
> On Sun, 24 Sep 2006, Scott Kitterman wrote:
> > I just got done fixing pySPF so that it does what RFC 4408 says for test:
> >
> > redirect-cancels-exp:
>
> ...
>
> > The problem is that since 'DEFAULT' is not the default fail explanation
> > in pySPF, the test still fails. It seems to me this is a test suite
> > issue. Ideally one would specify the result of this test as NOT
> > 'No-see-um'.
> >
> > Is there a way we can do that?
>
> Your fix for redirect-cancels-exp broke (or maybe my preliminary attempts
> to fix exp bugs broke):
>
> q.set_default_explanation('DEFAULT')
>
> You can see details of failing tests with -v:
>
> $ python testspf.py -v
> ...
> DEFAULT != SPF fail - not authorized
> redirect-cancels-exp in rfc4408-tests.yml failed, 6.2/13
> ...

Urg. What I did was set the EXP back to the default in pySPF just before
executing a redirect. I think that's the right way to do it.

What I had been thinking was the the test driver could pull the pySPF's
default fail EXP and use that as the expected result instead of default. I
haven't looked at testspf.py yet to get a real sense for if/how that would
work.

Scott K

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel [at] v2

stuart at bmsi

Sep 25, 2006, 11:18 AM

Post #6 of 9 (2198 views)
Permalink

Re: Test Suite Issue [In reply to]

On Mon, 25 Sep 2006, Scott Kitterman wrote:

> Urg. What I did was set the EXP back to the default in pySPF just before
> executing a redirect. I think that's the right way to do it.

I don't think that interacts correctly with an included policy that
contains a redirect. I'll add a test case for that.

> What I had been thinking was the the test driver could pull the pySPF's
> default fail EXP and use that as the expected result instead of default. I
> haven't looked at testspf.py yet to get a real sense for if/how that would
> work.

You don't want redirect to trash a callers default explanation that
they've gone to the trouble of setting with set_default_exp(). It
will likely contain an URL provided by or specific to the application
using pyspf.

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel [at] v2

scott at kitterman

Sep 25, 2006, 11:44 AM

Post #7 of 9 (2201 views)
Permalink

Re: Test Suite Issue [In reply to]

On Monday 25 September 2006 14:18, Stuart D. Gathman wrote:

> You don't want redirect to trash a callers default explanation that
> they've gone to the trouble of setting with set_default_exp(). It
> will likely contain an URL provided by or specific to the application
> using pyspf.

Makes sense. If you'll add the test case to the built in test, I'll clean up
the fix in the next couple of days.

Scott K

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel [at] v2

stuart at bmsi

Sep 25, 2006, 12:12 PM

Post #8 of 9 (2200 views)
Permalink

Re: Test Suite Issue [In reply to]

On Mon, 25 Sep 2006, Scott Kitterman wrote:

> > You don't want redirect to trash a callers default explanation that
> > they've gone to the trouble of setting with set_default_exp(). It
> > will likely contain an URL provided by or specific to the application
> > using pyspf.
>
> Makes sense. If you'll add the test case to the built in test, I'll clean up
> the fix in the next couple of days.

I cleaned up the fix. You needed to reset self.exp to dict(self.defexp),
not dict(self.EXPLANATIONS).

I also added another test that flunks pyspf for using "Macro Error"
as a sentinel - despite that being a legal expansion of %{l}.

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel [at] v2

scott at kitterman

Sep 25, 2006, 12:39 PM

Post #9 of 9 (2211 views)
Permalink

Re: Test Suite Issue [In reply to]

On Monday 25 September 2006 15:12, Stuart D. Gathman wrote:
> On Mon, 25 Sep 2006, Scott Kitterman wrote:
> > > You don't want redirect to trash a callers default explanation that
> > > they've gone to the trouble of setting with set_default_exp(). It
> > > will likely contain an URL provided by or specific to the application
> > > using pyspf.
> >
> > Makes sense. If you'll add the test case to the built in test, I'll
> > clean up the fix in the next couple of days.
>
> I cleaned up the fix. You needed to reset self.exp to dict(self.defexp),
> not dict(self.EXPLANATIONS).
>
Thanks. That's where I was headed if I'd had time.

> I also added another test that flunks pyspf for using "Macro Error"
> as a sentinel - despite that being a legal expansion of %{l}.

I assume you saw what I added for the 'c' macro?

It doesn't sound to me like there is a test in the suite to determine if all
Macro types are implemented. This would seem like a good thing to test
for...

Scott K

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?listname=spf-devel [at] v2

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads