Mailing List Archive: SPF: Devel

Re: Upcoming new test-suite release

1 2

View All

Index | Next | Previous | View Threaded

stuart at bmsi

Mar 26, 2008, 6:35 PM

Post #26 of 47 (7529 views)
Permalink

Re: Re: Upcoming new test-suite release [In reply to]

On Thu, 27 Mar 2008, Frank Ellermann wrote:

> <http://article.gmane.org/gmane.mail.spam.spf.devel/1831>
> <http://article.gmane.org/gmane.mail.spam.spf.discuss/23632>
> <http://article.gmane.org/gmane.mail.spam.spf.discuss/24048>
> <http://article.gmane.org/gmane.mail.spam.spf.discuss/24052>
>
> Inconclusive, two for "no match", two for "PermError", the
> latter based on 2.5.7.

I was for TempError myself. But no match is ok. The test suite
allowed no match or TempError until the latest change. It never
allowed PermError. I guess we all though it was just "so wrong",
at least for the macro expansion case. There should probably be
an errata to modify 2.5.7 if we go with no match.

For reference, 2.5.7/3 says:
"Be aware that if the domain owner uses macros (Section 8), it is possible that
this result is due to the checked identities having an unexpected format."

This does not say specifically that 'example..com' should be a PermError,
just that some unspecified condition might result in one.

Section 5 says:
"For these DNS queries, except where noted, if the DNS server returns an error
(RCODE other than 0 or 3) or the query times out, the mechanism throws the
exception "TempError". If the server returns "domain does not exist" (RCODE 3),
then evaluation of the mechanism continues as if the server returned no error
(RCODE 0) and zero answer records."

Since 'example..com' cannot actually be sent to a DNS server, it cannot
result in an RCODE other than 0 or 3, and cannot result in a timeout,
and hence cannot be a TempError. Besides, in the non-macro case, the error
is decidedly permanent, no temporary. That leaves 'no match' or PermError.

A PermError is wrong because in the macro case, the error is *not*
permanent, but may depend on the sender.

So 'no match' is the only consistent result. But the spec is far from
clear on the point, despite what Julian says.

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

julian at mehnle

Mar 28, 2008, 9:47 AM

Post #27 of 47 (7544 views)
Permalink

Re: Upcoming new test-suite release [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frank Ellermann wrote:
> Stuart D. Gathman wrote:
> > I was for TempError myself. But no match is ok.
>
> Okay, let's say TempError is out, depending on how we read RFC 4408 that
> alone is not necessarily an erratum.

Right. The issues of how RFC 4408 as-is is to be interpreted and whether
RFC 4408 should be amended via an erratum are generally separate.

> > There should probably be an errata to modify 2.5.7 if we go with no
> > match.
>
> ACK, "no match" is not what 2.5.7 proposes, if we want "no match" it
> needs an erratum.

<RFC authoring mode>
Again, I'm pretty sure that 2.5.7/1/3 was a mistake. Nowhere else
does RFC 4408 suggest an "after-macro-expansion syntax checking"
concept.
</RFC authoring mode>

<RFC interpretation mode>
The question however is: Given the present wording of RFC 4408,
should we accept implementations generating a PermError on "broken"
(i.e., valid syntax per RFC 4408 but invalid per RFCs 1034/1035/1123)
<target-name>s, as wrong as it may occur to us based on the RFC 4408
authoring background knowledge we posses?
</RFC interpretation mode>

<RFC authoring mode>
If we decide to tolerate PermError on broken <target-name>s, we cannot
simply remove 2.5.7/1/3 ...

| Be aware that if the domain owner uses macros (Section 8), it is
| possible that this result is due to the checked identities having an
| unexpected format.

... because we will then want implementors to be aware of the issue as
well! So we will either have to leave it as it is or amend it to say
that PermError is not an intended result for those cases, but that
there may be implementations working like that due to a legitimate
misinterpretation of earlier versions of the spec (including the
unamended RFC 4408).

If, however, we decide NOT to tolerate PermError, the best amendment
is probably to simply delete 2.5.7/1/3.
</RFC authoring mode>

I think we should make decisions in this order. The RFC 4408 test suite
depends solely on how we interpret RFC 4408 as it is now. Any RFC 4408
errata should then amend the RFC to clarify what it already says (albeit
contradictorily or otherwise ambiguously). In any case, RFC 4408 errata
are not supposed to modify the (originally intended) semantics of RFC
4408.

> > A PermError is wrong because in the macro case, the error is *not*
> > permanent, but may depend on the sender.
>
> Here we have a disconnect, the same MAIL FROM and HELO combo from the
> same IP with the same policy will always, permanently, reproducible
> fail.

Let's face it, PermError is far from clearly defined per RFC 4408. From
an entirely utilitarian point of view, all that can be said is that it
covers all error conditions that are not already covered by TempError,
whose semantics are a lot better defined in RFC 4408.

The real question then is: Does a "broken" <target-name> (either verba-
tim, e.g., "<more-than-63-chars>.com", or via macro expansion) need to be
_flagged_as_an_error_condition_ in the first place?

Had you asked me that question two years ago, I'd have said: definitely
yes! You may recall me having fought long and hard for making
SPF(NXDOMAIN) to be considered a case of PermError. It was all in vain;
the rough consensus was that checking for whether the authority domain
(whose policy is to be evaluated) actually exists was not SPF's problem
and that None should be returned. Likewise, consider 5/10/3 in section
5, "Mechanism Definitions":

| Several mechanisms rely on information fetched from DNS. [...] If the
| server returns "domain does not exist" (RCODE 3), then evaluation of the
| mechanism continues as if the server returned no error (RCODE 0) and
| zero answer records.

Now I think we ought to stick to that school of thought and treat "broken"
<target-name>s as no-match, too.

After all, given a policy using %{h}, why should "HELO whacky..spammer"
result in "PermError" when "HELO whacky.spammer" results in "Fail" or
"SoftFail"? (If anyone argues that "whacky..spammer" isn't even a valid
HELO per RFC 2821, consider %{l} and "MAIL FROM:<whacky..spammer@...>".)

As for the "mech:<more-than-63-chars>.com" case, I could be convinced that
this is a separate issue from the macro expansion one, however since it
clearly is NOT a syntax error per RFC 4408, it would still be hard to
justify a PermError result.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFH7SEXwL7PKlBZWjsRAmLiAJ0bri0uFkfAHMJYkiwo4djI5McfPQCeLqSM
/gC6CxQ1qZgeB1fNXVm4D7Y=
=YsYr
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

nobody at xyzzy

Mar 28, 2008, 1:44 PM

Post #28 of 47 (7542 views)
Permalink

Re: Upcoming new test-suite release [In reply to]

Julian Mehnle wrote:

> I think we should make decisions in this order. The RFC 4408 test
> suite depends solely on how we interpret RFC 4408 as it is now.

Yes, and as I never tried to implement my interpretation it's only
that, an interpretation, not running code.

I'd disagree with the point where you say that RFC 4408 considered
adjacent dots and overlong labels as valid <target-name>, I think
it simply forgot to discuss invalid <target-name>s apart from the
2.5.7 note.

IOW we screwed up, fortunately only in a totally obscure corner
case mostly relevant for test suites and errata. Maybe it could
be a vulnerability if spammers find servers where they like an
intentionally provoked Permerror better than no match, or v.v.,
depending on the receiver policy.

> Any RFC 4408 errata should then amend the RFC to clarify what it
> already says (albeit contradictorily or otherwise ambiguously).

Yep, additional 2.5.7 clarification for PermError, or stripping
the questionable statement for "no match" are good. I'm less
thrilled by "clarification that implemenations are free to pick
either PermError or no match, certainly not TempError", but it's
also possible.

Based on that, one thing is sure, just removing the section from
the errata page as "rejected" is out, whatever we pick, it needs
some text.

I think I can come up with a proposal for the three choices:

PermError => add note that this is about a syntactically invalid
<target-name> for DNS, for example adjacent dots.

No match => say that the 2.5.7 statement was wrong and should
be ignored, reflecting common "no match" practice.

Pick => say that some implementations skip a syntactically
invalid DNS <target-name>, e.g., adjacent dots.

> In any case, RFC 4408 errata are not supposed to modify the
> (originally intended) semantics of RFC 4408.

I fear we simply forgot to think about this corner case, and so
"originally intended semantics" equals a blind spot in our pre-
4408 reviews.

> Let's face it, PermError is far from clearly defined per RFC
> 4408.

Yeah, things can get tricky if many receivers decide to accept
PermError. In that case the spammers have to "educate" them ;-)

After that education, if PermError for legit mail simply means
"fix your policy", it's IMO fine. Admittedly the plan was to
limit it to "fix the policy because it's syntactically broken".

"Fix your policy" covering invalid <target-name> is still KISS.

> You may recall me having fought long and hard for making
> SPF(NXDOMAIN) to be considered a case of PermError.

Yes, but your concept was not simply "fix your policy", or was
it ? At that time we also had discussions about rejecting a
PermError at all, until we finally decided that it is receiver
policy.

> Now I think we ought to stick to that school of thought and
> treat "broken" <target-name>s as no-match, too.

I don't insist on it, I take the easy way out: Whatever the
test suite decides, and I'm aware that it already decided it,
but you could still change your mind, I have an idea what to
do with the "last erratum" as noted above. For TempError I'd
have no idea, but we have consensus that TempError is out.

> given a policy using %{h}, why should "HELO whacky..spammer"
> result in "PermError" when "HELO whacky.spammer" results in
> "Fail" or "SoftFail"?

Or "Neutral". The best decision depends on what you consider
as the possible *future* best common practice for receivers:

A rejected PermError is better than an accepted Neutral, and
a rejected Fail is better than an accepted PermError. As you
know I like SPF FAIL and hope that more domains will use it.

It is okay if you decide that "no match" does the trick.

We were not wasting time, until today I didn't have it clear
that a consistent erratum or clarification for 2.5.7 matching
any decision in the test suite is possible and required.

> for the "mech:<more-than-63-chars>.com" case, I could be
> convinced that this is a separate issue from the macro
> expansion one

Different, you could find it without instantiating macros.

> since it clearly is NOT a syntax error per RFC 4408, it
> would still be hard to justify a PermError result.

In the sense of "fix your policy" that is not hard, and it
is clearly an invalid label. Either a typo or a malicious
policy attacking the time resources of folks discussing
SPF test suites and errata, or another seriously bad idea.

I don't see how this case could ever result in an erratum.

Better decide the <target-name> handling first ignoring
this. After that decision it's easier, e.g. it could make
sense to pick the same solution for KISS and consistency,
it could also make sense to allow PermError or "no match"
for the overlong label.

Frank

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: http://www.listbox.com/member/archive/1007/=now
RSS Feed: http://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

julian at mehnle

Aug 15, 2008, 5:34 PM

Post #29 of 47 (7176 views)
Permalink

Re: Upcoming new test-suite release [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yet another attempt:

http://www.openspf.org/source/project/test-suite/rfc4408-tests.yml?view=diff&r1=98&r2=89
http://www.openspf.org/source/project/test-suite/rfc4408-tests.CHANGES?rev=98&view=auto

This should incorporate the resolutions of all the errata that were under
discussion since the 2007.05 test-suite release.

Any objections to making that a 2008.08 release?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkimIHoACgkQwL7PKlBZWju59ACgjyx423Mk6wH0C6RKn78IXuls
gIUAnRqLk+8JBMk8/QxN9FsTzKop7FAc
=fBIR
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

hmdmhdfmhdjmzdtjmzdtzktdkztdjz at gmail

Aug 16, 2008, 11:06 AM

Post #30 of 47 (7172 views)
Permalink

Re: Upcoming new test-suite release [In reply to]

Julian Mehnle wrote:

> Any objections to making that a 2008.08 release?

Looking only in the change log I see no problem.

Copied from my "last erratum" reply, because the
end was wrong; please add the "macro mania" tests:

| <http://article.gmane.org/gmane.mail.spam.spf.devel/1938>
|
| The rationale for the new "macro mania" tests was a
| developer trying to use a seriously limited .NET DNS
| API.
|
| The "quoted string" + "quoted pair" business will be
| a bit more complex, it could go into an update of the
| test suite when we have that clear.
|
| Using upper case %{S} in a <domain-spec> can also have
| odd effects, especially in conjunction with the "last"
| erratum limit 63:
|
- mail from: <123...25 [at] 789>
+ mail from: <123...31 [at] 345>
|
| "v=spf1 exists:%{S} ?all"
|
- .example (8) + .com (4) + 25 + @ (1) + 24 => 62. But
+ local part (31) + @ (1) + left label (30) => 62. But
| if the @ is URL encoded for an upper case S it is 64.

The "..." is not for three dots, it is an abbreviation
for omitted LDH fill characters (31 + @ + 30)

Frank

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

julian at mehnle

Aug 17, 2008, 5:30 AM

Post #31 of 47 (7168 views)
Permalink

Re: Upcoming new test-suite release [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frank Ellermann wrote:
> Julian Mehnle wrote:
> > Any objections to making that a 2008.08 release?
>
> Looking only in the change log I see no problem.

Good, thanks for lending your eyeballs. Release impending.

> Copied from my "last erratum" reply, because the end was wrong; please
> add the "macro mania" tests:

I added them, but rolled them up into a single test case ("macro-mania-
in-domain"). No need to be extra redundant.

Next stop: 2008.08 release announcement.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkioGeMACgkQwL7PKlBZWjtexgCg45tTST8dgzsdtFV4IJ56ZHxm
xeQAnR1TkiIRpooqhlzOZMSqfFQPi62F
=qhZP
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

hmdmhdfmhdjmzdtjmzdtzktdkztdjz at gmail

Aug 18, 2008, 7:00 AM

Post #32 of 47 (7169 views)
Permalink

Re: Upcoming new test-suite release [In reply to]

Julian Mehnle wrote:

> Next stop: 2008.08 release announcement.

Good, maybe add it to the "news" on the Web site,
we didn't have many news so far this year. Just
in case I mentioned the test suite in an SPF EAI
draft some weeks ago, with a link to the errata.

My SEO instincts suggest that the errata should
now get a link to the test suite, after all they
are covered (or at least supposed to be covered).

Frank
--
http://tools.ietf.org/html/draft-ellermann-spf-eai-03#section-3

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

stuart at bmsi

Aug 18, 2008, 9:18 AM

Post #33 of 47 (7154 views)
Permalink

Re: Re: Upcoming new test-suite release [In reply to]

On Mon, 18 Aug 2008, Frank Ellermann wrote:

> > Next stop: 2008.08 release announcement.

IMO, any test-suite changes should be double checked by running
against as many existing implementations as possible. I am running
against pyspf right now. Any new failures need to be diagnosed
as an implementation or test-suite problem. It is not enough just
to eyeball the changes.

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

stuart at bmsi

Aug 18, 2008, 10:18 AM

Post #34 of 47 (7164 views)
Permalink

Re: Re: Upcoming new test-suite release [In reply to]

On Mon, 18 Aug 2008, Stuart D. Gathman wrote:

> IMO, any test-suite changes should be double checked by running
> against as many existing implementations as possible. I am running
> against pyspf right now. Any new failures need to be diagnosed

Ok, looks like pyspf has picked up 3 new warnings - permerror is now
the preferred result for invalid domain. Ideally, the spec should
pick either nomatch or permerror so that results will be consistent. However,
what we ended up with isn't so bad. Most implementations will nomatch an
invalid domain, usually resulting in FAIL. Some will permerror instead,
hopefully providing some diagnostics to the domain owner. I am planning to add
a config option to pyspf for selecting the nomatch or permerror behaviour.

--
Stuart D. Gathman <stuart [at] bmsi>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

apache at bago

Aug 18, 2008, 11:02 AM

Post #35 of 47 (7162 views)
Permalink

Re: Re: Upcoming new test-suite release [In reply to]

Stuart D. Gathman ha scritto:
> On Mon, 18 Aug 2008, Frank Ellermann wrote:
>
>>> Next stop: 2008.08 release announcement.
>
> IMO, any test-suite changes should be double checked by running
> against as many existing implementations as possible. I am running
> against pyspf right now. Any new failures need to be diagnosed
> as an implementation or test-suite problem. It is not enough just
> to eyeball the changes.

I tested 2008.08 against jSPF and everything works.
I also tested current head for the test suite and we fail the macromania
test.

We didn't correctly expanded the %_ and %- (they was explanded to "_"
and "-" :-( ).

I have already an easy fix for that (It's very good to have this test
suite!!) ready to be committed.

At Apache JAMES project we are also planning to extract the test suite
for jSPF and publish it as a separate project (we currently have it in
our tests).

The jSPF tester have the ability to load a zone configuration from the
yaml file, set up a "fake" dsn server on a given port, read tests, run
the spf checker and check the results.

One issue I've found when trying our tester against other
implementations is that it is not easy to tell other implementations to
use a specific dns server/port for the resolution.

IMHO the jSPF tester could be a good way for OpenSPF to certify the
implementations, otherwise you will have to manually check if the test
suite for each implementation is correct or not.

Stefano

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

julian at mehnle

Aug 18, 2008, 1:47 PM

Post #36 of 47 (7157 views)
Permalink

Re: Upcoming new test-suite release [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Mon, 18 Aug 2008, Frank Ellermann wrote:
> > > Next stop: 2008.08 release announcement.
>
> IMO, any test-suite changes should be double checked by running
> against as many existing implementations as possible. I am running
> against pyspf right now. Any new failures need to be diagnosed
> as an implementation or test-suite problem. It is not enough just
> to eyeball the changes.

I agree with your thought, and I had run it against Mail::SPF (2.005 and
2.006) as well as pyspf (2.0.4 and 2.0.5) before pushing it out.
I didn't run it against any other implementations, though, mostly because
I don't trust any others as much as I trust Mail::SPF and pyspf.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkip38YACgkQwL7PKlBZWjvn4ACeOBQ6aBBYf5FmXe2cMhEfbVfU
jz4Animy6hDegTlTW0IAxadabxEuIu3W
=Na3k
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

julian at mehnle

Aug 18, 2008, 1:59 PM

Post #37 of 47 (7163 views)
Permalink

Re: Upcoming new test-suite release [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> I tested 2008.08 against jSPF and everything works.

Very good.

> I also tested current head for the test suite and we fail the
> macromania test.
>
> We didn't correctly expanded the %_ and %- (they was explanded to "_"
> and "-" :-( ).
>
> I have already an easy fix for that (It's very good to have this test
> suite!!) ready to be committed.

Even better! Good to know that the test suite helped expose those
mistakes. Mail::SPF 2.005 had a similar bug by expanding "%-" to "-", by
the way.

> The jSPF tester have the ability to load a zone configuration from the
> yaml file, set up a "fake" dsn server on a given port, read tests, run
> the spf checker and check the results.
>
> One issue I've found when trying our tester against other
> implementations is that it is not easy to tell other implementations to
> use a specific dns server/port for the resolution.

FWIW, it's all but trivial with Mail::SPF. One can specify a custom DNS
resolver object, e.g., a Net::DNS::Resolver object configured to always
use a specific resolving server.

> IMHO the jSPF tester could be a good way for OpenSPF to certify the
> implementations, otherwise you will have to manually check if the test
> suite for each implementation is correct or not.

How will this help getting rid of any manual procedures in testing the
conformity of a specification?

Besides, running an SPF implementations self-tests could be made (by
authors) as easy as `make test`. It even almost works with Mail::SPF --
you just have to run `perl Makefile.PL` beforehand.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkip4sgACgkQwL7PKlBZWjsJvACg/qM5EDSQepznc3TwAajuqGo9
njoAoNWmS/iDrp4LVauY7p0XLq3v54P+
=wEag
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

apache at bago

Aug 19, 2008, 1:26 AM

Post #38 of 47 (7160 views)
Permalink

Re: Re: Upcoming new test-suite release [In reply to]

Julian Mehnle ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Stefano Bagnara wrote:
>> I tested 2008.08 against jSPF and everything works.
>
> Very good.
>
>> I also tested current head for the test suite and we fail the
>> macromania test.
>>
>> We didn't correctly expanded the %_ and %- (they was explanded to "_"
>> and "-" :-( ).
>>
>> I have already an easy fix for that (It's very good to have this test
>> suite!!) ready to be committed.
>
> Even better! Good to know that the test suite helped expose those
> mistakes. Mail::SPF 2.005 had a similar bug by expanding "%-" to "-", by
> the way.
>
>> The jSPF tester have the ability to load a zone configuration from the
>> yaml file, set up a "fake" dsn server on a given port, read tests, run
>> the spf checker and check the results.
>>
>> One issue I've found when trying our tester against other
>> implementations is that it is not easy to tell other implementations to
>> use a specific dns server/port for the resolution.
>
> FWIW, it's all but trivial with Mail::SPF. One can specify a custom DNS
> resolver object, e.g., a Net::DNS::Resolver object configured to always
> use a specific resolving server.

I'm not a perl programmer, so I won't do this. A configuration option to
override the dns servers would do the trick.

>> IMHO the jSPF tester could be a good way for OpenSPF to certify the
>> implementations, otherwise you will have to manually check if the test
>> suite for each implementation is correct or not.
>
> How will this help getting rid of any manual procedures in testing the
> conformity of a specification?

This won't get rid of *any* manual procedure, but this at least save you
from checking the code quality correctness for the testsuite runner.

> Besides, running an SPF implementations self-tests could be made (by
> authors) as easy as `make test`. It even almost works with Mail::SPF --
> you just have to run `perl Makefile.PL` beforehand.

The difference between this test and "selftests" is big, IMHO.
Most "self-tests" I've seen around don't actually run a real UDP/TCP dns
query, but they mock the whole resolution mechanism at one or another layer.
Otherwise we could have a "selftest" returning "ok" for any test without
checking the real result.

e.g: jSPF supports synchronous, asynchronous and SEDA (Staged
Event-Driven Architecture) resolvers and testing against a real dns
server raised much more issues than our previous testsuite.

Having an external testsuite allow you to check bugs in the testsuites
too. Often people that wrote the code is the same that wrote the tester
code, this can result in obvious bugs to be left there because of
misunderstandings from the author.

BTW I don't care if you will or won't use the jSPF tester once we'll
publish it, but I'm a bit annoyed that OpenSPF page keep saying that the
only compliant implementations are pyspf 2 and Mail::SPF only because
OpenSPF members wrote that while jSPF has an "the jSPF library is
currently being evaluated by the project for RFC 4408 compliance" since
years. At the end of 2006 jSPF was listed as "The jSPF library is also
reported to pass all tests in the test suite, however this has not yet
been officially confirmed" and then on 2006-12-10 you removed the whole
sentence. On 2007-02-01 then the "currently being evaluated" has been
written there... 18 months passed since that...

IMHO it is bad for OpenSPF to fail to tell how implementations can be
listed between compliant implementation and ignore anyone but your own
implementations.

Stefano

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

julian at mehnle

Aug 20, 2008, 2:06 AM

Post #39 of 47 (7145 views)
Permalink

Re: Upcoming new test-suite release [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Julian Mehnle ha scritto:
> > Stefano Bagnara wrote:
> > > The jSPF tester have the ability to load a zone configuration from
> > > the yaml file, set up a "fake" dsn server on a given port, read
> > > tests, run the spf checker and check the results.
> > >
> > > One issue I've found when trying our tester against other
> > > implementations is that it is not easy to tell other implementations
> > > to use a specific dns server/port for the resolution.
> >
> > FWIW, it's all but trivial with Mail::SPF. One can specify a custom
> > DNS resolver object, e.g., a Net::DNS::Resolver object configured to
> > always use a specific resolving server.
>
> I'm not a perl programmer, so I won't do this. A configuration option
> to override the dns servers would do the trick.

Actually, I'd rather have the online version of the test suite driver
append a given zone name to every lookup performed, e.g.,
2008_08.rfc4408.test-suite.openspf.org. That way you wouldn't have to
specify a custom resolver server. That zone can then be delegated to
your fake DNS server.

> > Besides, running an SPF implementations self-tests could be made (by
> > authors) as easy as `make test`. It even almost works with Mail::SPF
> > -- you just have to run `perl Makefile.PL` beforehand.
>
> The difference between this test and "selftests" is big, IMHO.
> Most "self-tests" I've seen around don't actually run a real UDP/TCP
> dns query, but they mock the whole resolution mechanism at one or
> another layer. Otherwise we could have a "selftest" returning "ok" for
> any test without checking the real result.

I don't see the difference as being all that big, given that the test
suite drivers are probably rather trivial, but I certainly don't object
to have SPF implementations tested against live DNS. If you're willing
to help with setting this up, or even lead the effort, all the better!

> BTW I don't care if you will or won't use the jSPF tester once we'll
> publish it, but I'm a bit annoyed that OpenSPF page keep saying that
> the only compliant implementations are pyspf 2 and Mail::SPF only
> because OpenSPF members wrote that while jSPF has an "the jSPF library
> is currently being evaluated by the project for RFC 4408 compliance"
> since years. At the end of 2006 jSPF was listed as "The jSPF library is
> also reported to pass all tests in the test suite, however this has not
> yet been officially confirmed" and then on 2006-12-10 you removed the
> whole sentence. On 2007-02-01 then the "currently being evaluated" has
> been written there... 18 months passed since that...
>
> IMHO it is bad for OpenSPF to fail to tell how implementations can be
> listed between compliant implementation and ignore anyone but your own
> implementations.

As Stuart said, this isn't in any way due to a desire for self-promotion.
The reason that the website still says "The jSPF library is currently
being evaluated by the project for RFC 4408 compliance" simply is that
when I tried to run the (offline) test-suite against jSPF over a year
ago, it was very complicated to get it to work for someone who isn't an
experienced Java/Maven developer, and that the generated output was
practically unreadable, and no one else capable of editing the website
has since tried checking out jSPF's compliance. I admit that it was my
fault not to have announced that I had given up on trying to get useful
output from my jSPF test-suite runs, and hadn't asked that someone else
give it a try.

So please don't be mad, but let's now try to get this resolved instead.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkir3rIACgkQwL7PKlBZWjt+hQCgvaUqjODGqBHoBGCj6DbWJIzo
qQUAn1lV0LJR/sbXPowmNdrKSfu1YRG1
=jZrW
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

apache at bago

Aug 20, 2008, 2:27 AM

Post #40 of 47 (7147 views)
Permalink

Re: Re: Upcoming new test-suite release [In reply to]

Julian Mehnle ha scritto:
> Stefano Bagnara wrote:
>> IMHO it is bad for OpenSPF to fail to tell how implementations can be
>> listed between compliant implementation and ignore anyone but your own
>> implementations.
>
> As Stuart said, this isn't in any way due to a desire for self-promotion.
> The reason that the website still says "The jSPF library is currently
> being evaluated by the project for RFC 4408 compliance" simply is that
> when I tried to run the (offline) test-suite against jSPF over a year
> ago, it was very complicated to get it to work for someone who isn't an
> experienced Java/Maven developer, and that the generated output was
> practically unreadable, and no one else capable of editing the website
> has since tried checking out jSPF's compliance. I admit that it was my
> fault not to have announced that I had given up on trying to get useful
> output from my jSPF test-suite runs, and hadn't asked that someone else
> give it a try.

Probably running java applications is difficult for you as running perl
applications for me.
How easy is for me (let's say I only understand java) to run the
testsuite for other implementations?? Hard. It's not jSPF that is hard,
it is simply that the test suite for a library is usually written in the
library own language and using that very language testing methodologies.
We use jUnit and we wrapped the yaml tests into jUnit tests, so it is
really easy to run our test suite graphically in any jUnit compatible
environment.

> So please don't be mad, but let's now try to get this resolved instead.

My complain is that either you publish a procedure that I will try to
follow or you say that jSPF is compliant. It doesn't work and it's not
fair for me that you say "I don't know Java so your implementation will
not be listed".

Stefano

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

julian at mehnle

Aug 20, 2008, 3:39 AM

Post #41 of 47 (7140 views)
Permalink

Re: Upcoming new test-suite release [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Probably running java applications is difficult for you as running perl
> applications for me.
> How easy is for me (let's say I only understand java) to run the
> testsuite for other implementations?? Hard. It's not jSPF that is hard,
> it is simply that the test suite for a library is usually written in
> the library own language and using that very language testing
> methodologies. We use jUnit and we wrapped the yaml tests into jUnit
> tests, so it is really easy to run our test suite graphically in any
> jUnit compatible environment.
>
> > So please don't be mad, but let's now try to get this resolved
> > instead.
>
> My complain is that either you publish a procedure that I will try to
> follow or you say that jSPF is compliant. It doesn't work and it's not
> fair for me that you say "I don't know Java so your implementation will
> not be listed".

We don't have a formal procedure, and I don't think establishing one will
be worthwhile to any extent. We should simply strive to be fair. If we
aren't, please hit us on the head.

The issue with your implementation wasn't that I don't _speak_ Java, but
that there was no good documentation on how to set it up (cf.
http://search.cpan.org/src/JMEHNLE/Mail-SPF-v2.006/INSTALL if you will)
for someone who isn't familiar with know how to _use_ Java, Maven, and
jUnit, and, even more importantly, the test output was incomprehensible
to me. Perhaps that has since changed, but back then that was just how
it was.

Can you produce somethink like the following?

http://julian.io.link-m.de/tmp/mail-spf-perl-rfc4408-tests.log

You'll see that this isn't exactly good proof that an implementation
passes the test suite, but it at least demonstrates that the test suite
has been run in some way.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkir9GwACgkQwL7PKlBZWjuJagCfYnpMHnUoz0aMjZuBdVpxHmAM
MJgAoLFZHUwmJck3W/GIgrxJ2Xp26xaj
=l5+w
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

apache at bago

Aug 20, 2008, 3:54 AM

Post #42 of 47 (7154 views)
Permalink

Re: Re: Upcoming new test-suite release [In reply to]

Julian Mehnle ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Stefano Bagnara wrote:
>> Probably running java applications is difficult for you as running perl
>> applications for me.
>> How easy is for me (let's say I only understand java) to run the
>> testsuite for other implementations?? Hard. It's not jSPF that is hard,
>> it is simply that the test suite for a library is usually written in
>> the library own language and using that very language testing
>> methodologies. We use jUnit and we wrapped the yaml tests into jUnit
>> tests, so it is really easy to run our test suite graphically in any
>> jUnit compatible environment.
>>
>>> So please don't be mad, but let's now try to get this resolved
>>> instead.
>> My complain is that either you publish a procedure that I will try to
>> follow or you say that jSPF is compliant. It doesn't work and it's not
>> fair for me that you say "I don't know Java so your implementation will
>> not be listed".
>
> We don't have a formal procedure, and I don't think establishing one will
> be worthwhile to any extent. We should simply strive to be fair. If we
> aren't, please hit us on the head.
>
> The issue with your implementation wasn't that I don't _speak_ Java, but
> that there was no good documentation on how to set it up (cf.
> http://search.cpan.org/src/JMEHNLE/Mail-SPF-v2.006/INSTALL if you will)
> for someone who isn't familiar with know how to _use_ Java, Maven, and
> jUnit, and, even more importantly, the test output was incomprehensible
> to me. Perhaps that has since changed, but back then that was just how
> it was.

In every distribution we have BUILDING.txt:
-----
#############################################################################
# BUILDING JSPF
#############################################################################

You will need to install maven2 as well as acquire jSPF source from
subversion
or a source tarball.

Steps:

1) Install maven2 (v2.0.7 as of the time of this writing)

2) Add maven2 to your path. For me, I do the following:
$ tar zxvf maven-2.0.7.tar.gz
$ mv maven-2.0.7 /usr/local
$ ln -sf /usr/local/maven-2.0.7/bin/mvn /usr/local/bin/mvn

3) Change directory to jspf source dir
$ cd jspf-x/

4) Run the build (the "-Plocal" restrict maven to use bundled dependencies)
$ mvn -Plocal package
-----------------

It doesn't seems more difficult than the instructions you give for the
perl implementation.

> Can you produce somethink like the following?
>
> http://julian.io.link-m.de/tmp/mail-spf-perl-rfc4408-tests.log

A list of tests with an "ok" on their side??
The above "mvn package" command will run tests. The output will be more
verbose, but if any failure happens the package is not build and a
"BUILD FAILURE" appears.

> You'll see that this isn't exactly good proof that an implementation
> passes the test suite, but it at least demonstrates that the test suite
> has been run in some way.

We run it as part of our continuos integration build.
We run it at least 200 times per day and at least once per each commit.

If you just need a log of the test run to certificate jSPF I'm happy..
I'll send you the next output.

Stefano

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

apache at bago

Aug 20, 2008, 4:00 AM

Post #43 of 47 (7148 views)
Permalink

Re: Re: Upcoming new test-suite release [In reply to]

Stefano Bagnara ha scritto:
> Julian Mehnle ha scritto:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Stefano Bagnara wrote:
>>> Probably running java applications is difficult for you as running perl
>>> applications for me.
>>> How easy is for me (let's say I only understand java) to run the
>>> testsuite for other implementations?? Hard. It's not jSPF that is hard,
>>> it is simply that the test suite for a library is usually written in
>>> the library own language and using that very language testing
>>> methodologies. We use jUnit and we wrapped the yaml tests into jUnit
>>> tests, so it is really easy to run our test suite graphically in any
>>> jUnit compatible environment.
>>>
>>>> So please don't be mad, but let's now try to get this resolved
>>>> instead.
>>> My complain is that either you publish a procedure that I will try to
>>> follow or you say that jSPF is compliant. It doesn't work and it's not
>>> fair for me that you say "I don't know Java so your implementation will
>>> not be listed".
>>
>> We don't have a formal procedure, and I don't think establishing one
>> will be worthwhile to any extent. We should simply strive to be
>> fair. If we aren't, please hit us on the head.
>>
>> The issue with your implementation wasn't that I don't _speak_ Java,
>> but that there was no good documentation on how to set it up (cf.
>> http://search.cpan.org/src/JMEHNLE/Mail-SPF-v2.006/INSTALL if you
>> will) for someone who isn't familiar with know how to _use_ Java,
>> Maven, and jUnit, and, even more importantly, the test output was
>> incomprehensible to me. Perhaps that has since changed, but back then
>> that was just how it was.
>
> In every distribution we have BUILDING.txt:
> -----
> #############################################################################
>
> # BUILDING JSPF
> #############################################################################
>
>
> You will need to install maven2 as well as acquire jSPF source from
> subversion
> or a source tarball.
>
> Steps:
>
> 1) Install maven2 (v2.0.7 as of the time of this writing)
>
> 2) Add maven2 to your path. For me, I do the following:
> $ tar zxvf maven-2.0.7.tar.gz
> $ mv maven-2.0.7 /usr/local
> $ ln -sf /usr/local/maven-2.0.7/bin/mvn /usr/local/bin/mvn
>
> 3) Change directory to jspf source dir
> $ cd jspf-x/
>
> 4) Run the build (the "-Plocal" restrict maven to use bundled dependencies)
> $ mvn -Plocal package
> -----------------
>
> It doesn't seems more difficult than the instructions you give for the
> perl implementation.
>
>> Can you produce somethink like the following?
>>
>> http://julian.io.link-m.de/tmp/mail-spf-perl-rfc4408-tests.log
>
> A list of tests with an "ok" on their side??
> The above "mvn package" command will run tests. The output will be more
> verbose, but if any failure happens the package is not build and a
> "BUILD FAILURE" appears.
>
>> You'll see that this isn't exactly good proof that an implementation
>> passes the test suite, but it at least demonstrates that the test
>> suite has been run in some way.
>
> We run it as part of our continuos integration build.
> We run it at least 200 times per day and at least once per each commit.
>
> If you just need a log of the test run to certificate jSPF I'm happy..
> I'll send you the next output.

Here is the last successful build test result. I think it's more fancy
than your perl log, isn't it?

http://hudson.zones.apache.org/hudson/view/James/job/jspf-trunk/402/org.apache.james.jspf$resolver/testReport/org.apache.james.jspf/RFC4408YamlTest/

We also run more tests as you can see here:
http://hudson.zones.apache.org/hudson/view/James/job/jspf-trunk/402/org.apache.james.jspf$resolver/testReport/org.apache.james.jspf/

Because we also have an asynchronous resolver.

The test above is for our latest trunk tested against the latest yaml
testsuite from your trunk.

We only make releases when our product passes all of the last yaml suite
tests.

In fact our build tools reject to build jSPF at all if there are failures.

Stefano

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

julian at mehnle

Aug 20, 2008, 5:00 AM

Post #44 of 47 (7143 views)
Permalink

Re: Upcoming new test-suite release [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Stefano Bagnara ha scritto:
> > Julian Mehnle ha scritto:
> > > You'll see that this isn't exactly good proof that an implementation
> > > passes the test suite, but it at least demonstrates that the test
> > > suite has been run in some way.
> >
> > We run it as part of our continuos integration build.
> > We run it at least 200 times per day and at least once per each
> > commit.
> >
> > If you just need a log of the test run to certificate jSPF I'm
> > happy.. I'll send you the next output.
>
> Here is the last successful build test result. I think it's more fancy
> than your perl log, isn't it?

This isn't about being fancy.

> http://hudson.zones.apache.org/hudson/view/James/job/jspf-trunk/402/org.apache.james.jspf$resolver/testReport/org.apache.james.jspf/RFC4408YamlTest/

That looks acceptable (though as I said it isn't exactly good proof). Is
there a _release_ version of jSPF that passes _that_release_ of the test
suite? And which release of the test suite was that? 2007.05?

> The test above is for our latest trunk tested against the latest yaml
> testsuite from your trunk.

Please don't use "trunk" versions of our test suite (i.e., do not use the
file named "rfc4408-tests.yml"). That's the development version. Please
use the files named "rfc4408-tests-YYYY.MM.yml" instead.

> We only make releases when our product passes all of the last yaml
> suite tests.
>
> In fact our build tools reject to build jSPF at all if there are
> failures.

That's certainly good practice.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkisB0AACgkQwL7PKlBZWjvj5wCgm9HzVgFYUcmNN+ugjhmrK1d+
JRQAn1xdxtdfo3HR7rn7hESbpq9q1gph
=lJxP
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

apache at bago

Aug 20, 2008, 5:10 AM

Post #45 of 47 (7140 views)
Permalink

Re: Re: Upcoming new test-suite release [In reply to]

Julian Mehnle ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Stefano Bagnara wrote:
>> Stefano Bagnara ha scritto:
>>> Julian Mehnle ha scritto:
>>>> You'll see that this isn't exactly good proof that an implementation
>>>> passes the test suite, but it at least demonstrates that the test
>>>> suite has been run in some way.
>>> We run it as part of our continuos integration build.
>>> We run it at least 200 times per day and at least once per each
>>> commit.
>>>
>>> If you just need a log of the test run to certificate jSPF I'm
>>> happy.. I'll send you the next output.
>> Here is the last successful build test result. I think it's more fancy
>> than your perl log, isn't it?
>
> This isn't about being fancy.
>
>> http://hudson.zones.apache.org/hudson/view/James/job/jspf-trunk/402/org.apache.james.jspf$resolver/testReport/org.apache.james.jspf/RFC4408YamlTest/
>
> That looks acceptable (though as I said it isn't exactly good proof). Is
> there a _release_ version of jSPF that passes _that_release_ of the test
> suite? And which release of the test suite was that? 2007.05?

Out last release (jSPF 0.9.6) has been done when 2007.05 was out and the
tests passed against 2007.05. jSPF 0.9.6 also fully pass 2007.08, it has
1 issue in macromania that you added after 2007.08 in trunk.

Our macromania bug has been fixed in our trunk and the above report is
for the trunk version and your trunk testsuite including the macromania
test and our fix.

>> The test above is for our latest trunk tested against the latest yaml
>> testsuite from your trunk.
>
> Please don't use "trunk" versions of our test suite (i.e., do not use the
> file named "rfc4408-tests.yml"). That's the development version. Please
> use the files named "rfc4408-tests-YYYY.MM.yml" instead.

I use trunk in our trunk, so I can spot any issue as soon as possible.
In past it happened that openspf added tests that we didn't pass and we
complained for that and you finally removed them because they was
controversial, so I think it is good to test against trunk in our dev
snapshot.

I'll take care to add both the latest official and the latest trunk in
our test tree, so that we always check both suites.

Stefano

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

julian at mehnle

Aug 20, 2008, 7:35 AM

Post #46 of 47 (7148 views)
Permalink

Re: Upcoming new test-suite release [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Out last release (jSPF 0.9.6) has been done when 2007.05 was out and
> the tests passed against 2007.05. jSPF 0.9.6 also fully pass 2007.08,
> it has 1 issue in macromania that you added after 2007.08 in trunk.

There was no 2007.08. Do you mean 2008.08? If that's what you meant,
then there is nothing that has been added after the release of 2008.08
other than a cosmetic change to the spec reference field of the
"multitxt2" (not macromania) test.

> Our macromania bug has been fixed in our trunk and the above report is
> for the trunk version and your trunk testsuite including the macromania
> test and our fix.

Usually people use only release versions, so it is important that we tell
them which test suite release your latest release version conforms to.

Can the jSPF trunk version be downloaded other than by checking it out
from the Subversion repository? Or is the fix included in a nightly
build (assuming there even is a standalone nightly build for jSPF)?

If not, I suppose we should record it as follows:

2007.05: since 0.9.6

> >> The test above is for our latest trunk tested against the latest
> >> yaml testsuite from your trunk.
> >
> > Please don't use "trunk" versions of our test suite (i.e., do not use
> > the file named "rfc4408-tests.yml"). That's the development version.
> > Please use the files named "rfc4408-tests-YYYY.MM.yml" instead.
>
> I use trunk in our trunk, so I can spot any issue as soon as possible.
> In past it happened that openspf added tests that we didn't pass and we
> complained for that and you finally removed them because they was
> controversial, so I think it is good to test against trunk in our dev
> snapshot.
>
> I'll take care to add both the latest official and the latest trunk in
> our test tree, so that we always check both suites.

This may mislead you because sometimes we add or modify tests in the test
suite's trunk in a temporary fashion, e.g., when an issue is still under
discussion. Thus the change may get reversed later in a proper release
of the test suite. It has happened before.

In any case, compliance statements should only ever refer to release
versions of the test suite, so trying to comply with a trunk version
isn't going to be of much value.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkisK7sACgkQwL7PKlBZWju3xACgkmcToY1dKr8Smauumhu4ddAQ
macAnAqC7WO2toadGGQBE7pIIDBcLRxI
=xlt1
-----END PGP SIGNATURE-----

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

apache at bago

Aug 20, 2008, 10:22 AM

Post #47 of 47 (7144 views)
Permalink

Re: Re: Upcoming new test-suite release [In reply to]

Julian Mehnle ha scritto:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Stefano Bagnara wrote:
>> Out last release (jSPF 0.9.6) has been done when 2007.05 was out and
>> the tests passed against 2007.05. jSPF 0.9.6 also fully pass 2007.08,
>> it has 1 issue in macromania that you added after 2007.08 in trunk.
>
> There was no 2007.08. Do you mean 2008.08? If that's what you meant,
> then there is nothing that has been added after the release of 2008.08
> other than a cosmetic change to the spec reference field of the
> "multitxt2" (not macromania) test.

sorry, you are right.

with 2008.08 jSPF 0.9.6 fails the macromania test because %- and %_ are
incorrectly expanded to - and _.
I fixed that as soon as I saw the macromania test in openspf testsuite.

>> Our macromania bug has been fixed in our trunk and the above report is
>> for the trunk version and your trunk testsuite including the macromania
>> test and our fix.
>
> Usually people use only release versions, so it is important that we tell
> them which test suite release your latest release version conforms to.

jSPF 0.9.6 conforms to (and includes) rfc4408-tests-2007.05.yml
The next release will pass the current specification. The only known
issue with jSPF 0.9.6 is the macro expansion above.

> Can the jSPF trunk version be downloaded other than by checking it out
> from the Subversion repository? Or is the fix included in a nightly
> build (assuming there even is a standalone nightly build for jSPF)?

Our continuos integration server produce "lastStableBuild" artifacts:
http://hudson.zones.apache.org/hudson/view/James/job/jspf-trunk/lastStableBuild/org.apache.james.jspf$resolver/
They includes binaries and sources for the last version that passed
every test (in this case the current trunk is working).

At ASF (Apache Software Foundation) we have a strict release process and
we never recommend the use of snapshots and we never point users to
nightlies or snapshots.

> If not, I suppose we should record it as follows:
>
> 2007.05: since 0.9.6

If you tell me that it won't take 2 more years to update the page I'll
ask a new run as soon as we'll release 0.9.7. We'll probably wait a bit
more because we want to include newer dns libraries so to fix any
Kaminsky DNS vulnerability and a race issue in the asynchronous version
of our resolver.

>>>> The test above is for our latest trunk tested against the latest
>>>> yaml testsuite from your trunk.
>>> Please don't use "trunk" versions of our test suite (i.e., do not use
>>> the file named "rfc4408-tests.yml"). That's the development version.
>>> Please use the files named "rfc4408-tests-YYYY.MM.yml" instead.
>> I use trunk in our trunk, so I can spot any issue as soon as possible.
>> In past it happened that openspf added tests that we didn't pass and we
>> complained for that and you finally removed them because they was
>> controversial, so I think it is good to test against trunk in our dev
>> snapshot.
>>
>> I'll take care to add both the latest official and the latest trunk in
>> our test tree, so that we always check both suites.
>
> This may mislead you because sometimes we add or modify tests in the test
> suite's trunk in a temporary fashion, e.g., when an issue is still under
> discussion. Thus the change may get reversed later in a proper release
> of the test suite. It has happened before.

I know it happened. I was one of the people complaining to have it
reverted ;-)

I check trunk by purpose because I want to be aware of this issues ASAP
so that I can bring my opinion to this list before you release a bad
testsuite.

> In any case, compliance statements should only ever refer to release
> versions of the test suite

I agree.

> so trying to comply with a trunk version
> isn't going to be of much value.

I found it invaluable in past and it is not a big cost for us.

Stefano

-------------------------------------------
Sender Policy Framework: http://www.openspf.org
Modify Your Subscription: http://www.listbox.com/member/
Archives: https://www.listbox.com/member/archive/1007/=now
RSS Feed: https://www.listbox.com/member/archive/rss/1007/
Powered by Listbox: http://www.listbox.com

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads