Mailing List Archive: SPF: Devel

Testing jSPF's RFC conformance (was: jSPF-0.9b3 was released)

Index | Next | Previous | View Threaded

julian at mehnle

Dec 14, 2006, 5:00 AM

Post #1 of 16 (2237 views)
Permalink

Testing jSPF's RFC conformance (was: jSPF-0.9b3 was released)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Norman Maurer wrote:
> just want to let you know that we finally release the third beta of our
> pure java spf implementation called jSPF. You can grab the release from:
> http://people.apache.org/dist/james/jspf/
>
> This release pass all YAML tests published by openspf.org (still under
> development). So it should be fully RFC 4408 conform.
>
> For more information see:
> http://james.apache.org/jspf/

Norman, as you might have noticed, the test suite has been released as
revision 2006.11 recently. We would like to list jSPF as confirmedly RFC-
compliant. To that end, can you give us instructions for how to run the
test suite against jSPF ourselves and see if it (still) passes all the
tests?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFgUrqwL7PKlBZWjsRAnRyAJ0VTdVm8yPtqr7aCI2Sg/xaLS4QgQCgjT0y
LmP1Wjihkbs4W/ScyQHSiPA=
=/Esx
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

nm at spam-box

Dec 14, 2006, 5:24 AM

Post #2 of 16 (2185 views)
Permalink

Re: Testing jSPF's RFC conformance [In reply to]

Hi Julian,

just checkout the svn version of jspf and run the junit tests. We
include the testsuite (revision 2006.11) in our junit tests. you can
run the junit tests even with eclipse or with maven. And yes we pass all
at the moment :-)

After you checked out the svn you can run the junit tests with :
mvn test

If you need more instruction howto use maven etc just ask :-)

bye
Norman

Julian Mehnle schrieb:
> Norman Maurer wrote:
> > just want to let you know that we finally release the third beta of our
> > pure java spf implementation called jSPF. You can grab the release from:
> > http://people.apache.org/dist/james/jspf/
>
> > This release pass all YAML tests published by openspf.org (still under
> > development). So it should be fully RFC 4408 conform.
>
> > For more information see:
> > http://james.apache.org/jspf/
>
> Norman, as you might have noticed, the test suite has been released as
> revision 2006.11 recently. We would like to list jSPF as confirmedly
> RFC-
> compliant. To that end, can you give us instructions for how to run the
> test suite against jSPF ourselves and see if it (still) passes all the
> tests?
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe[at]james.apache.org
For additional commands, e-mail: server-dev-help[at]james.apache.org

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

julian at mehnle

Dec 20, 2006, 5:27 AM

Post #3 of 16 (2178 views)
Permalink

Re: Testing jSPF's RFC conformance [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Norman wrote:
> just checkout the svn version of jspf and run the junit tests. We
> include the testsuite (revision 2006.11) in our junit tests. you can
> run the junit tests even with eclipse or with maven. And yes we pass all
> at the moment :-)
>
> After you checked out the svn you can run the junit tests with :
> mvn test
>
> If you need more instruction howto use maven etc just ask :-)

I have installed neither Maven nor Eclipse. Can the tests be run using JRE
only, or is that non-trivial?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFiTpPwL7PKlBZWjsRAgSfAKCZPOXnU31FASKVvVtvVBDDyDvY2wCgyKlK
PgGzbLdejGG5gJteAjBPLek=
=1FiT
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

nm at byteaction

Dec 20, 2006, 5:31 AM

Post #4 of 16 (2178 views)
Permalink

Re: Re: Testing jSPF's RFC conformance [In reply to]

Hi Julian,
thats impossible (if im not wrong). Cause maven is used as building tool
and junit as test framework.

But if you want i can send you a "test protocol" of the the rfc4404
tests, so you can see all tests pass.

bye
Norman

Julian Mehnle schrieb:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Norman wrote:
>
>> just checkout the svn version of jspf and run the junit tests. We
>> include the testsuite (revision 2006.11) in our junit tests. you can
>> run the junit tests even with eclipse or with maven. And yes we pass all
>> at the moment :-)
>>
>> After you checked out the svn you can run the junit tests with :
>> mvn test
>>
>> If you need more instruction howto use maven etc just ask :-)
>>
>
> I have installed neither Maven nor Eclipse. Can the tests be run using JRE
> only, or is that non-trivial?
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.5 (GNU/Linux)
>
> iD8DBQFFiTpPwL7PKlBZWjsRAgSfAKCZPOXnU31FASKVvVtvVBDDyDvY2wCgyKlK
> PgGzbLdejGG5gJteAjBPLek=
> =1FiT
> -----END PGP SIGNATURE-----
>
> -------
> To unsubscribe, change your address, or temporarily deactivate your subscription,
> please go to http://v2.listbox.com/member/?list_id=1007
> !EXCUBATOR:1,45893a7244671120617705!
>

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

stuart at bmsi

Dec 20, 2006, 7:46 AM

Post #5 of 16 (2176 views)
Permalink

Re: Re: Testing jSPF's RFC conformance [In reply to]

On Wed, 20 Dec 2006, Norman Maurer wrote:

> thats impossible (if im not wrong). Cause maven is used as building tool
> and junit as test framework.

It should be possible for someone to install junit (just a jar) and run the
tests. Could you include the top level driver in the jSPF jar? That is what I
do for Java libraries. For a package com.bmsi.foo, I include a
com.bmsi.foo.Test class that will run all junit tests for that package.
Provides basic assurance when installing Java applications.

--
Stuart D. Gathman <stuart[at]bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

julian at mehnle

Dec 21, 2006, 10:12 AM

Post #6 of 16 (2168 views)
Permalink

Re: Testing jSPF's RFC conformance [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Norman Maurer wrote:
> just checkout the svn version of jspf and run the junit tests. We
> include the testsuite (revision 2006.11) in our junit tests. you can
> run the junit tests even with eclipse or with maven. And yes we pass all
> at the moment :-)
>
> After you checked out the svn you can run the junit tests with :
> mvn test

I think I managed to run `mvn test`, but it produced too much debug output
for me to read through. It also seemed to run a lot of other tests
besides the official RFC 4408 test suite.

Is there a way to disable the debug output (such as the internal warnings
on SPF syntax errors, etc.) and the other, non-official tests and just get
the official test results with the result produced by jSPF and some basic
diagnostic output (such as result explanations and {Temp,Perm}Error
messages)?

> But if you want i can send you a "test protocol" of the the rfc4404
> tests, so you can see all tests pass.

No, thank you. I'd really like to reproduce the test results myself (in a
readable format). :-)

BTW, Maven seems to have downloaded several packages during my first `mvn
test` run, but I can't find where it put them. I'd like to get rid of
them after the testing, so could you perhaps give me a hint where the
stuff got downloaded to? I checked jSPF out to "D:\Storage\Temporary\jSPF
\0.9b3" and ran `mvn test` in that same dir.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFis57wL7PKlBZWjsRAlLZAKCTBWQjoHMiPnWzDR8Al4zZqhyzxgCg8Koe
C9yPPlObGBt6Zl8aJoPTsI0=
=ZuHq
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

julian at mehnle

Dec 21, 2006, 11:14 AM

Post #7 of 16 (2168 views)
Permalink

Re: Testing jSPF's RFC conformance [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Norman Maurer wrote:
> > I think I managed to run `mvn test`, but it produced too much debug
> > output for me to read through. It also seemed to run a lot of other
> > tests besides the official RFC 4408 test suite.
>
> I'm not sure, but you don't need to read the result. It is only debug
> stuff. You just need to know that tests passes ;-)

Well, then does the following SPF implementation pass the test suite, too?

| #!/usr/bin/perl
| { local $/; <> }
| print("Test #$_: PASS\n") for 1..278;

I'm not saying that you're trying to trick us, but a little less noise and
a little more information would go a long way towards helping us see how
well jSPF implements RFC 4408.

> > Is there a way to disable the debug output (such as the internal
> > warnings on SPF syntax errors, etc.) and the other, non-official tests
> > and just get the official test results with the result produced by
> > jSPF and some basic diagnostic output (such as result explanations and
> > {Temp,Perm}Error messages)?
>
> No without changing the test code.
> Why do you need this?
> [...]
> Test result is pass/fail. If it finish without errors then it passed all
> the tests...

For example, this is how the output of a Mail::SPF test script looks:

http://julian.io.link-m.de/spf/mail-spf-rfc4408-tests.log

> You can change the test class if you want some other output, but this is
> not the goal of the unit-tests, so you shouldn't expect this from the
> tests.

I know that this is not the goal of the _unit_tests_. I am talking about a
_different_ goal.

Of course Mail::SPF uses a different method for automatic testing, too, but
said test script has been very helpful to me when I reviewed the RFC comp-
liance of Mail::SPF.

> Btw you can run a single test with maven, something like this:
> mvn test -Dtest=org.apache.james.jspf.RFC4408YamlTest

Thanks for the tip. I will try it.

> > BTW, Maven seems to have downloaded several packages during my first
> > `mvn test` run, but I can't find where it put them. I'd like to get
> > rid of them after the testing, so could you perhaps give me a hint
> > where the stuff got downloaded to? I checked jSPF out to
> > "D:\Storage\Temporary\jSPF \0.9b3" and ran `mvn test` in that same
> > dir.
>
> Check for .m2 folder in your home directory.

Found them there, thanks!

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFit0GwL7PKlBZWjsRAu3AAKCkMySRNIKLOFisD+xhbK9Z/97PSQCfZ8wy
Tw2UOrJ0LMzk8LIZUSoh2/Y=
=WHJO
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

julian at mehnle

Dec 22, 2006, 2:53 AM

Post #8 of 16 (2162 views)
Permalink

Re: Testing jSPF's RFC conformance [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Julian Mehnle wrote:
> I'm not saying that you're trying to trick us, but a little less noise
> and a little more information

... in the test log ...

(in case that wasn't clear)

> would go a long way towards helping us see how well jSPF implements RFC
> 4408.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFi7kzwL7PKlBZWjsRAndAAJ97vQkj7nyuWi4JJGr02H8ZAQUH5QCeMx4p
hlUpqVPnS2bnTQadDX0CD48=
=kaZM
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

apache at bago

Dec 22, 2006, 11:16 AM

Post #9 of 16 (2168 views)
Permalink

Re: Testing jSPF's RFC conformance [In reply to]

Julian Mehnle wrote:
> I'm not saying that you're trying to trick us, but a little less noise
> and a little more information
>
> ... in the test log ...
>
> (in case that wasn't clear)
>
> would go a long way towards helping us see how well jSPF implements RFC
> 4408.

Sorry I'm not subscribed to the spf list, Norman forwarded a message to
me and I tried to reply. Please keep me in CC.

Please tell us what you want to see and we'll try to make an option to
let you see what you want to see.

Currently they are simple unit tests, they are not intended to provide
any useful result but pass/fail :-) so what you ask is not possible,
unless we code it ;-)

We considered not useful to output the result and the explanation
because the tests pass *if* they equal the expected results.

In detail, they pass if expected result and explanation (defined in the
YAML file) equal the resulting result and explanation. We only added a
couple of checks: if expected explanation is DEFAULT then we check for
"http://www.openspf.org/why.html?sender=*", if expected explanation is
"cafe:babe::1 is queried as
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.E.B.A.B.E.F.A.C.ip6.arpa"
we check for "cafe:babe:0:0:0:0:0:1 is queried as
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.E.B.A.B.E.F.A.C.ip6.arpa"
because cafe:babe::1 and cafe:babe:0:0:0:0:0:1 are equal, but java give
us the expanded one.

Stefano

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

apache at bago

Dec 22, 2006, 11:34 AM

Post #10 of 16 (2158 views)
Permalink

Re: Testing jSPF's RFC conformance [In reply to]

Julian Mehnle wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Stefano Bagnara wrote:
>> Norman Maurer wrote:
>>> I think I managed to run `mvn test`, but it produced too much debug
>>> output for me to read through. It also seemed to run a lot of other
>>> tests besides the official RFC 4408 test suite.
>> I'm not sure, but you don't need to read the result. It is only debug
>> stuff. You just need to know that tests passes ;-)
>
> Well, then does the following SPF implementation pass the test suite, too?
>
> | #!/usr/bin/perl
> | { local $/; <> }
> | print("Test #$_: PASS\n") for 1..278;

NO! And this is why your shouldn't look for the output to see if a
testsuite is implemented or not ;-)

We can give you an output that is almost identical to the input (the
yaml file) but I think this would not give you more informations than
the perl script you wrote before.

Unfortunately our architecture does not let us to provide more
information about what rule matched what and what was the cause of the
result within the result: we can only return the plain result and the
explanation.
We get the further informations from the BIG debug output that provide a
lot of step-by-step information (as you probably saw).

> I'm not saying that you're trying to trick us

I believe this, because we don't earn anything tricking you :-)

> For example, this is how the output of a Mail::SPF test script looks:
>
> http://julian.io.link-m.de/spf/mail-spf-rfc4408-tests.log

Ok, we can do something similar but we can't easily provide the
additional informations within the result. So we would only output what
we got in output, that simply is what we had in input.

Is this what you are looking for? (This make no sense to me, but maybe
this is still useful to you, so I want to be sure, before working on this).

>> You can change the test class if you want some other output, but this is
>> not the goal of the unit-tests, so you shouldn't expect this from the
>> tests.
>
> I know that this is not the goal of the _unit_tests_. I am talking about a
> _different_ goal.

Ok, then what I wrote in the previous message is superflous :-)

I believe I didn't understand the goal: is your goal to oversight our
work and certify that our tests/implementation are correctly
implemented? or is your goal to test why a given test that you may have
not committed to the "official" yaml is not passing? or is it something
else?

Stefano

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

julian at mehnle

Dec 22, 2006, 1:15 PM

Post #11 of 16 (2169 views)
Permalink

Re: Testing jSPF's RFC conformance [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefano Bagnara wrote:
> Please tell us what you want to see and we'll try to make an option to
> let you see what you want to see.
>
> Currently they are simple unit tests, they are not intended to provide
> any useful result but pass/fail :-) so what you ask is not possible,
> unless we code it ;-)

Does jSPF support "Received-SPF" header generation? If it does, outputting
that one for each test would be a good start.

Does jSPF generate some sort of explanatory text message for the result?
(I do not mean the authority/"exp=" explanation.) E.g., the Mail::SPF
Perl implementation puts such a message in the comment field of the
"Received-SPF" header. For example:

| Result: PermError (Missing required domain-spec in
| '-exists:%(ir).sbl.example.com')
| Received-SPF: permerror (e1.example.com: Missing required domain-spec in
| '-exists:%(ir).sbl.example.com') receiver=io.link-m.de; identity=mfrom;
| envelope-from="test[at]e1.example.com"; helo=msgbas2x.cos.example.com;
| client-ip=192.168.218.40

> Unfortunately our architecture does not let us to provide more
> information about what rule matched what and what was the cause of the
> result within the result: we can only return the plain result and the
> explanation.

Oh. That's unfortunate.

> [ http://julian.io.link-m.de/spf/mail-spf-rfc4408-tests.log ]
> Ok, we can do something similar but we can't easily provide the
> additional informations within the result. So we would only output what
> we got in output, that simply is what we had in input.
>
> Is this what you are looking for? (This make no sense to me, but maybe
> this is still useful to you, so I want to be sure, before working on
> this).

No. I agree that this wouldn't be useful. Something like the explanatory
message described above would be most helpful, though. If you can't
"easily provide the additional informations within the result", could you
at least send that information (and not all the warnings and other debug
output) to STDOUT when jSPF is run in some special testing mode?

> I believe I didn't understand the goal: is your goal to oversight our
> work and certify that our tests/implementation are correctly
> implemented? or is your goal to test why a given test that you may have
> not committed to the "official" yaml is not passing? or is it something
> else?

The goal is understanding (or at least being able to understand) _why_ jSPF
passes each test in the test suite. An SPF implementation can return any
result code for a number of different reasons. It just giving the right
results may make it pass the test suite but isn't very illuminating
otherwise.

BTW, thank you for staying in touch and for your willingness to help us
with assessing the RFC conformance of jSPF!

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFjErpwL7PKlBZWjsRAo//AKDWuhsPH501AoUHcSd0/AyTDn9wBACeOAfF
Z2GFLW8nvCEj5VipR4rl7QE=
=Q99o
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

stuart at bmsi

Dec 22, 2006, 2:04 PM

Post #12 of 16 (2161 views)
Permalink

Re: Re: Testing jSPF's RFC conformance [In reply to]

On Fri, 22 Dec 2006, Julian Mehnle wrote:

> The goal is understanding (or at least being able to understand) _why_ jSPF
> passes each test in the test suite. An SPF implementation can return any
> result code for a number of different reasons. It just giving the right
> results may make it pass the test suite but isn't very illuminating
> otherwise.

I think you are asking too much. The test suite is just a tool. It
can only check that an implementation gives the right result for a strategic
set of inputs. It cannot check that it gave the right result for the right
reason.

Ultimately, if an implementation gives the right result for all inputs,
it doesn't really matter what its "motives" were.

--
Stuart D. Gathman <stuart[at]bmsi.com>
Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

julian at mehnle

Dec 22, 2006, 2:31 PM

Post #13 of 16 (2152 views)
Permalink

Re: Testing jSPF's RFC conformance [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stuart D. Gathman wrote:
> On Fri, 22 Dec 2006, Julian Mehnle wrote:
> > The goal is understanding (or at least being able to understand) _why_
> > jSPF passes each test in the test suite. An SPF implementation can
> > return any result code for a number of different reasons. It just
> > giving the right results may make it pass the test suite but isn't
> > very illuminating otherwise.
>
> I think you are asking too much. The test suite is just a tool. It
> can only check that an implementation gives the right result for a
> strategic set of inputs. It cannot check that it gave the right result
> for the right reason.

I didn't say this was a goal of the test suite. It is a separate goal that
I think should be worked on.

> Ultimately, if an implementation gives the right result for all inputs,
> it doesn't really matter what its "motives" were.

But then it might give the right results for the programmed tests for the
wrong reasons, and be incompliant beyond the ground covered by the test
suite. Knowing that an implementation gives the correct results for the
right reasons allows you to be more confident about its conformance.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFjFy6wL7PKlBZWjsRApMJAJ9DrkboLCmrX9A++pn/rSkCCwJQpQCbB19v
dGxpT8rx9WnS2GD0iCnNh+U=
=ASbR
-----END PGP SIGNATURE-----

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

apache at bago

Dec 22, 2006, 2:42 PM

Post #14 of 16 (2158 views)
Permalink

Re: Testing jSPF's RFC conformance [In reply to]

Julian Mehnle wrote:
> No. I agree that this wouldn't be useful. Something like the explanatory
> message described above would be most helpful, though. If you can't
> "easily provide the additional informations within the result", could you
> at least send that information (and not all the warnings and other debug
> output) to STDOUT when jSPF is run in some special testing mode?

I will review our logging ASAP to understand what we can do easily and
I'll report to you. (And maybe I hope I will have a solution for you)

>> I believe I didn't understand the goal: is your goal to oversight our
>> work and certify that our tests/implementation are correctly
>> implemented? or is your goal to test why a given test that you may have
>> not committed to the "official" yaml is not passing? or is it something
>> else?
>
> The goal is understanding (or at least being able to understand) _why_ jSPF
> passes each test in the test suite. An SPF implementation can return any
> result code for a number of different reasons. It just giving the right
> results may make it pass the test suite but isn't very illuminating
> otherwise.

I think this is not a good direction: tests are used to see *what* an
application do, to see *why* it do that you have to read and understand
its source code and running environment :-)

So if what you want to do is certification you should really read the
code or write a TCK including much more tests so it should not be easy
to randomly guess the right result without implementing the right algorythm.

If you instead want to analyze if there are "holes" in the current test
suite and some implementation pass it because of this then I understand
this questions, but I believe this is more a test for the yaml file than
a test for jSPF's RFC conformance :-)

> BTW, thank you for staying in touch and for your willingness to help us
> with assessing the RFC conformance of jSPF!

We have a common goal, so it is good to help each other :-)
Thank you,

Stefano

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

apache at bago

Dec 22, 2006, 4:10 PM

Post #15 of 16 (2162 views)
Permalink

Re: Testing jSPF's RFC conformance [In reply to]

Stefano Bagnara wrote:
> I will review our logging ASAP to understand what we can do easily and
> I'll report to you. (And maybe I hope I will have a solution for you)

I just committed to our trunk some improvement that I hope will help you.

In detail I added categories to loggers also when we are in test, and
made the ConsoleLogger (the one that is used by default to output
EVERYTHING to console) to also output the category.

This should give you much more fine control on the output using grep
against the level and the category.

I also added a main to the org.apache.james.jspf.RFC4408YamlTest
testcase so you can execute it directly and it will use Log4J instead of
our ConsoleLogger. Using Log4J you can tune the output
(level/category/format) by writing your own log4j.properties

I don't know if you know java and specifically log4j, so I didn't know
what was the best solution for you (strings to be able to grep or log4j
usage). Hope you can find everything you need there.

I have not removed/added logs because what you defined NOISE is subject
to what you are looking in a specific moment, so it does not make sense
to remove part of it. Sometimes you want to check dns queries done,
sometimes you may want to look what happenend parsing the rule, and so
on. If you think you need something MORE then feel free to ask and I'll
try to add the requested debug logging.

The test execution is logged in the "test" category: in this category
you can see what test is being executed and what is the result. Then
there are categories for the parser, for the dns service (in this case a
mocked dns service), for the directives and for each modifier. But I
guess this will more clear once you execute it.

Let me know if you are able to download and build the last trunk or if
you want me to publish a test build for you, or you prefer to wait for
our next release.

Stefano

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

nobody at xyzzy

Dec 23, 2006, 3:14 AM

Post #16 of 16 (2149 views)
Permalink

Re: Testing jSPF's RFC conformance [In reply to]

Julian Mehnle wrote:

> But then it might give the right results for the programmed tests for the
> wrong reasons, and be incompliant beyond the ground covered by the test
> suite. Knowing that an implementation gives the correct results for the
> right reasons allows you to be more confident about its conformance.

When I learned that stuff conformance tests were "black box" tests, you
couldn't tell why a test case passed, failed, or was inconclusive. Your
question "why" is a "white box" test, you need to know the code.

I wouldn't try to mix the concepts, "knowing the code" allows many useful
tests (coverage, etc.), but it can also have undesirable side-effects.

Frank

-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?list_id=1007

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com