Mailing List Archive: exim: dev

test suite out of order results

Index | Next | Previous | View Threaded

tlyons at ivenue

May 1, 2012, 6:36 AM

Post #1 of 3 (189 views)
Permalink

test suite out of order results

When I run the test suite, some tests give me out of order results in
the log file. A good example is test #5. Ever seen this before? Can
anybody suggest changes I can or should make to get these log lines
appearing in the right order? This is running on a real machine, not
a vm. It's Ubuntu 11.04. App is configured to be owned by user
"exim" and builder/tester user is "exim-build":

[exim-build [at] tlyon ~/projects/exim/test (master)]$ id exim
uid=480(exim) gid=500(pubwww) groups=500(pubwww)
[exim-build [at] tlyon ~/projects/exim/test (master)]$ id exim-build
uid=1000(exim-build) gid=136(exim-build) groups=136(exim-build),500(pubwww)

Does anything above look wrong?

Here is a description I sent offline to someone else:

---------------- Forwarded Message ----------------

I'll start with a simple issue:
With current master, tests 1-4 work fine. test 5 errors, caused by
what I assume is a simple timing issue. The munging is working fine,
it's just that the lines end up out of order. log/0005 basically has:
queueID_1 <= sender
queueID_1 => recipient
queueID_1 Completed
queueID_2 <= sender
queueID_2 => recipient
queueID_2 Completed
queueID_3 <= sender
Start queue run
queueID_3 => recipient
queueID_3 Completed
End queue run

Whereas mine ends up looking like:
queueID_1 <= sender
queueID_2 <= sender
queueID_3 <= sender
Start queue run
queueID_1 => recipient
queueID_1 Completed
queueID_2 => recipient
queueID_2 Completed
queueID_3 => recipient
queueID_3 Completed
End queue run

Does yours do the same? Or does the ordering end up exactly the way
the test log appears? I may need to look at what features I'm
enabling on this build. It's a plain build with almost nothing extra
enabled beside the bare requirements. I can provide my Local/Makefile
that I use to build exim if necessary.

...Todd

--
Always code as if the guy who ends up maintaining your code will be a
violent psychopath who knows where you live. -- Martin Golding

--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

tlyons at ivenue

May 1, 2012, 1:34 PM

Post #2 of 3 (175 views)
Permalink

Re: test suite out of order results [In reply to]

On Tue, May 1, 2012 at 6:36 AM, Todd Lyons <tlyons [at] ivenue> wrote:
> When I run the test suite, some tests give me out of order results in
> the log file. A good example is test #5. Ever seen this before? Can
> anybody suggest changes I can or should make to get these log lines
> appearing in the right order? This is running on a real machine, not

Jeremy Harris caught up to me in IRC and figured it out. User
"exim-build" is where my git checkout and my test framework is
located. User "exim" is what Local/Makefile expects the config and
spool directories to be owned by (EXIM_USER and EXIM_GROUP). I had
incorrectly also configured CONFIGURE_OWNER and CONFIGURE_GROUP to
"exim". Once I changed it to "exim-build", rebuilt the binary, and
re-ran the test, my system successfully runs test 5 now.

So a FAQ item for running the test suite:
Q: Why does running the test suite fail on test 0005 with the log
lines all present, but just out of order?
A: The config file that the test suite uses is owned by a different
user than what is configured in the exim version the test suite is
testing. If you look at the stderr output log file, you'll likely see
it's complaining about the ownership of the config file. Modify the
src/Local/Makefile, setting CONFIGURE_OWNER and CONFIGURE_GROUP to the
user you are running the test as, rebuild exim, and repeat the test.

** I don't know _why_ that caused the problem, but I do know that
changing the settings _fixed_ the problem. **

Next step is to figure out why test #22 is failing for me. For some
reason my exim is "using cached result" when looking up password data,
but the stderr/0022 expects "cache not available". More testing...
:-)

...Todd
--
Always code as if the guy who ends up maintaining your code will be a
violent psychopath who knows where you live. -- Martin Golding

--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

pdp at exim

May 1, 2012, 8:13 PM

Post #3 of 3 (176 views)
Permalink

Re: test suite out of order results [In reply to]

On 2012-05-01 at 13:34 -0700, Todd Lyons wrote:
> So a FAQ item for running the test suite:

If you have got your new commit bit set up with Nigel, per his mail to
you, then feel free to add this to the README in the test subdirectory.
:)

--
## List details at https://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads