Mailing List Archive: Python: Bugs

[issue6896] Intermittent failures in test_mailbox

Index | Next | Previous | View Threaded

report at bugs

Oct 31, 2009, 3:21 PM

Post #1 of 13 (195 views)
Permalink

[issue6896] Intermittent failures in test_mailbox

Antoine Pitrou <pitrou[at]free.fr> added the comment:

Looking at the frequent buildbot failures, they always seem to happen in
TestMaildir. Therefore, it may have to do with the metadata-updating
behaviour of the filesystem (since maildir uses one file per message).

Unfortunately, while I tried to reproduce the failures on my home system
with a variety of filesystems (ext3, ext4, reiserfs, xfs and even vfat),
I never got any failure here.

Ezio, could you give details on your system, your filesystem and its
mount options? (if you don't know them, look in /proc/mounts)

----------
nosy: +pitrou
versions: +Python 3.1

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue6896>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 31, 2009, 4:42 PM

Post #2 of 13 (177 views)
Permalink

[issue6896] Intermittent failures in test_mailbox [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

Ezio, could you give a try to the following patch?

----------
keywords: +patch
Added file: http://bugs.python.org/file15243/test_mailbox.patch

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue6896>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 31, 2009, 4:56 PM

Post #3 of 13 (177 views)
Permalink

[issue6896] Intermittent failures in test_mailbox [In reply to]

Changes by Antoine Pitrou <pitrou[at]free.fr>:

----------
nosy: +r.david.murray

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue6896>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 31, 2009, 5:14 PM

Post #4 of 13 (178 views)
Permalink

[issue6896] Intermittent failures in test_mailbox [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

Another possible explanation is that the logic for computing
Maildir._last_time is wrong. It should be computed before refreshing the
internal cache, not after. Here is a patch.

----------
Added file: http://bugs.python.org/file15244/test_mailbox2.patch

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue6896>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 31, 2009, 6:20 PM

Post #5 of 13 (177 views)
Permalink

[issue6896] Intermittent failures in test_mailbox [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

According to testers, both patches fail at addressing the issue.

----------

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue6896>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 31, 2009, 6:20 PM

Post #6 of 13 (177 views)
Permalink

[issue6896] Intermittent failures in test_mailbox [In reply to]

Changes by Antoine Pitrou <pitrou[at]free.fr>:

Removed file: http://bugs.python.org/file15243/test_mailbox.patch

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue6896>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 31, 2009, 6:20 PM

Post #7 of 13 (177 views)
Permalink

[issue6896] Intermittent failures in test_mailbox [In reply to]

Changes by Antoine Pitrou <pitrou[at]free.fr>:

Removed file: http://bugs.python.org/file15244/test_mailbox2.patch

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue6896>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Oct 31, 2009, 8:01 PM

Post #8 of 13 (172 views)
Permalink

[issue6896] Intermittent failures in test_mailbox [In reply to]

R. David Murray <rdmurray[at]bitdance.com> added the comment:

I hacked mailbox.py so that a _toc refresh is always done and ran the
tests in a loop, and did not get a failure in 30+ runs, whereas without
that hack I would get failures almost every run. So I think Antoine is
on the right track.

----------

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue6896>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 1, 2009, 6:50 AM

Post #9 of 13 (158 views)
Permalink

[issue6896] Intermittent failures in test_mailbox [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

The explanation seems to be that some systems (including RDM's buildbot)
have slightly bizarre mtime behaviour:

$ date && python -c 'import os; os.link("setup.py", "t/c")' && stat t &&
date
Sun Nov 1 09:49:04 EST 2009
File: `t'
Size: 144 Blocks: 0 IO Block: 4096 directory
Device: 811h/2065d Inode: 223152 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 1001/ pitrou) Gid: ( 1005/ pitrou)
Access: 2009-11-01 09:10:11.000000000 -0500
Modify: 2009-11-01 09:49:03.000000000 -0500
Change: 2009-11-01 09:49:03.000000000 -0500
Sun Nov 1 09:49:04 EST 2009

As you see, the mtime of the directory is set a full one second before
the date at which its contents are modified.
I guess the only safe solution is to explicitly flush the internal cache
when we do any modifications.

----------

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue6896>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 1, 2009, 7:12 AM

Post #10 of 13 (158 views)
Permalink

[issue6896] Intermittent failures in test_mailbox [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

Ok, here is a patch which seems to fix the problem reliably on RDM's
buildbot. Ezio, can you give it a try?

----------
Added file: http://bugs.python.org/file15246/maildir.patch

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue6896>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 1, 2009, 7:31 AM

Post #11 of 13 (158 views)
Permalink

[issue6896] Intermittent failures in test_mailbox [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

PS: this is the kind of traces I had when running TestMaildir on David's
buildbot:

at 1257086460.18, linking/renaming ... mtime of
'/home/pitrou/trunk/@test_23145_tmp/new' is now 1257086459.0
at 1257086460.21, linking/renaming ... mtime of
'/home/pitrou/trunk/@test_23145_tmp/new' is now 1257086459.0
at 1257086460.24, linking/renaming ... mtime of
'/home/pitrou/trunk/@test_23145_tmp/new' is now 1257086459.0
at 1257086460.27, linking/renaming ... mtime of
'/home/pitrou/trunk/@test_23145_tmp/new' is now 1257086459.0
at 1257086460.3, linking/renaming ... mtime of
'/home/pitrou/trunk/@test_23145_tmp/new' is now 1257086459.0
at 1257086460.33, linking/renaming ... mtime of
'/home/pitrou/trunk/@test_23145_tmp/new' is now 1257086459.0
at 1257086460.36, linking/renaming ... mtime of
'/home/pitrou/trunk/@test_23145_tmp/new' is now 1257086460.0
at 1257086460.38, linking/renaming ... mtime of
'/home/pitrou/trunk/@test_23145_tmp/new' is now 1257086460.0
at 1257086460.4, linking/renaming ... mtime of
'/home/pitrou/trunk/@test_23145_tmp/new' is now 1257086460.0

As you see, the mtime is set to 1257086459 even when time.time() returns
1257086460.00 and higher, and it's only set to 1257086460 when
time.time() reaches 1257086460.36.

----------

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue6896>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 1, 2009, 5:06 PM

Post #12 of 13 (145 views)
Permalink

[issue6896] Intermittent failures in test_mailbox [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

Ezio indicated that the patch suppressed all failures for him, so I
committed it on trunk.

----------

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue6896>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Nov 2, 2009, 4:23 AM

Post #13 of 13 (137 views)
Permalink

[issue6896] Intermittent failures in test_mailbox [In reply to]

Antoine Pitrou <pitrou[at]free.fr> added the comment:

Now merged into py3k and 3.1, where it has solved the buildbot issues.
2.6 doesn't apply, since the caching scheme isn't present there.
If there's any problem with the patch, please reopen.

----------
resolution: -> fixed
status: open -> closed

_______________________________________
Python tracker <report[at]bugs.python.org>
<http://bugs.python.org/issue6896>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com