Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

Re: cpython (2.7): #9559: Append data to single-file mailbox files if messages are only added

 

 

Python dev RSS feed   Index | Next | Previous | View Threaded


solipsis at pitrou

Jun 28, 2012, 4:07 AM

Post #1 of 5 (302 views)
Permalink
Re: cpython (2.7): #9559: Append data to single-file mailbox files if messages are only added

On Thu, 28 Jun 2012 12:59:02 +0200
petri.lehtinen <python-checkins [at] python> wrote:
> http://hg.python.org/cpython/rev/c37cb11b546f
> changeset: 77832:c37cb11b546f
> branch: 2.7
> parent: 77823:73710ae9fedc
> user: Petri Lehtinen <petri [at] digip>
> date: Thu Jun 28 13:48:17 2012 +0300
> summary:
> #9559: Append data to single-file mailbox files if messages are only added
>
> If messages were only added, a new file is no longer created and
> renamed over the old file when flush() is called on an mbox, MMDF or
> Babyl mailbox.

Why so? Appending is not atomic and, if it fails in the middle, you
could get a corrupt mbox file.
Furthermore, I disagree that it's a bugfix: IMO it should wait for 3.4.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


petri at digip

Jun 28, 2012, 6:16 AM

Post #2 of 5 (287 views)
Permalink
Re: cpython (2.7): #9559: Append data to single-file mailbox files if messages are only added [In reply to]

Antoine Pitrou wrote:
> > If messages were only added, a new file is no longer created and
> > renamed over the old file when flush() is called on an mbox, MMDF or
> > Babyl mailbox.
>
> Why so? Appending is not atomic and, if it fails in the middle, you
> could get a corrupt mbox file.
> Furthermore, I disagree that it's a bugfix: IMO it should wait for 3.4.

The code previosly already appended messages to the end of the file
when calling add(). This patch just changed it to not do a full
rewrite when flush() is called. Having a partially written message in
the end of your mailbox doesn't seem like a fatal corruption to me.

Furthermore, I (and R. David Murray) think this is not so surprising
for users. Most (or all) other implementations always write changes
in-place without renaming, as this makes it possible to find out
whether new mail has arrived.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


rdmurray at bitdance

Jun 28, 2012, 8:08 AM

Post #3 of 5 (291 views)
Permalink
Re: cpython (2.7): #9559: Append data to single-file mailbox files if messages are only added [In reply to]

On Thu, 28 Jun 2012 16:16:45 +0300, Petri Lehtinen <petri [at] digip> wrote:
> Antoine Pitrou wrote:
> > > If messages were only added, a new file is no longer created and
> > > renamed over the old file when flush() is called on an mbox, MMDF or
> > > Babyl mailbox.
> >
> > Why so? Appending is not atomic and, if it fails in the middle, you
> > could get a corrupt mbox file.
> > Furthermore, I disagree that it's a bugfix: IMO it should wait for 3.4.
>
> The code previosly already appended messages to the end of the file
> when calling add(). This patch just changed it to not do a full
> rewrite when flush() is called. Having a partially written message in
> the end of your mailbox doesn't seem like a fatal corruption to me.
>
> Furthermore, I (and R. David Murray) think this is not so surprising
> for users. Most (or all) other implementations always write changes
> in-place without renaming, as this makes it possible to find out
> whether new mail has arrived.

It is true, however, that Petri found that mutt (I think?) does some extra
gymnastics to provide recovery where the write fails part way through,
and it would be worth adding that as an enhanced bugfix if someone
has the motivation (basically, make a copy of the unmodified mailbox
and mv it back into place if the write fails).

Even that fix won't prevent corruption in the case of a system crash,
but, then, not much will in that case.

--David
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


solipsis at pitrou

Jun 28, 2012, 9:07 AM

Post #4 of 5 (284 views)
Permalink
Re: cpython (2.7): #9559: Append data to single-file mailbox files if messages are only added [In reply to]

On Thu, 28 Jun 2012 16:16:45 +0300
Petri Lehtinen <petri [at] digip> wrote:
> Antoine Pitrou wrote:
> > > If messages were only added, a new file is no longer created and
> > > renamed over the old file when flush() is called on an mbox, MMDF or
> > > Babyl mailbox.
> >
> > Why so? Appending is not atomic and, if it fails in the middle, you
> > could get a corrupt mbox file.
> > Furthermore, I disagree that it's a bugfix: IMO it should wait for 3.4.
>
> The code previosly already appended messages to the end of the file
> when calling add(). This patch just changed it to not do a full
> rewrite when flush() is called.

Ok, I agree it sounds good then. Thanks for explaining.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


petri at digip

Jun 28, 2012, 11:09 AM

Post #5 of 5 (282 views)
Permalink
Re: cpython (2.7): #9559: Append data to single-file mailbox files if messages are only added [In reply to]

R. David Murray wrote:
> It is true, however, that Petri found that mutt (I think?) does some extra
> gymnastics to provide recovery where the write fails part way through,
> and it would be worth adding that as an enhanced bugfix if someone
> has the motivation (basically, make a copy of the unmodified mailbox
> and mv it back into place if the write fails).

This is not what mutt does. It just writes the modified part of the
mailbox to a temporary file, and then copies the data from the
temporary file to the mailbox file. If this last step fails, the
temporary file is left behind for recovery.

Copying the whole mailbox before making modifications might be clever,
though. It's just quite a lot of writing, especially for big
mailboxes. OTOH, the whole file is rewritten by the current code, too.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.