Mailing List Archive: Python: Dev

transitioning from % to {} formatting

1 2 3 4 5 6

View All

Index | Next | Previous | View Threaded

steven.bethard at gmail

Sep 29, 2009, 5:27 PM

Post #1 of 141 (1371 views)
Permalink

transitioning from % to {} formatting

There's a lot of code already out there (in the standard library and
other places) that uses %-style formatting, when in Python 3.0 we
should be encouraging {}-style formatting. We should really provide
some sort of transition plan. Consider an example from the logging
docs:

logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")

We'd like to support both this style as well as the following style:

logging.Formatter("{asctime} - {name} - {levelname} - {message}")

Perhaps we'd eventually deprecate the %-style formatting, but at least
in the intervening time, we'd have to support both. I see a few
possibilities:

* Add a new class, NewFormatter, which uses the {}-style formatting.
This is ugly because it makes the formatting we're trying to encourage
look like the alternative implementation instead of the standard one.
It also means we have to come up with new names for every API that
uses format strings.

* Have Formatter try to guess whether it got %-style formatting or
{}-style formatting, and then delegate to the appropriate one. I don't
know how accurately we can guess. We also end up still relying on both
formatting styles under the hood.

* Have Formatter convert all %-style formatting strings to {}-style
formatting strings (automatically). This still involves some guessing,
and involves some serious hacking to translate from one to the other
(maybe it wouldn't even always be possible?). But at least we'd only
be using {}-style formatting under the hood.

Thoughts?

Steve
--
Where did you get that preposterous hypothesis?
Did Steve tell you that?
--- The Hiphopopotamus
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

steven.bethard at gmail

Sep 29, 2009, 6:16 PM

Post #2 of 141 (1336 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

On Tue, Sep 29, 2009 at 5:54 PM, Benjamin Peterson <benjamin [at] python> wrote:
> * Provide a flag to Formatter which controls whether new or old
> formatting is used. Emit a warning when it's not true.

So then the transition strategy is something like:

version N: Add formatting flag which uses {}-style formatting on True
and %-style formatting on False, which defaults to False
version N + 1: Deprecate False value for formatting flag (all code
should start specifying flag=True)
version N + 2: Raise error on False value for formatting flag (all
code must specify flag=True)
version N + 3: Deprecate formatting flag
version N + 4: Remove formatting flag

?

Steve
--
Where did you get that preposterous hypothesis?
Did Steve tell you that?
--- The Hiphopopotamus
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Sep 29, 2009, 8:15 PM

Post #3 of 141 (1333 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

Steven Bethard wrote:
> There's a lot of code already out there (in the standard library and
> other places) that uses %-style formatting, when in Python 3.0 we
> should be encouraging {}-style formatting.

I don't agree that we should do that. I see nothing wrong with using
% substitution.

> We should really provide some sort of transition plan.

-1.

> Consider an example from the logging
> docs:
>
> logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
>
> We'd like to support both this style as well as the following style:
>
> logging.Formatter()

Now, that's different. IIUC, you are not actually performing the
substitution here, but only at a later place.

So changing to the new formatting mechanism is an API change.
I agree that the new placeholder syntax needs to be supported.

> * Add a new class, NewFormatter, which uses the {}-style formatting.
> This is ugly because it makes the formatting we're trying to encourage
> look like the alternative implementation instead of the standard one.

It's also ugly because the class has the word "new" in its name, which
no class should ever have. In a few years, it would still be around,
but not new anymore.

> * Have Formatter convert all %-style formatting strings to {}-style
> formatting strings (automatically). This still involves some guessing,
> and involves some serious hacking to translate from one to the other
> (maybe it wouldn't even always be possible?). But at least we'd only
> be using {}-style formatting under the hood.

I don't see the point of having a converter. The tricky part, as you
say, is the guessing. Whether the implementation then converts the
string or has two alternative formatting algorithms is an implementation
detail. I would favor continued use of the actual % substitution
code.

I would propose that the format argument gets an argument name,
according to the syntax it is written in. For PEP 3101 format,
I would call the argument "format" (like the method name of the
string type), i.e.

logging.Formatter(
format="{asctime} - {name} - {levelname} - {message}")

For the % formatting, I suggest "dicttemplate" (assuming that
you *have* to use dictionary %(key)s style currently).

The positional parameter would also mean dicttemplate, and
would be deprecated (eventually requiring a keyword-only
parameter).

Regards,
Martin

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

steven.bethard at gmail

Sep 29, 2009, 9:35 PM

Post #4 of 141 (1334 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

On Tue, Sep 29, 2009 at 8:15 PM, "Martin v. Löwis" <martin [at] v> wrote:
> Steven Bethard wrote:
>> Consider an example from the logging
>> docs:
>>
>> logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
>>
>> We'd like to support both this style as well as the following style:
>>
>> logging.Formatter()
>
> Now, that's different. IIUC, you are not actually performing the
> substitution here, but only at a later place.
>
> So changing to the new formatting mechanism is an API change.
> I agree that the new placeholder syntax needs to be supported.

Just to be clear, I don't think logging is the only place these kind
of things happen. Some others I found looking around:

BaseHTTPServer.BaseHTTPRequestHandler.error_message_format
http://docs.python.org/library/basehttpserver.html#BaseHTTPServer.BaseHTTPRequestHandler.error_message_format

BaseHTTPServer.BaseHTTPRequestHandler.log_message
http://docs.python.org/3.1/library/http.server.html#http.server.BaseHTTPRequestHandler.log_message

email.generator.DecodedGenerator
http://docs.python.org/library/email.generator.html#email.generator.DecodedGenerator

There may be more.

> I would propose that the format argument gets an argument name,
> according to the syntax it is written in. For PEP 3101 format,
> I would call the argument "format" (like the method name of the
> string type), i.e.
>
> logging.Formatter(
> format="{asctime} - {name} - {levelname} - {message}")
>
> For the % formatting, I suggest "dicttemplate" (assuming that
> you *have* to use dictionary %(key)s style currently).
>
> The positional parameter would also mean dicttemplate, and
> would be deprecated (eventually requiring a keyword-only
> parameter).

This is a nice solution for the cases where we can be confident that
the parameter is currently only used positionally. However, at least
in Python 3.1, "fmt" is already documented as a keyword parameter:

http://docs.python.org/3.1/library/logging.html#logging.Formatter

I guess we could follow the same approach though, and have fmt= be the
%-style formatting, and use some other keyword argument for {}-style
formatting.

We've got a similar problem for the
BaseHTTPRequestHandler.error_message_format attribute. I guess we'd
want to introduce some other attribute which is the error message
format for the {}-style formatting?

Steve
--
Where did you get that preposterous hypothesis?
Did Steve tell you that?
--- The Hiphopopotamus
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

foom at fuhm

Sep 29, 2009, 10:04 PM

Post #5 of 141 (1334 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

I'm resending a message I sent in June, since it seems the same thread
has come up again, and I don't believe anybody actually responded
(positively or negatively) to the suggestion back then.

http://mail.python.org/pipermail/python-dev/2009-June/090176.html

On Jun 21, 2009, at 5:40 PM, Eric Smith wrote:
> I've basically come to accept that %-formatting can never go away,
> unfortunately. There are too many places where %-formatting is used,
> for example in logging Formatters. %-formatting either has to exist
> or it has to be emulated.

It'd possibly be helpful if there were builtin objects which forced
the format style to be either newstyle or oldstyle, independent of
whether % or format was called on it.

E.g.
x = newstyle_formatstr("{} {} {}")
x % (1,2,3) == x.format(1,2,3) == "1 2 3"

and perhaps, for symmetry:
y = oldstyle_formatstr("%s %s %s")
y.format(1,2,3) == x % (1,2,3) == "1 2 3"

This allows the format string "style" decision is to be made external
to the API actually calling the formatting function. Thus, it need not
matter as much whether the logging API uses % or .format() internally
-- that only affects the *default* behavior when a bare string is
passed in.

This could allow for a controlled switch towards the new format string
format, with a long deprecation period for users to migrate:

1) introduce the above feature, and recommend in docs that people only
ever use new-style format strings, wrapping the string in
newstyle_formatstr() when necessary for passing to an API which uses %
internally.
2) A long time later...deprecate str.__mod__; don't deprecate
newstyle_formatstr.__mod__.
3) A while after that (maybe), remove str.__mod__ and replace all
calls in Python to % (used as a formatting operator) with .format() so
that the default is to use newstyle format strings for all APIs from
then on.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ncoghlan at gmail

Sep 29, 2009, 10:48 PM

Post #6 of 141 (1328 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

James Y Knight wrote:
> I'm resending a message I sent in June, since it seems the same thread
> has come up again, and I don't believe anybody actually responded
> (positively or negatively) to the suggestion back then.
>
> http://mail.python.org/pipermail/python-dev/2009-June/090176.html
>
> On Jun 21, 2009, at 5:40 PM, Eric Smith wrote:
>> I've basically come to accept that %-formatting can never go away,
>> unfortunately. There are too many places where %-formatting is used,
>> for example in logging Formatters. %-formatting either has to exist or
>> it has to be emulated.
>
> It'd possibly be helpful if there were builtin objects which forced the
> format style to be either newstyle or oldstyle, independent of whether %
> or format was called on it.
>
> E.g.
> x = newstyle_formatstr("{} {} {}")
> x % (1,2,3) == x.format(1,2,3) == "1 2 3"
>
> and perhaps, for symmetry:
> y = oldstyle_formatstr("%s %s %s")
> y.format(1,2,3) == x % (1,2,3) == "1 2 3"
>
> This allows the format string "style" decision is to be made external to
> the API actually calling the formatting function. Thus, it need not
> matter as much whether the logging API uses % or .format() internally --
> that only affects the *default* behavior when a bare string is passed in.
>
> This could allow for a controlled switch towards the new format string
> format, with a long deprecation period for users to migrate:
>
> 1) introduce the above feature, and recommend in docs that people only
> ever use new-style format strings, wrapping the string in
> newstyle_formatstr() when necessary for passing to an API which uses %
> internally.
> 2) A long time later...deprecate str.__mod__; don't deprecate
> newstyle_formatstr.__mod__.
> 3) A while after that (maybe), remove str.__mod__ and replace all calls
> in Python to % (used as a formatting operator) with .format() so that
> the default is to use newstyle format strings for all APIs from then on.

I must have missed this suggestion when it went past the first time. I
certainly like this approach - it has the virtue of only having to solve
the problem once, and then application developers can use it to adapt
any existing use of %-mod formatting to str.format formatting.

Something like:

class formatstr(str):
def __mod__(self, other):
if isinstance(other, dict):
return self.format(**dict)
if isinstance(other, tuple)
return self.format(*other)
return self.format(other)

APIs that did their own parsing based on %-formatting codes would still
break, as would any that explicitly called "str" on the object (or
otherwise stripped the subclass away, such as via "'%s' % fmt"), but
most things should pass a string subclass through transparently.

I wouldn't bother with a deprecation plan for 'normal' %-formatting
though. I don't think it is going to be practical to actually get rid of
that short of creating Python 4.0.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

steven.bethard at gmail

Sep 30, 2009, 12:35 AM

Post #7 of 141 (1326 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

On Tue, Sep 29, 2009 at 10:04 PM, James Y Knight <foom [at] fuhm> wrote:
> It'd possibly be helpful if there were builtin objects which forced the
> format style to be either newstyle or oldstyle, independent of whether % or
> format was called on it.
>
> E.g.
> x = newstyle_formatstr("{} {} {}")
> x % (1,2,3) == x.format(1,2,3) == "1 2 3"
>
> and perhaps, for symmetry:
> y = oldstyle_formatstr("%s %s %s")
> y.format(1,2,3) == x % (1,2,3) == "1 2 3"
>
> This allows the format string "style" decision is to be made external to the
> API actually calling the formatting function. Thus, it need not matter as
> much whether the logging API uses % or .format() internally -- that only
> affects the *default* behavior when a bare string is passed in.
>
> This could allow for a controlled switch towards the new format string
> format, with a long deprecation period for users to migrate:
>
> 1) introduce the above feature, and recommend in docs that people only ever
> use new-style format strings, wrapping the string in newstyle_formatstr()
> when necessary for passing to an API which uses % internally.
> 2) A long time later...deprecate str.__mod__; don't deprecate
> newstyle_formatstr.__mod__.
> 3) A while after that (maybe), remove str.__mod__ and replace all calls in
> Python to % (used as a formatting operator) with .format() so that the
> default is to use newstyle format strings for all APIs from then on.

So I understand how this might help a user to move from %-style
formatting to {}-style formatting, but it's still not clear to me how
to use this to evolve an API. In particular, if the goal is for the
API to move from accepting %-style format strings to {}-style format
strings, how should that API use newstyle_formatstr or
oldstyle_formatstr to make this transition?

Steve
--
Where did you get that preposterous hypothesis?
Did Steve tell you that?
--- The Hiphopopotamus
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

p.f.moore at gmail

Sep 30, 2009, 2:18 AM

Post #8 of 141 (1327 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

2009/9/30 Steven Bethard <steven.bethard [at] gmail>:
> On Tue, Sep 29, 2009 at 10:04 PM, James Y Knight <foom [at] fuhm> wrote:
>> It'd possibly be helpful if there were builtin objects which forced the
>> format style to be either newstyle or oldstyle, independent of whether % or
>> format was called on it.
> So I understand how this might help a user to move from %-style
> formatting to {}-style formatting, but it's still not clear to me how
> to use this to evolve an API. In particular, if the goal is for the
> API to move from accepting %-style format strings to {}-style format
> strings, how should that API use newstyle_formatstr or
> oldstyle_formatstr to make this transition?

IIUC, the API doesn't change. It's just that the internal code goes as follows:

1. (Current) Use %. str and oldformat objects work as now, newformat
objects work (using .format).
2. Convert the code to use .format - oldformat and newformat objects
work as before, str objects must be in new format.

The problem is, there's a point at which bare str arguments change
behaviour. So unless people wrap their arguments when calling the API,
there's still a point when things break (albeit with a simple
workaround available).

So maybe we need transition steps - wrap bare str objects in oldformat
classes, and warn, then wrap str objects in newformat objects and
warn, then stop wrapping.

That sounds to me like "normal" usage (bare strings) will be
annoyingly painful for a substantial transition period.

I'm assuming that the oldformat and newformat classes are intended to
be transitional things, and there's no intention that users should be
using the wrapper objects always. (And of course better names than
"oldformat" and "newformat" are needed - as Martin pointed out, having
"old" and "new" in the names is inappropriate). Otherwise, I'm a
strong -1 on the whole idea.

Paul.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

ncoghlan at gmail

Sep 30, 2009, 3:43 AM

Post #9 of 141 (1298 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

Steven Bethard wrote:
> So I understand how this might help a user to move from %-style
> formatting to {}-style formatting, but it's still not clear to me how
> to use this to evolve an API. In particular, if the goal is for the
> API to move from accepting %-style format strings to {}-style format
> strings, how should that API use newstyle_formatstr or
> oldstyle_formatstr to make this transition?

The problem is that many (most?) of the problematic APIs (such as
logging) will have multiple users in a given application, so getting the
granularity of any behavioural switches right is going to be difficult.

Providing a formatstr() type that makes .__mod__() work based on a
.format() style string (and a formatmod() type that does the opposite)
would allow for extremely fine-grained decision making, since every
format string will either be an ordinary str instance or else an
instance of the formatting subclass.

(Note that the primary use case for both proposed types is an
application developer adapting between two otherwise incompatible third
party libraries - the choice of class just changes based on whether the
old code being adapted is the code invoking mod on a format string or
the code providing a format string that expects to be used with the mod
operator).

I don't see any way for delayed formatting of "normal" strings in any
existing API to move away from %-formatting except via a long and
painful deprecation process (i.e. first warning when bare strings are
used for a release or two, then switching entirely to the new formatting
method) or by duplicating the API and maintaining the two interfaces in
parallel for the foreseeable future.

As Paul noted, the two proposed classes may also be useful to the
library developer during such a transition process - they could accept
strings in the "wrong" format just by wrapping them appropriately rather
than having to maintain the parallel APIs all the way through the
software stack.

Probably worth letting these concepts bake for a while longer, but it
definitely be nice to do *something* to help enable this transition in
2.7/3.2.

Cheers,
Nick.

--
Nick Coghlan | ncoghlan [at] gmail | Brisbane, Australia
---------------------------------------------------------------
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

eric at trueblade

Sep 30, 2009, 4:15 AM

Post #10 of 141 (1324 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

Martin v. L�wis wrote:
> Steven Bethard wrote:
>> There's a lot of code already out there (in the standard library and
>> other places) that uses %-style formatting, when in Python 3.0 we
>> should be encouraging {}-style formatting.
>
> I don't agree that we should do that. I see nothing wrong with using
> % substitution.

It's a maintenance burden. There are several outstanding bugs with it,
admittedly not of any great significance. I've been putting time into
fixing at least one of them. When Mark and I did short-float-repr, at
least half of my time was consumed with %-formatting, mostly because of
how it does memory management.

On the plus side, %-formatting is (and always will be) faster than
str.format(). Its very limitations make it possible for it to be fast.

I'd note that PEP 3101 calls str.format() a replacement for
%-formatting, not an alternate mechanism to achieve the same end.

>> * Add a new class, NewFormatter, which uses the {}-style formatting.
>> This is ugly because it makes the formatting we're trying to encourage
>> look like the alternative implementation instead of the standard one.
>
> It's also ugly because the class has the word "new" in its name, which
> no class should ever have. In a few years, it would still be around,
> but not new anymore.
>
>> * Have Formatter convert all %-style formatting strings to {}-style
>> formatting strings (automatically). This still involves some guessing,
>> and involves some serious hacking to translate from one to the other
>> (maybe it wouldn't even always be possible?). But at least we'd only
>> be using {}-style formatting under the hood.
>
> I don't see the point of having a converter. The tricky part, as you
> say, is the guessing. Whether the implementation then converts the
> string or has two alternative formatting algorithms is an implementation
> detail. I would favor continued use of the actual % substitution
> code.

Having a converter and guessing are 2 distinct issues. I believe a
convert from %-formatting specification strings to str.format() strings
is possible. I point this out not because I think a converter solves
this problem, but because in the past there's been a debate on whether a
converter is even possible.

Eric.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

barry at python

Sep 30, 2009, 5:21 AM

Post #11 of 141 (1323 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

On Sep 29, 2009, at 11:15 PM, Martin v. L�wis wrote:

> I would propose that the format argument gets an argument name,
> according to the syntax it is written in. For PEP 3101 format,
> I would call the argument "format" (like the method name of the
> string type), i.e.
>
> logging.Formatter(
> format="{asctime} - {name} - {levelname} - {message}")
>
> For the % formatting, I suggest "dicttemplate" (assuming that
> you *have* to use dictionary %(key)s style currently).
>
> The positional parameter would also mean dicttemplate, and
> would be deprecated (eventually requiring a keyword-only
> parameter).

Although I hate the name 'dicttemplate', this seems like the best
solution to me. Maybe it's good that 'dicttemplate' is so ugly though
so that people will naturally prefer 'format' :). But I like this
because there's not really any magic, it's explicit, and the decision
is made by the coder at the call site. The implementation does not
need to guess at all.

If this is adopted, it should become a common idiom across Python so
that once you've learned how to transition between the format strings,
you pretty much know how to do it for any supporting API. So we
should adopt it across all of the standard library.

-Barry

Attachments:

PGP.sig (0.81 KB)

p.f.moore at gmail

Sep 30, 2009, 5:29 AM

Post #12 of 141 (1324 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

2009/9/30 Barry Warsaw <barry [at] python>:
> Although I hate the name 'dicttemplate', this seems like the best solution
> to me. �Maybe it's good that 'dicttemplate' is so ugly though so that people
> will naturally prefer 'format' :). �But I like this because there's not
> really any magic, it's explicit, and the decision is made by the coder at
> the call site. �The implementation does not need to guess at all.

Ignoring the "make it ugly to persuade people not to use it" aspect,
why not just use 'template'?

Paul
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

steve at pearwood

Sep 30, 2009, 7:34 AM

Post #13 of 141 (1317 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

On Wed, 30 Sep 2009 03:04:05 pm James Y Knight wrote:

> It'd possibly be helpful if there were builtin objects which forced
> the format style to be either newstyle or oldstyle, independent of
> whether % or format was called on it.
>
> E.g.
> x = newstyle_formatstr("{} {} {}")
> x % (1,2,3) == x.format(1,2,3) == "1 2 3"

People will want this formatstr object to behave like strings, with
concatenation, slicing, etc.:

>>> x = newstyle_formatstr("x{} {} : ")
>>> y = newstyle_formatstr("{}")
>>> (x[1:] + y) % (1, 2, 3)
'1 2 : 3'

Instead of having to support one type with %-formatting and
{}-formatting (str), now the std lib will have two classes
with %-formatting and {}-formatting. How is this an improvement?

Moving along, let's suppose the newstyle_formatstr is introduced. What's
the intention then? Do we go through the std lib and replace every call
to (say)

somestring % args

with

newstyle_formatstr(somestring) % args

instead? That seems terribly pointless to me -- it does nothing about
getting rid of % but adds a layer of indirection which slows down the
code. Things are no better if the replacement code is:

newstyle_formatstr(somestring).format(*args)

(or similar). If we can do that, then why not just go all the way and
use this as the replacement instead?

somestring.format(*args)

> and perhaps, for symmetry:
> y = oldstyle_formatstr("%s %s %s")
> y.format(1,2,3) == x % (1,2,3) == "1 2 3"

Now we have three classes that support both % and {} formatting. Great.

[...]
> This could allow for a controlled switch towards the new format
> string format, with a long deprecation period for users to migrate:
>
> 1) introduce the above feature, and recommend in docs that people
> only ever use new-style format strings, wrapping the string in
> newstyle_formatstr() when necessary for passing to an API which uses
> % internally.

And how are people supposed to know what the API uses internally?

Personally, I think your chances of getting people to write:

logging.Formatter(newstyle_formatstr("%(asctime)s - %(name)s - %(level)s - %(msg)s"))

instead of

logging.Formatter("%(asctime)s - %(name)s - %(level)s - %(msg)s")

is slim to none -- especially when the second call still works. You'd
better off putting the call to newstyle_formatstr() inside
logging.Formatter, and not even telling the users.

Instead of wrapping strings in a class that makes .__mod__()
and .format() behave the same, at some cost on every call presumably,
my preferred approach would be a converter function (perhaps taken from
2to3?) which modified strings like "%(asctime)s" to "{asctime}". That
cost gets paid *once*, rather than on every call.

(Obviously the details will need to be ironed out, and it will depend on
the external API. If the external API depends on the caller using %
explicitly, then this approach may not work.)

> 2) A long time later...deprecate str.__mod__;

How long? I hope that's not until I'm dead and buried.

--
Steven D'Aprano
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

steven.bethard at gmail

Sep 30, 2009, 8:22 AM

Post #14 of 141 (1314 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

On Wed, Sep 30, 2009 at 5:21 AM, Barry Warsaw <barry [at] python> wrote:
> On Sep 29, 2009, at 11:15 PM, Martin v. Löwis wrote:
>
>> I would propose that the format argument gets an argument name,
>> according to the syntax it is written in. For PEP 3101 format,
>> I would call the argument "format" (like the method name of the
>> string type), i.e.
>>
>> logging.Formatter(
>> format="{asctime} - {name} - {levelname} - {message}")
>>
>> For the % formatting, I suggest "dicttemplate" (assuming that
>> you *have* to use dictionary %(key)s style currently).
>>
>> The positional parameter would also mean dicttemplate, and
>> would be deprecated (eventually requiring a keyword-only
>> parameter).
>
> Although I hate the name 'dicttemplate', this seems like the best solution
> to me. Maybe it's good that 'dicttemplate' is so ugly though so that people
> will naturally prefer 'format' :). But I like this because there's not
> really any magic, it's explicit, and the decision is made by the coder at
> the call site. The implementation does not need to guess at all.

Could you comment on what you think we should do when the parameter is
not positional? As I mentioned upthread, in the case of
logging.Formatter, it's already documented as taking the keyword
parameter "fmt", so we'd have to use the name "fmt" for % formatting.

Steve
--
Where did you get that preposterous hypothesis?
Did Steve tell you that?
--- The Hiphopopotamus
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

barry at python

Sep 30, 2009, 8:31 AM

Post #15 of 141 (1314 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

On Sep 30, 2009, at 11:22 AM, Steven Bethard wrote:

> On Wed, Sep 30, 2009 at 5:21 AM, Barry Warsaw <barry [at] python>
> wrote:
>> On Sep 29, 2009, at 11:15 PM, Martin v. L�wis wrote:
>>
>>> I would propose that the format argument gets an argument name,
>>> according to the syntax it is written in. For PEP 3101 format,
>>> I would call the argument "format" (like the method name of the
>>> string type), i.e.
>>>
>>> logging.Formatter(
>>> format="{asctime} - {name} - {levelname} - {message}")
>>>
>>> For the % formatting, I suggest "dicttemplate" (assuming that
>>> you *have* to use dictionary %(key)s style currently).
>>>
>>> The positional parameter would also mean dicttemplate, and
>>> would be deprecated (eventually requiring a keyword-only
>>> parameter).
>>
>> Although I hate the name 'dicttemplate', this seems like the best
>> solution
>> to me. Maybe it's good that 'dicttemplate' is so ugly though so
>> that people
>> will naturally prefer 'format' :). But I like this because there's
>> not
>> really any magic, it's explicit, and the decision is made by the
>> coder at
>> the call site. The implementation does not need to guess at all.
>
> Could you comment on what you think we should do when the parameter is
> not positional? As I mentioned upthread, in the case of
> logging.Formatter, it's already documented as taking the keyword
> parameter "fmt", so we'd have to use the name "fmt" for % formatting.

I'm okay with fmt==%-formatting and format=={}-formatting, but I'd
also be okay with transitioning 'fmt' to 'dicttemplate' or whatever.
I think the important thing is to be explicit in the method signature
which one you want (secondary would be trying to standardize this
across the stdlib).

-Barry

Attachments:

PGP.sig (0.81 KB)

steven.bethard at gmail

Sep 30, 2009, 8:39 AM

Post #16 of 141 (1314 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

On Wed, Sep 30, 2009 at 8:31 AM, Barry Warsaw <barry [at] python> wrote:
> On Sep 30, 2009, at 11:22 AM, Steven Bethard wrote:
>> On Wed, Sep 30, 2009 at 5:21 AM, Barry Warsaw <barry [at] python> wrote:
>>> On Sep 29, 2009, at 11:15 PM, Martin v. Löwis wrote:
>>>> I would propose that the format argument gets an argument name,
>>>> according to the syntax it is written in. For PEP 3101 format,
>>>> I would call the argument "format" (like the method name of the
>>>> string type), i.e.
>>>>
>>>> logging.Formatter(
>>>> format="{asctime} - {name} - {levelname} - {message}")
>>>>
>>>> For the % formatting, I suggest "dicttemplate" (assuming that
>>>> you *have* to use dictionary %(key)s style currently).
>>
>> Could you comment on what you think we should do when the parameter is
>> not positional? As I mentioned upthread, in the case of
>> logging.Formatter, it's already documented as taking the keyword
>> parameter "fmt", so we'd have to use the name "fmt" for % formatting.
>
> I'm okay with fmt==%-formatting and format=={}-formatting, but I'd also be
> okay with transitioning 'fmt' to 'dicttemplate' or whatever. I think the
> important thing is to be explicit in the method signature which one you want
> (secondary would be trying to standardize this across the stdlib).

Thanks for the clarification. I generally like this approach, though
it's not so convenient for argparse which already takes format strings
like this::

parser = ArgumentParser(usage='%(prog)s [--foo]')
parser.add_argument(
'--foo', type=int, default=42,
help='A foo of type %(type)s, defaulting to %(42)s)

That is, existing keyword arguments that already have good names (and
are pretty much always used as keyword arguments) take format strings.
I'm not sure that changing the name of usage= or help= here is really
an option.

I guess in this case I'm stuck with something like Benjamin's
suggestion of adding an additional flag to control which type of
formatting, and the corresponding 4 versions of cleanup. Ew.

Steve
--
Where did you get that preposterous hypothesis?
Did Steve tell you that?
--- The Hiphopopotamus
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

barry at python

Sep 30, 2009, 8:50 AM

Post #17 of 141 (1313 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

On Sep 30, 2009, at 11:39 AM, Steven Bethard wrote:

> Thanks for the clarification. I generally like this approach, though
> it's not so convenient for argparse which already takes format strings
> like this::
>
> parser = ArgumentParser(usage='%(prog)s [--foo]')
> parser.add_argument(
> '--foo', type=int, default=42,
> help='A foo of type %(type)s, defaulting to %(42)s)
>
> That is, existing keyword arguments that already have good names (and
> are pretty much always used as keyword arguments) take format strings.
> I'm not sure that changing the name of usage= or help= here is really
> an option.

Ah right.

> I guess in this case I'm stuck with something like Benjamin's
> suggestion of adding an additional flag to control which type of
> formatting, and the corresponding 4 versions of cleanup. Ew.

I missed Benjamin's suggestion, but in this case I would say add a
flag to ArgumentParser. I'm either going to want {} formatting all or
nothing. E.g.

import argparse
parser = ArgumentParser(usage='{prog} [--foo]', format=argparse.BRACES)
parser.add_argument(
'--foo', type=int, default=42,
help='A foo of type {type}, defaulting to {42}')

(although that last looks weird ;).

-Barry

Attachments:

PGP.sig (0.81 KB)

foom at fuhm

Sep 30, 2009, 9:19 AM

Post #18 of 141 (1315 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

On Sep 30, 2009, at 10:34 AM, Steven D'Aprano wrote:
>> E.g.
>> x = newstyle_formatstr("{} {} {}")
>> x % (1,2,3) == x.format(1,2,3) == "1 2 3"
>
> Moving along, let's suppose the newstyle_formatstr is introduced.
> What's
> the intention then? Do we go through the std lib and replace every
> call
> to (say)
> somestring % args
> with
> newstyle_formatstr(somestring) % args
> instead? That seems terribly pointless to me

Indeed, that *would* be terribly pointless! Actually, more than
pointless, it would be broken, as you've changed the API from taking
oldstyle format strings to newstyle format strings.

That is not the suggestion. The intention is to change /nearly
nothing/ in the std lib, and yet allow users to use newstyle string
substitution with every API.

Many Python APIs (e.g. logging) currently take a %-type formatting
string. It cannot simply be changed to take a {}-type format string,
because of backwards compatibility concerns. Either a new API can be
added to every one of those functions/classes, or, a single API can be
added to inform those places to use newstyle format strings.

>> This could allow for a controlled switch towards the new format
>> string format, with a long deprecation period for users to migrate:
>>
>> 1) introduce the above feature, and recommend in docs that people
>> only ever use new-style format strings, wrapping the string in
>> newstyle_formatstr() when necessary for passing to an API which uses
>> % internally.
>
> And how are people supposed to know what the API uses internally?

It's documented, (as it already must be, today!).

> Personally, I think your chances of getting people to write:
> logging.Formatter(newstyle_formatstr("%(asctime)s - %(name)s - %
> (level)s - %(msg)s"))
> instead of
> logging.Formatter("%(asctime)s - %(name)s - %(level)s - %(msg)s")

That's not my proposal.

The user could write either:
logging.Formatter("%(asctime)s - %(name)s - %(level)s - %(msg)s")
(as always -- that can't be changed without a long deprecation
period), or:
logging.Formatter(newstyle_formatstr("{asctime} - {name} - {level} -
{msg}")

This despite the fact that logging has not been changed to use {}-
style formatting internally. It should continue to call "self._fmt %
record.__dict__" for backward compatibility.

That's not to say that this proposal would allow no work to be done to
check the stdlib for issues. The Logging module presents one: it
checks if the format string contains "%{asctime}" to see if it should
bother to calculate the time. That of course would need to be changed.
Best would be to stick an instance which lazily generates its string
representation into the dict. The other APIs mentioned on this thread
(BaseHTTPServer, email.generator) will work immediately without
changes, however.

James
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

python at rcn

Sep 30, 2009, 9:36 AM

Post #19 of 141 (1312 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

>>> The positional parameter would also mean dicttemplate, and
>>> would be deprecated (eventually requiring a keyword-only
>>> parameter).
>>
>> Although I hate the name 'dicttemplate', this seems like the best solution
>> to me. Maybe it's good that 'dicttemplate' is so ugly though so that people
>> will naturally prefer 'format' :). But I like this because there's not
>> really any magic, it's explicit, and the decision is made by the coder at
>> the call site. The implementation does not need to guess at all.
>
> Could you comment on what you think we should do when the parameter is
> not positional? As I mentioned upthread, in the case of
> logging.Formatter, it's already documented as taking the keyword
> parameter "fmt", so we'd have to use the name "fmt" for % formatting.

Unless there is a firm decision to switch to kill %-formatting across the board,
I don't think anything should be done at all.

Creating Py3.x was all about removing cruft and clutter. I don't think it
would be improved by adding two ways to do it for everything in the
standard library. That is a lot of additional code, API expansion, and
new testing, fatter docs, and extra maintenance, but giving us no new
functionality.

Anytime we start hearing about newstyle/oldstyle combinations, I think
a flag should go up. Anytime there is a proposal to make sweeping
additions that do not add new capabilities, a flag should go up.

I understand the desire to have all formatting support both ways,
but I don't think it is worth the costs. People *never* need both ways
though they may have differing preferences about which *one* to use.

my-two-cents,

Raymond

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Sep 30, 2009, 9:48 AM

Post #20 of 141 (1312 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

>>> There's a lot of code already out there (in the standard library and
>>> other places) that uses %-style formatting, when in Python 3.0 we
>>> should be encouraging {}-style formatting.
>>
>> I don't agree that we should do that. I see nothing wrong with using
>> % substitution.
>
> It's a maintenance burden.

Well - that's the cost of keeping it in the language. It's not a problem
with using it while it *is* in the language.

So if a decision was made to eventually remove % formatting, it would
be reasonable to start migrating code to PEP 3101. However, no such
decision has been made (and hopefully won't be throughout 3.x), so
as the mechanism *is* available, there is no need to start changing
existing code (except the for actual issue Steven discusses, namely
libraries that expect strings in % template form).

> I'd note that PEP 3101 calls str.format() a replacement for
> %-formatting, not an alternate mechanism to achieve the same end.

I think this is a mis-wording; the intent of the PEP apparently is
to propose this mechanism as an option, not as an actual replacement.
This becomes clear when reading the "Backwards Compatibility" section:

# Backwards compatibility can be maintained by leaving the existing
# mechanisms in place. The new system does not collide with any of
# the method names of the existing string formatting techniques, so
# both systems can co-exist until it comes time to deprecate the
# older system.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

tjreedy at udel

Sep 30, 2009, 9:49 AM

Post #21 of 141 (1313 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

James Y Knight wrote:
> allow users to use newstyle string substitution with every API.

However it is done, I think someone (like new Python programmers) should
be able to program in Python3, and use everything in the stdlib, without
ever learning % formatting -- and that I should be able to forget about
it ;-).

+10 on the goal.

Terry Jan Reedy

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

martin at v

Sep 30, 2009, 9:50 AM

Post #22 of 141 (1315 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

> Although I hate the name 'dicttemplate', this seems like the best
> solution to me. Maybe it's good that 'dicttemplate' is so ugly though
> so that people will naturally prefer 'format' :). But I like this
> because there's not really any magic, it's explicit, and the decision is
> made by the coder at the call site. The implementation does not need to
> guess at all.

Unfortunately, as Steven pointed out, the parameter is *already*
documented with the name "fmt". So one option would be to call it
"fmt" and "format"; the other option would be to not only deprecate
the positional passing, but also the passing under the name fmt=.

As for calling it "dicttemplate" - I'm sure people can and will propose
alternative spellings :-)

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

solipsis at pitrou

Sep 30, 2009, 10:01 AM

Post #23 of 141 (1314 views)
Permalink

Re: transitioning from % to {} formattin g [In reply to]

James Y Knight <foom <at> fuhm.net> writes:
>
> The user could write either:
> logging.Formatter("%(asctime)s - %(name)s - %(level)s - %(msg)s")
> (as always -- that can't be changed without a long deprecation
> period), or:
> logging.Formatter(newstyle_formatstr("{asctime} - {name} - {level} -
> {msg}")

Why not allow logging.Formatter to take a callable, which would in turn call the
callable with keyword arguments?

Therefore, you could write:
logging.Formatter("{asctime} - {name} - {level} - {msg}".format)

and then:
logging.critical(name="Python", msg="Buildbots are down")

All this without having to learn about a separate "compatibility wrapper object".

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

python at rcn

Sep 30, 2009, 10:05 AM

Post #24 of 141 (1314 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

[Terry Reedy]
> However it is done, I think someone (like new Python programmers) should
> be able to program in Python3, and use everything in the stdlib, without
> ever learning % formatting -- and that I should be able to forget about
> it ;-).
>
> +10 on the goal.

If that were possible, it would be nice. But as long as the language supports
%-formatting, it is going to be around in one form or another. Any non-casual
user will bump into %-formatting in books, in third-party modules, in ASPN recipes,
on the newsgroup, and in our own source code. If they maintain any exising
software, they will likely encounter too. It doesn't seem to be a subject that
can be ignored.

Also, I think it premature to say that {}-formatting has been proven in battle.
AFAICT, there has been very little uptake. I've personally made an effort
to use {}-formatting more often but find that I frequently have to lookup
the syntax and need to experiment with the interactive interpreter to get it right.
I haven't found it easy to teach or to get other people to convert. This is
especially true if the person has encountered %-formatting in other languages
(it is a popular approach).

Raymond
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

steven.bethard at gmail

Sep 30, 2009, 10:14 AM

Post #25 of 141 (1313 views)
Permalink

Re: transitioning from % to {} formatting [In reply to]

On Wed, Sep 30, 2009 at 8:50 AM, Barry Warsaw <barry [at] python> wrote:
> On Sep 30, 2009, at 11:39 AM, Steven Bethard wrote:
>
>> Thanks for the clarification. I generally like this approach, though
>> it's not so convenient for argparse which already takes format strings
>> like this::
>>
>> parser = ArgumentParser(usage='%(prog)s [--foo]')
>> parser.add_argument(
>> '--foo', type=int, default=42,
>> help='A foo of type %(type)s, defaulting to %(42)s)

Yep, sorry, typo, that should have been %(default)s, not %(42)s.

>> I guess in this case I'm stuck with something like Benjamin's
>> suggestion of adding an additional flag to control which type of
>> formatting, and the corresponding 4 versions of cleanup. Ew.
>
> I missed Benjamin's suggestion, but in this case I would say add a flag to
> ArgumentParser. I'm either going to want {} formatting all or nothing.
> E.g.
>
> import argparse
> parser = ArgumentParser(usage='{prog} [--foo]', format=argparse.BRACES)

Yeah, that's basically Benjamin's suggestion, with the transition path being:

(1) Introduce format= keyword argument, defaulting to PERCENTS
(2) Deprecate format=PERCENTS
(3) Error on format=PERCENTS (Benjamin suggested just changing the
default here, but that would give a release where errors would pass
silently)
(4) Deprecate format= keyword argument.
(5) Remove format= keyword argument.

It's a little sad that it takes 5 versions to do this, but I guess if
a user is on top of things, at version (1) they add format=BRACES to
all their code, and then remove those at version (4). So even though
there are 5 versions, there are only two code changes required.

At least in the case of argparse, this can be a constructor argument
as you suggest, and we only have to introduce this flag in one place.

Steve
--
Where did you get that preposterous hypothesis?
Did Steve tell you that?
--- The Hiphopopotamus
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

1 2 3 4 5 6

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads