Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

A new way to configure logging

 

 

Python dev RSS feed   Index | Next | Previous | View Threaded


vinay_sajip at yahoo

Oct 7, 2009, 7:49 AM

Post #1 of 17 (1296 views)
Permalink
A new way to configure logging

At present, configuration of Python's logging package can be done in one of two
ways:

1. Create a ConfigParser-readable configuration file and use
logging.config.fileConfig() to read and implement the configuration therein.
2. Use the logging API to programmatically configure logging using getLogger(),
addHandler() etc.

The first of these works for simple cases but does not cover all of the logging
API (e.g. Filters). The second of these provides maximal control, but besides
requiring users to write the configuration code, it fixes the configuration in
Python code and does not facilitate changing it easily at runtime.

In addition, the ConfigParser format appears to engender dislike (sometimes
strong dislike) in some quarters. Though it was chosen because it was the only
configuration format supported in the stdlib at that time, many people regard it
(or perhaps just the particular schema chosen for logging's configuration) as
'crufty' or 'ugly', in some cases apparently on purely aesthetic grounds. Recent
versions of Python of course support an additional battery-included format which
can be used for configuration - namely, JSON. Other options, such as YAML, are
also possible ways of configuring systems, Google App Engine-style, and PyYAML
has matured nicely.

There has also been talk on the django-dev mailing list about providing better
support for using Python logging in Django. When it happens (as of course I hope
it does) this has the consequence that many new users who use Django but are
relatively inexperienced in Python (e.g. in PHP shops which are moving to
Django) will become exposed to Python logging. As Django is configured using a
Python module and use of ConfigParser-style files is not a common approach in
that ecosystem, users will find either of the two approaches outlined above a
particular pain point when configuring logging for their Django applications and
websites, unless something is done to avoid it.

All three of the contenders for the title of "commonly found configuration
mechanism" - JSON, YAML and Python code - will be expressible, in Python, as
Python dicts. So it seems to make sense to add, to logging.config, a new
callable bound to "dictConfig" which will take a single dictionary argument and
configure logging from that dictionary.

An important facet of implementing such a scheme will be the format or schema
which the dictionary has to adhere to. I have started working on what such a
schema would look like, and if people here think it's a good idea to go ahead
with this, I'll provide the details of the schema on a separate post which I'll
also cross-post on comp.lang.python so that I can get feedback from there, too.

In outline, the scheme I have in mind will look like this, in terms of the new
public API:

class DictConfigurator:
def __init__(self, config): #config is a dict-like object (duck-typed)
import copy
self.config = copy.deepcopy(config)

def configure(self):
# actually do the configuration here using self.config

dictConfigClass = DictConfigurator

def dictConfig(config):
dictConfigClass(config).configure()

This allows easy replacement of DictConfigurator with a suitable subclass where
needed.

What's the general feeling here about this proposal? All comments and
suggestions will be gratefully received.

Regards,


Vinay Sajip


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


p.f.moore at gmail

Oct 7, 2009, 8:05 AM

Post #2 of 17 (1237 views)
Permalink
Re: A new way to configure logging [In reply to]

2009/10/7 Vinay Sajip <vinay_sajip [at] yahoo>:
> What's the general feeling here about this proposal? All comments and
> suggestions will be gratefully received.

+1

One option I would have found useful in some code I wrote would be to
extend the configuration -

class DictConfigurator:
...
def extend(self, moreconfig):
import copy
more = copy.deepcopy(moreconfig) # Not sure if this is needed?
self.config.update(more)

Paul.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


olemis at gmail

Oct 7, 2009, 8:19 AM

Post #3 of 17 (1255 views)
Permalink
Re: A new way to configure logging [In reply to]

On Wed, Oct 7, 2009 at 9:49 AM, Vinay Sajip <vinay_sajip [at] yahoo> wrote:
> At present, configuration of Python's logging package can be done in one of two
> ways:
>
> 1. Create a ConfigParser-readable configuration file and use
> logging.config.fileConfig() to read and implement the configuration therein.
> 2. Use the logging API to programmatically configure logging using getLogger(),
> addHandler() etc.
>
[...]

Wow ! Great explanation !
;o)

>
> All three of the contenders for the title of "commonly found configuration
> mechanism" - JSON, YAML and Python code - will be expressible, in Python, as
> Python dicts.

.INI files too

{
'section1' :
{'opt11' : 'val1', 'opt12' : 32323},
'section1' :
{'opt21' : 'val2', 'opt22' : 32323},
'section1' :
{'opt31' : 'val3', 'opt32' : 32323},
...
}

In fact it seems that this is a typical characteristic of config formats (CMIIW)

> So it seems to make sense to add, to logging.config, a new
> callable bound to "dictConfig" which will take a single dictionary argument and
> configure logging from that dictionary.
>

This kind of problems is similar to the one mentioned in another
thread about modifying config options after executing commands. In
that case I mentioned that the same dict-like interface also holds for
WinReg and so on ...

So thinking big (yes ! I have a big head ! ) I think that the best
approach in this case is to build an adaptation (simple) layer on top
of ConfigParser, JSON, WinReg, PyCode, YAML, ... and build specific
extensions for these formats . Perhaps the proper interfaces are
already there (e.g. `dict`, `shelve` ) and I'm just blind and looking
somewhere else ;o)

If `dict` (possibly nested ;o) is enough to represent config files
then the new method should be ok ... IMHO

[...]
>
> In outline, the scheme I have in mind will look like this, in terms of the new
> public API:
>
> class DictConfigurator:
>    def __init__(self, config): #config is a dict-like object (duck-typed)

========

>        import copy
>        self.config = copy.deepcopy(config)

Why ?

>
>    def configure(self):
>        #  actually do the configuration here using self.config
>
> dictConfigClass = DictConfigurator
>
> def dictConfig(config):
>    dictConfigClass(config).configure()
>
> This allows easy replacement of DictConfigurator with a suitable subclass where
> needed.
>

extension is cool ... what's the point about adding the new method
instead of using `DictConfigurator` directly ?

> What's the general feeling here about this proposal?
>

I'm feeling things ... :)

--
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
Nabble - Trac Users - Coupling trac and symfony framework -
http://feedproxy.google.com/~r/TracGViz-full/~3/hlNmupEonF0/Coupling-trac-and-symfony-framework-td25431579.html
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


vinay_sajip at yahoo

Oct 7, 2009, 8:49 AM

Post #4 of 17 (1244 views)
Permalink
Re: A new way to configure logging [In reply to]

Olemis Lang <olemis <at> gmail.com> writes:

> This kind of problems is similar to the one mentioned in another
> thread about modifying config options after executing commands. In
> that case I mentioned that the same dict-like interface also holds for
> WinReg and so on ...
>
> So thinking big (yes ! I have a big head ! ) I think that the best
> approach in this case is to build an adaptation (simple) layer on top
> of ConfigParser, JSON, WinReg, PyCode, YAML, ... and build specific
> extensions for these formats . Perhaps the proper interfaces are
> already there (e.g. `dict`, `shelve` ) and I'm just blind and looking
> somewhere else ;o)

Sorry, you've lost me :-)

> >        import copy
> >        self.config = copy.deepcopy(config)
>
> Why ?

So I'm free to mutate self.config as I see fit.

> extension is cool ... what's the point about adding the new method
> instead of using `DictConfigurator` directly ?

When you say "the new method", if you mean "configure" - the pattern is so that
a subclass can override __init__ and do additional setup before configure() is
called.

Regards,

Vinay Sajip


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


paul at rudin

Oct 7, 2009, 9:06 AM

Post #5 of 17 (1255 views)
Permalink
Re: A new way to configure logging [In reply to]

Vinay Sajip <vinay_sajip [at] yahoo> writes:


> What's the general feeling here about this proposal? All comments and
> suggestions will be gratefully received.
>

How about the global logging configuration being available as an object
supporting the usual dictionary interface? So you could, for example, do
something like: "logging.dictconfig.update(partialconfig)"

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


olemis at gmail

Oct 7, 2009, 9:26 AM

Post #6 of 17 (1250 views)
Permalink
Re: A new way to configure logging [In reply to]

On Wed, Oct 7, 2009 at 10:49 AM, Vinay Sajip <vinay_sajip [at] yahoo> wrote:
> Olemis Lang <olemis <at> gmail.com> writes:
>
>> This kind of problems is similar to the one mentioned in another
>> thread about modifying config options after executing commands. In
>> that case I mentioned that the same dict-like interface also holds for
>> WinReg and so on ...
>>
>> So thinking big (yes ! I have a big head ! ) I think that the best
>> approach in this case is to build an adaptation (simple) layer on top
>> of ConfigParser, JSON, WinReg, PyCode, YAML, ... and build specific
>> extensions for these formats . Perhaps the proper interfaces are
>> already there (e.g. `dict`, `shelve` ) and I'm just blind and looking
>> somewhere else ;o)
>
> Sorry, you've lost me :-)
>

Never mind . I was just trying to say that using `dict`, an adapter
could be implemented (if not already there ;o) for multiple formats
like the ones I mentioned above and the solution would cover many
config formats ... and also I was saying that I was not sure about
whether this is correct , I mean for *ANY* config formats, but
definitely will work for many ;o)

>> >        import copy
>> >        self.config = copy.deepcopy(config)
>>
>> Why ?
>
> So I'm free to mutate self.config as I see fit.
>

why not to let the user do it if he | she thinks so. I mean if
somebody supplies in a temporary mapping that can be written then why
to spend that time cloning the dict ? If the function has
side-effects, well just document it ;o)

>> extension is cool ... what's the point about adding the new method
>> instead of using `DictConfigurator` directly ?
>
> When you say "the new method", if you mean "configure" - the pattern is so that
> a subclass can override __init__ and do additional setup before configure() is
> called.
>

Sorry I was confused. I was talking about `dictConfig` *FUNCTION* (to
be added to logging.config ... isn't it ? ;o). I mean if dictConfig is
sematically equivalent to a simple `dc.configure` why to add such
function ?

PS: Not a big problem anyway ;o)

--
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
Looking for a technique to create flexible, graphical dashboards ...
- http://feedproxy.google.com/~r/TracGViz-full/~3/71kRhT34BgU/43789
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


olemis at gmail

Oct 7, 2009, 9:29 AM

Post #7 of 17 (1248 views)
Permalink
Re: A new way to configure logging [In reply to]

On Wed, Oct 7, 2009 at 11:06 AM, Paul Rudin <paul [at] rudin> wrote:
> Vinay Sajip <vinay_sajip [at] yahoo> writes:
>
>
>> What's the general feeling here about this proposal? All comments and
>> suggestions will be gratefully received.
>>
>
> How about the global logging configuration being available as an object
> supporting the usual dictionary interface? So you could, for example, do
> something like: "logging.dictconfig.update(partialconfig)"
>

Seems interesting ... yes ! ;o)

--
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
Search for milestone change in change history - Trac Users ... -
http://feedproxy.google.com/~r/TracGViz-full/~3/Gsj4xLjuCtk/578d40b734e5e753
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


vinay_sajip at yahoo

Oct 7, 2009, 9:55 AM

Post #8 of 17 (1257 views)
Permalink
Re: A new way to configure logging [In reply to]

Paul Rudin <paul@...> writes:

> How about the global logging configuration being available as an object
> supporting the usual dictionary interface? So you could, for example, do
> something like: "logging.dictconfig.update(partialconfig)"

A "partial configuration" only makes sense under certain limited conditions. For
example, a dict used for configuration will map to a graph of objects - filters,
formatters, handlers and loggers - which by necessity need to be referred to via
a string key in the dict - because you can't easily encode a graph in a dict
which comes from, say, a YAML or JSON file. These keys are temporary and just
used to indicate e.g. which handlers to attach to which loggers for that
particular configuration call:

{
"handlers": {
"h1": {
configuration for a handler
},
"h2": {
configuration for another handler
},
"h3": {
configuration for yet another handler
},
},
"loggers": {
"foo": {
"level": "DEBUG",
"handlers": ["h1", "h3"],
},
"bar.baz": {
"level": "INFO",
"handlers": ["h2", "h3"],
},
"spam.eggs": {
"level": "ERROR",
"handlers": ["h1", "h3"],
},
}
}

Here, the keys "h1", "h2" etc. are just used to encode connections in the graph,
and are useless outside the context of the specific configuration call. If you
pass a partial configuration, to be implemented incrementally, there's no way of
knowing if (in general) it's going to be consistent with the existing
configuration. Nor can you refer to e.g. handlers, formatters or filters you
passed in earlier configuration calls.

Since logger names such as "foo", "bar.baz" in the above example *are*
meaningful across multiple configuration calls, it *would be* possible to make
certain changes in the configuration - such as changing levels of loggers - by
passing in a partial configuration. However, if this "partial" functionality is
provided without restriction, it's hard to ensure that a user doesn't
inadvertently misconfigure logging. And while that is equally true with a
one-shot configuration, it's at least easier to debug. Things can get pretty
murky from a diagnostic point of view if code can make arbitrary changes to the
logging configuration during application execution, so that's not a good
practice to encourage :-)

Also, providing a readable dict-like interface to the internal configuration
will almost certainly lead to backward-compatibility headaches in the future.

Regards,


Vinay Sajip


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


v+python at g

Oct 7, 2009, 6:46 PM

Post #9 of 17 (1234 views)
Permalink
Re: A new way to configure logging [In reply to]

On approximately 10/7/2009 7:49 AM, came the following characters from
the keyboard of Vinay Sajip:

> In outline, the scheme I have in mind will look like this, in terms of the new
> public API:
>
> class DictConfigurator:
> def __init__(self, config): #config is a dict-like object (duck-typed)
> import copy
> self.config = copy.deepcopy(config)
>
> def configure(self):
> # actually do the configuration here using self.config
>
> dictConfigClass = DictConfigurator
>
> def dictConfig(config):
> dictConfigClass(config).configure()
>
> This allows easy replacement of DictConfigurator with a suitable subclass where
> needed.


Concept sounds good, and the idea of separating the syntax of the
configuration file from the action of configuring is a clever way of
avoiding the "syntax of the (day|platform|environment)" since everyone
seems to invent new formats. So people that want to expose a text file
to their users have the choice of syntaxes to expose, and then they do

logCfg = readMyFavoriteSyntax( logCfgFile )
# or extract a subset of a larger config file for their project
DictConfigurator( logCfg )

But DictConfigurator the name seems misleading... like you are
configuring how dicts work, rather than how logs work. Maybe with more
context this is not a problem, but if it is a standalone class, it is
confusing what it does, by name alone.

--
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


vinay_sajip at yahoo

Oct 7, 2009, 10:43 PM

Post #10 of 17 (1231 views)
Permalink
Re: A new way to configure logging [In reply to]

Paul Moore <p.f.moore <at> gmail.com> writes:

> One option I would have found useful in some code I wrote would be to
> extend the configuration -
>
> class DictConfigurator:
> ...
> def extend(self, moreconfig):
> import copy
> more = copy.deepcopy(moreconfig) # Not sure if this is needed?
> self.config.update(more)

See my response to Paul Rudin's post about incremental configuration.

Regards,

Vinay Sajip


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


vinay_sajip at yahoo

Oct 7, 2009, 10:45 PM

Post #11 of 17 (1227 views)
Permalink
Re: A new way to configure logging [In reply to]

Glenn Linderman <v+python <at> g.nevcal.com> writes:

> But DictConfigurator the name seems misleading... like you are
> configuring how dicts work, rather than how logs work. Maybe with more
> context this is not a problem, but if it is a standalone class, it is
> confusing what it does, by name alone.

Of course the fully qualified name would be logging.config.DictConfigurator
which does provide the requisite context, but if I think of/someone suggests a
better name I'll certainly use it.




_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


v+python at g

Oct 8, 2009, 12:44 AM

Post #12 of 17 (1224 views)
Permalink
Re: A new way to configure logging [In reply to]

On approximately 10/7/2009 10:45 PM, came the following characters from
the keyboard of Vinay Sajip:
> Glenn Linderman <v+python <at> g.nevcal.com> writes:
>
>> But DictConfigurator the name seems misleading... like you are
>> configuring how dicts work, rather than how logs work. Maybe with more
>> context this is not a problem, but if it is a standalone class, it is
>> confusing what it does, by name alone.
>
> Of course the fully qualified name would be logging.config.DictConfigurator
> which does provide the requisite context, but if I think of/someone suggests a
> better name I'll certainly use it.


Yes, I suspected it might be something like that, and that does provide
the context, so I'm content. Thanks for verifying it, and the rest is
bikeshedding...

In thinking more about this, I guess I would have the same comment about
the existing fileConfig() API also... it isn't file that is being
configured, but the logs. readConfigFromFile would have been more
descriptive (and lots longer :( ).

Had fileConfig been called "bulkConfig" or "configAll" (these names
attempt to contrast the function of this API versus the individual APIs
that configure one thing at a time) taking a file parameter, then it
could just simply/suddenly also allow a dict parameter, and no new API
would have to be created. Too late for this, however, and with the new
API, it would be possible to deprecate the fileConfig API, and tell the
user to use the ConfigParser (or JSON or YAML or whatever) to create the
dict, and then pass that to DictConfigurator (or bulkConfig or whatever).


--
Glenn -- http://nevcal.com/
===========================
A protocol is complete when there is nothing left to remove.
-- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


doug.hellmann at gmail

Oct 8, 2009, 5:21 AM

Post #13 of 17 (1221 views)
Permalink
Re: A new way to configure logging [In reply to]

On Oct 7, 2009, at 10:49 AM, Vinay Sajip wrote:

> All three of the contenders for the title of "commonly found
> configuration
> mechanism" - JSON, YAML and Python code - will be expressible, in
> Python, as
> Python dicts. So it seems to make sense to add, to logging.config, a
> new
> callable bound to "dictConfig" which will take a single dictionary
> argument and
> configure logging from that dictionary.

I've had bad experiences in the past with dictionary-based APIs. They
seem "simpler" in the short run, because the user "only needs to
create some dictionaries". Once the complexity of that nested
dictionary grows to a certain point, though, one has to refer back to
documentation constantly to make sure the structure conforms to the
"schema". Building a simple config tree using light-weight classes
with documented APIs tends to be more sustainable in the long run.

Doug

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


olemis at gmail

Oct 8, 2009, 7:26 AM

Post #14 of 17 (1224 views)
Permalink
Re: A new way to configure logging [In reply to]

On Thu, Oct 8, 2009 at 2:44 AM, Glenn Linderman <v+python [at] g> wrote:
> On approximately 10/7/2009 10:45 PM, came the following characters from the
> keyboard of Vinay Sajip:
>>
>> Glenn Linderman <v+python <at> g.nevcal.com> writes:
>>
>>> But DictConfigurator the name seems misleading... like you are
>>> configuring how dicts work, rather than how logs work.  Maybe with more
>>> context this is not a problem, but if it is a standalone class, it is
>>> confusing what it does, by name alone.
>>
>> Of course the fully qualified name would be
>> logging.config.DictConfigurator
>> which does provide the requisite context, but if I think of/someone
>> suggests a
>> better name I'll certainly use it.
>
[...]
>
> In thinking more about this, I guess I would have the same comment about the
> existing fileConfig() API also... it isn't file that is being configured,
> but the logs.  readConfigFromFile would have been more descriptive (and lots
> longer :( ).
>

cfgFromIni | cfgFromFile ? I prefer the former because YAML, shelves,
JSON & Co can also be stored in a file , so at least I get a better
understanding of the format in use. Otherwise I'd had to read the docs
or figure out which one

> Had fileConfig been called "bulkConfig" or "configAll" (these names attempt
> to contrast the function of this API versus the individual APIs that
> configure one thing at a time) taking a file parameter, then it could just
> simply/suddenly also allow a dict parameter, and no new API would have to be
> created.  Too late for this, however, and with the new API, it would be
> possible to deprecate the fileConfig API, and tell the user to use the
> ConfigParser (or JSON or YAML or whatever) to create the dict, and then pass
> that to DictConfigurator (or bulkConfig or whatever).
>

similar and consistent with what I mentioned before but IMO fileConfig
should remain for backwards compatibility

Objections ?

--
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
Looking for a technique to create flexible, graphical dashboards ...
- http://feedproxy.google.com/~r/TracGViz-full/~3/71kRhT34BgU/43789
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


vinay_sajip at yahoo

Oct 8, 2009, 7:47 AM

Post #15 of 17 (1223 views)
Permalink
Re: A new way to configure logging [In reply to]

> I've had bad experiences in the past with dictionary-based APIs. They seem

> "simpler" in the short run, because the user "only needs to create some
> dictionaries". Once the complexity of that nested dictionary grows to a certain
> point, though, one has to refer back to documentation constantly to make sure

Fair point, and agreed that the schema needs some care, here's roughly what I'm thinking of as an example configuration (YAML):

formatters:
brief:
format: '%(levelname)-8s: %(name)-15s: %(message)s'
precise:
format: '%(asctime)s %(name)-15s %(levelname)-8s %(message)s'
filters:
allow_foo:
name: foo
handlers:
console:
class : logging.StreamHandler
formatter: brief
level : INFO
stream : sys.stdout
filters: [allow_foo]
file:
class : logging.handlers.RotatingFileHandler
formatter: precise
filename: logconfig.log
maxBytes: 1024
backupCount: 3
debugfile:
class : logging.FileHandler
formatter: precise
filename: logconfig-detail.log
mode: a
loggers:
foo:
level : ERROR
handlers: [debugfile]
spam:
level : CRITICAL
handlers: [debugfile]
propagate: no
bar.baz:
level: WARNING

root:
level : DEBUG
handlers : [console, file]

It's not too deeply nested, and I can't see any need for it being more deeply nested. It more or less mirrors the way you'd have to write the corresponding ConfigParser-compatible configuration, but if you use a dict as a common format, you have the benefits which I mentioned of supporting JSON, YAML or Python code in a Django settings.py.

> the structure conforms to the "schema". Building a simple config tree using
> light-weight classes with documented APIs tends to be more sustainable in the
> long run.

When you say 'config tree using light-weight classes', I presume you mean a parallel set of classes to the actual logging classes which will be instantiated? ISTM logging classes are already reasonably usable for programmatic configuration, but this doesn't address e.g. updating configurations easily without program code changes, or the ability to configure from say a YAML node, where YAML may be being used for application configuration as a whole. (Unless of course I've misunderstood what you're getting at.)

Regards,

Vinay Sajip




_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


doug.hellmann at gmail

Oct 8, 2009, 9:00 AM

Post #16 of 17 (1223 views)
Permalink
Re: A new way to configure logging [In reply to]

On Oct 8, 2009, at 10:47 AM, Vinay Sajip wrote:

>> I've had bad experiences in the past with dictionary-based APIs.
>> They seem
>
>> "simpler" in the short run, because the user "only needs to create
>> some
>> dictionaries". Once the complexity of that nested dictionary grows
>> to a certain
>> point, though, one has to refer back to documentation constantly to
>> make sure
>
> Fair point, and agreed that the schema needs some care, here's
> roughly what I'm thinking of as an example configuration (YAML):
>
> formatters:
> brief:
> format: '%(levelname)-8s: %(name)-15s: %(message)s'
> precise:
> format: '%(asctime)s %(name)-15s %(levelname)-8s %(message)s'
> filters:
> allow_foo:
> name: foo
> handlers:
> console:
> class : logging.StreamHandler
> formatter: brief
> level : INFO
> stream : sys.stdout
> filters: [allow_foo]
> file:
> class : logging.handlers.RotatingFileHandler
> formatter: precise
> filename: logconfig.log
> maxBytes: 1024
> backupCount: 3
> debugfile:
> class : logging.FileHandler
> formatter: precise
> filename: logconfig-detail.log
> mode: a
> loggers:
> foo:
> level : ERROR
> handlers: [debugfile]
> spam:
> level : CRITICAL
> handlers: [debugfile]
> propagate: no
> bar.baz:
> level: WARNING
>
> root:
> level : DEBUG
> handlers : [console, file]
>
> It's not too deeply nested, and I can't see any need for it being
> more deeply nested. It more or less mirrors the way you'd have to
> write the corresponding ConfigParser-compatible configuration, but
> if you use a dict as a common format, you have the benefits which I
> mentioned of supporting JSON, YAML or Python code in a Django
> settings.py.

Yes, that makes sense.

>> the structure conforms to the "schema". Building a simple config
>> tree using
>> light-weight classes with documented APIs tends to be more
>> sustainable in the
>> long run.
>
> When you say 'config tree using light-weight classes', I presume you
> mean a parallel set of classes to the actual logging classes which
> will be instantiated? ISTM logging classes are already reasonably
> usable for programmatic configuration, but this doesn't address e.g.
> updating configurations easily without program code changes, or the
> ability to configure from say a YAML node, where YAML may be being
> used for application configuration as a whole. (Unless of course
> I've misunderstood what you're getting at.)

The probably are light-weight enough, but I didn't want to assume
those classes would be used directly. I suspect they could be, though.

I guess there aren't *so* many levels or keys that the data structure
will become unwieldy.

Doug

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


olemis at gmail

Nov 2, 2009, 1:43 PM

Post #17 of 17 (1024 views)
Permalink
Re: A new way to configure logging [In reply to]

On Wed, Oct 7, 2009 at 11:26 AM, Olemis Lang <olemis [at] gmail> wrote:
> On Wed, Oct 7, 2009 at 10:49 AM, Vinay Sajip <vinay_sajip [at] yahoo> wrote:
>> Olemis Lang <olemis <at> gmail.com> writes:
>>
>>> This kind of problems is similar to the one mentioned in another
>>> thread about modifying config options after executing commands. In
>>> that case I mentioned that the same dict-like interface also holds for
>>> WinReg and so on ...
>>>
>>> So thinking big (yes ! I have a big head ! ) I think that the best
>>> approach in this case is to build an adaptation (simple) layer on top
>>> of ConfigParser, JSON, WinReg, PyCode, YAML, ... and build specific
>>> extensions for these formats . Perhaps the proper interfaces are
>>> already there (e.g. `dict`, `shelve` ) and I'm just blind and looking
>>> somewhere else ;o)
>>
>> Sorry, you've lost me :-)
>>
>
> Never mind . I was just trying to say that using `dict`, an adapter
> could be implemented (if not already there ;o) for multiple formats
> like the ones I mentioned above and the solution would cover many
> config formats ... and also I was saying that I was not sure about
> whether this is correct , I mean for *ANY* config formats,

Intention = answer myself + sharing with you => for a better Pyland

Maybe we could learn something from Augeas [1]_ and use their
experience to include some of those features to `stdlib` . It seems to
be an ambitious attempt to put all the pieces in config-land together;
and AFAICS they handle (a lof of) config formats and have an API for
that, and ... CMIIW anyway

.. [1] Augeas
(http://augeas.net/)

--
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.