Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Dev

Things to Know About Super

 

 

First page Previous page 1 2 Next page Last page  View All Python dev RSS feed   Index | Next | Previous | View Threaded


michele.simionato at gmail

Aug 24, 2008, 3:19 AM

Post #1 of 37 (14168 views)
Permalink
Things to Know About Super

Recently I have opened a blog on Artima and I am publishing a
few Python-related essays I had in store. In particular a trilogy
of papers about "super". From the foreword:

"""
In 2004 I decided to write a comprehensive paper documenting
``super`` pitfalls and traps, with the goal of publishing it on the
Python web site, just as I had published my essay on multiple
inheritance and the `Method Resolution Order`_. With time the paper
grew longer and longer but I never had the feeling that I had covered
everything I needed to say: moreover I have a full time job, so I
never had the time to fully revise the paper as a whole. As a
consequence, four years have passed and the paper is still in draft
status. This is a pity, since it documents issues that people
encounter and that regularly come out on the Python newsgroups and
forums.

Keeping the draft sitting on my hard disk is doing a disservice to the
community. Still, I lack to time to finish it properly. To come out
from the impasse, I decided to split the long paper in a series of
short blog posts, which I do have the time to review properly.
Moreover people are free to post comments and corrections in case I am
making
mistakes (speaking about ``super`` this is always possible). Once I
finish the series, I may integrate the corrections, put it together
again and possibly publish it as whole on the Python website.
In other words, in order to finish the task,
I am trying the strategies of *divide et conquer*
and *release early, release often*. We will see how it goes.
"""

It seems the strategy worked since I have finished
the paper, by splitting it in three post. Actually, it it not
really finished because it does not say anything about
the new super in Python 3.0, but that could do in a separate
essay. Here I am asking for feedback/corrections.
Moreover if people think it is worthy idea, I can re-assemble
the paper again, donate it to the PSF and publish it on the
Python website.

Michele Simionato
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


matt.giuca at gmail

Aug 24, 2008, 6:43 AM

Post #2 of 37 (13951 views)
Permalink
Re: Things to Know About Super [In reply to]

Hi Michele,

Do you have a URL for this blog?

Matt


michele.simionato at gmail

Aug 24, 2008, 7:12 AM

Post #3 of 37 (13987 views)
Permalink
Re: Things to Know About Super [In reply to]

On Aug 24, 3:43 pm, "Matt Giuca" <matt.gi...@gmail.com> wrote:
> Hi Michele,
>
> Do you have a URL for this blog?
>

Sorry, here it is:
http://www.artima.com/weblogs/index.jsp?blogger=micheles
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


python at rcn

Aug 26, 2008, 8:32 AM

Post #4 of 37 (13937 views)
Permalink
Re: Things to Know About Super [In reply to]

[Michele Simionato]
> Recently I have opened a blog on Artima and I am publishing a
> few Python-related essays I had in store. In particular a trilogy
> of papers about "super". From the foreword:
>
> """
> In 2004 I decided to write a comprehensive paper documenting
> ``super`` pitfalls and traps, ...

Thanks for posting these blogs. I think they serve as a nice
compilation of docs, original analysis, and various discussions on super().

What I would really like to see is a fourth blog entry that shows
how to use super() reliably and correctly. In general, I opposed
to creating documentation in the form of "danger, danger this
could explode ...." IMO, there is not much point in dwelling on
bugs that have already been fixed, nor is there an advantage to
showing all the ways the tool can be misused.

In applications without multiple inheritance, it is a straight-forward
to use super() as a way to avoid directly naming an immediate
parent class. This is useful in and of itself.

For cooperative multiple inheritance, I take issue with the abstracted
examples provided (i.e. inconsistent signatures). In a real app that
actually needs cooperative multiple inheritance, it becomes self-evident
what "cooperative" actually means -- the methods *have* to be
designed to interoperate -- it is intrinsic to the problem at hand.

Your reasoning is akin to saying that cooperative multitasking is intrinsically
flawed because one of the tasks could be designed to not cooperate
(never yield).

Cooperative multiple inheritance is *not* about mixing two unrelated
parents that just happen to use the same method name but have
different semantics and were not designed to cooperate with each other.

The A-B-C-D diagrams and foo/bar methods in the examples are
deceptive because they silently drop the precondition of cooperation
while attempting to demonstrate a supposed flaw in the tool.

If I understand the problem correctly, in the rare cases where you do
need cooperative multiple inheritance, then super() is the only workable
solution short of designing some equivalent using composition instead
of inheritance.

I do agree with you that the docs can be improved. I'll work on patch
that makes clear that super() returns a proxy-object (not an actual
parent class) that dispatches explict method calls (not syntax driven)
to one of multiple parents designed to interact cooperatively.

Also, it may be controversial, but there may be some merit in
de-documenting the "unbound" case. It seems to add more
confusion than it's worth.

Lastly, I take issue with one other part of the blogs. While they show
a clear dislike for cooperative multiple inheritance, they take a potshot
at multiple inheritance in general. I don't follow the logic here. IMO,
mixin classes like DictMixin have proven themselves as being very
useful. Plenty of frameworks share this approach. Likewise, the
new ABCs offer mixin capabilities that are really nice.

I think it is a non-sequiter to reason from "diamond diagrams are
complicated" to "mixins should be disallowed". Instead, I think it
better to simply recommend that a key to happiness is to keep
various mixin classes completely orthogonal to one another
(no overlapping method names).

Lest I sound negative, let me thank you again for the blog entries
and for compiling the most complete discussion of it in one place.


Raymond

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


michele.simionato at gmail

Aug 26, 2008, 9:13 AM

Post #5 of 37 (13933 views)
Permalink
Re: Things to Know About Super [In reply to]

On Tue, Aug 26, 2008 at 5:32 PM, Raymond Hettinger <python [at] rcn> wrote:
> What I would really like to see is a fourth blog entry that shows
> how to use super() reliably and correctly.

That could be arranged.

> In general, I opposed
> to creating documentation in the form of "danger, danger this
> could explode ...." IMO, there is not much point in dwelling on
> bugs that have already been fixed, nor is there an advantage to
> showing all the ways the tool can be misused.

Yep. The parts about the bugs of super in 2.2 and 2.3 were written years
ago, when they were relevant. Nowadays they are less relevant, but since they
were already written and since there are still people using older versions
of Python I decided to keep them. I would not keep them in a revised version
intended as "semi-official" documentation of super. Still, I think they are
fine as a blog post.

> For cooperative multiple inheritance, I take issue with the abstracted
> examples provided (i.e. inconsistent signatures). In a real app that
> actually needs cooperative multiple inheritance, it becomes self-evident
> what "cooperative" actually means -- the methods *have* to be
> designed to interoperate -- it is intrinsic to the problem at hand.

> Cooperative multiple inheritance is *not* about mixing two unrelated
> parents that just happen to use the same method name but have
> different semantics and were not designed to cooperate with each other.
>
> The A-B-C-D diagrams and foo/bar methods in the examples are
> deceptive because they silently drop the precondition of cooperation
> while attempting to demonstrate a supposed flaw in the tool.

They just show that the tool is delicate and not easy to use.

> If I understand the problem correctly, in the rare cases where you do
> need cooperative multiple inheritance, then super() is the only workable
> solution short of designing some equivalent using composition instead
> of inheritance.

In my experience, one can go a long way using composition instead of
inheritance.
I also think that Python would not lose much without cooperative
multiple inheritance.
This is however a personal opinion and in any case the point is moot
because the language is the way it is. Still, in a blog post personal
opinions and even rants have their place. That part could be removed in an
"semi-official" document.

> Also, it may be controversial, but there may be some merit in de-documenting
> the "unbound" case. It seems to add more confusion than it's worth.

Fine with me.

> Lastly, I take issue with one other part of the blogs. While they show
> a clear dislike for cooperative multiple inheritance, they take a potshot
> at multiple inheritance in general. I don't follow the logic here. IMO,
> mixin classes like DictMixin have proven themselves as being very
> useful. Plenty of frameworks share this approach. Likewise, the
> new ABCs offer mixin capabilities that are really nice.
>
> I think it is a non-sequiter to reason from "diamond diagrams are
> complicated" to "mixins should be disallowed". Instead, I think it better
> to simply recommend that a key to happiness is to keep various mixin classes
> completely orthogonal to one another (no overlapping method names).

I not completely against multiple inheritance. I am against multiple inheritance
as it is now. A restricted form of multiple inheritance in which mixins classes
are guaranteed to be orthogonal would be fine with me (provided it is
not abused).
This concept exists already in other languages, the orthogonal mixins
are called "traits".

I have written a trilogy of papers about mixins: if you read them, you will see
where I come from (Zope, which I do not like) and you will also see
that I like DictMixin
instead.
I will publish the papers in the blog soon or later, but you can find
the Italian version here:

http://stacktrace.it/articoli/2008/06/i-pericoli-della-programmazione-con-i-mixin1/
http://stacktrace.it/articoli/2008/07/i-pericoli-della-programmazione-con-i-mixin2/
http://stacktrace.it/articoli/2008/08/i-pericoli-della-programmazione-con-i-mixin3/


Michele Simionato
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


python at rcn

Aug 26, 2008, 11:56 AM

Post #6 of 37 (13927 views)
Permalink
Re: Things to Know About Super [In reply to]

[Raymond]
>> Cooperative multiple inheritance is *not* about mixing two unrelated
>> parents that just happen to use the same method name but have
>> different semantics and were not designed to cooperate with each other.
>>
>> The A-B-C-D diagrams and foo/bar methods in the examples are
>> deceptive because they silently drop the precondition of cooperation
>> while attempting to demonstrate a supposed flaw in the tool.

[Michele]
> They just show that the tool is delicate and not easy to use.

To me, they miss the point. Simply, if you don't have diamonds,
then super() is easy to use and if you do have have diamonds,
then super() is the *only* way to do it.

Diamonds impose a set of design constraints (making the classes
cooperative). The A-B-C-D diagrams ignore this point and
make super() seem like an accident waiting to happen. In contrast,
with a concrete example, accidental non-cooperativeness would be
more self-evident.

The problem isn't that super() is fragile. The problem is that
a cooperative design pattern requires you to actually make
the classes cooperate. This is no different than a visitor design
pattern requiring that you attach a conformant visit() method for
on each of the visited classes.


[Michele]
> In my experience, one can go a long way using composition instead of
> inheritance.

While that is often true, I don't think it applies to the case of
cooperative multiple inheritence. To achieve substantially the
same functionality using composition, you would likely have to
re-invent much of what super() does for us automatically, and
you would still be imposing constraits on the composed classes
that are substantially the same as what you have with inheritance.


[Michele]
> I also think that Python would not lose much without cooperative
> multiple inheritance.

I would state this differently: "The use cases for cooperative multiple
inheritence don't arise often in practice; so, if we dropped support
for those cases, you probably wouldn't notice until you encountered
one of the rare occasions where it was the right answer to your problem."

There was some quote floating around that expressed the situation
well -- it went something like: "Python makes most problems easy
and hard problems possible". The use cases for cooperative multiple
inheritance fall in the latter category.

BTW, I really like your paper explaining the MRO. Excellent work.


Raymond


_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


steve at holdenweb

Aug 26, 2008, 2:10 PM

Post #7 of 37 (13942 views)
Permalink
Re: Things to Know About Super [In reply to]

Michele Simionato wrote:
> On Tue, Aug 26, 2008 at 5:32 PM, Raymond Hettinger <python [at] rcn> wrote:
[...]
>> I think it is a non-sequiter to reason from "diamond diagrams are
>> complicated" to "mixins should be disallowed". Instead, I think it better
>> to simply recommend that a key to happiness is to keep various mixin classes
>> completely orthogonal to one another (no overlapping method names).
>
> I not completely against multiple inheritance. I am against multiple inheritance
> as it is now. A restricted form of multiple inheritance in which mixins classes
> are guaranteed to be orthogonal would be fine with me (provided it is
> not abused).
> This concept exists already in other languages, the orthogonal mixins
> are called "traits".
>
If you aren't aware of it you should take a look at Enthought's traits
package. It's part of the Enthought Tool Suite (ETS).

https://svn.enthought.com/enthought/wiki

While I too appreciate your comments about super I believe you have
perhaps overdone it. I do look forward to seeing the edited edition as a
part of the documentation. [Hint: the *docs* aren't in feature freeze ;-)]

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC http://www.holdenweb.com/

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


michele.simionato at gmail

Aug 26, 2008, 6:16 PM

Post #8 of 37 (13933 views)
Permalink
Re: Things to Know About Super [In reply to]

On Tue, Aug 26, 2008 at 8:56 PM, Raymond Hettinger <python [at] rcn> wrote:
> I would state this differently: "The use cases for cooperative multiple
> inheritence don't arise often in practice; so, if we dropped support
> for those cases, you probably wouldn't notice until you encountered
> one of the rare occasions where it was the right answer to your problem."
>
> There was some quote floating around that expressed the situation
> well -- it went something like: "Python makes most problems easy
> and hard problems possible". The use cases for cooperative multiple
> inheritance fall in the latter category.

It is just a matter of how rare the use cases really are. Cooperative
methods has been introduced 6+ years ago. In all this time surely
they must have been used. How many compelling uses of cooperation
we can find in real life code? For instance in the standard library or
in some well known framework? This is a serious question I have been
wanting to ask for years. I am sure people here can find some example,
so just give me a pointer and we will see.

> BTW, I really like your paper explaining the MRO. Excellent work.

The issue with that paper is that I wrote it when my Python experience
was reduced to six month and my experience with real life large object oriented
frameworks was zero. Nowadays I value simplicity more.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


aleaxit at gmail

Aug 26, 2008, 6:30 PM

Post #9 of 37 (13931 views)
Permalink
Re: Things to Know About Super [In reply to]

On Tue, Aug 26, 2008 at 6:16 PM, Michele Simionato
<michele.simionato [at] gmail> wrote:
...
> It is just a matter of how rare the use cases really are. Cooperative
> methods has been introduced 6+ years ago. In all this time surely
> they must have been used. How many compelling uses of cooperation
> we can find in real life code? For instance in the standard library or
> in some well known framework? This is a serious question I have been
> wanting to ask for years. I am sure people here can find some example,
> so just give me a pointer and we will see.

http://www.koders.com/default.aspx?s=super&btn=&la=Python&li=* finds
over 5,000 hits, but it would take substantial work to sift through
them (in particular because not all refer to the built-in super, as
you'll see even in the first page!) -- and a random hit I found by
going to p.7 is really bad...:

"""Mixin to enable reification."""
def __init__(self):
super(ReificationStore, self).__init__()

[there's *nothing else* in this __init__!].


Alex
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


pje at telecommunity

Aug 26, 2008, 8:15 PM

Post #10 of 37 (13943 views)
Permalink
Re: Things to Know About Super [In reply to]

At 03:16 AM 8/27/2008 +0200, Michele Simionato wrote:
>It is just a matter of how rare the use cases really are. Cooperative
>methods has been introduced 6+ years ago. In all this time surely
>they must have been used. How many compelling uses of cooperation
>we can find in real life code? For instance in the standard library or
>in some well known framework? This is a serious question I have been
>wanting to ask for years. I am sure people here can find some example,
>so just give me a pointer and we will see.

ISTR pointing out on more than one occasion that a major use case for
co-operative super() is in the implementation of metaclasses. The
__init__ and __new__ signatures are fixed, multiple inheritance is
possible, and co-operativeness is a must (as the base class methods
*must* be called). I'm hard-pressed to think of a metaclass
constructor or initializer that I've written in the last half-decade
or more where I didn't use super() to make it co-operative.

That, IMO, is a compelling use case even if there were not a single
other example of the need for super. However, I'm pretty sure I've
had other cases where it was necessary to co-operate in cases where
multiple inheritance occurred later; ie. where it was possible for a
subclass to add a new class between parents. Remember that
subclasses of a new-style class do not always have the same MRO tail
as the original class; i.e., a subclass of "class A(B, C):" is only
constrained to have [A...B...C] in its MRO; semi-arbitrary classes
may be inserted between e.g. A and B. So, a new-style class cannot,
as a general rule, statically determine what base class
implementation of a method should be invoked. I personally consider
the rare case where I have to force such static knowledge to be an
unfortunate wart in the design (of that code, not Python).

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


michele.simionato at gmail

Aug 26, 2008, 8:18 PM

Post #11 of 37 (13936 views)
Permalink
Re: Things to Know About Super [In reply to]

On Wed, Aug 27, 2008 at 3:30 AM, Alex Martelli <aleaxit [at] gmail> wrote:
> On Tue, Aug 26, 2008 at 6:16 PM, Michele Simionato
> <michele.simionato [at] gmail> wrote:
> ...
>> It is just a matter of how rare the use cases really are. Cooperative
>> methods has been introduced 6+ years ago. In all this time surely
>> they must have been used. How many compelling uses of cooperation
>> we can find in real life code? For instance in the standard library or
>> in some well known framework? This is a serious question I have been
>> wanting to ask for years. I am sure people here can find some example,
>> so just give me a pointer and we will see.
>
> http://www.koders.com/default.aspx?s=super&btn=&la=Python&li=* finds
> over 5,000 hits, but it would take substantial work to sift through
> them (in particular because not all refer to the built-in super, as
> you'll see even in the first page!)

Yep. Notice (I am sure you understood the point correctly, but just to clarify)
that I am not interested in random occurrences of super, but in
code/frameworks expressly designed to leverage on cooperation
and doing it in a compelling way. IOW, I want to see cases where using
cooperation
is really better than relying on other techniques. Guido gives an example in
http://www.python.org/download/releases/2.2.3/descrintro/#cooperation
with a .save method, so in theory there are good use cases, but I
wonder in practice how common they are and if they are
frequent enough to justify the added complication.

M.S.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


michele.simionato at gmail

Aug 26, 2008, 8:48 PM

Post #12 of 37 (13936 views)
Permalink
Re: Things to Know About Super [In reply to]

On Tue, Aug 26, 2008 at 6:13 PM, Michele Simionato
<michele.simionato [at] gmail> wrote:
> I not completely against multiple inheritance. I am against multiple inheritance
> as it is now. A restricted form of multiple inheritance in which mixins classes
> are guaranteed to be orthogonal would be fine with me (provided it is
> not abused).
> This concept exists already in other languages, the orthogonal mixins
> are called "traits".

I must correct myself here. Even if for practical purposes traits look
like a restricted multiple
inheritance, in principle it is better to think of them as of an
enhanced single inheritance.
With traits there is always a single superclass: traits are just
single inheritance with a nice
syntax to include methods (like in Ruby) and a guarantee that methods
will not be overridden
silently (this one is missing in Ruby).


M.S.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


michele.simionato at gmail

Aug 26, 2008, 10:24 PM

Post #13 of 37 (13940 views)
Permalink
Re: Things to Know About Super [In reply to]

On Wed, Aug 27, 2008 at 5:15 AM, Phillip J. Eby <pje [at] telecommunity> wrote:
> ISTR pointing out on more than one occasion that a major use case for
> co-operative super() is in the implementation of metaclasses. The __init__
> and __new__ signatures are fixed, multiple inheritance is possible, and
> co-operativeness is a must (as the base class methods *must* be called).
> I'm hard-pressed to think of a metaclass constructor or initializer that
> I've written in the last half-decade or more where I didn't use super() to
> make it co-operative.
>
> That, IMO, is a compelling use case even if there were not a single other
> example of the need for super.
I have been giving a lot of thought to this use case, at least
from the time of the metaclass conflict recipe. I have always wondered
why the recipe had to be so complicated. At the end, I have come to
the conclusion that the problem was not with the recipe but with
multiple inheritance itself.
Let me explain the argument.

A possible use case for multiple inheritance of metaclasses is the
following: suppose I have a DebugMeta metaclass which adds some
debugging support to classes; now I want to apply it to a third party
framework which uses a FrameworkMeta metaclass internally. Let us
suppose the framework author wrote its metaclass correctly, by
supporting cooperation:

.. code-block:: python

class FrameworkMeta(type):
def __new__(mcl, name, bases, dic):
print "Adding framework features to %s" % name
return super(FrameworkMeta, mcl).__new__(mcl, name, bases, dic)


>>> class FrameworkClass(object):
... __metaclass__ = FrameworkMeta
Adding framework features to FrameworkClass

Moreover, suppose I wrote my DebugMeta to support cooperation
correctly:

.. code-block:: python

class DebugMeta(type):
def __new__(mcl, name, bases, dic):
print "Adding debugging features to %s" % name
return super(DebugMeta, mcl).__new__(mcl, name, bases, dic)


Now I can add the debugging features to a class in this way:

.. code-block:: python

class DebugFrameworkMeta(DebugMeta, FrameworkMeta):
pass


>>> class DebugFrameworkClass(FrameworkClass):
... __metaclass__ = DebugFrameworkMeta
Adding debugging features to DebugFrameworkClass
Adding framework features to DebugFrameworkClass

As you see everything works fine. Now, lets travel in the fictional
world of a fictional language called T-Python which is just like
Python, except it lacks multiple inheritance but has some support for
traits. By this I mean that there is an ``include_mixin`` function
working more or less like this (it could be enhanced but I am keeping
it dead simple here for the sake of the argument):

.. code-block:: python

def include_mixin(mixin, cls): # could be extended to use more mixins
# traits as in Squeak take the precedence over the base class
dic = vars(mixin).copy() # could be extended to walk the ancestors
return type(mixin.__name__ + cls.__name__, (cls,), dic)


I will argue that T-Python is not worse than Python for this use
case (composition of metaclasses).

In the fictional world there is not need for super since
all hierarchies are linear and you can just call the base class;
FrameworkMeta could have been written as

.. code-block:: python

class FrameworkMeta2(type):
def __new__(mcl, name, bases, dic):
print "Adding framework features to %s" % name
return type.__new__(mcl, name, bases, dic)


and DebugMetas as

.. code-block:: python

class DebugMeta2(type):
def __new__(mcl, name, bases, dic):
print "Adding debugging features to %s" % name
return mcl.__base__.__new__(mcl, name, bases, dic)


Notice that DebugMeta2 is performing a sort of cooperative call here
(``mcl.__base__.__new__``) but dead simple since there is just one base class.

The analogous of FrameworkClass can be defined as

>>> class FrameworkClass2(object):
... __metaclass__ = FrameworkMeta2
Adding framework features to FrameworkClass2

and the analogous of DebugFrameworkClass as

>>> class DebugFrameworkClass2(FrameworkClass2):
... __metaclass__ = DebugFrameworkMeta2
Adding debugging features to DebugFrameworkClass2
Adding framework features to DebugFrameworkClass2

So, as you see, it works. Checks of the kind
``isinstance(DebugFrameworkClass2, DebugMeta2)`` would fail, but this
is not a big issue (isinstance should not be used, or you could
register DebugMeta2 as a base class even if it is not by using
Python 2.6 ABC's).

Now, I am not claiming that I have thought of all possible usages of
multiple inheritance and metaclasses: however I have not found yet a
use case that I could not rewrite by using single-inheritance + traits
as the one I have just shown. Possibly there are cases which are
difficult to rewrite: but how common are they?

Notice that I am not advocating rewriting Python. The argument here is
purely hyphotetic and concerning a fictional language: I just want to
understand if full multiple inheritance is really that useful or not,
as a matter of principle.


M.S.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


michele.simionato at gmail

Aug 26, 2008, 11:02 PM

Post #14 of 37 (13922 views)
Permalink
Re: Things to Know About Super [In reply to]

On Tue, Aug 26, 2008 at 11:10 PM, Steve Holden <steve [at] holdenweb> wrote:
> If you aren't aware of it you should take a look at Enthought's traits
> package. It's part of the Enthought Tool Suite (ETS).

I know of the existence of that framework, however it is quite
large and I don't see the relation with the concept of traits
I have in mind, which is more or less the one described here:
http://www.iam.unibe.ch/%7Escg/Archive/Papers/Scha03aTraits.pdf

Basically, these are the properties of traits:

1. the methods/attributes in a trait go logically together;
2. if a trait enhances a class, then all subclasses are enhanced too;
3. if a trait has methods in common with the class, then the
methods defined in the class have the precedence;
4. the ordering of traits is not important, i.e. enhancing a class
first with trait T1 and then with trait T2 or viceversa is the same;
5. if traits T1 and T2 have names in common, enhancing a class both
with T1 and T2 raises an error unless there is an explicitoverriding;
6. if a trait has methods in common with the base class, then the
trait methods have the precedence;

Properties from 4 to 6 are the distinguishing properties of traits
with respect to multiple inheritance and mixins.
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


greg.ewing at canterbury

Aug 27, 2008, 12:54 AM

Post #15 of 37 (13922 views)
Permalink
Re: Things to Know About Super [In reply to]

Phillip J. Eby wrote:

> ISTR pointing out on more than one occasion that a major use case for
> co-operative super() is in the implementation of metaclasses. The
> __init__ and __new__ signatures are fixed, multiple inheritance is
> possible, and co-operativeness is a must

Do you have a real-life example of this where multiple
inheritance is actually used?

A non-contrived example or two would be a good thing to
have in tutorials etc. where super() is discussed. It
would help to convey the kinds of situations in which
use of super() is and is not appropriate.

--
Greg
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


mal at egenix

Aug 27, 2008, 3:53 AM

Post #16 of 37 (13934 views)
Permalink
Re: Things to Know About Super [In reply to]

On 2008-08-27 09:54, Greg Ewing wrote:
> Do you have a real-life example of this where multiple
> inheritance is actually used?
>
> A non-contrived example or two would be a good thing to
> have in tutorials etc. where super() is discussed. It
> would help to convey the kinds of situations in which
> use of super() is and is not appropriate.

The typical use is in mixin classes that can be used to
add functionality to base classes, something you often find
in application frameworks, e.g.

class NewComponent(Feature1Mixin, Feature2Mixin, BaseComponent):
...

If the mixin classes have to override one of the methods defined
in BaseComponent, then they must pay attention to all other mixin
classes used to define the NewComponent.

Without super() (or some other mechanism of accessing the base
method, like e.g. mxTools' basemethod() for classic classes), the
mixins could potentially override methods defined by other mixin
classes which would then not get called.

As example, think of a typical application server method

def process_request(self, request):
...

To work properly, each implementation of the method in the mixin classes
and base class will have to be called - in the order they were defined
in the class definition.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Aug 27 2008)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


aleaxit at gmail

Aug 27, 2008, 7:30 AM

Post #17 of 37 (13925 views)
Permalink
Re: Things to Know About Super [In reply to]

On Tue, Aug 26, 2008 at 10:24 PM, Michele Simionato
<michele.simionato [at] gmail> wrote:
...
> .. code-block:: python
>
> def include_mixin(mixin, cls): # could be extended to use more mixins
> # traits as in Squeak take the precedence over the base class
> dic = vars(mixin).copy() # could be extended to walk the ancestors
> return type(mixin.__name__ + cls.__name__, (cls,), dic)

I don't see any use of this in your following example so I assume
you're introducing it just to be able to say that:

> In the fictional world there is not need for super since
> all hierarchies are linear and you can just call the base class;

Nevertheless I must be missing something in the following example:

> FrameworkMeta could have been written as
>
> .. code-block:: python
>
> class FrameworkMeta2(type):
> def __new__(mcl, name, bases, dic):
> print "Adding framework features to %s" % name
> return type.__new__(mcl, name, bases, dic)
>
>
> and DebugMetas as
>
> .. code-block:: python
>
> class DebugMeta2(type):
> def __new__(mcl, name, bases, dic):
> print "Adding debugging features to %s" % name
> return mcl.__base__.__new__(mcl, name, bases, dic)
>
>
> Notice that DebugMeta2 is performing a sort of cooperative call here
> (``mcl.__base__.__new__``) but dead simple since there is just one base class.
>
> The analogous of FrameworkClass can be defined as
>
>>>> class FrameworkClass2(object):
> ... __metaclass__ = FrameworkMeta2
> Adding framework features to FrameworkClass2
>
> and the analogous of DebugFrameworkClass as
>
>>>> class DebugFrameworkClass2(FrameworkClass2):
> ... __metaclass__ = DebugFrameworkMeta2

What's DebugFrameworkMeta2? I assume it's some kind of mix but I
don't see it defined anywhere so I'm having to guess.

> Adding debugging features to DebugFrameworkClass2
> Adding framework features to DebugFrameworkClass2

But where does DebugMeta2 get injected or otherwise get into the
picture in this example, so that the "Adding debugging features" print
executes? I don't see any code in your example that performs this
injection.

Maybe you're missing a key bit where you build DebugFrameworkMeta2 by
using that example include_mixin you have?

I'd like to understand what, in this example, removes the apparent
"fragility" (or, lack of flexibility) where DebugMeta2 specifically
uses mcl.__base__ -- IOW, if I have another "mixin metaclass" just
like DebugMeta2 but called (say) RemotableMeta which does a print
"Adding remoting features" and then also calls
mcl.__base__.__new__(mcl ... just like DebugMeta2, what gets both of
their __new__ methods called in the right order?

Maybe you could help me understand this by giving a fully executable
Python snippet using __bases__[0] instead of the hypothetical
__base__?


A;ex
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


michele.simionato at gmail

Aug 27, 2008, 7:36 AM

Post #18 of 37 (13904 views)
Permalink
Re: Things to Know About Super [In reply to]

On Wed, Aug 27, 2008 at 4:30 PM, Alex Martelli <aleaxit [at] gmail> wrote:
> Maybe you could help me understand this by giving a fully executable
> Python snippet using __bases__[0] instead of the hypothetical
> __base__?

Sorry Alex, I have the fully functional snippet but evidently I have
sent some other blurb instead (it was early in the morning)
It is on my machine at home and now I am at work, so have patience ;)

Michele
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


greg.ewing at canterbury

Aug 27, 2008, 5:14 PM

Post #19 of 37 (13895 views)
Permalink
Re: Things to Know About Super [In reply to]

M.-A. Lemburg wrote:

> The typical use is in mixin classes that can be used to
> add functionality to base classes...

But this is just another waffly made-up example. I'm
talking about real-life use cases from actual code
that's in use.

--
Greg
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


michele.simionato at gmail

Aug 27, 2008, 9:35 PM

Post #20 of 37 (13924 views)
Permalink
Re: Things to Know About Super [In reply to]

Alex Martelli wrote:
> What's DebugFrameworkMeta2? I assume it's some kind of mix but I
> don't see it defined anywhere so I'm having to guess.

Sorry Alex, here is definition which got lost in cut&paste:

DebugFrameworkMeta2 = include_mixin(DebugMeta2, FrameworkMeta2)

> I'd like to understand what, in this example, removes the apparent
> "fragility" (or, lack of flexibility) where DebugMeta2 specifically
> uses mcl.__base__ -- IOW, if I have another "mixin metaclass" just
> like DebugMeta2 but called (say) RemotableMeta which does a print
> "Adding remoting features" and then also calls
> mcl.__base__.__new__(mcl ... just like DebugMeta2, what gets both of
> their __new__ methods called in the right order?

If you want to reimplement full cooperation between mixins classes
you must rework a bit the example, but it does not take a big effort
(see later). However my main point is: do we really want cooperative
methods? Multiple inheritance of metaclasses is perhaps
the strongest use case for multiple inheritance, but is it strong
enough? I mean, in real code how many times did I need that?
I would not mind make life harder for gurus and simpler for
application programmers. I do not think removing cooperation
would be so bad in practice. In many practical cases, one could just write
the metaclass by hand, in this example something like

class RemotableDebugFrameworkMeta(FrameworkMeta):
def __new__(mcl, name, bases, dic):
print "Adding remoting features to %s" % name
print "Adding debugging features to %s" % name
return FrameworkMeta.__new__(mcl, name, bases, dic)

Maybe you would need to duplicate a couple of lines and/or to introduce
an helper function, but you would have the benefit of having a more
explicit metaclass, with a simpler hierarchy (see the appendix for
an alternative solution).

> Maybe you could help me understand this by giving a fully executable
> Python snippet using __bases__[0] instead of the hypothetical
> __base__?

To the best of my knowledge __base__ is a valid class attribute,
it denotes the "right" class to use as base. Usually it is the same
as bases[0], but there is at least one case when it is different,
when composing old style and new style classes:

>>> class OldStyle: pass
>>> class NewStyle(object): pass
>>> class New(OldStyle, NewStyle): pass
>>> New.__bases__[0]
<class __main__.OldStyle at 0x777060>
>>> New.__base__
<class '__main__.NewStyle'>

Appendix: how to reimplement cooperation in a single-inheritance world
----------------------------------------------------------------------------

Quoting Raymond: "To achieve substantially the
same functionality, you would likely have to
re-invent much of what super() does for us automatically, and
you would still be imposing constraits on the composed classes
that are substantially the same as what you have with inheritance."

Raymond of course is right, but I am arguing that one does not really
need to re-invent cooperation because the use case for cooperation
are exceedingly rare. Still, if one really wants to reimplement
cooperation, here is my take at the challenge:

$ cat cooperative_mixins.py
"Implements cooperative mixins using multiple-inheritance only"

## everything in three lines
def include_mixin(mixin, cls): # could be extended to use more mixins
# traits as in Squeak take the precedence over the base class
dic = vars(mixin).copy() # could be extended to walk the ancestors
dic['_%s__super' % mixin.__name__] = cls
return type(mixin.__name__ + cls.__name__, (cls,), dic)

## examples:

class FrameworkMeta(type): # example metaclass
def __new__(mcl, name, bases, dic):
print "Adding framework features to %s" % name
return type.__new__(mcl, name, bases, dic)

class DebugMeta(type): # mixin metaclass
def __new__(mcl, name, bases, dic):
print "Adding debugging features to %s" % name
return mcl.__super.__new__(mcl, name, bases, dic)

class RemotableMeta(type): # another mixin metaclass
def __new__(mcl, name, bases, dic):
print "Adding remoting features to %s" % name
return mcl.__super.__new__(mcl, name, bases, dic)

class FrameworkClass(object):
__metaclass__ = FrameworkMeta

DebugFrameworkMeta = include_mixin(DebugMeta, FrameworkMeta)

print '**************** creating DebugFrameworkClass'
class DebugFrameworkClass(FrameworkClass):
__metaclass__ = DebugFrameworkMeta

RemotableDebugFrameworkMeta = include_mixin(
RemotableMeta, DebugFrameworkMeta)

print '**************** creating RemotableDebugFrameworkClass'
class RemotableDebugFrameworkClass(FrameworkClass):
__metaclass__ = RemotableDebugFrameworkMeta
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


pje at telecommunity

Aug 28, 2008, 8:30 AM

Post #21 of 37 (13888 views)
Permalink
Re: Things to Know About Super [In reply to]

At 06:35 AM 8/28/2008 +0200, Michele Simionato wrote:
>Multiple inheritance of metaclasses is perhaps
>the strongest use case for multiple inheritance, but is it strong
>enough? I mean, in real code how many times did I need that?
>I would not mind make life harder for gurus and simpler for
>application programmers.

Then you need to leave MI and co-operation the hell alone. Right
now, an application programmer can mix metaclasses like this:

class FooBar(Foo, Bar):
class __metaclass__(Foo.__class__, Bar.__class__): pass
...

Or, in 3.x:

class FooBarClass(Foo.__class__, Bar.__class__): pass

class FooBar(Foo, Bar, metaclass=FooBarClass):
...

Either way, this is useful in cases where Foo and Bar come from
different frameworks. That's the *only* way to get such things to
co-operate, in fact.


>I do not think removing cooperation
>would be so bad in practice. In many practical cases, one could just write
>the metaclass by hand,

How is that making things easier for application programmers?


>Maybe you would need to duplicate a couple of lines and/or to introduce
>an helper function,

...which then has to have an agreed-upon protocol that all metaclass
authors have to follow... which we already have... but which you're
proposing to get rid of... so we can re-invent it lots of
times... in mutually incompatible ways. :)

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


michele.simionato at gmail

Aug 28, 2008, 8:50 AM

Post #22 of 37 (13882 views)
Permalink
Re: Things to Know About Super [In reply to]

On Aug 28, 5:30 pm, "Phillip J. Eby" <p...@telecommunity.com> wrote:
> How is that making things easier for application programmers?

We have different definitions of "application programmer". For me a typical
application programmer is somebody who never fiddles with metaclasses,
which are the realm of framework builders. But the borders are fluid, I agree.

> >Maybe you would need to duplicate a couple of lines and/or to introduce
> >an helper function,
>
> ...which then has to have an agreed-upon protocol that all metaclass
> authors have to follow... which we already have... but which you're
> proposing to get rid of... so we can re-invent it lots of
> times... in mutually incompatible ways. :)

Notice that I was discussing an hypothetical language. I was arguing
that in principle
one could write a different language from Python, with single inheritance only,
and not lose much expressivity. I am not advocating any change to
current Python.
My point was in language design: I want to know how much I can remove
from a language
and still have something useful, in the spirit of the famous
Saint-Exupery quote.

Michele Simionato
_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


jjb5 at cornell

Aug 28, 2008, 11:36 AM

Post #23 of 37 (13886 views)
Permalink
Re: Things to Know About Super [In reply to]

Greg,


> Do you have a real-life example of this where multiple
> inheritance is actually used?

I have built a framework that I have called the "capability pattern"
which uses multiple inheritance in a way that might be unique (I'm not
familiar enough with other frameworks to know for sure).

There are two classes, a Collector and a Capability. The result of
calling a function of the collector is a list of results of calling the
functions of the bound capabilities. For example, these three are
capability classes:

class X(Capability):
def f(self): return 'X.f'

class Y(Capability):
def g(self): return 'Y.g'

class Z(Capability):
def f(self): return 'Z.f'
def g(self): return 'Z.g'

Now to create a sample collector:

class A(Collector, X, Y, Z): pass

Calling A().f() returns ['X.f', 'Z.f'].

I use this pattern in a web application. The do_GET call is mapped into
do_SHOW, and each 'capability' can return something from its do_SHOW
(usually a <div> element) and the results are sent back to the user. In
my case I have lots of combinations of capabilities that can be mixed
together.

I decided to use multiple inheritance over other patterns because I
wanted to leverage isinstance(obj,Y) to indicate that some object has
some capability, and not having to duplicate the method resolution order
code for other kinds of methods is really nice.

> A non-contrived example or two would be a good thing to
> have in tutorials etc. where super() is discussed. It
> would help to convey the kinds of situations in which
> use of super() is and is not appropriate.

So this is a collection of cooperative classes, and super() isn't used.


Joel

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


pje at telecommunity

Aug 28, 2008, 11:54 AM

Post #24 of 37 (13940 views)
Permalink
Re: Things to Know About Super [In reply to]

At 05:50 PM 8/28/2008 +0200, Michele Simionato wrote:
>On Aug 28, 5:30 pm, "Phillip J. Eby" <p...@telecommunity.com> wrote:
> > How is that making things easier for application programmers?
>
>We have different definitions of "application programmer". For me a typical
>application programmer is somebody who never fiddles with metaclasses,
>which are the realm of framework builders.

Application programmers use frameworks, and sometimes more than
one. If they're subclassing from two different frameworks, each
using a different metaclass, they will need to also multiple-inherit
the metaclass.

This is in fact so annoying that I created a "universal metaclass" in
DecoratorTools whose sole function is to delegate metaclass __new__,
__init__, and __call__ to class-level methods (e.g. __class_new__,
__class_call__, etc.), thereby eliminating the need to have custom
metaclasses for most use cases in the first place. Now, wherever
possible, I use that single metaclass in my frameworks, so that
there's no need to mix them.

That, IMO, would be a more useful change than getting rid of super();
it would get rid of the explicit metaclass mixing. (It would still
not remove the need for co-operative methods, as the class-delegated
methods still need to be co-operative for MI to work.)

There are, of course, other ways to create co-operative function
calls besides super(), and I've certainly created more a few of them
in my time. (E.g. generic function method combination,
instancemethod() chains, and next-method-iterators, to name the ones
that occur to me right off.) But these are more for cases where
super() is wholly inadequate to the purpose, and none are anywhere
near as convenient as super().

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com


tjreedy at udel

Aug 28, 2008, 12:59 PM

Post #25 of 37 (13897 views)
Permalink
Re: Things to Know About Super [In reply to]

Michele Simionato wrote:

> Notice that I was discussing an hypothetical language. I was arguing
> that in principle
> one could write a different language from Python, with single inheritance only,
> and not lose much expressivity. I am not advocating any change to
> current Python.

Since this is a list for discussing changing current Python, you will
have to excuse misunderstanding on that point. I do realize that you
began with a concrete offer of improving the documentation of the super
you hypothetically would like to be rid of ;-)

_______________________________________________
Python-Dev mailing list
Python-Dev [at] python
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/list-python-dev%40lists.gossamer-threads.com

First page Previous page 1 2 Next page Last page  View All Python dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.