Mailing List Archive: Python: Bugs

[issue15528] Better support for finalization with weakrefs

Index | Next | Previous | View Threaded

report at bugs

Aug 1, 2012, 9:18 AM

Post #1 of 13 (198 views)
Permalink

[issue15528] Better support for finalization with weakrefs

New submission from Richard Oudkerk:

A patch with docs and tests for the idea I suggested on python-ideas:

http://comments.gmane.org/gmane.comp.python.ideas/15863

To repeat what I wrote there, the current issues with weakref callbacks include:

1. They are rather low level, and working out how to use them
correctly requires a bit of head scratching. One must find
somewhere to store the weakref till after the referent is dead, and
without accidentally keeping the referent alive. Then one must
ensure that the callback frees the weakref (without leaving any
remnant ref-cycles).

When it is an option, using a __del__ method is far less hassle.

2. Callbacks invoked during interpreter shutdown are troublesome. For
instance they can throw exceptions because module globals have been
replaced by None.

3. Sometimes you want the callback to be called at program exit even
if the referent is still alive.

4. Exceptions thrown in callbacks do not produce a traceback. This
can make debugging rather awkward. (Maybe this is because printing
tracebacks is problematic during interpreter shutdown?)

(Note that problems 2-4 also apply to using __del__ methods.)

Trivial usage of the class looks like

>>> class Kenny: pass
...
>>> kenny = Kenny()
>>> finalize(kenny, print, "you killed kenny!")
<finalize.finalize object at 0xffed4f2c>
>>> del kenny
you killed kenny!

----------
files: finalize.patch
keywords: patch
messages: 167142
nosy: sbt
priority: normal
severity: normal
stage: patch review
status: open
title: Better support for finalization with weakrefs
type: enhancement
versions: Python 3.4
Added file: http://bugs.python.org/file26649/finalize.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15528>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 9:39 AM

Post #2 of 13 (190 views)
Permalink

[issue15528] Better support for finalization with weakrefs [In reply to]

Serhiy Storchaka added the comment:

I don't quite understand the purpose of your suggestions. What can you do with it help, what you can not do with contextlib.ExitStack, atexit, __del__ method, weakref.WeakKeyDictionary or weakref.ref? I read the documentation, but the meaning eludes me.

----------
nosy: +storchaka

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15528>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 12:24 PM

Post #3 of 13 (191 views)
Permalink

[issue15528] Better support for finalization with weakrefs [In reply to]

Richard Oudkerk added the comment:

> I don't quite understand the purpose of your suggestions. What can you do
> with it help, what you can not do with contextlib.ExitStack, atexit,
> __del__ method, weakref.WeakKeyDictionary or weakref.ref? I read the
> documentation, but the meaning eludes me.

finalize does not "compete" with contextlib.ExitStack, atexit and
weakref.WeakKeyDictionary. It only competes with __del__ and weakref
callbacks.

Points 1 and 2 in my first message are the main points. Also, read the
warning at

http://docs.python.org/py3k/reference/datamodel.html#object.__del__

which also applies to weakref callbacks.

Other problems with __del__:

* Ref cycles which contain an object with a __del__ method are immortal

* __del__ methods can "ressurect" the object.

There was actually a proposal to remove or replace __del__ methods in
Python 3000. See the "Removing __del__" thread(s):

http://mail.python.org/pipermail/python-3000/2006-September/thread.html#3797

As for weakref callbacks, I think they are just too difficult to use correctly
unless you are very familiar with them.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15528>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 1:36 PM

Post #4 of 13 (191 views)
Permalink

[issue15528] Better support for finalization with weakrefs [In reply to]

Serhiy Storchaka added the comment:

> finalize does not "compete" with contextlib.ExitStack, atexit and
> weakref.WeakKeyDictionary. It only competes with __del__ and weakref
> callbacks.

What kind of problems you solve with __del__ and weakref callbacks? For most
of them contextlib.ExitStack and atexit are safer and better fit.

> Points 1 and 2 in my first message are the main points. Also, read the
> warning at

For point 1: global weakref.WeakKeyDictionary is good store for weak refs with
callbacks.

global weakdict
weakdict[kenny] = weakref.ref(kenny, lambda _: print("you killed kenny!"))

If you do not want to work at a low level, in most cases fit the above-
mentioned high-level tools.

For point 2 Antoine has noted that it is a known issue and there is a solution
(issue812369).

> http://docs.python.org/py3k/reference/datamodel.html#object.__del__
>
> which also applies to weakref callbacks.

Is this your point 2?

> Other problems with __del__:
>
> * Ref cycles which contain an object with a __del__ method are immortal

It is use case for weakrefs which break reference loops.

> * __del__ methods can "ressurect" the object.

What you are meaning?

Relying on GC is dangerous thing for resource releasing. And it even more
dangerous for alternative implementations without reference counting. That is
why in recent times in Python there has been a tendency to RAII strategy
(context managers).

Can you give examples, where the use of finalize necessary or more convenient
the other ways?

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15528>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 4, 2012, 3:58 PM

Post #5 of 13 (191 views)
Permalink

[issue15528] Better support for finalization with weakrefs [In reply to]

Richard Oudkerk added the comment:

> For point 1: global weakref.WeakKeyDictionary is good store for weak refs with
> callbacks.
>
> global weakdict
> weakdict[kenny] = weakref.ref(kenny, lambda _: print("you killed kenny!"))

That depends on kenny being hashable.

It also surprises me a bit that it works. It seems to depend on unguaranteed
implementation details: PyObject_ClearWeakRefs() copies all weakrefs with
callbacks to a tuple before invoking callbacks. If this were not the case
then I think the weakref you created (which is scheduled to fire second) would
be garbage collected before its callback could be invoked.

I think users should have a documented way to schedule callbacks without having to
explicitly save the weakref.

> For point 2 Antoine has noted that it is a known issue and there is a solution
> (issue812369).

That has been languishing for 9 years. I would not hold your breath.

> > http://docs.python.org/py3k/reference/datamodel.html#object.__del__
> >
> > which also applies to weakref callbacks.
>
> Is this your point 2?

Yes.

> Relying on GC is dangerous thing for resource releasing. And it even more
> dangerous for alternative implementations without reference counting. That is
> why in recent times in Python there has been a tendency to RAII strategy
> (context managers).

I agree that explicit clean up using context managers to be strongly encouraged.
But resources should still be released if the object is garbage collected. Or
do you think that file objects should stop bothering to close their fds when they
are deallocated?

> Can you give examples, where the use of finalize necessary or more convenient
> the other ways?

multiprocessing (which I orginally wrote) makes extensive use of finalizers,
(although with a different implementation and api). In some cases that is because
at the time I wanted to support versions of Python which did not have the with
statement.

But there are various things there that depend on finalizers, and on being able
to guarantee that they get called on exit (in a predictable order).

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15528>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 5, 2012, 6:55 AM

Post #6 of 13 (189 views)
Permalink

[issue15528] Better support for finalization with weakrefs [In reply to]

Richard Oudkerk added the comment:

Updated patch.

get() and put() replaced by peek() and detach(). Added isalive(). Now finalizer class has no state, i.e. __slots__ == ().

----------
Added file: http://bugs.python.org/file26701/finalize.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15528>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 5, 2012, 10:03 AM

Post #7 of 13 (191 views)
Permalink

[issue15528] Better support for finalization with weakrefs [In reply to]

Antoine Pitrou added the comment:

I don't understand why you have two names: finalize and Finalizer. Let's keep only one of them.

isalive() could be a read-only property named "alive" instead.

In _make_callback, I still think the default error reporting mechanism should be kept. It can be improved separately.

----------
nosy: +pitrou

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15528>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 6, 2012, 5:35 AM

Post #8 of 13 (187 views)
Permalink

[issue15528] Better support for finalization with weakrefs [In reply to]

Richard Oudkerk added the comment:

> In _make_callback, I still think the default error reporting mechanism
> should be kept. It can be improved separately.

New patch. This time I have got rid of _make_callback, and just given __call__ an ignored optional argument.

----------
Added file: http://bugs.python.org/file26708/finalize.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15528>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 10, 2012, 6:32 AM

Post #9 of 13 (173 views)
Permalink

[issue15528] Better support for finalization with weakrefs [In reply to]

Changes by Andrew Svetlov <andrew.svetlov [at] gmail>:

----------
nosy: +asvetlov

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15528>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 11, 2012, 5:24 AM

Post #10 of 13 (168 views)
Permalink

[issue15528] Better support for finalization with weakrefs [In reply to]

Antoine Pitrou added the comment:

In the latest patch, what are "broken" and "priority" for?

Also, I would question why atexit is false by default. I would find it personally less surprising to be true.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15528>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 11, 2012, 12:17 PM

Post #11 of 13 (166 views)
Permalink

[issue15528] Better support for finalization with weakrefs [In reply to]

Richard Oudkerk added the comment:

> In the latest patch, what are "broken" and "priority" for?

They are for a subclass used by multiprocessing. But it is premature to worry about subclassing, so I will take them out.

> Also, I would question why atexit is false by default. I would find it
> personally less surprising to be true.

OK.

One thing I do worry about is having the loop in the exit function to run any finalizers created during the exit function. The current implementation will run these extra finalizers at the wrong time. Fixing this could probably be done by using a dirty flag and disabling gc while running the finalizers.

I wonder if it would be better to not call finalizers created during the exit function. We cannot guarantee that every finalizer created during shutdown is run, so is a best effort attempt really worth the effort?

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15528>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 17, 2012, 6:07 AM

Post #12 of 13 (155 views)
Permalink

[issue15528] Better support for finalization with weakrefs [In reply to]

Richard Oudkerk added the comment:

Updated patch.

----------
Added file: http://bugs.python.org/file26879/finalize.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15528>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Aug 19, 2012, 12:38 PM

Post #13 of 13 (143 views)
Permalink

[issue15528] Better support for finalization with weakrefs [In reply to]

Richard Oudkerk added the comment:

New patch.

----------
Added file: http://bugs.python.org/file26900/finalize.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15528>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads