Mailing List Archive: Python: Bugs

[issue3329] API for setting the memory allocator used by Python

Index | Next | Previous | View Threaded

report at bugs

Jul 9, 2008, 12:48 PM

Post #1 of 7 (387 views)
Permalink

[issue3329] API for setting the memory allocator used by Python

New submission from Jukka Laurila <jukka.p.laurila [at] nokia>:

Currently Python always uses the C library malloc/realloc/free as the
underlying mechanism for requesting memory from the OS, but especially
on memory-limited platforms it is often desirable to be able to override
the allocator and to redirect all Python's allocations to use a special
heap. This will make it possible to free memory back to the operating
system without restarting the process, and to reduce fragmentation by
separating Python's allocations from the rest of the program.

The proposal is to make it possible to set the allocator used by the
Python interpreter by calling the following function before Py_Initialize():

void Py_SetAllocator(void* (*alloc)(size_t), void* (*realloc)(void*,
size_t), void (*free)(void*))

Direct function calls to malloc/realloc/free in obmalloc.c must be
replaced with calls through the function pointers set through this
function. By default these would of course point to the C stdlib
malloc/realloc/free.

----------
components: Interpreter Core
messages: 69482
nosy: jlaurila
severity: normal
status: open
title: API for setting the memory allocator used by Python
type: feature request
versions: Python 2.6, Python 3.0

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue3329>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 9, 2008, 12:56 PM

Post #2 of 7 (384 views)
Permalink

[issue3329] API for setting the memory allocator used by Python [In reply to]

Brett Cannon <brett [at] python> added the comment:

Is registering pointers to functions really necessary, or would defining
macros work as well? From a performance perspective I would like to
avoid having a pointer indirection step every time malloc/realloc/free
is called.

I guess my question becomes, Jukka, is this more for alternative
implementations of Python where changes to source are already expected,
or for apps that embed Python where a change of malloc/realloc/free
varies from app to app that dynamically loads Python?

----------
nosy: +brett.cannon

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue3329>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 9, 2008, 1:07 PM

Post #3 of 7 (384 views)
Permalink

[issue3329] API for setting the memory allocator used by Python [In reply to]

Adam Olsen <rhamph [at] gmail> added the comment:

How would this allow you to free all memory? The interpreter will still
reference it, so you'd have to have called Py_Finalize already, and
promise not to call Py_Initialize afterwords. This further supposes the
process will live a long time after killing off the interpreter, but in
that case I recommend putting python in a child process instead.

----------
nosy: +Rhamphoryncus

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue3329>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 10, 2008, 1:05 AM

Post #4 of 7 (367 views)
Permalink

[issue3329] API for setting the memory allocator used by Python [In reply to]

Jukka Laurila <jukka.p.laurila [at] nokia> added the comment:

Brett, the ability to define the allocator dynamically at runtime could
be a compile time option, turned on by default only on small memory
platforms. On most platforms you can live with plain old malloc and may
want to avoid the indirection. If no other platform is interested in
this, we can just make it a Symbian-specific extension but I wanted to
see if there's general interest in this.

The application would control the lifecycle of the Python heap, and this
seemed like the most natural way for the application to tell the
interpreter which heap instance to use.

Adam, the cleanup would work by freeing the entire heap used by Python
after calling Py_Finalize. In the old PyS60 code we made Python 2.2.2
clean itself completely by freeing the Python-specific heap and making
sure all pointers to heap-allocated items are properly reinitialized.

Yes, there are various static pointers that are initially set to NULL,
initialized to point at things on the heap and not reset to NULL at
Py_Finalize, and these are currently an obstacle to calling
Py_Initialize again. I'm considering submitting a separate ticket about
that since it seems like the ability to free the heap combined with the
ability to reinitialize the static pointers could together make full
cleanup possible.

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue3329>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 10, 2008, 2:59 AM

Post #5 of 7 (355 views)
Permalink

[issue3329] API for setting the memory allocator used by Python [In reply to]

Nick Coghlan <ncoghlan [at] gmail> added the comment:

Given where we are in the release cycle, I've bumped the target releases
to 2.7/3.1. So Symbian are probably going to have to do something
port-specific anyway in order to get 2.6/3.0 up and running.

And in terms of hooking into this kind of thing, some simple macros that
can be overriden in pyport.h (as Brett suggested) may be a better idea
than baking any specific approach into the core interpreter.

----------
nosy: +ncoghlan
versions: +Python 2.7, Python 3.1 -Python 2.6, Python 3.0

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue3329>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 10, 2008, 3:12 AM

Post #6 of 7 (362 views)
Permalink

[issue3329] API for setting the memory allocator used by Python [In reply to]

Raymond Hettinger <rhettinger [at] users> added the comment:

I think it is reasonable to get a macro definition change into 2.6.
The OP's request is essential for his application (running Python
on Nokia phones) and it would be a loss to wait two years for this.
Also, his request for a macro will enable another important piece
of functionality -- allowing a build to intercept and instrument all
calls to the memory allocator.

Barry, can you rule on whether to keep this open for consideration in
2.6. It seems daft to postpone this discussion indefinitely. If we
can agree to a simple, non-invasive solution while there is still yet
another beta, then it makes sense to proceed.

----------
assignee: -> barry
nosy: +barry, rhettinger

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue3329>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

report at bugs

Jul 10, 2008, 9:57 AM

Post #7 of 7 (352 views)
Permalink

[issue3329] API for setting the memory allocator used by Python [In reply to]

Adam Olsen <rhamph [at] gmail> added the comment:

Basically you just want to kick the malloc implementation into doing
some housekeeping, freeing its caches? I'm kinda surprised you don't
add the hook directly to your libc's malloc.

IMO, there's no use-case for this until Py_Finalize can completely tear
down the interpreter, which requires a lot of special work (killing(!)
daemon threads, unloading C modules, etc), and nobody intends to do that
at this point.

The practical alternative, as I said, is to run python in a subprocess.
Let the OS clean up after us.

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue3329>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads