Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Bugs

[issue15490] Correct __sizeof__ support for StringIO

 

 

Python bugs RSS feed   Index | Next | Previous | View Threaded


report at bugs

Jul 29, 2012, 12:54 PM

Post #1 of 18 (379 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO

Changes by Serhiy Storchaka <storchaka [at] gmail>:


Added file: http://bugs.python.org/file26588/stringio_sizeof-2.7.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jul 29, 2012, 12:53 PM

Post #2 of 18 (373 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

New submission from Serhiy Storchaka:

Here is a patch that implements correct __sizeof__ for C implementation of io.StringIO.

I haven't tested the patch on narrow build.

----------
components: IO, Library (Lib)
files: stringio_sizeof-3.3.patch
keywords: patch
messages: 166807
nosy: benjamin.peterson, hynek, pitrou, storchaka, stutzbach
priority: normal
severity: normal
status: open
title: Correct __sizeof__ support for StringIO
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file26586/stringio_sizeof-3.3.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jul 29, 2012, 12:53 PM

Post #3 of 18 (381 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Changes by Serhiy Storchaka <storchaka [at] gmail>:


Added file: http://bugs.python.org/file26587/stringio_sizeof-3.2.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jul 29, 2012, 12:58 PM

Post #4 of 18 (380 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Antoine Pitrou added the comment:

> Here is a patch that implements correct __sizeof__ for C implementation of io.StringIO.

For some value of "correct", since the internal accumulator could hold
alive some unicode strings.

> I haven't tested the patch on narrow build.

There's no narrow build anymore on 3.3.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jul 29, 2012, 1:54 PM

Post #5 of 18 (370 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Serhiy Storchaka added the comment:

> For some value of "correct", since the internal accumulator could hold
> alive some unicode strings.

This is not a concern of __sizeof__, because these lists and strings are
managed by GC.

> There's no narrow build anymore on 3.3.

I mean 2.7 and 3.2. I have written test for such case, but can't check if it
correct.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jul 29, 2012, 2:04 PM

Post #6 of 18 (379 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Antoine Pitrou added the comment:

> > For some value of "correct", since the internal accumulator could hold
> > alive some unicode strings.
>
> This is not a concern of __sizeof__, because these lists and strings are
> managed by GC.

It is, since they are private and not known by the user.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jul 29, 2012, 2:11 PM

Post #7 of 18 (373 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Martin v. Löwis added the comment:

The question really is what memory blocks can "leak out" of the object, and those don't really belong to the container. Those shouldn't be accounted for, since somebody else may get hold of them, and pass them to sys.sizeof.

For the PyAccu, AFAICT, objects cannot leak out of it (except for gc.getobjects in debug mode). So I think the memory allocated by the accu really belongs to the StringIO, and needs to be accounted there. The same goes for readnl, writenl, and weakreflist.

On a related note, I wonder why the tp_clear and tp_traverse functions for stringio are so short. Shouldn't it also visit "decoder"?

----------
nosy: +loewis

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jul 30, 2012, 1:14 AM

Post #8 of 18 (373 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Serhiy Storchaka added the comment:

> For the PyAccu, AFAICT, objects cannot leak out of it (except for gc.getobjects in debug mode).

Not only in debug mode.

>>> import io, gc
>>> s=io.StringIO()
>>> s.write('12345')
5
>>> s.write('qwerty')
6
>>> for o in gc.get_objects():
... if "'123" in repr(o) and len(repr(o)) < 1000:
... print(type(o), repr(o))
...
<class 'list'> ['12345', 'qwerty']
<class 'list'> [.'o', 'gc', 'get_objects', 'repr', 'o', "'123", 1000, 'len', 'repr', 'o', 'print', 'type', 'o', 'repr', 'o']
<class 'tuple'> ("'123", 1000, None)

Someone can summarize sys.getsizeof() for all managed by GC objects and therefore count internal objects twice.

I think the standard library should provide a method for recursive calculation of memory (in some sense, perhaps using different strategies), but that should wait up to 3.4. __sizeof__ should count non-object memory that is not available for GC. As I understand it.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jul 30, 2012, 2:01 AM

Post #9 of 18 (369 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Martin v. Löwis added the comment:

>> For the PyAccu, AFAICT, objects cannot leak out of it (except for
>> gc.getobjects in debug mode).
>
> Not only in debug mode.

I see. I meant sys.getobjects, which is in debug mode only, but
I now see that gc.get_objects will get the list (but not the strings)
of the Accu.

That still leaves readnl and writenl.

> I think the standard library should provide a method for recursive
> calculation of memory (in some sense, perhaps using different
> strategies), but that should wait up to 3.4.

Actually, this is (and should be) a separate project:

http://guppy-pe.sourceforge.net/

> __sizeof__ should count non-object memory that is not available for
> GC. As I understand it.

I think you misunderstand slightly; the GC relevance is only a side
effect. __sizeof__ should only account for memory that isn't separately
accessible. The notion of "separately accessible" is somewhat weak, since
it may depend on the container.

Non-PyObject blocks clearly need to be accounted for.

PyObject blocks normally don't need to be accounted for, as they are typically
accessible separately, by the following means:
- gc.get_objects (for GC objects)
- gc.get_referents (for contained non-GC objects)

There are also irregular ways to get objects:
- in debug mode only: sys.getobjects
- customer access functions or attributes

For memory accounting, it would really be best if the latter two categories
wouldn't exist, i.e. if all non-GC objects were still visible through
tp_traverse.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jul 31, 2012, 8:52 AM

Post #10 of 18 (360 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Serhiy Storchaka added the comment:

Do I understand correctly that for each Python subobject must be one of the following:
1) or it is visited in tp_traverse;
2) or it is accounted in __sizeof__ together with all its subobjects (if they cannot leak out)?

PyAccu can contains not only Unicode objects, but at least an instance of a Unicode subtype, which can has a reference to StringIO object, creating a cycle. Does this mean that PyAccu should be visited in tp_traverse and not be accounted in __sizeof__? Or should we tighten the control and ensure that PyAccu contains only exact Unicode objects?

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jul 31, 2012, 12:34 PM

Post #11 of 18 (359 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Antoine Pitrou added the comment:

> Does this mean that PyAccu should be visited in tp_traverse and not be
> accounted in __sizeof__? Or should we tighten the control and ensure
> that PyAccu contains only exact Unicode objects?

IMO, the latter.

----------

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Jul 31, 2012, 5:56 PM

Post #12 of 18 (360 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Changes by Jesús Cea Avión <jcea [at] jcea>:


----------
nosy: +jcea

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 5, 2012, 4:01 AM

Post #13 of 18 (359 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Changes by Serhiy Storchaka <storchaka [at] gmail>:


----------
assignee: -> storchaka

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 14, 2012, 1:57 PM

Post #14 of 18 (328 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Serhiy Storchaka added the comment:

Patch updated. Now private pyobjects (readnl, accu) counted.

Note all three patches rather different.

----------
Added file: http://bugs.python.org/file26810/stringio_sizeof-3.3_2.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 14, 2012, 1:58 PM

Post #15 of 18 (318 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Changes by Serhiy Storchaka <storchaka [at] gmail>:


Added file: http://bugs.python.org/file26811/stringio_sizeof-3.2_2.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 14, 2012, 1:59 PM

Post #16 of 18 (318 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Changes by Serhiy Storchaka <storchaka [at] gmail>:


Added file: http://bugs.python.org/file26812/stringio_sizeof-2.7_2.patch

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 15, 2012, 1:57 PM

Post #17 of 18 (318 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Changes by Serhiy Storchaka <storchaka [at] gmail>:


----------
assignee: storchaka ->
keywords: +needs review
stage: -> patch review

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com


report at bugs

Aug 29, 2012, 5:02 AM

Post #18 of 18 (266 views)
Permalink
[issue15490] Correct __sizeof__ support for StringIO [In reply to]

Changes by Aaron Iles <aaron.iles [at] gmail>:


----------
nosy: +aliles

_______________________________________
Python tracker <report [at] bugs>
<http://bugs.python.org/issue15490>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com

Python bugs RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.