Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Python: Python

Which uses less memory?

 

 

Python python RSS feed   Index | Next | Previous | View Threaded


bret.wortman at gmail

Nov 15, 2007, 9:46 AM

Post #1 of 3 (43 views)
Permalink
Which uses less memory?

I'm not sure if this is as easy a question as I'd like it to be, but
here goes....

I'm working on an application that is very memory intensive, so we're
trying to reduce the memory footprint of classes wherever possible. I
have a need for a class which is able to have a type identifier which
can be examined at run-time to determine whether that class (called
Domain) contains data I care about or not.

I've thought of two ways to implement this:

1. Add a type attribute and set it to a descriptive string.
2. Create marker classes and use multiple inheritance to "attach"
these markers to specific Domains.

Here's the kicker: I need to serialize these Domains and send them to/
from Java code as well as work on them using Python. We're looking to
use Hessian and pyactivemq to handle the Java/Python interfaces.
Which, I guess, leads to the following group of questions:

1. Which method has the smaller footprint within the Python engine?
2. Do these protocols (Hessian and Stomp) preserve the class
information when the class is serialized?

Any input would be most welcome. Thanks!


Bret Wortman
--
http://mail.python.org/mailman/listinfo/python-list


nick at craig-wood

Nov 17, 2007, 2:30 PM

Post #2 of 3 (36 views)
Permalink
Re: Which uses less memory? [In reply to]

bret.wortman[at]gmail.com <bret.wortman[at]gmail.com> wrote:
> I'm not sure if this is as easy a question as I'd like it to be, but
> here goes....
>
> I'm working on an application that is very memory intensive, so we're
> trying to reduce the memory footprint of classes wherever possible. I
> have a need for a class which is able to have a type identifier which
> can be examined at run-time to determine whether that class (called
> Domain) contains data I care about or not.
>
> I've thought of two ways to implement this:
>
> 1. Add a type attribute and set it to a descriptive string.
> 2. Create marker classes and use multiple inheritance to "attach"
> these markers to specific Domains.
>
> Here's the kicker: I need to serialize these Domains and send them to/
> from Java code as well as work on them using Python. We're looking to
> use Hessian and pyactivemq to handle the Java/Python interfaces.
> Which, I guess, leads to the following group of questions:
>
> 1. Which method has the smaller footprint within the Python engine?
> 2. Do these protocols (Hessian and Stomp) preserve the class
> information when the class is serialized?
>
> Any input would be most welcome. Thanks!

I don't know the answers to your specific questions but we managed to
1/3 the memory requirement of our app by identifying the class having
the most instances (about 1,000,000 in our case) and adding __slots__
to it!

I'd guess that if you __slot__-ed the Domain class then you'll find
the overhead of a type attribute is minimal (4 bytes per instance I
think).

No idea about Hessian or Stomp (never heard of them!) but classes with
__slot__s are normal classes which would pickle or unpickle.

--
Nick Craig-Wood <nick[at]craig-wood.com> -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list


gagsl-py2 at yahoo

Nov 18, 2007, 9:20 AM

Post #3 of 3 (32 views)
Permalink
Re: Which uses less memory? [In reply to]

En Sat, 17 Nov 2007 19:30:04 -0300, Nick Craig-Wood <nick[at]craig-wood.com>
escribi�:

>> I'm working on an application that is very memory intensive, so we're
>> trying to reduce the memory footprint of classes wherever possible. I
>
> I'd guess that if you __slot__-ed the Domain class then you'll find
> the overhead of a type attribute is minimal (4 bytes per instance I
> think).
>
> No idea about Hessian or Stomp (never heard of them!) but classes with
> __slot__s are normal classes which would pickle or unpickle.

Actually classes with __slots__ require that you write your own
__getstate__ and __setstate__. The following implementation may be enough
in simple cases:

py> class X(object):
... __slots__ = ('foo','bar')
...
py> x = X()
py> x.foo = 123
py> dumps(x)
Traceback (most recent call last):
...
TypeError: a class that defines __slots__ without defining __getstate__
cannot b
e pickled

class X(object):
__slots__ = ('foo','bar')
def __getstate__(self):
return dict((name, getattr(self, name))
for name in self.__slots__
if hasattr(self, name))
def __setstate__(self, state):
for name,value in state.iteritems():
setattr(self, name, value)

py> x = X()
py> x.foo = 123
py> p = dumps(x)
py> x2 = loads(p)
py> type(x2)
<class '__main__.X'>
py> x2.foo
123
py> x2.bar
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: bar

--
Gabriel Genellina

--
http://mail.python.org/mailman/listinfo/python-list

Python python RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact lists@gossamer-threads.com
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.