Mailing List Archive: Python: Python

tiny script has memory leak

Index | Next | Previous | View Threaded

georgeryoung at gmail

May 11, 2012, 2:29 PM

Post #1 of 6 (227 views)
Permalink

tiny script has memory leak

sys.version --> '2.6 (r26:66714, Feb 21 2009, 02:16:04) \n[GCC 4.3.2
[gcc-4_3-branch revision 141291]]
I thought this script would be very lean and fast, but with a large
value for n (like 150000), it uses 26G of virtural memory, and things
start to crumble.

#!/usr/bin/env python
'''write a file of random integers. args are: file-name how-many'''
import sys, random

f = open(sys.argv[1], 'w')
n = int(sys.argv[2])
for i in xrange(n):
print >>f, random.randint(0, sys.maxint)
f.close()

What's using so much memory?
What would be a better way to do this? (aside from checking arg
values and types, I know...)
--
http://mail.python.org/mailman/listinfo/python-list

ian.g.kelly at gmail

May 14, 2012, 5:19 PM

Post #2 of 6 (225 views)
Permalink

Re: tiny script has memory leak [In reply to]

On Fri, May 11, 2012 at 3:29 PM, gry <georgeryoung [at] gmail> wrote:
> sys.version --> '2.6 (r26:66714, Feb 21 2009, 02:16:04) \n[GCC 4.3.2
> [gcc-4_3-branch revision 141291]]
> I thought this script would be very lean and fast, but with a large
> value for n (like 150000), it uses 26G of virtural memory, and things
> start to crumble.
>
> #!/usr/bin/env python
> '''write a file of random integers. args are: file-name how-many'''
> import sys, random
>
> f = open(sys.argv[1], 'w')
> n = int(sys.argv[2])
> for i in xrange(n):
> print >>f, random.randint(0, sys.maxint)
> f.close()
>
> What's using so much memory?

I don't know, I'm not able to replicate the problem you're reporting.
When I try your script with a value of 150000, it runs in under a
second and does not appear to consume any more virtual memory than
what is normally used by the Python interpreter. I suspect there is
something else at play here.

> What would be a better way to do this? (aside from checking arg
> values and types, I know...)

I don't see anything wrong with the way you're currently doing it,
assuming you can solve your memory leak issue.

Ian
--
http://mail.python.org/mailman/listinfo/python-list

rosuav at gmail

May 14, 2012, 5:45 PM

Post #3 of 6 (223 views)
Permalink

Re: tiny script has memory leak [In reply to]

On Sat, May 12, 2012 at 7:29 AM, gry <georgeryoung [at] gmail> wrote:
> f = open(sys.argv[1], 'w')

What are you passing as the file name argument? Could that device be
the cause of your memory usage spike?

ChrisA
--
http://mail.python.org/mailman/listinfo/python-list

iainking at gmail

May 17, 2012, 2:06 AM

Post #4 of 6 (214 views)
Permalink

Re: tiny script has memory leak [In reply to]

On Friday, 11 May 2012 22:29:39 UTC+1, gry wrote:
> sys.version --> '2.6 (r26:66714, Feb 21 2009, 02:16:04) \n[GCC 4.3.2
> [gcc-4_3-branch revision 141291]]
> I thought this script would be very lean and fast, but with a large
> value for n (like 150000), it uses 26G of virtural memory, and things
> start to crumble.
>
> #!/usr/bin/env python
> '''write a file of random integers. args are: file-name how-many'''
> import sys, random
>
> f = open(sys.argv[1], 'w')
> n = int(sys.argv[2])
> for i in xrange(n):
> print >>f, random.randint(0, sys.maxint)
> f.close()
>
> What's using so much memory?
> What would be a better way to do this? (aside from checking arg
> values and types, I know...)

Ran OK for me, python 2.4.1 on Windows 7

Iain
--
http://mail.python.org/mailman/listinfo/python-list

alain at dpt-info

May 17, 2012, 2:50 AM

Post #5 of 6 (217 views)
Permalink

Re: tiny script has memory leak [In reply to]

gry <georgeryoung [at] gmail> writes:

> sys.version --> '2.6 (r26:66714, Feb 21 2009, 02:16:04) \n[GCC 4.3.2
> [gcc-4_3-branch revision 141291]]

> I thought this script would be very lean and fast, but with a large
> value for n (like 150000), it uses 26G of virtural memory, and things
> start to crumble.
>
> #!/usr/bin/env python
> '''write a file of random integers. args are: file-name how-many'''
> import sys, random
>
> f = open(sys.argv[1], 'w')
> n = int(sys.argv[2])
> for i in xrange(n):
> print >>f, random.randint(0, sys.maxint)
> f.close()

sys.version is '2.6.6 (r266:84292, Sep 15 2010, 16:22:56) \n[GCC 4.4.5]'
here, and your script works like a charm. BTW, I would use f.write()
instead of print >> f (which I think is deprecated).

-- Alain.
--
http://mail.python.org/mailman/listinfo/python-list

tjreedy at udel

May 17, 2012, 6:42 AM

Post #6 of 6 (216 views)
Permalink

Re: tiny script has memory leak [In reply to]

On 5/17/2012 5:50 AM, Alain Ketterlin wrote:
> gry<georgeryoung [at] gmail> writes:
>
>> sys.version --> '2.6 (r26:66714, Feb 21 2009, 02:16:04) \n[GCC 4.3.2
>> [gcc-4_3-branch revision 141291]]
>
>> I thought this script would be very lean and fast, but with a large
>> value for n (like 150000), it uses 26G of virtural memory, and things
>> start to crumble.
>>
>> #!/usr/bin/env python
>> '''write a file of random integers. args are: file-name how-many'''
>> import sys, random
>>
>> f = open(sys.argv[1], 'w')
>> n = int(sys.argv[2])
>> for i in xrange(n):
>> print>>f, random.randint(0, sys.maxint)
>> f.close()
>
> sys.version is '2.6.6 (r266:84292, Sep 15 2010, 16:22:56) \n[GCC 4.4.5]'
> here, and your script works like a charm. BTW, I would use f.write()

That would have to be f.write(str(random.randint(0, sys.maxint))+end)
where above end would be '\n'.

> instead of print>> f (which I think is deprecated).

In the sense that in Py3, print is a function with a file parameter:

print(random.randint(0, sys.maxint), file=f)

The idiosyncratic ugliness of >>file was one reason for the change.
Adding the option to specify separator and terminator was another.

--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads