Mailing List Archive: Catalyst: Users

How to reduce the memory footprint?

Index | Next | Previous | View Threaded

julien at sobrier

Nov 23, 2009, 10:18 AM

Post #1 of 7 (1349 views)
Permalink

How to reduce the memory footprint?

Hello,
I have quite a small Catalyst application that runs with FastCGI +
FCGI::ProcManager::MaxRequests

I run 5 instances of FastCGI. Each instance was taking about 90MB of memory.
I tried to reduce the memory fooprint by reducing the number of libraries I
used. The memory usage is now 120MB per instance! The memory increase is
probably due to other changes.

I am wondering how to profile the memory usage, and how to reduce it:
* how can I figure out what is talking how much memory?
* is there a list of common things to do do to reduce the memory usage?
* is 100+MB per instance for small application usual? I hope not...
* any tip to reduce or profile my memory usage?

Thank you
Julien

bobtfish at bobtfish

Nov 23, 2009, 1:00 PM

Post #2 of 7 (1303 views)
Permalink

Re: How to reduce the memory footprint? [In reply to]

On 23 Nov 2009, at 18:18, Julien Sobrier wrote:

> Hello,
> I have quite a small Catalyst application that runs with FastCGI +
> FCGI::ProcManager::MaxRequests
>
> I run 5 instances of FastCGI. Each instance was taking about 90MB of
> memory. I tried to reduce the memory fooprint by reducing the number
> of libraries I used. The memory usage is now 120MB per instance! The
> memory increase is probably due to other changes.

When you say '5 instances', you mean you have nprocs = 5, right?

This _isn't_ using 100Mb per instance - all of the application is
loaded into memory before fork() is called, so it will mostly be
shared (by copy on write) memory.

Cheers
t0m

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

julien at sobrier

Nov 23, 2009, 1:25 PM

Post #3 of 7 (1301 views)
Permalink

Re: How to reduce the memory footprint? [In reply to]

Right, nprocs=5

I thought 100M RES/300M VIRT meanst 100M per process, and 300M shard among
perl processes.

I don't have many other processes running on the box. If I turn off
catalyst, I use less than 250MB or memory. With Catalyst, I'm over 800MB
with some occasional swapping.

On Mon, Nov 23, 2009 at 1:00 PM, Tomas Doran <bobtfish [at] bobtfish> wrote:

>
> On 23 Nov 2009, at 18:18, Julien Sobrier wrote:
>
> Hello,
>> I have quite a small Catalyst application that runs with FastCGI +
>> FCGI::ProcManager::MaxRequests
>>
>> I run 5 instances of FastCGI. Each instance was taking about 90MB of
>> memory. I tried to reduce the memory fooprint by reducing the number of
>> libraries I used. The memory usage is now 120MB per instance! The memory
>> increase is probably due to other changes.
>>
>
> When you say '5 instances', you mean you have nprocs = 5, right?
>
> This _isn't_ using 100Mb per instance - all of the application is loaded
> into memory before fork() is called, so it will mostly be shared (by copy on
> write) memory.
>
> Cheers
> t0m
>
>
> _______________________________________________
> List: Catalyst [at] lists
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst [at] lists/
> Dev site: http://dev.catalyst.perl.org/
>

peter at peknet

Nov 23, 2009, 1:44 PM

Post #4 of 7 (1294 views)
Permalink

Re: How to reduce the memory footprint? [In reply to]

Julien Sobrier wrote on 11/23/2009 03:25 PM:
> Right, nprocs=5
>
> I thought 100M RES/300M VIRT meanst 100M per process, and 300M shard
> among perl processes.
>
> I don't have many other processes running on the box. If I turn off
> catalyst, I use less than 250MB or memory. With Catalyst, I'm over 800MB
> with some occasional swapping.

that amount of memory use doesn't seem excessive to me. I've had 250+MB
per process before for larger apps.

If you have only that much memory (~1G) available, do you really need to
be running 5 procs? Why not 2 or 3? If you run into bottlenecks with 3
preforked processes on your current hw, you'll need to be looking at
upgrading your hardware anyway.

--
Peter Karman . http://peknet.com/ . peter [at] peknet

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

jshirley at gmail

Nov 23, 2009, 1:45 PM

Post #5 of 7 (1295 views)
Permalink

Re: How to reduce the memory footprint? [In reply to]

On Mon, Nov 23, 2009 at 1:25 PM, Julien Sobrier <julien [at] sobrier> wrote:

> Right, nprocs=5
>
> I thought 100M RES/300M VIRT meanst 100M per process, and 300M shard among
> perl processes.
>
> I don't have many other processes running on the box. If I turn off
> catalyst, I use less than 250MB or memory. With Catalyst, I'm over 800MB
> with some occasional swapping.
>
>
>
How are you profiling your app (and determining how much memory is
"free")? You can profile the memory usage of your app using the
development server and get a very good idea of where the memory usage is at.

I wouldn't trust the output of top/ps to determine how much memory is used.

-J

autarch at urth

Nov 23, 2009, 2:39 PM

Post #6 of 7 (1294 views)
Permalink

Re: How to reduce the memory footprint? [In reply to]

On Mon, 23 Nov 2009, Julien Sobrier wrote:

> I run 5 instances of FastCGI. Each instance was taking about 90MB of memory.
> I tried to reduce the memory fooprint by reducing the number of libraries I
> used. The memory usage is now 120MB per instance! The memory increase is
> probably due to other changes.

If you can, take a look at smem (http://www.selenic.com/smem/) to see how
much memory these procs are _really_ using.

This tool looks at how much memory is shared between procs and gives you a
much better sense of how much memory it costs _per process_.

But you need to be on a relatively recent Linux kernel to use it.

-dave

/*============================================================
http://VegGuide.org http://blog.urth.org
Your guide to all that's veg House Absolute(ly Pointless)
============================================================*/

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

rodrigolive at gmail

Nov 23, 2009, 3:39 PM

Post #7 of 7 (1294 views)
Permalink

Re: How to reduce the memory footprint? [In reply to]

On Mon, Nov 23, 2009 at 10:45 PM, J. Shirley <jshirley [at] gmail> wrote:

>
> I wouldn't trust the output of top/ps to determine how much memory is used.
>
> -J
>
>
I do.
Not for individual processes, though. But for the totals top is perfectly
fine for me.

I don't have many other processes running on the box. If I turn off
> catalyst, I use less than 250MB or memory. With Catalyst, I'm over 800MB
> with some occasional swapping.
>
>

When you say you go from less than 250 to over 800MB, it seems a little too
much. What does "small app" mean in this case?

My ubuntu box does not have swapping (it's OpenVZ), when it goes over my
allocated 256MB ram, out-of-memory hell breaks loose. So what I'm watching
for is actually my total free memory.

Before starting up my Catalyst app:

top - 23:16:01 up 14 days, 4:57, 1 user, load average: 0.02, 0.02, 0.00
Tasks: 23 total, 2 running, 21 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 262144k total, 25720k used, *236424k* *free*, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

After startup (out-of-the-box HTTP::Prefork, small app), 30MB are gone:

top - 23:14:36 up 14 days, 4:56, 1 user, load average: 0.07, 0.03, 0.01
Tasks: 29 total, 2 running, 27 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 262144k total, 55448k used, *206696k* *free*, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND

3637 catw 18 0 28448 24m 1524 S 0.0 9.5 0:00.00
perl

3673 catw 20 0 28448 23m 748 S 0.0 9.2 0:00.00
perl

3675 catw 20 0 28448 23m 628 S 0.0 9.2 0:00.00
perl

3676 catw 20 0 28448 23m 628 S 0.0 9.2 0:00.00
perl

3677 catw 24 0 28448 23m 628 S 0.0 9.2 0:00.00
perl

3678 catw 25 0 28448 23m 628 S 0.0 9.2 0:00.00 perl

After a few thousand hits, total mem usage seems to level out at 62MB:

top - 23:48:13 up 14 days, 5:29, 1 user, load average: 0.02, 0.02, 0.00
Tasks: 29 total, 2 running, 27 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 262144k total, 87264k used, *174880k free*, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND

3637 catw 15 0 28560 24m 1524 S 0.0 9.5 0:00.01
perl

3673 catw 15 0 29320 25m 1616 S 0.0 9.9 0:00.24
perl

3675 catw 15 0 29308 25m 1616 S 0.0 9.9 0:00.28
perl

3676 catw 15 0 29308 25m 1616 S 0.0 9.9 0:00.25
perl

3677 catw 15 0 29308 25m 1616 S 0.0 9.9 0:00.20
perl

3678 catw 15 0 29244 25m 1616 S 0.0 9.9 0:00.22 perl

Maybe the Catalyst wiki should have some profiling and performance oriented
info. That way people can have an idea on how to measure and what to expect
from their apps. And what to expect after using some of the most popular
modules (ie DBIC).

-rodrigo

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads