Mailing List Archive: Catalyst: Users

Running Catalyst apps with start_server

Index | Next | Previous | View Threaded

orasnita at gmail

Jan 21, 2012, 2:16 PM

Post #1 of 7 (473 views)
Permalink

Running Catalyst apps with start_server

Hi,

Is Server-Starter a good way for starting a Catalyst app?

I am asking this because I don't have success with using start_server with the --daemonize option.
I use Perl 5.14.2 under Ubuntu 11.

Here is the command I used:

start_server \
--port :5000 \
-- starman \
--workers 15 \
--max-requests 1000 \
--pid /srv/log/site.pid \
--access-log /srv/log/access.log \
--error-log /srv/log/error.log \
--daemonize \
/srv/BRK/brk.psgi

The command is not executed in the background, but it keeps printing the following errors:

new worker 7423 seems to have failed to start, exit status:256
starting new worker 9044
Pid_file already exists for running process (27684)... aborting
new worker 9044 seems to have failed to start, exit status:256
starting new worker 10428
Pid_file already exists for running process (27684)... aborting
new worker 10428 seems to have failed to start, exit status:256
....

So I need to break it. After breaking it, I can see that there are still some starman processes running, and their number seems to decrease, then increase again and so on, but the Starman web server doesn't answer to requests:

teddy 12843 0.0 0.0 0 0 pts/2 Z+ 23:05 0:00 [starman] <defunct>
teddy 14076 0.0 0.1 9820 5096 ? R 23:05 0:00 starman master --workers 15 --max-requests 1000 --pid /srv/log/site.pid --access-log /srv/log/access.log --error-log /srv/log/error.log --daemonize
teddy 14080 0.0 0.0 0 0 ? R 23:05 0:00 [starman master ]
teddy 14082 0.0 0.0 0 0 ? Z 23:05 0:00 [starman master ] <defunct>
teddy 14083 0.0 0.1 9820 5020 ? R 23:05 0:00 starman master --workers 15 --max-requests 1000 --pid /srv/log/site.pid --acce
ss-log /srv/log/access.log --error-log /srv/log/error.log --daemonize
...

On the other hand, if I use the same command, but without the --daemonize option, start_server seems to work fine and the web server can be also accessed with no problems. It prints:

start_server (pid:17145) starting now...
starting new worker 17146

But of course, it doesn't go in the background...
I can put it to run in the background if I add a " &" after the command line start_server ... and I've seen that if I do that it works and I can also add or remove workers with kill -TTIN|TTOU, but... is it a good way? Or it is better recommended to run starman directly?

Octavian

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

bobtfish at bobtfish

Jan 22, 2012, 3:59 AM

Post #2 of 7 (457 views)
Permalink

Re: Running Catalyst apps with start_server [In reply to]

On 21 Jan 2012, at 22:16, Octavian Rasnita wrote:
> But of course, it doesn't go in the background...
> I can put it to run in the background if I add a " &" after the
> command line start_server ... and I've seen that if I do that it
> works and I can also add or remove workers with kill -TTIN|TTOU,
> but... is it a good way? Or it is better recommended to run starman
> directly?

Any way you like is fine - as we're a ubuntu shop at work, we're
mostly deploying things just using upstart to start a (not
daemonizing) server - which works well for us.

You could alternatively try using CatalystX::Script::Server::Starman -
this should make your normal myapp_server be starman, and then try the
deamonize and pidfile options from the Catalyst script? Does that work
for you any better (And I'll be more willing help to debug that as
it's my code, and I don't use Server::Starter - sorry!)?

Cheers
t0m

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

orasnita at gmail

Jan 23, 2012, 1:34 PM

Post #3 of 7 (461 views)
Permalink

Re: Running Catalyst apps with start_server [In reply to]

Hi t0m,

From: "Tomas Doran" <bobtfish [at] bobtfish>
>
> On 21 Jan 2012, at 22:16, Octavian Rasnita wrote:
>> But of course, it doesn't go in the background...
>> I can put it to run in the background if I add a " &" after the
>> command line start_server ... and I've seen that if I do that it
>> works and I can also add or remove workers with kill -TTIN|TTOU,
>> but... is it a good way? Or it is better recommended to run starman
>> directly?
>
> Any way you like is fine - as we're a ubuntu shop at work, we're
> mostly deploying things just using upstart to start a (not
> daemonizing) server - which works well for us.

I intend to use my app under Ubuntu, so I'll probably also want to use upstart, but for the moment I am testing it and not with very good results.

> You could alternatively try using CatalystX::Script::Server::Starman -
> this should make your normal myapp_server be starman, and then try the
> deamonize and pidfile options from the Catalyst script? Does that work

I will try that, because it would be easier, but for the moment I have a problem with starman which seems to be leaking memory and I don't know how I can solve it.

I use Ubuntu 11 and Perl 5.14.2 and latest Catalyst and Starman and I am testing a simple MyApp Catalyst app created with:

catalyst.pl MyApp

I have compared Catalyst test server with starman and I am just following the used memory as it is reported by the top command and here are the results of my tests:

Before running the Catalyst test server:
340508k used memory

After running the Cat test server:
360992k

After the first request:
360992k

After 1000 more requests (with ab -n 1000 -c 1):
360992k

So Catalyst seems to be working fine, with no leaks.

After 1000 more requests (with ab -n 1000 -c 20):
361116k

After ~ 1 minute with no activity:
360868k

After killing Catalyst test server:
340532k

after ~ 2 minutes of inactivity:
340516k

...which is almost as before running the Catalyst test server.

After running again Catalyst test server:
360984k

After ~ 10000 requests (with ab -n 100000 -c 50):
361248k

After ~ 90000 requests:
361248k

After closing Catalyst test server:
341052k

After ~ 1 minute of inactivity:
340060k

So I can say that Catalyst and MyApp app don't leak.

After reboot, before running starman:
110192k

After running starman in a memory-friendly style, by preloading some modules, using:

starman \
--listen :5000 \
--workers 1 \
--max-requests 1000 \
--pid /srv/log/myapp.pid \
--access-log /srv/log/myapp_access.log \
--error-log /srv/log/myapp_error.log \
--daemonize \
-I /home/teddy/perl5/lib1/lib/perl5 \
-I /home/teddy/perl5/perlbrew/perls/perl-5.14.2/lib/5.14.2 \
-MMoose \
-Mnamespace::autoclean \
-MCatalyst \
-MCatalyst::Plugin::ConfigLoader \
-MCatalyst::Plugin::Static::Simple \
/srv/MyApp/myapp.psgi

144800k

After the first request:
145296k

After the second request:
145296k

After 1000 more requests (with ab -n 1000 -c 1):
146412k

After ~ 5 minutes of inactivity:
146412k

After another 1000 requests (with ab -n 1000 -c 20):
146784k

After ~ 5 minutes of inactivity:
146412k

After 10000 requests (with ab -n 100000 -c 50):
156188k

After 90000 requests:
221836k

After 100000 requests:
228572k

After ~ 15 minutes of inactivity:
227696k

After 15 more minutes of inactivity:
227696k

After one more hour of inactivity:
227944k

So it seems that there is a memory leak if the memory is not freed even after 1 hour.

I made a second test with Starman:

Before starman starts:
109944k

After starman started:
145252k

After the first request:
145392k

After 1000 requests (with ab -n 1000 -c 1):
146012k

After 1000 more requests:
146400k

After 100000 requests (with ab -n 100000 -c 1):
229864k

After 100000 more requests (with ab -n 100000 -c 50):
320128k

After 1000000 more requests (with ab -n 1000000 -c 100):
1129368k

After ~ 1 hour of inactivity:
1123904k

After 1000000 more requests (with ab -n 1000000 -c 100):
1933104k

After ~ 1 hour of inactivity:
1933096k

So something's obviously wrong if so much memory is occupied even after 1 hour of inactivity.

After closing starman with kill -quit `cat myapp.pid`
1905312k

After ~ 1 hour of inactivity:
1905560k

This is even stranger, because the memory is not free even though Starman was closed, and the single solution is to reboot.

And before closing Starman I have also tried to use kill -HUP `cat myapp.pid` to reload the workers but the memory didn't decrease.

I have also tried to run starman with 5 workers and and also tried without putting it to run in the background, but it still leaks.

Does this happends only to me and others can run starman without leaks?
Or at least is there a way of limiting the memory size and make Starman restart the workers that pass that limit?

I have also tried to use:

builder {
enable "Plack::Middleware::SizeLimit" => (
max_unshared_size_in_kb => 1024 * 4,
check_every_n_requests => 2
);
$app;
};

But it doesn't seem to have any effect.
I have read that "This middleware only works when the environment psgix.harakiri is set to a true value by the Plack server. If it's set to false, then this middleware simply does nothing." but I don't know if Starman sets it and supports it.

Thanks

Octavian

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

bobtfish at bobtfish

Jan 24, 2012, 4:24 AM

Post #4 of 7 (450 views)
Permalink

Re: Running Catalyst apps with start_server [In reply to]

On 23 Jan 2012, at 21:34, Octavian Rasnita wrote:
>
> So something's obviously wrong if so much memory is occupied even
> after 1 hour of inactivity.

To start with, you're testing entirely wrong.

Testing the free RAM on the machine is bullshit - as the kernel is
going to cache data for you, so the 'free' RAM figure means nothing.
>
> And before closing Starman I have also tried to use kill -HUP `cat
> myapp.pid` to reload the workers but the memory didn't decrease.

Why were you thinking it would?

The only figures really of note is the VSZ of each process. (And this
doesn't account for memory sharing).

> I have also tried to run starman with 5 workers and and also tried
> without putting it to run in the background, but it still leaks.

You haven't given any information from which we could conclude it leaks.

>
> Does this happends only to me and others can run starman without
> leaks?

It isn't leaking.

Your model of how memory works is broken.

What will (appear) to happen is that starman pre-loads all your bits
(lets say that's 20Mb for the sake of argument). It then forks, giving
you 5 workers... So you now have 6 x 20Mb (VSZ) - there is memory
sharing going on here, so you're not actually using that memory, but
lets ignore that...

Then you do a load of (the same) request, which generates a 1Mb output
document, but generating that document involves the user of 10Mb of RAM.

After 5 requests (one to each worker), you will now be (appearing to
be) using 20 + 5 * (20+10) Mb of RAM (combined VSZ).

Now, if you continue making the same request, memory useage should not
go up significantly (although as your workers process more requests,
they're more likely to become un-shared, so 'real' memory use in the
background goes up.. but again, let's ignore this).

You stop making requests... Nothing changes.. Perl _never_ gives RAM
back to the system, until it restarts. If you come back and do another
web request, the memory perl has internally free will be re-used, but
it won't be released back to the operating system.

If you now kill Starman, then the operating system _may_, at _it's
discression_ free up all the pages from which perl code was cached,
and it may not. Measuring the OS free memory is just wrong...

>
> 144800k
>
> After the first request:
> 145296k
>
> After the second request:
> 145296k
>
> After 1000 more requests (with ab -n 1000 -c 1):
> 146412k
>
> After ~ 5 minutes of inactivity:
> 146412k
>
> After another 1000 requests (with ab -n 1000 -c 20):
> 146784k
>
> After ~ 5 minutes of inactivity:
> 146412k
>
> After 10000 requests (with ab -n 100000 -c 50):
> 156188k
>
> After 90000 requests:
> 221836k
>
> After 100000 requests:
> 228572k
>
> After ~ 15 minutes of inactivity:
> 227696k
>
> After 15 more minutes of inactivity:
> 227696k
>
> After one more hour of inactivity:
> 227944k
>
> So it seems that there is a memory leak if the memory is not freed
> even after 1 hour.

No, this (the 'after 1 hour' thing) is not a leak - this is perl not
giving the OS memory back, by design. (And yes - you may have a tiny
leak in there somewhere due to the small continuing RAM increase per
request - although I'd be more likely to blame your app than Starman
for this)

This is why your generally arrange for workers to restart after N
requests, as if they serve a _massive_ page, then they won't give that
memory back ever...

So just set children to die after a few thousand requests and stop
worrying?

Cheers
t0m

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

orasnita at gmail

Jan 25, 2012, 5:41 AM

Post #5 of 7 (450 views)
Permalink

Re: Running Catalyst apps with start_server [In reply to]

Hi t0m,

From: "Tomas Doran" <bobtfish [at] bobtfish>
>
> On 23 Jan 2012, at 21:34, Octavian Rasnita wrote:
>>
>> So something's obviously wrong if so much memory is occupied even
>> after 1 hour of inactivity.
>
> To start with, you're testing entirely wrong.

Well, this is good news. :-)

> Testing the free RAM on the machine is bullshit - as the kernel is
> going to cache data for you, so the 'free' RAM figure means nothing.

I know that, but a problem I had was that when the "used" memory reported by top reached to be as big as the total memory, the system started to swap. This is why I was looking at it.

> The only figures really of note is the VSZ of each process. (And this
> doesn't account for memory sharing).

OK, I will make some tests to see if the size of that memory increases.

> What will (appear) to happen is that starman pre-loads all your bits
> (lets say that's 20Mb for the sake of argument). It then forks, giving
> you 5 workers... So you now have 6 x 20Mb (VSZ) - there is memory
> sharing going on here, so you're not actually using that memory, but
> lets ignore that...
>
> Then you do a load of (the same) request, which generates a 1Mb output
> document, but generating that document involves the user of 10Mb of RAM.
>
> After 5 requests (one to each worker), you will now be (appearing to
> be) using 20 + 5 * (20+10) Mb of RAM (combined VSZ).
>
> Now, if you continue making the same request, memory useage should not
> go up significantly (although as your workers process more requests,
> they're more likely to become un-shared, so 'real' memory use in the
> background goes up.. but again, let's ignore this).
>
> You stop making requests... Nothing changes.. Perl _never_ gives RAM
> back to the system, until it restarts. If you come back and do another
> web request, the memory perl has internally free will be re-used, but
> it won't be released back to the operating system.
>
> If you now kill Starman, then the operating system _may_, at _it's
> discression_ free up all the pages from which perl code was cached,
> and it may not. Measuring the OS free memory is just wrong...

Thanks for your explanations. It is helpful.
I was also thinking that Perl should reuse the memory has free internally, but I have seen that the "used" memory as reported by top continues to increase, so the free memory looked like it was not reused, but now I understand that this memory size is wrong.

I have made harder tests that should use the entire memory and make the system swap, but I've seen that the "used" memory reported by top continued to increase, but only up to a certain size that never reached the total memory size, so that increase looks to be false indeed.

> No, this (the 'after 1 hour' thing) is not a leak - this is perl not
> giving the OS memory back, by design. (And yes - you may have a tiny
> leak in there somewhere due to the small continuing RAM increase per
> request - although I'd be more likely to blame your app than Starman
> for this)

For doing those tests I used a simple Catalyst app made with catalyst.pl MyApp and nothing more.
And I also tested it with Catalyst's test server and it doesn't seems to be leaking.

> This is why your generally arrange for workers to restart after N
> requests, as if they serve a _massive_ page, then they won't give that
> memory back ever...
>
> So just set children to die after a few thousand requests and stop
> worrying?

Yep, I am doing that.

The test app wasn't leaking, but my real app might have some leaks.
I have used CatalystX::LeackChecker and I didn't find any leaks, but anyway, if the app has some leaks that I can't find, what do you suggest it is the solution for not consuming the entire memory?
Is closing and starting starman now and then the only solution?

Thanks.

Octavian

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

bobtfish at bobtfish

Jan 25, 2012, 6:59 AM

Post #6 of 7 (446 views)
Permalink

Re: Running Catalyst apps with start_server [In reply to]

On 25 Jan 2012, at 13:41, Octavian Rasnita wrote:

> Is closing and starting starman now and then the only solution?

No, and I didn't suggest that at any point.

You ask starman to restart child processes after N requests. This is
entirely different to restarting the entire Starman (as there is no
interruption in service - one worker quits after finishing handling
it's current request).

And memory leakage aside - this is the only possible way to maximise
memory sharing - as pages which were shared with the parent _will_ get
un-shared as you continue processing requests - there isn't anything
you can do about this, it's just how copy on write works...

(As perl is interpreted, your perl code lives in 'data' pages, rather
than executable pages - so you don't and can't get the same memory
sharing you can in C - where the code pages are always shared with
your child processes - as you're executing the same program...

Or, rather - you do get exactly the same semantics - the perl binary
itself, and any shared objects (.so files) you have loaded in the
parent will be shared with children forever - but this is generally a
small proportion of your memory use, compared to your perl code and
data structures, which all live in 'data' pages - meaning that your
program running causes static perl code in data pages to become
unshared..

Cheers
t0m

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

pagaltzis at gmx

Jan 26, 2012, 4:07 PM

Post #7 of 7 (447 views)
Permalink

Re: Running Catalyst apps with start_server [In reply to]

* Tomas Doran <bobtfish [at] bobtfish> [2012-01-25 16:05]:
> (As perl is interpreted, your perl code lives in 'data' pages, rather
> than executable pages - so you don't and can't get the same memory
> sharing you can in C - where the code pages are always shared with
> your child processes - as you're executing the same program...
>
> Or, rather - you do get exactly the same semantics - the perl binary
> itself, and any shared objects (.so files) you have loaded in the
> parent will be shared with children forever - but this is generally
> a small proportion of your memory use, compared to your perl code and
> data structures, which all live in 'data' pages - meaning that your
> program running causes static perl code in data pages to become
> unshared..

There is an accessible explanation of this here:
http://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html

In other notes, I have seen weblog posts (maybe from Reini Urban, but
I do not recall for certain) discuss the possibility of restructuring
the Perl op tree such that the op data that needs to be mutable is
stored separately in memory from that which never changes. That would
allow forked Perl programs to not inevitably slowly grow unshared over
time, just as C programs don’t. But I understood it to be a non-trivial
undertaking (much though a feasible one). Maybe someday perl will be
implemented that way.

Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>

_______________________________________________
List: Catalyst [at] lists
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst [at] lists/
Dev site: http://dev.catalyst.perl.org/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads