Mailing List Archive: Apache: Users

Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) !

1 2

View All

Index | Next | Previous | View Threaded

dbucherml at hsolutions

Aug 17, 2012, 4:46 PM

Post #1 of 26 (895 views)
Permalink

Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) !

Dear all,

That's an unbelievable issue but we have single apache
process that takes 5 GB of memory ! And it doesn't happens always with
the same URLs, it's unpredictable and we don't understand why it is
happening at all !!!

Any help would be greatly appreciated (as well as
the users of the website and the owners) !

We even developed for days
some software to analyze what is happening !

We do a pmap PID each
second when the problems comes up, but the offending line is
:

00002aad145c2000 2929376K rw--- [ anon ]

so what can we do with
that, is there a way to know what it is ?

We also analyze open files
with lsof -p.

I am NOW CONVINCED that it only happens when the
connection COMES FROM GOOGLE BOT (????)

Do you have suggestions on how
to analyze ?

Notes :

* The config is Linux+Apache+PHP+Postgresql
*
Of course PHP memory_limit was the first point we checked but this is
not the problem : grep memory_limit /etc/php.ini => memory_limit = 32M

* By the way we tried rlimitmem but either it doesn't work, either we
don't used it correctly, as it's no use : still using more than 5
gigabytes
* That unanswered thread looks similar to our problem :
http://serverfault.com/questions/161478/apache-process-consuming-all-memory-on-the-server

Thanks
a lot for any help and/or suggestion !

Denis

P.S.

> # httpd -V
>
Server version: Apache/2.2.3
> Server built: Jun 6 2012 10:00:42
>
Server's Module Magic Number: 20051115:3
> Server loaded: APR 1.2.7,
APR-Util 1.2.7
> Compiled using: APR 1.2.7, APR-Util 1.2.7
>
Architecture: 64-bit
> Server MPM: Prefork
> threaded: no
> forked: yes
(variable process count)
> Server compiled with....
> -D
APACHE_MPM_DIR="server/mpm/prefork"
> -D APR_HAS_SENDFILE
> -D
APR_HAS_MMAP
> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
> -D
APR_USE_SYSVSEM_SERIALIZE
> -D APR_USE_PTHREAD_SERIALIZE
> -D
SINGLE_LISTEN_UNSERIALIZED_ACCEPT
> -D APR_HAS_OTHER_CHILD
> -D
AP_HAVE_RELIABLE_PIPED_LOGS
> -D DYNAMIC_MODULE_LIMIT=128
> -D
HTTPD_ROOT="/etc/httpd"
> -D SUEXEC_BIN="/usr/sbin/suexec"
> -D
DEFAULT_PIDLOG="run/httpd.pid"
> -D
DEFAULT_SCOREBOARD="logs/apache_runtime_status"
> -D
DEFAULT_LOCKFILE="logs/accept.lock"
> -D
DEFAULT_ERRORLOG="logs/error_log"
> -D
AP_TYPES_CONFIG_FILE="conf/mime.types"
> -D
SERVER_CONFIG_FILE="conf/httpd.conf"

brett.maxfield at gmail

Aug 17, 2012, 8:55 PM

Post #2 of 26 (876 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

On 18/08/2012, at 6:46 AM, Denis BUCHER <dbucherml [at] hsolutions> wrote:

> Dear all,
>
> That's an unbelievable issue but we have single apache process that takes 5 GB of memory ! And it doesn't happens always with the same URLs, it's unpredictable and we don't understand why it is happening at all !!!
>
> Any help would be greatly appreciated (as well as the users of the website and the owners) !
>
> We even developed for days some software to analyze what is happening !
>
> We do a pmap PID each second when the problems comes up, but the offending line is :
>
> 00002aad145c2000 2929376K rw--- [ anon ]
>
> so what can we do with that, is there a way to know what it is ?
>
> We also analyze open files with lsof -p.
>
> I am now convinced that it only happens when the connection comes from Google Bot (????)
>
> Do you have suggestions on how to analyze ?
>
> Notes :
>
> The config is Linux+Apache+PHP+Postgresql
> Of course PHP memory_limit was the first point we checked but this is not the problem : grep memory_limit /etc/php.ini => memory_limit = 32M
> By the way we tried rlimitmem but either it doesn't work, either we don't used it correctly, as it's no use : still using more than 5 gigabytes
> That unanswered thread looks similar to our problem : http://serverfault.com/questions/161478/apache-process-consuming-all-memory-on-the-server
> Thanks a lot for any help and/or suggestion !
>
> Denis
>
Do you have a robots.txt ? Maybe you have some dynamic page that when spidered returns a very deep structure or some script that returns a temporary failure, causing a loop?

Look at the http log for googlebot, and see if there is any pattern to it. If you have the access logs it will tell you the script name at least.

If you find anything suspicious exclude it from spidering with robots.txt

If you don't maybe it is some sort of non-google bit attack, and you can block it with apache (as a hacker or snoop bot wont likely listen to robots.txt

In any case if you identify the path, maybe you can help identify / fix the bug..

Cheers
Brett

dbucherml at hsolutions

Aug 18, 2012, 1:04 AM

Post #3 of 26 (874 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

Le 18.08.2012 05:55, Brett Maxfield a écrit :

> On 18/08/2012, at
6:46 AM, Denis BUCHER wrote:
>
>> Dear all,
>>
>> That's an
unbelievable issue but we have single apache process that takes 5 GB of
memory ! And it doesn't happens always with the same URLs, it's
unpredictable and we don't understand why it is happening at all !!!
>>

>> Any help would be greatly appreciated (as well as the users of the
website and the owners) !
>>
>> We even developed for days some
software to analyze what is happening !
>>
>> We do a pmap PID each
second when the problems comes up, but the offending line is :
>>
>>
00002aad145c2000 2929376K rw--- [ anon ]
>>
>> so what can we do with
that, is there a way to know what it is ?
>>
>> We also analyze open
files with lsof -p.
>>
>> I am NOW CONVINCED that it only happens when
the connection COMES FROM GOOGLE BOT (????)
>>
>> Do you have
suggestions on how to analyze ?
>>
>> Notes :
>>
>> * The config is
Linux+Apache+PHP+Postgresql
>> * Of course PHP memory_limit was the
first point we checked but this is not the problem : grep memory_limit
/etc/php.ini => memory_limit = 32M
>> * By the way we tried rlimitmem
but either it doesn't work, either we don't used it correctly, as it's
no use : still using more than 5 gigabytes
>> * That unanswered thread
looks similar to our problem :
http://serverfault.com/questions/161478/apache-process-consuming-all-memory-on-the-server
[1]
>>
>> Thanks a lot for any help and/or suggestion !
>>
>>
Denis
>
> Do you have a robots.txt ? Maybe you have some dynamic page
that when spidered returns a very deep structure or some script that
returns a temporary failure, causing a loop?
>
> Look at the http log
for googlebot, and see if there is any pattern to it. If you have the
access logs it will tell you the script name at least.
> If you find
anything suspicious exclude it from spidering with robots.txt
> If you
don't maybe it is some sort of non-google bit attack, and you can block
it with apache (as a hacker or snoop bot wont likely listen to
robots.txt
> In any case if you identify the path, maybe you can help
identify / fix the bug..
> Cheers
> Brett

Dear Brett,

Thanks for
your reply, but there is no pattern at all, Googlebots just seems to
fetch "normal" pages on our webserver !

We have taken at least 150-200
hours analyzing logs and memory usage, and file usage, and the only
pattern we found was that the client is Googlebot !

And whatever page
it is, is there some explanation why apache can takes all the server
memory ? PHP is limited, so how is it possible for Apache to do that ?

Under which circumstances is Apache capable of using so much memory, I
mean why, where, how ?

Thanks a lot for any further help !

Denis

P. S. Our robots.txt file contains only two lines :

> User-agent:
*
> Disallow: /newsite_temp2011_old/

Links:
------
[1]
http://serverfault.com/questions/161478/apache-process-consuming-all-memory-on-the-server
[2]
mailto:dbucherml [at] hsolutions

alex at alex

Aug 18, 2012, 1:29 AM

Post #4 of 26 (869 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

--On 18 August 2012 10:04:12 +0200 Denis BUCHER <dbucherml [at] hsolutions>
wrote:

> And whatever page it is, is there some explanation why apache can takes
> all the server memory ? PHP is limited, so how is it possible for Apache
> to do that ?

Right, but it's taking lots of virtual memory, and that appears to
be larger than your physical memory. The pmap manpage is woefully
short but I think that's what it's saying. That itself isn't a problem.
Perhaps it's just mmap'ing a large file, for instance? Or allocating
3G of virtual memory with an anonymous mmap (which might be
what '[anon]' means). Assuming you are on a 64 bit server, you are
never likely to run out of virtual memory. So, is it writing to
that an actually using huge quantities of physical memory? IE
does RSIZE on 'top' grow huge? If not, I wouldn't worry. If so,
I'd guess something is allocating a large or several large objects,
as IIRC libc uses anonymous mmap rather than brk only for large
allocations. I wouldn't necessarily trust php_memory_limit. So,
either a memory leak in php, apache or the postgres client, or
you have a script sending back a huge reply, or a DB query with
a huge reply, or something else. What I'd do is wait until it
happens again and strace the pid concerned and see what it's
doing.

--
Alex Bligh

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

dbucherml at hsolutions

Aug 18, 2012, 2:01 AM

Post #5 of 26 (870 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

Le 18.08.2012 10:29, Alex Bligh a écrit :

> --On 18 August 2012
10:04:12 +0200 Denis BUCHER
> wrote:
>
>> And whatever page it is, is
there some explanation why apache can takes all the server memory ? PHP
is limited, so how is it possible for Apache to do that ?
>
> Right,
but it's taking lots of virtual memory, and that appears to
> be larger
than your physical memory.

Yes the process is using SWAP (it takes all
physical memory and all SWAP just for itself !)

> The pmap manpage is
woefully
> short but I think that's what it's saying. That itself isn't
a problem.
> Perhaps it's just mmap'ing a large file, for instance? Or
allocating
> 3G of virtual memory with an anonymous mmap (which might
be
> what '[anon]' means). Assuming you are on a 64 bit server, you
are
> never likely to run out of virtual memory.

I don't know what you
mean with "just mmap'ing a large file" ? I mean whatever it is, it's a
HUGE problem !

So, is it writing to
that an actually using huge
quantities of physical memory? IE
does RSIZE on 'top' grow huge?

How
can I see RSIZE in top ? I don't see it ?

But when it is happening,
%MEM is something like 99% in top

> : 142 total, 1 running, 140
sleeping, 1 stopped, 0 zombie
> Cpu(s): 1.6%us, 3.7%s
ni, 0.0%id,
94.3%wa, 0.0%hi, 0.5%si, 0.0%st
Mem: 4041804k total, 4020816k used,
20988k free, 368k buffers
Swap: 4192956k total, 857500k used, 3335456k
free, 36332k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
25178 apache 18 0 4109m 3.0g 2968 D 7.0 78.8 0:09.13 httpd
234
root 10 -5 0 0 0 D 3.0 0.0 203:22.33 kswapd0
1 root 15 0 10364 80 56 S
0.0 0.0 0:10.65 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0

3 root 34 19 0 0 0 S 0.0 0.0 0:00.05 ksoftirqd/0
4 root RT -5 0 0 0 S
0.0 0.0 0:00.00 watchdog/0
5 root RT -5 0 0 0 S 0.0 0.0 0:00.05
migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:00.12 ksoftirqd/1
7 root RT
-5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root 10 -5 0 0 0 S 0.0 0.0
0:00.07 events/0
9 root 10 -5 0 0 0 S 0.0 0.0 0:00.26 events/1
10 root
10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper

If not, I wouldn't worry.

???!!!! Maybe I forgot to tell you that the server is COMPLETELY DOWN
AND UNRESPONDING for two minutes every five minutes, nobody is able to
it, neither users, neither the employees and we can't even type a letter
in SSH during all this time !!! All webpages come with "connection
reset" or with no answ

> -left:#1010ff 2px solid; margin-left:5px;
w

If so,
I'd guess something is allocating a large or several large
objects,
as IIRC libc uses anonymous mmap rather than brk only for
large
allocations. I wouldn't necessarily trust php_memory_limit.
So,
either a memory leak in php, apache or the postgres client, or
you
have a script sending back a huge reply, or a DB query with
a huge
reply, or something else. What I'd do is wait until it
happens again and
strace the pid concerned and see what it's
doing.

Yes it's very ea

>
can I strace it ?
>
> I really don't know how to do it ?
>
> Thanks
a lot for your help, it's very nice !
>
> Denis

Links:
------
[1]
mailto:dbucherml [at] hsolutions

alex at alex

Aug 18, 2012, 2:05 AM

Post #6 of 26 (877 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

> How can I see RSIZE in top ? I don't see it ?

Sorry, that's 'RES' in your top display.

> If not, I wouldn't worry.

You need to worry, because it is.

> But how can I strace it ?

strace -fp [pid]

so "strace -fp 25178" in the case you gave.

and see what syscalls it's making. Might give you a hint.

--
Alex Bligh

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

dbucherml at hsolutions

Aug 18, 2012, 2:35 AM

Post #7 of 26 (875 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

Le 18.08.2012 11:05, Alex Bligh a écrit :

>> But how can I strace
it ?
>
> strace -fp [pid]
>
> so "strace -fp 25178" in the case you
gave.
>
> and see what syscalls it's making. Might give you a
hint.

OK, but my problem is that when I detect an "offending" process
(using too much memory), it's already too late !

The strace will only
show mmaps until "[crit] Memory allocation failed" !

Do you have a
suggestion on how I could "intercept" these processes before the problem
happens ?

Denis

alex at alex

Aug 18, 2012, 2:56 AM

Post #8 of 26 (871 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

--On 18 August 2012 11:35:29 +0200 Denis BUCHER <dbucherml [at] hsolutions>
wrote:

> OK, but my problem is that when I detect an "offending" process (using
> too much memory), it's already too late !
>
> The strace will only show mmaps until "[crit] Memory allocation failed" !
>
> Do you have a suggestion on how I could "intercept" these processes
> before the problem happens ?

rlimit maybe.

I'd suggest something is trying to allocate a 'negative' amount of memory.
The real answer is to locate the bug. If you really want to do that,
build/install debug versions of your s/w, attach gdb to the running
process, and ask for a backtrace.

--
Alex Bligh

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

dbucherml at hsolutions

Aug 18, 2012, 3:41 AM

Post #9 of 26 (879 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

Le 18.08.2012 11:56, Alex Bligh a écrit :

> --On 18 August 2012
11:35:29 +0200 Denis BUCHER
> wrote:
>
>> OK, but my problem is that
when I detect an "offending" process (using too much memory), it's
already too late ! The strace will only show mmaps until "[crit] Memory
allocation failed" ! Do you have a suggestion on how I could "intercept"
these processes before the problem happens ?
>
> rlimit maybe.
>
> I'd
suggest something is trying to allocate a 'negative' amount of memory.
>
The real answer is to locate the bug. If you really want to do that,
>
build/install debug versions of your s/w, attach gdb to the running
>
process, and ask for a backtrace.

Mmmm I don't know how to to all
that...

I just send you the end of strace, just by chance, maybe you
can see something interesting ?

> mmap(NULL, 1048576,
PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x2b92e9929000
> brk(0x2b91bbe72000) = 0x2b91bbe16000
> mmap(NULL,
1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x2b92e9a29000
> brk(0x2b91bbe72000) = 0x2b91bbe16000
> mmap(NULL,
1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x2b92e9b29000
> brk(0x2b91bbe72000) = 0x2b91bbe16000
> mmap(NULL,
1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x2b92e9c29000
> brk(0x2b91bbe72000) = 0x2b91bbe16000
> mmap(NULL,
1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x2b92e9d29000
> brk(0x2b91bbe72000) = 0x2b91bbe16000
> mmap(NULL,
1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1
ENOMEM (Cannot allocate memory)
> mmap(NULL, 134217728, PROT_NONE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x2b92e9f29000
>
munmap(0x2b92e9f29000, 34435072) = 0
> munmap(0x2b92f0000000, 32673792)
= 0
> mprotect(0x2b92ec000000, 376832, PROT_READ|PROT_WRITE) = -1 ENOMEM
(Cannot allocate memory)
> munmap(0x2b92ec000000, 67108864) = 0
>
mmap(NULL, 134217728, PROT_NONE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x2b92e9e29000
>
munmap(0x2b92e9e29000, 35483648) = 0
> munmap(0x2b92f0000000, 31625216)
= 0
> mprotect(0x2b92ec000000, 135168, PROT_READ|PROT_WRITE) = -1 ENOMEM
(Cannot allocate memory)
> munmap(0x2b92ec000000, 67108864) = 0
>
write(2, "[crit] Memory allocation failed,"..., 51) = 51
>
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
> tgkill(17282, 17282,
SIGABRT) = 0
> --- SIGABRT (Aborted) @ 0 (0) ---
> chdir("/etc/httpd") =
0
> rt_sigaction(SIGABRT, {SIG_DFL, [], SA_RESTORER|SA_INTERRUPT,
0x2b91b7f33be0}, {SIG_DFL, [], SA_RESTORER|SA_RESETHAND,
0x2b91b7f33be0}, 8) = 0
> kill(17282, SIGABRT) = 0
>
rt_sigreturn(0x4382) = 0
> --- SIGABRT (Aborted) @ 0 (0) ---
> Process
17282 detached

Links:
------
[1] mailto:dbucherml [at] hsolutions

dbucherml at hsolutions

Aug 18, 2012, 3:43 AM

Post #10 of 26 (875 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

dbucherml at hsolutions

Aug 18, 2012, 4:06 AM

Post #11 of 26 (871 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

dbucherml at hsolutions

Aug 18, 2012, 4:18 AM

Post #12 of 26 (870 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

Le 18.08.2012 11:56, Alex Bligh a écrit :

> --On 18 August 2012
11:35:29 +0200 Denis BUCHER
> wrote:
>
>> OK, but my problem is that
when I detect an "offending" process (using too much memory), it's
already too late ! The strace will only show mmaps until "[crit] Memory
allocation failed" ! Do you have a suggestion on how I could "intercept"
these processes before the problem happens ?
>
> rlimit maybe.
>
> I'd
suggest something is trying to allocate a 'negative' amount of memory.
>
The real answer is to locate the bug. If you really want to do that,
>
build/install debug versions of your s/w, attach gdb to the running
>
process, and ask for a backtrace.

And I think I was able to capture the
strace :

> accept(4, {sa_family=AF_INET6, sin6_port=htons(42635),
inet_pton(AF_INET6, "::ffff:66.249.66.5", &sin6_addr), sin6_flowinfo=0,
sin6_scope_id=0}, [68719476764]) = 18
> semop(7438351, 0x2b91b7f24da6,
1) = 0
> getsockname(18, {sa_family=AF_INET6, sin6_port=htons(80),
inet_pton(AF_INET6, "::ffff:213.162.5.179", &sin6_addr),
sin6_flowinfo=0, sin6_scope_id=0}, [68719476764]) = 0
> fcntl(18,
F_GETFL) = 0x2 (flags O_RDWR)
> fcntl(18, F_SETFL, O_RDWR|O_NONBLOCK) =
0
> read(18, "GET /search/Delight%20combinaiso"..., 8000) = 280
>
stat("/var/www/vhosts/ourserver.ch/www/http/search/Delight
combinaison/metas-12-00088/1/de", 0x7fff2c6fe8c0) = -1 ENOENT (No such
file or directory)
> lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096,
...}) = 0
> lstat("/var/www", {st_mode=S_IFDIR|0755, st_size=4096, ...})
= 0
> lstat("/var/www/vhosts", {st_mode=S_IFDIR|0755, st_size=4096,
...}) = 0
> lstat("/var/www/vhosts/ourserver.ch",
{st_mode=S_IFDIR|S_ISGID|0755, st_size=4096, ...}) = 0
>
lstat("/var/www/vhosts/ourserver.ch/www", {st_mode=S_IFDIR|S_ISGID|0755,
st_size=4096, ...}) = 0
> lstat("/var/www/vhosts/ourserver.ch/www/http",
{st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
>
open("/var/www/vhosts/ourserver.ch/www/http/.htaccess", O_RDONLY) = 19
>
fstat(19, {st_mode=S_IFREG|0440, st_size=7995, ...}) = 0
> read(19,
"nn##############################"..., 4096) = 4096
> read(19,
"t_bw=$4&inc=$6&slang=$8 [P]nRewr"..., 4096) = 3899
> read(19, "", 4096)
= 0
> close(19) = 0
>
lstat("/var/www/vhosts/ourserver.ch/www/http/search", 0x7fff2c6fe8c0) =
-1 ENOENT (No such file or directory)
> ) = 0
> epoll_wait(17,
brk(0x2b91bbe3c000) = 0x2b91bbe16000
>
> Then, it only does mmap/brk
until "Cannot allocate memory"

Links:
------
[1]
mailto:dbucherml [at] hsolutions

alex at alex

Aug 18, 2012, 4:38 AM

Post #13 of 26 (862 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

--On 18 August 2012 13:18:11 +0200 Denis BUCHER <dbucherml [at] hsolutions>
wrote:

> And I think I was able to capture the strace :

If you use strace with (say) "-s2048" as a parameter you'd be able to
see the whole URL here:

> read(18, "GET /search/Delight%20combinaiso"..., 8000) = 280

and you might be able to replicate at will.

--
Alex Bligh

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

icicimov at gmail

Aug 18, 2012, 4:40 AM

Post #14 of 26 (861 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

What does your mpm_prefork config look like? Since you use prefork you
shoud have more than one process running for sure. And also check the
kernel settings with ulimit command.
Any errors on the database site or the applications? Anything in the apache
logs?
On Aug 18, 2012 9:46 AM, "Denis BUCHER" <dbucherml [at] hsolutions> wrote:

> **
>
> Dear all,
>
> That's an unbelievable issue but we have single apache process that takes
> 5 GB of memory ! And it doesn't happens always with the same URLs, it's
> unpredictable and we don't understand why it is happening at all !!!
>
> Any help would be greatly appreciated (as well as the users of the website
> and the owners) !
>
> We even developed for days some software to analyze what is happening !
>
> We do a pmap PID each second when the problems comes up, but the offending
> line is :
>
> 00002aad145c2000 2929376K rw--- [ anon ]
>
> so what can we do with that, is there a way to know what it is ?
>
> We also analyze open files with lsof -p.
>
> I am *now convinced* that it only happens when the connection* comes from
> Google Bot* (????)
>
> Do you have suggestions on how to analyze ?
>
> Notes :
>
> - The config is Linux+Apache+PHP+Postgresql
> - Of course PHP memory_limit was the first point we checked but this
> is not the problem : grep memory_limit /etc/php.ini => memory_limit = 32M
> - By the way we tried rlimitmem but either it doesn't work, either we
> don't used it correctly, as it's no use : still using more than 5 gigabytes
> - That unanswered thread looks similar to our problem :
> http://serverfault.com/questions/161478/apache-process-consuming-all-memory-on-the-server
>
> Thanks a lot for any help and/or suggestion !
>
> Denis
>
> P.S.
>
> # httpd -V
> Server version: Apache/2.2.3
> Server built: Jun 6 2012 10:00:42
> Server's Module Magic Number: 20051115:3
> Server loaded: APR 1.2.7, APR-Util 1.2.7
> Compiled using: APR 1.2.7, APR-Util 1.2.7
> Architecture: 64-bit
> Server MPM: Prefork
> threaded: no
> forked: yes (variable process count)
> Server compiled with....
> -D APACHE_MPM_DIR="server/mpm/prefork"
> -D APR_HAS_SENDFILE
> -D APR_HAS_MMAP
> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
> -D APR_USE_SYSVSEM_SERIALIZE
> -D APR_USE_PTHREAD_SERIALIZE
> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
> -D APR_HAS_OTHER_CHILD
> -D AP_HAVE_RELIABLE_PIPED_LOGS
> -D DYNAMIC_MODULE_LIMIT=128
> -D HTTPD_ROOT="/etc/httpd"
> -D SUEXEC_BIN="/usr/sbin/suexec"
> -D DEFAULT_PIDLOG="run/httpd.pid"
> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
> -D DEFAULT_LOCKFILE="logs/accept.lock"
> -D DEFAULT_ERRORLOG="logs/error_log"
> -D AP_TYPES_CONFIG_FILE="conf/mime.types"
> -D SERVER_CONFIG_FILE="conf/httpd.conf"
>
>
>
>
>
>
>

icicimov at gmail

Aug 19, 2012, 5:26 AM

Post #15 of 26 (844 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

Are you saying you are seeing only one apache process? Since you are using
mpm_prefork there shoul be more than one for sure. Can you show us your
prefork settings? Any errors maybe in the database or the php application?
Anything in the apache logs?
On Aug 18, 2012 7:01 PM, "Denis BUCHER" <dbucherml [at] hsolutions> wrote:

> **
>
> Le 18.08.2012 10:29, Alex Bligh a �crit :
>
> --On 18 August 2012 10:04:12 +0200 Denis BUCHER <dbucherml [at] hsolutions>
> wrote:
>
> And whatever page it is, is there some explanation why apache can takes
> all the server memory ? PHP is limited, so how is it possible for Apache to
> do that ?
>
> Right, but it's taking lots of virtual memory, and that appears to
> be larger than your physical memory.
>
> Yes the process is using SWAP (it takes all physical memory and all SWAP
> just for itself !)
>
> The pmap manpage is woefully
> short but I think that's what it's saying. That itself isn't a problem.
> Perhaps it's just mmap'ing a large file, for instance? Or allocating
> 3G of virtual memory with an anonymous mmap (which might be
> what '[anon]' means). Assuming you are on a 64 bit server, you are
> never likely to run out of virtual memory.
>
> I don't know what you mean with "just mmap'ing a large file" ? I mean
> whatever it is, it's a HUGE problem !
>
> So, is it writing to
> that an actually using huge quantities of physical memory? IE
> does RSIZE on 'top' grow huge?
>
> How can I see RSIZE in top ? I don't see it ?
>
> But when it is happening, %MEM is something like 99% in top :
>
> top - 10:57:33 up 51 days, 16:17, 2 users, load average: 1.29, 0.34, 0.16
> Tasks: 142 total, 1 running, 140 sleeping, 1 stopped, 0 zombie
> Cpu(s): 1.6%us, 3.7%sy, 0.0%ni, 0.0%id, 94.3%wa, 0.0%hi, 0.5%si, 0.0%st
> Mem: 4041804k total, 4020816k used, 20988k free, 368k buffers
> Swap: 4192956k total, 857500k used, 3335456k free, 36332k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 25178 apache 18 0 4109m 3.0g 2968 D 7.0 78.8 0:09.13 httpd
> 234 root 10 -5 0 0 0 D 3.0 0.0 203:22.33 kswapd0
> 1 root 15 0 10364 80 56 S 0.0 0.0 0:10.65 init
> 2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
> 3 root 34 19 0 0 0 S 0.0 0.0 0:00.05 ksoftirqd/0
> 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
> 5 root RT -5 0 0 0 S 0.0 0.0 0:00.05 migration/1
> 6 root 34 19 0 0 0 S 0.0 0.0 0:00.12 ksoftirqd/1
> 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
> 8 root 10 -5 0 0 0 S 0.0 0.0 0:00.07 events/0
> 9 root 10 -5 0 0 0 S 0.0 0.0 0:00.26 events/1
> 10 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
>
>
> If not, I wouldn't worry.
>
> ???!!!! Maybe I forgot to tell you that the server is COMPLETELY DOWN
> AND UNRESPONDING for two minutes every five minutes, nobody is able to it,
> neither users, neither the employees and we can't even type a letter in SSH
> during all this time !!! All webpages come with "connection reset" or with
> no answer at all... In fact, I am _extremely_ worried !
>
> If so,
> I'd guess something is allocating a large or several large objects,
> as IIRC libc uses anonymous mmap rather than brk only for large
> allocations. I wouldn't necessarily trust php_memory_limit. So,
> either a memory leak in php, apache or the postgres client, or
> you have a script sending back a huge reply, or a DB query with
> a huge reply, or something else. What I'd do is wait until it
> happens again and strace the pid concerned and see what it's
> doing.
>
> Yes it's very easy to make it happen again, I just need to remove Google
> Bots IPs from firewall DROP list ;-)
>
> But how can I strace it ?
>
> I really don't know how to do it ?
>
> Thanks a lot for your help, it's very nice !
>
> Denis
>

dbucherml at hsolutions

Aug 19, 2012, 8:58 AM

Post #16 of 26 (829 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

Dear Igor,

In fact I see plenty of apache processes, but only ONE
takes out the whole memory ! All others are OK.

And the most
surprising is that it only happens with Google bot and other search
engines crawler but never with users ???
Denis

Le 19.08.2012 14:26,
Igor Cicimov a écrit :

> Are you saying you are seeing only one apache
process? Since you are using mpm_prefork there shoul be more than one
for sure. Can you show us your prefork settings? Any errors maybe in the
database or the php application? Anything in the apache logs?
> On Aug
18, 2012 7:01 PM, "Denis BUCHER" wrote:
>
>> Le 18.08.2012 10:29, Alex
Bligh a écrit :
>>
>>> --On 18 August 2012 10:04:12 +0200 Denis BUCHER

>>> wrote:
>>>
>>>> And whatever page it is, is there some explanation
why apache can takes all the server memory ? PHP is limited, so how is
it possible for Apache to do that ?
>>>
>>> Right, but it's taking lots
of virtual memory, and that appears to
>>> be larger than your physical
memory.
>>
>> Yes the process is using SWAP (it takes all physical
memory and all SWAP just for itself !)
>>
>>> The pmap manpage is
woefully
>>> short but I think that's what it's saying. That itself
isn't a problem.
>>> Perhaps it's just mmap'ing a large file, for
instance? Or allocating
>>> 3G of virtual memory with an anonymous mmap
(which might be
>>> what '[anon]' means). Assuming you are on a 64 bit
server, you are
>>> never likely to run out of virtual memory.
>>
>> I
don't know what you mean with "just mmap'ing a large file" ? I mean
whatever it is, it's a HUGE problem !
>>
>> So, is it writing to
>>
that an actually using huge quantities of physical memory? IE
>> does
RSIZE on 'top' grow huge?
>>
>> How can I see RSIZE in top ? I don't
see it ?
>>
>>> ys, 16:17, 2 users, load average: 1.29, 0.34, 0.16
>>>
Tasks: 142 total, 1 running, 140 sleeping, &nbs
>> 0 zombie
>> Cpu(s):
1.6%us, 3.7%sy, 0.0%ni, 0.0%id, 94.3%wa, 0.0%hi, 0.5%si, 0.0%st
>> Mem:
4041804k total, 4020816k used, 20988k free, 368k buffers
>> Swap:
4192956k total, 857500k used, 3335456k free, 36332k cached
>>
>> PID
USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 25178 apache 18 0
4109m 3.0g 2968 D 7.0 78.8 0:09.13 httpd
>> 234 root 10 -5 0 0 0 D 3.0
0.0 203:22.33 kswapd0
>> 1 root 15 0 10364 80 56 S 0.0 0.0 0:10.65
init
>> 2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
>> 3 root 34 19
0 0 0 S 0.0 0.0 0:00.05 ksoftirqd/0
>> 4 root RT -5 0 0 0 S 0.0 0.0
0:00.00 watchdog/0
>> 5 root RT -5 0 0 0 S 0.0 0.0 0:00.05
migration/1
>> 6 root 34 19 0 0 0 S 0.0 0.0 0:00.12 ksoftirqd/1
>> 7
root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
>> 8 root 10 -5 0 0 0 S
0.0 0.0 0:00.07 events/0
>> 9 root 10 -5 0 0 0 S 0.0 0.0 0:00.26
events/1
>> 10 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
>>
>> If not,
I wouldn't worry.
>>
>> ???!!!! Maybe I forgot to tell you that the
server is COMPLETELY DOWN AND UNRESPONDING for two minutes every five
minutes, nobody is able to it, neither users, neither the employees and
we can't even type a letter in SSH during al
>>
>>> remely_ worried
!
>> 5px; border-left: #1010ff 2px solid; margin-left: 5px; width:
100%;">
>>
>> If so,
>> I'd guess something is allocating a large or
several large objects,
>> as IIRC libc uses anonymous mmap rather than
brk only for large
>> allocations. I wouldn't necessarily trust
php_memory_limit. So,
>> either a memory leak in php, apache or the
postgres client, or
>> you have a script sending back a huge reply, or a
DB query with
>> a huge reply, or something else. What I'd do is wait
until it
>> happens aga> to make it happen again, I just need to remove
Google Bots IPs from firewall DROP list ;-)
>>>
>>> But how can I
strace it ?
>>>
>>> I really don't know how to do it ?
>>>
>>>
Thanks a lot for your help, it's very nice !
>>>
>>> Denis

Links:
------
[1] mailto:dbucherml [at] hsolutions
[2]
mailto:dbucherml [at] hsolutions

alex at alex

Aug 19, 2012, 9:17 AM

Post #17 of 26 (830 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

--On 19 August 2012 17:58:34 +0200 Denis BUCHER <dbucherml [at] hsolutions>
wrote:

> And the most surprising is that it only happens with Google bot and other
> search engines crawler but never with users ???

I would bet it queries a URL that users do not.

--
Alex Bligh

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

dbucherml at hsolutions

Aug 19, 2012, 10:29 AM

Post #18 of 26 (832 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

Le 19.08.2012 18:17, Alex Bligh a écrit :

> --On 19 August 2012
17:58:34 +0200 Denis BUCHER
> wrote:
>
>> And the most surprising is
that it only happens with Google bot and other search engines crawler
but never with users ???
>
> I would bet it queries a URL that users do
not.

That was my first guess, but sadly no... If I query the same URL
from Firefox, no problem... I even tried with wget and w3m, only Google
(and some other crawlers) creates this problem.
Denis

Links:
------
[1] mailto:dbucherml [at] hsolutions

yehuda at ymkatz

Aug 19, 2012, 11:57 AM

Post #19 of 26 (827 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

On Sun, Aug 19, 2012 at 1:29 PM, Denis BUCHER <dbucherml [at] hsolutions>wrote:

> **
>
> Le 19.08.2012 18:17, Alex Bligh a �crit :
>
> --On 19 August 2012 17:58:34 +0200 Denis BUCHER <dbucherml [at] hsolutions>
> wrote:
>
> And the most surprising is that it only happens with Google bot and other
> search engines crawler but never with users ???
>
> I would bet it queries a URL that users do not.
>
> That was my first guess, but sadly no... If I query the same URL from
> Firefox, no problem... I even tried with wget and w3m, only Google (and
> some other crawlers) creates this problem.
> Denis
>
What happens when you change the user agent in Firefox to match GoogleBot?
(You can use this plugin:
https://addons.mozilla.org/en-US/firefox/addon/user-agent-switcher/?src=search
)

dbucherml at hsolutions

Aug 19, 2012, 12:27 PM

Post #20 of 26 (833 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

Le 19.08.2012 20:57, Yehuda Katz a écrit :

> On Sun, Aug 19, 2012
at 1:29 PM, Denis BUCHER wrote:
>
>> Le 19.08.2012 18:17, Alex Bligh a
écrit :
>>
>>> --On 19 August 2012 17:58:34 +0200 Denis BUCHER
>>>
wrote:
>>>
>>>> And the most surprising is that it only happens with
Google bot and other search engines crawler but never with users ???
>>>

>>> I would bet it queries a URL that users do not.
>>
>> That was my
first guess, but sadly no... If I query the same URL from Firefox, no
problem... I even tried with wget and w3m, only Google (and some other
crawlers) creates this problem.
>> Denis
> What happens when you change
the user agent in Firefox to match GoogleBot?
> (You can use this
plugin:
https://addons.mozilla.org/en-US/firefox/addon/user-agent-switcher/?src=search
[3])

Good idea ! I tried, but changing the User agent to Google Bot
doesn't seem to create/reproduce the problem...

Denis

Links:
------
[1] mailto:dbucherml [at] hsolutions
[2]
mailto:dbucherml [at] hsolutions
[3]
https://addons.mozilla.org/en-US/firefox/addon/user-agent-switcher/?src=search

yehuda at ymkatz

Aug 19, 2012, 12:34 PM

Post #21 of 26 (834 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

On Sun, Aug 19, 2012 at 3:27 PM, Denis BUCHER <dbucherml [at] hsolutions>wrote:

> **
>
> And the most surprising is that it only happens with Google bot and other
>> search engines crawler but never with users ???
>>
>> Good idea ! I tried, but changing the User agent to Google Bot doesn't
> seem to create/reproduce the problem...
>
One more thought, since you said you can't really get strace, to see what
the difference with a GoogleBot request. Can you do a packet capture (using
wireshark or similar) of one request to a URL from Firefox (limit wireshark
to capture from just your client IP).
Capture a similar request limited to GoogleBot IPs and see if there are any
major differences. (If you send me/the list the pcap files, someone can
probably help you check)

douglist at macnauchtan

Aug 19, 2012, 12:35 PM

Post #22 of 26 (832 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

Google, and perhaps others, access index.html with some options identified by a letter. I don't understand it fully but it's possible to look for filenames that are indexed but perhaps are not actually addressed in any page intended for public distribution.

--

--> From the U S of A, the only socialist country that refuses to admit it. <--

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

icicimov at gmail

Aug 19, 2012, 3:58 PM

Post #23 of 26 (825 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

Mind sharing the proxy rules? From your strace looks like you get file not
found error for the bot urls so might be something related. Also maybe more
detailed description of your setup might help like how many servers and
apps you are running is the database on the same box is the same apache a
proxy as well and any other third party stuff you might be using.
On Aug 18, 2012 9:07 PM, "Denis BUCHER" <dbucherml [at] hsolutions> wrote:

> **
>
> Le 18.08.2012 11:56, Alex Bligh a �crit :
>
>
> --On 18 August 2012 11:35:29 +0200 Denis BUCHER <dbucherml [at] hsolutions>
> wrote:
>
> OK, but my problem is that when I detect an "offending" process (using too
> much memory), it's already too late ! The strace will only show mmaps until
> "[crit] Memory allocation failed" ! Do you have a suggestion on how I could
> "intercept" these processes before the problem happens ?
>
> rlimit maybe.
>
> I'd suggest something is trying to allocate a 'negative' amount of memory.
> The real answer is to locate the bug. If you really want to do that,
> build/install debug versions of your s/w, attach gdb to the running
> process, and ask for a backtrace.
>
> Update :
>
> Just to let you know, we use proxying to serve http://requests/like/thatto customers while internally getting
> http://requests?like=that if you see what I mean
>
>
> Could it be the source of a problem ?
>
> Denis
>
>
>
>

jdmls at yahoo

Aug 20, 2012, 2:10 AM

Post #24 of 26 (810 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

From: Denis BUCHER <dbucherml [at] hsolutions>

># httpd -V
>>Server version: Apache/2.2.3

Tried with a more recent version of apache...?
There a re quite a few memory issues fixed since 2.2.3:

http://httpd.apache.org/security/vulnerabilities_22.html

JD

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe [at] httpd
For additional commands, e-mail: users-help [at] httpd

icicimov at gmail

Aug 20, 2012, 2:59 AM

Post #25 of 26 (808 views)
Permalink

Re: Unbelievable : one single apache process uses more than whole server memory (5 gigabytes) ! [In reply to]

On Aug 20, 2012 7:10 PM, "John Doe" <jdmls [at] yahoo> wrote:
>
> From: Denis BUCHER <dbucherml [at] hsolutions>
>
> ># httpd -V
> >>Server version: Apache/2.2.3
>
>
> Tried with a more recent version of apache...?

+1

> There a re quite a few memory issues fixed since 2.2.3:
>
> http://httpd.apache.org/security/vulnerabilities_22.html
>
>
> JD
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe [at] httpd
> For additional commands, e-mail: users-help [at] httpd
>

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads