Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: DBMail: users

FYI: how to prevent mysql from oom-killer

 

 

DBMail users RSS feed   Index | Next | Previous | View Threaded


h.reindl at thelounge

Apr 13, 2012, 5:17 AM

Post #1 of 4 (651 views)
Permalink
FYI: how to prevent mysql from oom-killer

the following may be useful for most server systems

OOM-killer acts if some process reclaims more and more
memory and the kernel randomly kills unimportant tasks
using hughe memory

in case of a running mysqld the classification "unimportant"
is nearly all time wrong and can cause hughe damage and work
in other words: you really never want killed a database server
randomly instead "dbmail-imapd" which can be restarted via
systemd without pain and may be the root-cause of OOM
_________________________________________

with one single command you can protect processes from get killed
i started to run this every 15 minutes to make sure it is also
active after restarts

i am considering include this in "mysqld.service" as
"ExecStartPost=-/usr/local/bin/mysql-no-oom.sh" in our
internal mysqld-packages and include also the script
_________________________________________

[root [at] mai:~]$ cat /etc/crontab | grep oom
0,15,30,45 * * * * root bash /usr/local/bin/mysql-no-oom.sh

[root [at] mai:~]$ cat /usr/local/bin/mysql-no-oom.sh
#!/bin/bash
pgrep -f "/usr/libexec/mysqld" | while read PID; do echo -1000 > /proc/$PID/oom_score_adj; done
Attachments: signature.asc (0.26 KB)


h.reindl at thelounge

Apr 13, 2012, 7:09 AM

Post #2 of 4 (626 views)
Permalink
Re: FYI: how to prevent mysql from oom-killer [In reply to]

Am 13.04.2012 15:47, schrieb Mihamina Rakotomandimby:
> On 04/13/2012 03:17 PM, Reindl Harald wrote:
>> with one single command you can protect processes from get killed
>> i started to run this every 15 minutes to make sure it is also
>> active after restarts
>
> I understand your issue, but isn'there a configuration way to just limit the memory usage of MySQL?

no, you did not understand the issue nor my intention
intention was not "i have a problem and needs help"
intention was "if someone may have a problem this is the help"

the problem is that ANOTHER process or many of them triggering
a OOM situation and after that the kernel killing randomly
processes to prevent from a system crash

the proc-setting is to prevent that this randomly process
is mysqld instead the process which is eating up your memory
_____________________

as real world example (i was there)

* machine has 9 GB RAM
* machine is a dbmail-server
* mysqld has assigend buffer-pools and so on up to 7 GB
* someone tryies to copy a 2 GB attachment in drafts folder
* the memory usage of imapd explodes
* the first times OOM killer is killing imapd
* at least the OOM is killing mysqld

you want NOT killed mysqld in this situation
you want not killed mysqld damaging tables becuase you are
running out of memory because too much httpd-processes
eating up your memory

at least - you never want to get randomly killed mysqld

pgrep -f "/usr/libexec/mysqld" | while read PID; do echo -1000 > /proc/$PID/oom_score_adj; done

is preventing you from this happening
maybe you want prevent such a situation before it happend

i searched how can i proctect our servers, found a solution and
thought it is a good idea to publish it on 3 mailing-lists
where many users could need it
Attachments: signature.asc (0.26 KB)


skraps at hushmail

Apr 13, 2012, 10:55 AM

Post #3 of 4 (628 views)
Permalink
Re: FYI: how to prevent mysql from oom-killer [In reply to]

What is hughe? Is that a common word in your area or region? Or a
typo? Just wondering. No harm meant.

On 04/13/2012 at 10:10 AM, Reindl Harald wrote:Am 13.04.2012 15:47,
schrieb Mihamina Rakotomandimby:
> On 04/13/2012 03:17 PM, Reindl Harald wrote:
>> with one single command you can protect processes from get killed
>> i started to run this every 15 minutes to make sure it is also
>> active after restarts
>
> I understand your issue, but isn'there a configuration way to just
limit the memory usage of MySQL?

no, you did not understand the issue nor my intention
intention was not "i have a problem and needs help"
intention was "if someone may have a problem this is the help"

the problem is that ANOTHER process or many of them triggering
a OOM situation and after that the kernel killing randomly
processes to prevent from a system crash

the proc-setting is to prevent that this randomly process
is mysqld instead the process which is eating up your memory
_____________________

as real world example (i was there)

* machine has 9 GB RAM
* machine is a dbmail-server
* mysqld has assigend buffer-pools and so on up to 7 GB
* someone tryies to copy a 2 GB attachment in drafts folder
* the memory usage of imapd explodes
* the first times OOM killer is killing imapd
* at least the OOM is killing mysqld

you want NOT killed mysqld in this situation
you want not killed mysqld damaging tables becuase you are
running out of memory because too much httpd-processes
eating up your memory

at least - you never want to get randomly killed mysqld

pgrep -f "/usr/libexec/mysqld" | while read PID; do echo -1000 >
/proc/$PID/oom_score_adj; done

is preventing you from this happening
maybe you want prevent such a situation before it happend

i searched how can i proctect our servers, found a solution and
thought it is a good idea to publish it on 3 mailing-lists
where many users could need it


h.reindl at thelounge

Apr 13, 2012, 11:03 AM

Post #4 of 4 (625 views)
Permalink
Re: FYI: how to prevent mysql from oom-killer [In reply to]

the "h" was indeed a typo

some links to show the meaning of "huge" in the IT
you can call it also "really big"

http://linuxgazette.net/155/krishnakumar.html
http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaattunhp.htm
http://www.nicolaskuttler.com/post/filesystem-with-huge-files-cannot-be-mounted-read-write-without-config_lbdaf/

Am 13.04.2012 19:55, schrieb skraps [at] hushmail:
> What is hughe? Is that a common word in your area or region? Or a typo? Just wondering. No harm meant.
>
> On 04/13/2012 at 10:10 AM, Reindl Harald <h.reindl [at] thelounge> wrote:
>
> Am 13.04.2012 15:47, schrieb Mihamina Rakotomandimby:
> > On 04/13/2012 03:17 PM, Reindl Harald wrote:
> >> with one single command you can protect processes from get killed
> >> i started to run this every 15 minutes to make sure it is also
> >> active after restarts
> >
> > I understand your issue, but isn'there a configuration way to just limit the memory usage of MySQL?
>
> no, you did not understand the issue nor my intention
> intention was not "i have a problem and needs help"
> intention was "if someone may have a problem this is the help"
>
> the problem is that ANOTHER process or many of them triggering
> a OOM situation and after that the kernel killing randomly
> processes to prevent from a system crash
>
> the proc-setting is to prevent that this randomly process
> is mysqld instead the process which is eating up your memory
> _____________________
>
> as real world example (i was there)
>
> * machine has 9 GB RAM
> * machine is a dbmail-server
> * mysqld has assigend buffer-pools and so on up to 7 GB
> * someone tryies to copy a 2 GB attachment in drafts folder
> * the memory usage of imapd explodes
> * the first times OOM killer is killing imapd
> * at least the OOM is killing mysqld
>
> you want NOT killed mysqld in this situation
> you want not killed mysqld damaging tables becuase you are
> running out of memory because too much httpd-processes
> eating up your memory
>
> at least - you never want to get randomly killed mysqld
>
> pgrep -f "/usr/libexec/mysqld" | while read PID; do echo -1000 > /proc/$PID/oom_score_adj; done
>
> is preventing you from this happening
> maybe you want prevent such a situation before it happend
>
> i searched how can i proctect our servers, found a solution and
> thought it is a good idea to publish it on 3 mailing-lists
> where many users could need it
Attachments: signature.asc (0.26 KB)

DBMail users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.