Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: OpenSSH: Dev

Fwd: Re: Fwd: cgroup OOM killer loop causes system to lockup (possible fix included) - now pinpointed to openssh-server

 

 

OpenSSH dev RSS feed   Index | Next | Previous | View Threaded


cal.leeming at simplicitymedialtd

May 30, 2011, 1:56 PM

Post #1 of 13 (1288 views)
Permalink
Fwd: Re: Fwd: cgroup OOM killer loop causes system to lockup (possible fix included) - now pinpointed to openssh-server

Just did some testing..

root [at] vick:~# cat /var/log/auth.log | grep "Set"
May 30 21:41:05 vicky sshd[1568]: Set /proc/self/oom_adj from -17 to -17
May 30 21:41:07 vicky sshd[1574]: Set /proc/self/oom_adj to -17

root [at] vick:~# ps faux | grep 1574
root 1574 0.0 0.0 70488 3404 ? Ss 21:41 0:00 \_
sshd: root [at] pt/1

root [at] vick:~# ps faux | grep "1568"
root 1568 0.0 0.0 49168 1152 ? Ss 21:41 0:00
/usr/sbin/sshd

In sshd.c there seems to be:
static int oom_adj_save = INT_MIN;

root [at] courtne:~/openssh-5.5p1# grep -R "Set %s to %d" .
./openbsd-compat/port-linux.c: verbose("Set %s to %d",
OOM_ADJ_PATH, oom_adj_save);

Then I tried on a server with different network card hardware (as shown
below), and got this from the logs:

root [at] courtne:~/openssh-5.5p1# cat /var/log/auth.log | grep "Set"
May 30 21:50:15 courtney sshd[4821]: Set /proc/self/oom_adj from 0 to -17
May 30 21:50:26 courtney sshd[4848]: Set /proc/self/oom_adj to 0

root [at] courtne:~/openssh-5.5p1# ps faux | grep "4848"
root 4848 0.0 0.0 70488 3372 ? Ss 21:50 0:00 \_
sshd: root [at] pt/1

root [at] courtne:~/openssh-5.5p1# ps faux | grep "4821"
root 4821 0.0 0.0 49168 1160 ? Ss 21:50 0:00
/usr/sbin/sshd

root [at] courtne:~/openssh-5.5p1# cat /var/log/auth.log | grep -e "Set" -e
"oom_adjust_restore"
May 30 21:50:15 courtney sshd[4821]: Set /proc/self/oom_adj from 0 to -17
May 30 21:50:26 courtney sshd[4848]: debug3: oom_adjust_restore
May 30 21:50:26 courtney sshd[4848]: Set /proc/self/oom_adj to 0




On 30/05/2011 21:30, Cal Leeming [Simplicity Media Ltd] wrote:
> Hi all,
>
> Please find below a complete transcript of the emails between
> debian/kernel-mm mailing lists.
>
> I've had a response back from someone on the deb mailing list stating:
>
> ====================================
> The bug seems to be that sshd does not reset the OOM adjustment before
> running the login shell (or other program). Therefore, please report a
> bug against openssh-server.
> ====================================
>
> Therefore, I am submitting this bug to you also.. If someone would be
> kind enough to have a flick thru all the below debug/logs, it'd be
> very much appreciated.
>
> Cal

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev


cal.leeming at simplicitymedialtd

May 30, 2011, 2:32 PM

Post #2 of 13 (1255 views)
Permalink
RE: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. [In reply to]

So I modified the code to try and repair this oom_adj problem...

port-linux.c:
line 235: //static int oom_adj_save = INT_MIN;
line 236: static int oom_adj_save = 0;
line 277: verbose("Set %s to %d - sleepycal", OOM_ADJ_PATH, oom_adj_save);


I then ran compiled the package, ran SSHd, and yet we still have -17 in
oom_adj_save. Wtf? Now, I'm not much of a C coder, but this is weird
even in my books...

May 30 22:18:19 vicky sshd[12825]: Set /proc/self/oom_adj to -17 - sleepycal

So, I went all out crazy, and did the following patch:

static int sleepycal_oom_adj_save = 0;
verbose("sleepycal_oom_adj_save=%d", sleepycal_oom_adj_save);

if (fprintf(fp, "%d\n", sleepycal_oom_adj_save) <= 0)
verbose("error writing %s: %s", OOM_ADJ_PATH,
strerror(errno));
else
verbose("Set %s to %d - sleepycal", OOM_ADJ_PATH,
sleepycal_oom_adj_save);

And it worked!!! :)

May 30 22:27:12 vicky sshd[2532]: sleepycal_oom_adj_save=0
May 30 22:27:12 vicky sshd[2532]: Set /proc/self/oom_adj to 0 - sleepycal

root [at] vick:~/openssh-5.5p1# cat /proc/2532/oom_adj
0

So, it turns out that it is actually OpenSSH which is broken, after
almost 3 days of frustrating digging through millions of lines of code
lol. Anyways, would appreciate if someone could get this merged into
master (obv rename the vars if you want).

Attached is the appropriate patch file as of openssh-5.5p1

Cal

On 30/05/2011 21:56, Cal Leeming [Simplicity Media Ltd] wrote:
> Just did some testing..
>
> root [at] vick:~# cat /var/log/auth.log | grep "Set"
> May 30 21:41:05 vicky sshd[1568]: Set /proc/self/oom_adj from -17 to -17
> May 30 21:41:07 vicky sshd[1574]: Set /proc/self/oom_adj to -17
>
> root [at] vick:~# ps faux | grep 1574
> root 1574 0.0 0.0 70488 3404 ? Ss 21:41 0:00 \_
> sshd: root [at] pt/1
>
> root [at] vick:~# ps faux | grep "1568"
> root 1568 0.0 0.0 49168 1152 ? Ss 21:41 0:00
> /usr/sbin/sshd
>
> In sshd.c there seems to be:
> static int oom_adj_save = INT_MIN;
>
> root [at] courtne:~/openssh-5.5p1# grep -R "Set %s to %d" .
> ./openbsd-compat/port-linux.c: verbose("Set %s to %d",
> OOM_ADJ_PATH, oom_adj_save);
>
> Then I tried on a server with different network card hardware (as
> shown below), and got this from the logs:
>
> root [at] courtne:~/openssh-5.5p1# cat /var/log/auth.log | grep "Set"
> May 30 21:50:15 courtney sshd[4821]: Set /proc/self/oom_adj from 0 to -17
> May 30 21:50:26 courtney sshd[4848]: Set /proc/self/oom_adj to 0
>
> root [at] courtne:~/openssh-5.5p1# ps faux | grep "4848"
> root 4848 0.0 0.0 70488 3372 ? Ss 21:50 0:00 \_
> sshd: root [at] pt/1
>
> root [at] courtne:~/openssh-5.5p1# ps faux | grep "4821"
> root 4821 0.0 0.0 49168 1160 ? Ss 21:50 0:00
> /usr/sbin/sshd
>
> root [at] courtne:~/openssh-5.5p1# cat /var/log/auth.log | grep -e "Set"
> -e "oom_adjust_restore"
> May 30 21:50:15 courtney sshd[4821]: Set /proc/self/oom_adj from 0 to -17
> May 30 21:50:26 courtney sshd[4848]: debug3: oom_adjust_restore
> May 30 21:50:26 courtney sshd[4848]: Set /proc/self/oom_adj to 0
>
>
>
>
> On 30/05/2011 21:30, Cal Leeming [Simplicity Media Ltd] wrote:
>> Hi all,
>>
>> Please find below a complete transcript of the emails between
>> debian/kernel-mm mailing lists.
>>
>> I've had a response back from someone on the deb mailing list stating:
>>
>> ====================================
>> The bug seems to be that sshd does not reset the OOM adjustment before
>> running the login shell (or other program). Therefore, please report a
>> bug against openssh-server.
>> ====================================
>>
>> Therefore, I am submitting this bug to you also.. If someone would be
>> kind enough to have a flick thru all the below debug/logs, it'd be
>> very much appreciated.
>>
>> Cal
>
Attachments: oom_patch_for_openssh-5.5p1_by_sleepycal.patch (1.75 KB)


gert at greenie

May 31, 2011, 12:25 AM

Post #3 of 13 (1234 views)
Permalink
Re: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. [In reply to]

Hi,

On Mon, May 30, 2011 at 10:32:24PM +0100, Cal Leeming [Simplicity Media Ltd] wrote:
> So, it turns out that it is actually OpenSSH which is broken, after

I would not second this. To me, this very much looks like:

> On 30/05/2011 21:56, Cal Leeming [Simplicity Media Ltd] wrote:
> > Just did some testing..
> >
> >root [at] vick:~# cat /var/log/auth.log | grep "Set"
> >May 30 21:41:05 vicky sshd[1568]: Set /proc/self/oom_adj from -17 to -17
> >May 30 21:41:07 vicky sshd[1574]: Set /proc/self/oom_adj to -17

... it's reading out the old value, saving it, setting it to "-17" (for
the sshd listener process, that one is not to be killed), and later on
*restoring* the old value (for all child processes). See the comments
in platform.c

The log messages look weird because the value is -17 already when sshd
starts - so it's adjusting "-17 to -17" and later on "restoring -17" -
looks stupid, but that's computers for you. But what it tells you is
that the value isn't set by sshd to "-17" but that sshd inherited that
from whoever started it.

The question here is why sshd is sometimes started with -17 and sometimes
with 0 - that's the bug, not that sshd keeps what it's given.

(Ask yourself: if sshd had no idea about oom_adj at all, would this make
it buggy by not changing the value?)


Anyway, as a workaround for your system, you can certainly set

oom_adj_save = 0;

in the beginning of port-linux.c / oom_adjust_restore(), to claim that
"hey, this was the saved value to start with" and "restore" oom_adj to 0
then - but that's just hiding the bug, not fixing it.

gert
--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert [at] greenie
fax: +49-89-35655025 gert [at] net
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev


cal.leeming at simplicitymedialtd

May 31, 2011, 4:11 AM

Post #4 of 13 (1229 views)
Permalink
Re: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. [In reply to]

Hi Gert,

Thank you for taking the time to respond! Further comments below:

On Tue, May 31, 2011 at 8:25 AM, Gert Doering <gert [at] greenie> wrote:
> Hi,
>
> On Mon, May 30, 2011 at 10:32:24PM +0100, Cal Leeming [Simplicity Media Ltd] wrote:
>> So, it turns out that it is actually OpenSSH which is broken, after
>
> I would not second this.  To me, this very much looks like:
>
>> On 30/05/2011 21:56, Cal Leeming [Simplicity Media Ltd] wrote:
>> > Just did some testing..
>> >
>> >root [at] vick:~# cat /var/log/auth.log | grep "Set"
>> >May 30 21:41:05 vicky sshd[1568]: Set /proc/self/oom_adj from -17 to -17
>> >May 30 21:41:07 vicky sshd[1574]: Set /proc/self/oom_adj to -17
>
> ... it's reading out the old value, saving it, setting it to "-17" (for
> the sshd listener process, that one is not to be killed), and later on
> *restoring* the old value (for all child processes).  See the comments
> in platform.c
>
> The log messages look weird because the value is -17 already when sshd
> starts - so it's adjusting "-17 to -17" and later on "restoring -17" -
> looks stupid, but that's computers for you.  But what it tells you is
> that the value isn't set by sshd to "-17" but that sshd inherited that
> from whoever started it.

Could you point out the line of code where oom_adj_save is set to the
original value, because I've looked everywhere, and from what I can
tell, it's only ever set to INT_MIN, and no where else is it changed.
(C is not my strongest language tho, so I most likely have overlooked
something). This is where I got thrown off.

>
> The question here is why sshd is sometimes started with -17 and sometimes
> with 0 - that's the bug, not that sshd keeps what it's given.
>
> (Ask yourself: if sshd had no idea about oom_adj at all, would this make
> it buggy by not changing the value?)

This was what I was trying to pinpoint down before. I had came to this
conclusion myself, sent it to the Debian bug list, and they dismissed
on the grounds that it was an openssh problem...

So far, the buck has been passed from kernel-mm to debian-kernel, to
openssh, and now back to debian-kernel lol. The most annoying thing,
is that you can't get this bug to happen unless you physically test on
a machine which requires the bnx2 firmwire, so I get the feeling this
won't get resolved :/

>
>
> Anyway, as a workaround for your system, you can certainly set
>
>  oom_adj_save = 0;
>
> in the beginning of port-linux.c / oom_adjust_restore(), to claim that
> "hey, this was the saved value to start with" and "restore" oom_adj to 0
> then - but that's just hiding the bug, not fixing it.

I'm disappointed this wasn't the correct fix, I honestly thought I had
patched it right :(

But, on the other hand, ssh users should really never have a default
oom_adj of -17, so maybe 0 should be set as default anyway? If this is
not the case, could you give reasons why??

>
> gert
> --
> USENET is *not* the non-clickable part of WWW!
>                                                           //www.muc.de/~gert/
> Gert Doering - Munich, Germany                             gert [at] greenie
> fax: +49-89-35655025                        gert [at] net
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev


cal.leeming at simplicitymedialtd

May 31, 2011, 4:26 AM

Post #5 of 13 (1228 views)
Permalink
Re: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. [In reply to]

Bug report submitted to Debian:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=628690

Should be interesting to see what happens with this lol.

On Tue, May 31, 2011 at 12:11 PM, Cal Leeming [Simplicity Media Ltd]
<cal.leeming [at] simplicitymedialtd> wrote:
> Hi Gert,
>
> Thank you for taking the time to respond! Further comments below:
>
> On Tue, May 31, 2011 at 8:25 AM, Gert Doering <gert [at] greenie> wrote:
>> Hi,
>>
>> On Mon, May 30, 2011 at 10:32:24PM +0100, Cal Leeming [Simplicity Media Ltd] wrote:
>>> So, it turns out that it is actually OpenSSH which is broken, after
>>
>> I would not second this.  To me, this very much looks like:
>>
>>> On 30/05/2011 21:56, Cal Leeming [Simplicity Media Ltd] wrote:
>>> > Just did some testing..
>>> >
>>> >root [at] vick:~# cat /var/log/auth.log | grep "Set"
>>> >May 30 21:41:05 vicky sshd[1568]: Set /proc/self/oom_adj from -17 to -17
>>> >May 30 21:41:07 vicky sshd[1574]: Set /proc/self/oom_adj to -17
>>
>> ... it's reading out the old value, saving it, setting it to "-17" (for
>> the sshd listener process, that one is not to be killed), and later on
>> *restoring* the old value (for all child processes).  See the comments
>> in platform.c
>>
>> The log messages look weird because the value is -17 already when sshd
>> starts - so it's adjusting "-17 to -17" and later on "restoring -17" -
>> looks stupid, but that's computers for you.  But what it tells you is
>> that the value isn't set by sshd to "-17" but that sshd inherited that
>> from whoever started it.
>
> Could you point out the line of code where oom_adj_save is set to the
> original value, because I've looked everywhere, and from what I can
> tell, it's only ever set to INT_MIN, and no where else is it changed.
> (C is not my strongest language tho, so I most likely have overlooked
> something). This is where I got thrown off.
>
>>
>> The question here is why sshd is sometimes started with -17 and sometimes
>> with 0 - that's the bug, not that sshd keeps what it's given.
>>
>> (Ask yourself: if sshd had no idea about oom_adj at all, would this make
>> it buggy by not changing the value?)
>
> This was what I was trying to pinpoint down before. I had came to this
> conclusion myself, sent it to the Debian bug list, and they dismissed
> on the grounds that it was an openssh problem...
>
> So far, the buck has been passed from kernel-mm to debian-kernel, to
> openssh, and now back to debian-kernel lol. The most annoying thing,
> is that you can't get this bug to happen unless you physically test on
> a machine which requires the bnx2 firmwire, so I get the feeling this
> won't get resolved :/
>
>>
>>
>> Anyway, as a workaround for your system, you can certainly set
>>
>>  oom_adj_save = 0;
>>
>> in the beginning of port-linux.c / oom_adjust_restore(), to claim that
>> "hey, this was the saved value to start with" and "restore" oom_adj to 0
>> then - but that's just hiding the bug, not fixing it.
>
> I'm disappointed this wasn't the correct fix, I honestly thought I had
> patched it right :(
>
> But, on the other hand, ssh users should really never have a default
> oom_adj of -17, so maybe 0 should be set as default anyway? If this is
> not the case, could you give reasons why??
>
>>
>> gert
>> --
>> USENET is *not* the non-clickable part of WWW!
>>                                                           //www.muc.de/~gert/
>> Gert Doering - Munich, Germany                             gert [at] greenie
>> fax: +49-89-35655025                        gert [at] net
>>
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev


dtucker at zip

May 31, 2011, 5:11 AM

Post #6 of 13 (1240 views)
Permalink
Re: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. [In reply to]

On Tue, May 31, 2011 at 9:11 PM, Cal Leeming [Simplicity Media Ltd]
<cal.leeming [at] simplicitymedialtd> wrote:
[...]
> Could you point out the line of code where oom_adj_save is set to the
> original value, because I've looked everywhere, and from what I can
> tell, it's only ever set to INT_MIN,

It's read from /proc/self/oom_adj at startup time in oom_adjust_setup():

if ((fp = fopen(oom_adj_path, "r+")) != NULL) {
if (fscanf(fp, "%d", &oom_adj_save) != 1)

This is the reason for the "Set /proc/self/oom_adj from -17 to -17"
probably why Gert commented on it.

Basically, sshd sets the listening process to -17, then restores
whatever was previously set for all forked processes. If oom_adj was
previously -17, sshd will restore that.

[...]
> This was what I was trying to pinpoint down before. I had came to this
> conclusion myself, sent it to the Debian bug list, and they dismissed
> on the grounds that it was an openssh problem...

I'd suggest "grep -rl oom_adj /etc" and see if one of your system
startup scripts sets it. Failing that, I'd try cold booting your
machine without your problem module, modprobe it and check
/proc/self/oom_adj and see if the modprobe or module loading somehow
changes that (I can't imagine that it would, but you seem to have a
really strange case here...).

--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev


cal.leeming at simplicitymedialtd

May 31, 2011, 5:18 AM

Post #7 of 13 (1230 views)
Permalink
Re: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. [In reply to]

On Tue, May 31, 2011 at 1:11 PM, Darren Tucker <dtucker [at] zip> wrote:
> On Tue, May 31, 2011 at 9:11 PM, Cal Leeming [Simplicity Media Ltd]
> <cal.leeming [at] simplicitymedialtd> wrote:
> [...]
>> Could you point out the line of code where oom_adj_save is set to the
>> original value, because I've looked everywhere, and from what I can
>> tell, it's only ever set to INT_MIN,
>
> It's read from /proc/self/oom_adj at startup time in oom_adjust_setup():
>
>               if ((fp = fopen(oom_adj_path, "r+")) != NULL) {
>                        if (fscanf(fp, "%d", &oom_adj_save) != 1)
>
> This is the reason for the "Set /proc/self/oom_adj from -17 to -17"
> probably why Gert commented on it.
>
> Basically, sshd sets the listening process to -17, then restores
> whatever was previously set for all forked processes.  If oom_adj was
> previously -17, sshd will restore that.
>
> [...]
>> This was what I was trying to pinpoint down before. I had came to this
>> conclusion myself, sent it to the Debian bug list, and they dismissed
>> on the grounds that it was an openssh problem...
>
> I'd suggest "grep -rl oom_adj /etc" and see if one of your system
> startup scripts sets it.  Failing that, I'd try cold booting your
> machine without your problem module, modprobe it and check
> /proc/self/oom_adj and see if the modprobe or module loading somehow
> changes that (I can't imagine that it would, but you seem to have a
> really strange case here...).

Oh trust me, I looked *everywhere*. Even to the extent of running
tripwire from a bare bones system, and looking manually at every
change made. I also looked for loads of different keywords (-17, oom,
proc, self) etc. Spent hours on it :/

As for the comment about the modprobe, I already did all this (full
debug can be found at
http://www.debianhelp.org/content/cgroup-oom-killer-loop-causes-system-lockup-possible-fix-included
), and found that when the bnx2 module isn't loaded, the problem goes
away.. When it is loaded, the problem comes back.

This is what I mean by it being a very VERY strange problem.

My guess is that the bnx2 firmware does some sort of
kernel-to-userspace weirdness, which causes user land apps (which have
to go through the bnx2 in the network stack) to somehow inherit the
-17 that all kernel processes get... Sadly, I don't know enough about
how the kernel works to even begin to debug the problem.. plus (from
what I can tell) the firmware (*.fw) is closed source..

I have a very strong feeling that the buck will probably just get
passed around until it eventually gets forgotten about :( I wish I
knew more about kernel development to try and fix this issue!

>
> --
> Darren Tucker (dtucker at zip.com.au)
> GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
>     Good judgement comes with experience. Unfortunately, the experience
> usually comes from bad judgement.
>
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev


gert at greenie

May 31, 2011, 5:25 AM

Post #8 of 13 (1227 views)
Permalink
Re: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. [In reply to]

Hi,

On Tue, May 31, 2011 at 12:11:13PM +0100, Cal Leeming [Simplicity Media Ltd] wrote:
> Could you point out the line of code where oom_adj_save is set to the
> original value, because I've looked everywhere, and from what I can
> tell, it's only ever set to INT_MIN, and no where else is it changed.
> (C is not my strongest language tho, so I most likely have overlooked
> something). This is where I got thrown off.

oom_adjust_setup() does this:

if ((fp = fopen(oom_adj_path, "r+")) != NULL) {
if (fscanf(fp, "%d", &oom_adj_save) != 1)
verbose("error reading %s: %s", oom_adj_path,
strerror(errno));

the "fscanf()" call will read an integer ("%d") from the file named,
and write that number into the variable being pointed to (&oom_adj_save).

The loop is a bit tricky to read as it takes different paths into
account, and will exit after the first successful update.

fscanf() will return the number of successful conversions, so if it
was able to read "one number", the return value is "1", and it will
jump to the else block

else {
rewind(fp);
if (fprintf(fp, "%d\n", value) <= 0)
verbose("error writing %s: %s",
oom_adj_path, strerror(errno));
else
verbose("Set %s from %d to %d",
oom_adj_path, oom_adj_save, value);
}

where it will overwrite what is in that file with the new value
("value"), and then print the "Set ... from -17 to -17" message that
you saw.


> > The question here is why sshd is sometimes started with -17 and sometimes
> > with 0 - that's the bug, not that sshd keeps what it's given.
> >
> > (Ask yourself: if sshd had no idea about oom_adj at all, would this make
> > it buggy by not changing the value?)
>
> This was what I was trying to pinpoint down before. I had came to this
> conclusion myself, sent it to the Debian bug list, and they dismissed
> on the grounds that it was an openssh problem...

I must admit that I have no idea what is causing it, but from the logs,
it very much looks like sshd is started with "-17" in there - but only
in the problem case.


> So far, the buck has been passed from kernel-mm to debian-kernel, to
> openssh, and now back to debian-kernel lol. The most annoying thing,
> is that you can't get this bug to happen unless you physically test on
> a machine which requires the bnx2 firmwire, so I get the feeling this
> won't get resolved :/

And *that* strongly points to a bug in the bnx2 stuff - if other programs
change their behaviour based on the existance of a given driver, that
does not smell very healthy.

[..]
> > Anyway, as a workaround for your system, you can certainly set
> >
> >  oom_adj_save = 0;
> >
> > in the beginning of port-linux.c / oom_adjust_restore(), to claim that
> > "hey, this was the saved value to start with" and "restore" oom_adj to 0
> > then - but that's just hiding the bug, not fixing it.
>
> I'm disappointed this wasn't the correct fix, I honestly thought I had
> patched it right :(

Well, that's the short hand - "just ignore everything that happened at
init / save time, and forcibly write back '0', no matter what was there
before".

> But, on the other hand, ssh users should really never have a default
> oom_adj of -17, so maybe 0 should be set as default anyway? If this is
> not the case, could you give reasons why??

Well, I would say "the default value in there is a matter of local policy",
so what if someone wants to make sure that whatever is run from sshd is
always privileged regarding oom, even if a local firefox etc. is running
amock and you need to ssh-in and kill the GUI stuff...

One might opt to run sshd (and all its children) at "-5" (slightly special
treatment), or "0" (no special treatment), or even at "-17" - but that's
local policy.


Mmmh.

Since this seems to be inherited, it might even work if you just change
the sshd startup script, and insert

echo 0 >/proc/self/oom_adj

in there, right before it starts the sshd... "local policy at work".

gert

--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert [at] greenie
fax: +49-89-35655025 gert [at] net
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev


dtucker at zip

May 31, 2011, 5:37 AM

Post #9 of 13 (1232 views)
Permalink
Re: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. [In reply to]

On Tue, May 31, 2011 at 10:18 PM, Cal Leeming [Simplicity Media Ltd]
<cal.leeming [at] simplicitymedialtd> wrote:
[...]
> Oh trust me, I looked *everywhere*. Even to the extent of running
> tripwire from a bare bones system, and looking manually at every
> change made. I also looked for loads of different keywords (-17, oom,
> proc, self) etc. Spent hours on it :/
>
> As for the comment about the modprobe, I already did all this (full
> debug can be found at
> http://www.debianhelp.org/content/cgroup-oom-killer-loop-causes-system-lockup-possible-fix-included
> ), and found that when the bnx2 module isn't loaded, the problem goes
> away.. When it is loaded, the problem comes back.

Did you check /proc/self/oom_adj before and after loading the module?
I don't see that in there, and it it *does* change it would eliminate
sshd as a variable.

As a workaround, you could add "echo 0 >/proc/self/oom_adj" to
/etc/default/ssh. It's a bit ugly, but at least you wouldn't need to
recompile anything.

--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev


cal.leeming at simplicitymedialtd

May 31, 2011, 5:42 AM

Post #10 of 13 (1236 views)
Permalink
Re: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. [In reply to]

On 31/05/2011 13:25, Gert Doering wrote:
> Hi,
>
> On Tue, May 31, 2011 at 12:11:13PM +0100, Cal Leeming [Simplicity Media Ltd] wrote:
>> Could you point out the line of code where oom_adj_save is set to the
>> original value, because I've looked everywhere, and from what I can
>> tell, it's only ever set to INT_MIN, and no where else is it changed.
>> (C is not my strongest language tho, so I most likely have overlooked
>> something). This is where I got thrown off.
> oom_adjust_setup() does this:
>
> if ((fp = fopen(oom_adj_path, "r+")) != NULL) {
> if (fscanf(fp, "%d",&oom_adj_save) != 1)
> verbose("error reading %s: %s", oom_adj_path,
> strerror(errno));
>
> the "fscanf()" call will read an integer ("%d") from the file named,
> and write that number into the variable being pointed to (&oom_adj_save).
>
> The loop is a bit tricky to read as it takes different paths into
> account, and will exit after the first successful update.
>
> fscanf() will return the number of successful conversions, so if it
> was able to read "one number", the return value is "1", and it will
> jump to the else block
>
> else {
> rewind(fp);
> if (fprintf(fp, "%d\n", value)<= 0)
> verbose("error writing %s: %s",
> oom_adj_path, strerror(errno));
> else
> verbose("Set %s from %d to %d",
> oom_adj_path, oom_adj_save, value);
> }
>
> where it will overwrite what is in that file with the new value
> ("value"), and then print the "Set ... from -17 to -17" message that
> you saw.

Ah, thank you for explaining this. Makes a lot more sense now :)

>
>>> The question here is why sshd is sometimes started with -17 and sometimes
>>> with 0 - that's the bug, not that sshd keeps what it's given.
>>>
>>> (Ask yourself: if sshd had no idea about oom_adj at all, would this make
>>> it buggy by not changing the value?)
>> This was what I was trying to pinpoint down before. I had came to this
>> conclusion myself, sent it to the Debian bug list, and they dismissed
>> on the grounds that it was an openssh problem...
> I must admit that I have no idea what is causing it, but from the logs,
> it very much looks like sshd is started with "-17" in there - but only
> in the problem case.
>
>
>> So far, the buck has been passed from kernel-mm to debian-kernel, to
>> openssh, and now back to debian-kernel lol. The most annoying thing,
>> is that you can't get this bug to happen unless you physically test on
>> a machine which requires the bnx2 firmwire, so I get the feeling this
>> won't get resolved :/
> And *that* strongly points to a bug in the bnx2 stuff - if other programs
> change their behaviour based on the existance of a given driver, that
> does not smell very healthy.

Agreed.. I was thinking of adding some debug into the fs/proc/ code
which does a kprint on every oom_adj read/write, but I couldn't figure
out how to extract the pid from the task (pointer?).

> [..]
>>> Anyway, as a workaround for your system, you can certainly set
>>>
>>> oom_adj_save = 0;
>>>
>>> in the beginning of port-linux.c / oom_adjust_restore(), to claim that
>>> "hey, this was the saved value to start with" and "restore" oom_adj to 0
>>> then - but that's just hiding the bug, not fixing it.
>> I'm disappointed this wasn't the correct fix, I honestly thought I had
>> patched it right :(
> Well, that's the short hand - "just ignore everything that happened at
> init / save time, and forcibly write back '0', no matter what was there
> before".
>
>> But, on the other hand, ssh users should really never have a default
>> oom_adj of -17, so maybe 0 should be set as default anyway? If this is
>> not the case, could you give reasons why??
> Well, I would say "the default value in there is a matter of local policy",
> so what if someone wants to make sure that whatever is run from sshd is
> always privileged regarding oom, even if a local firefox etc. is running
> amock and you need to ssh-in and kill the GUI stuff...
>
> One might opt to run sshd (and all its children) at "-5" (slightly special
> treatment), or "0" (no special treatment), or even at "-17" - but that's
> local policy.

Ah, okay that's make sense.

>
> Mmmh.
>
> Since this seems to be inherited, it might even work if you just change
> the sshd startup script, and insert
>
> echo 0>/proc/self/oom_adj
>
> in there, right before it starts the sshd... "local policy at work".

Yeah I was going to do this, but then I thought "well if this problem is
occurring for openssh, then what else could it be affecting?". As you
pointed out above, having the oom_adj changed based on the existence of
a driver is really not good.

I will paste this convo trail into the debian ticket, and hopefully
it'll help convince them this issue needs fixing.

> gert

Thanks again for taking the time to reply!

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev


cal.leeming at simplicitymedialtd

May 31, 2011, 5:58 AM

Post #11 of 13 (1228 views)
Permalink
Re: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. [In reply to]

On 31/05/2011 13:37, Darren Tucker wrote:
> On Tue, May 31, 2011 at 10:18 PM, Cal Leeming [Simplicity Media Ltd]
> <cal.leeming [at] simplicitymedialtd> wrote:
> [...]
>> Oh trust me, I looked *everywhere*. Even to the extent of running
>> tripwire from a bare bones system, and looking manually at every
>> change made. I also looked for loads of different keywords (-17, oom,
>> proc, self) etc. Spent hours on it :/
>>
>> As for the comment about the modprobe, I already did all this (full
>> debug can be found at
>> http://www.debianhelp.org/content/cgroup-oom-killer-loop-causes-system-lockup-possible-fix-included
>> ), and found that when the bnx2 module isn't loaded, the problem goes
>> away.. When it is loaded, the problem comes back.
> Did you check /proc/self/oom_adj before and after loading the module?
> I don't see that in there, and it it *does* change it would eliminate
> sshd as a variable.
>
> As a workaround, you could add "echo 0>/proc/self/oom_adj" to
> /etc/default/ssh. It's a bit ugly, but at least you wouldn't need to
> recompile anything.

Thank you for the response! I've put some replies back to Gerts email
(relating to this suggestion).
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev


openssh at roumenpetrov

May 31, 2011, 1:07 PM

Post #12 of 13 (1232 views)
Permalink
Re: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. [In reply to]

Darren Tucker wrote:
> On Tue, May 31, 2011 at 10:18 PM, Cal Leeming [Simplicity Media Ltd]
> <cal.leeming [at] simplicitymedialtd> wrote:
> [...]
>
>> Oh trust me, I looked *everywhere*. Even to the extent of running
>> tripwire from a bare bones system, and looking manually at every
>> change made. I also looked for loads of different keywords (-17, oom,
>> proc, self) etc. Spent hours on it :/
>>
>> As for the comment about the modprobe, I already did all this (full
>> debug can be found at
>> http://www.debianhelp.org/content/cgroup-oom-killer-loop-causes-system-lockup-possible-fix-included
>> ), and found that when the bnx2 module isn't loaded, the problem goes
>> away.. When it is loaded, the problem comes back.
>>
> Did you check /proc/self/oom_adj before and after loading the module?
> I don't see that in there, and it it *does* change it would eliminate
> sshd as a variable.
>
> As a workaround, you could add "echo 0>/proc/self/oom_adj" to
> /etc/default/ssh. It's a bit ugly, but at least you wouldn't need to
> recompile anything.
>

May is not related but /proc/self/oom_adjis is reported as deprecated:
syslog:<DATE> <HOST> kernel: udevd (<PID>): /proc/<PID>/oom_adj is
deprecated, please use /proc/<PID>/oom_score_adj instead, where kernel
is 2.6.38.6. I see oom_score_adj for first time in 2.6.36 .

Regards,
Roumen

--
Get X.509 certificates support in OpenSSH:
http://roumenpetrov.info/openssh/

_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev


gert at greenie

May 31, 2011, 1:43 PM

Post #13 of 13 (1248 views)
Permalink
Re: port-linux.c bug with oom_adjust_restore() - causes real bad oom_adj - which can cause DoS conditions. [In reply to]

Hi,

On Tue, May 31, 2011 at 11:07:38PM +0300, Roumen Petrov wrote:
> May is not related but /proc/self/oom_adjis is reported as deprecated:
> syslog:<DATE> <HOST> kernel: udevd (<PID>): /proc/<PID>/oom_adj is
> deprecated, please use /proc/<PID>/oom_score_adj instead, where kernel
> is 2.6.38.6. I see oom_score_adj for first time in 2.6.36 .

Unrelated. OpenSSH will use whatever is there.

gert

--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert [at] greenie
fax: +49-89-35655025 gert [at] net
_______________________________________________
openssh-unix-dev mailing list
openssh-unix-dev [at] mindrot
https://lists.mindrot.org/mailman/listinfo/openssh-unix-dev

OpenSSH dev RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.