Mailing List Archive: OpenSSH: Bugs

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled

Index | Next | Previous | View Threaded

bugzilla-daemon at mindrot

Jun 3, 2003, 4:47 AM

Post #1 of 16 (199 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled

http://bugzilla.mindrot.org/show_bug.cgi?id=585

Summary: sshd core dumping on IRIX 6.5.18 with
VerifyReverseMapping enabled
Product: Portable OpenSSH
Version: -current
Platform: MIPS
OS/Version: IRIX
Status: NEW
Severity: major
Priority: P2
Component: sshd
AssignedTo: openssh-bugs [at] mindrot
ReportedBy: ktaylor [at] daac

** I'm re-opening this case (it was bug #574). I don't think it got entered
correctly into the system **

Occasionally, we're noticing that sshd is core dumping on our IRIX 6.5.18f machine.

The only time we've noticed it is when users are logging in with putty
from offsite (although this is not really a client issue).

The user manages to log in, sshd apparently core dumps, but the user is not
logged out, the privilege separated user is still running their own personal
sshd spawn, and the parent is 1, so the root owned sshd process is gone.

wtmp is not updated, so the only way you can tell the user is logged in is by
listing their processes.

The end user doesn't notice that anything happened...and this doesn't ALWAYS
happen, but I can't correlate any system event and this. It will happen when the
system is first started, and it will happen when it's busier.

First core:

6 record_login(pid = 13759, ttyname = 0x1014a22c = "/dev/ttyq7", user =
0x101520d8 = "user1", uid = ####, host = 0x101522a8 =
"pcp01711145pcs.nrockv01.md.`omcast.net", addr = 0x7fff24b0, addrlen = 16)
["/usr/local/src/security/openssh-3.6.1p1/sshlogin.c":72, 0x1002be58]

Second core:

6 record_login(pid = 182438, ttyname = 0x1014a22c = "/dev/ttyq39", user =
0x101520d8 = "user2", uid = ####, host = 0x10152358 =
"toronto-hse-ppp3760148.symp`tico.ca", addr = 0x7fff24b0, addrlen = 16)
["/usr/local/src/security/openssh-3.6.1p1/sshlogin.c":72, 0x1002be58]

For some reason, the 29th character of the hostname is messed up. The first
hostname should be .comcast.net, the second hostname should be sympatico.ca

After looking through the source code, the actual problem may lie in
verify_reverse_mapping.

We had this option enabled in sshd_config, we disabled it and are currently
monitoring for the core dumps. If we don't see any, that may be the root of this
problem....hopefully it will point someone in the right direction towards fixing it.

After about 2 weeks, we have not had any core files, so it was definately this
option causing the crashing problem.

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 3, 2003, 5:02 AM

Post #2 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From djm [at] mindrot 2003-06-03 23:02 -------
It looks like the hostnames are being scribbled over by something. Perhaps a bug
in getaddrinfo()?

Is Irix using our getaddrinfo() replacement? (check for HAVE_GETADDRINFO in
config.h)

I doubt that the bug is in our canohost.c file, as it is used on all platforms.

Also, did you compile in 64-bit mode?

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 3, 2003, 5:05 AM

Post #3 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From ktaylor [at] daac 2003-06-03 23:05 -------

/* Define to 1 if you have the `getaddrinfo' function. */
#define HAVE_GETADDRINFO 1

We compile in n32 mode.

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 3, 2003, 5:09 AM

Post #4 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From ktaylor [at] daac 2003-06-03 23:09 -------

This was also in our config.h

/* getaddrinfo is broken (if present) */
/* #undef BROKEN_GETADDRINFO */

I'm not sure if it matters much that we're using openssh-3.6.1p1, not p2.

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 3, 2003, 2:41 PM

Post #5 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From djm [at] mindrot 2003-06-04 08:41 -------
Well, that indicates that you are using the system getaddrinfo function. We have
encountered bugs on some platforms' versions of these, but never ones leading to
crashes.

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 4, 2003, 12:59 AM

Post #6 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From djm [at] mindrot 2003-06-04 18:59 -------
*** Bug 574 has been marked as a duplicate of this bug. ***

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 4, 2003, 1:03 AM

Post #7 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From djm [at] mindrot 2003-06-04 19:03 -------
I just discovered your debugger output in bug #574 - this looks like things are
blowing up inside malloc(). This is usually an indication that memory has been
trashed before the call.

Consider building against ElectricFence[1] or some other malloc debugging
library. This would likely show up the error at the time the corruption happens.

[1] ftp://ftp.perens.com/pub/ElectricFence/ (I have no idea whether or not it
works on Irix)

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 4, 2003, 2:19 AM

Post #8 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From ktaylor [at] daac 2003-06-04 20:19 -------
not having any luck getting to sites with dmalloc tools. Unfortunately I'm not
very experienced with source debugging, so hopefully these things are easy to
implement.

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 4, 2003, 2:24 AM

Post #9 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From djm [at] mindrot 2003-06-04 20:24 -------
I should also warn you that electricfence drives up memory usage considerably

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 4, 2003, 2:29 AM

Post #10 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From ktaylor [at] daac 2003-06-04 20:29 -------
that could be a problem then, the system we're seeing the problems on may run
into troubles with high memory usage from sshd.

I may try forcing sshd to build using your getaddrinfo, and maybe that will
clear things up temporarily, although may not solve the actual problem.
Unfortunately we don't have a good test scenario that can generate this problem.
It has to happen on our main production box.

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 10, 2003, 7:16 AM

Post #11 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From ktaylor [at] daac 2003-06-11 01:16 -------
due to the security bug, we re-enabled verifyreversemapping, and immediately saw
core dumps again, so that just proves we're looking in the right spot.

Luckily the users are not inconvenienced by this.

Tomorrow, we're going to try using the sshd binary that uses the non-system
getaddrinfo function. (we rebuilt after unsetting HAVE_GETADDRINFO in config.h)
Hopefully that's all we needed to do.

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 11, 2003, 8:12 AM

Post #12 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From ktaylor [at] daac 2003-06-12 02:12 -------
Ok. After using the fake-getaddrinfo, sshd is still crashing. Here's the latest
dbx output.

Is there anything else we can look at without resorting to memory debugging?

> 0 realfree(0x10165f80, 0x10151490, 0x10165f60, 0x73706561, 0x73706560,
0x7ffed420, 0x10, 0x0)
["/xlv86/patches/5015/work/irix/lib/libc/libc_n32_M4/gen/malloc.c":527, 0xfb2466c]
1 cleanfree(0x0, 0x10151490, 0x10165f60, 0x73706561, 0x73706560, 0x7ffed420,
0x10, 0x0)
["/xlv86/patches/5015/work/irix/lib/libc/libc_n32_M4/gen/malloc.c":944, 0xfb24eac]
2 __malloc(0x260, 0x10151490, 0x10165f60, 0x73706561, 0x73706560, 0x7ffed420,
0x10, 0x0)
["/xlv86/patches/5015/work/irix/lib/libc/libc_n32_M4/gen/malloc.c":230, 0xfb240e0]
3 _malloc(0x0, 0x10151490, 0x10165f60, 0x73706561, 0x73706560, 0x7ffed420,
0x10, 0x0)
["/xlv86/patches/5015/work/irix/lib/libc/libc_n32_M4/gen/malloc.c":186, 0xfb23f4c]
4 xmalloc(size = 608)
["/usr/local/src/security/openssh-3.6.1p1/xmalloc.c":28, 0x10065994]
5 login_alloc_entry(pid = 20692179, username = 0x10151490 = "asdfa", hostname
= 0x10165f60 = "dsl093-055-063.blt1.dsl.spe`keasy.net", line = 0x1014a27c =
"/dev/ttyq25") ["/usr/local/src/security/openssh-3.6.1p1/loginrec.c":325,
0x10048b00]
6 record_login(pid = 20692179, ttyname = 0x1014a27c = "/dev/ttyq25", user =
0x10151490 = "asdf", uid = ####, host = 0x10165f60 =
"dsl093-055-063.blt1.dsl.spe`keasy.net", addr = 0x7ffed420, addrlen = 16)
["/usr/local/src/security/openssh-3.6.1p1/sshlogin.c":72, 0x1002beb8]
7 mm_record_login(s = 0x1014a248, pw = 0x1015dc08)
["/usr/local/src/security/openssh-3.6.1p1/monitor.c":1030, 0x10042c84]
8 mm_answer_pty(socket = 6, m = 0x7ffed510)
["/usr/local/src/security/openssh-3.6.1p1/monitor.c":1080, 0x10042f2c]
9 monitor_read(pmonitor = 0x101527c0, ent = 0x10137790, pent = (nil))
["/usr/local/src/security/openssh-3.6.1p1/monitor.c":371, 0x10040f54]
10 monitor_child_postauth(pmonitor = 0x101527c0)
["/usr/local/src/security/openssh-3.6.1p1/monitor.c":334, 0x10040dac]
11 privsep_postauth(authctxt = 0x101515b0)
["/usr/local/src/security/openssh-3.6.1p1/sshd.c":665, 0x10025f78]
12 main(ac = 1, av = 0x7ffedf14)
["/usr/local/src/security/openssh-3.6.1p1/sshd.c":1533, 0x10028a88]
13 __start()
["/xlv55/kudzu-apr12/work/irix/lib/libc/libc_n32_M4/csu/crt1text.s":177, 0x10024a48]

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 12, 2003, 2:41 AM

Post #13 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From dtucker [at] zip 2003-06-12 20:41 -------
Out of curiousity, what is MAXHOSTNAMELEN defined as on IRIX?

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 12, 2003, 2:50 AM

Post #14 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From ktaylor [at] daac 2003-06-12 20:50 -------
param.h:#define MAXHOSTNAMELEN 256 /* can't be longer than SYS_NMLN
- 1 */

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 12, 2003, 2:55 AM

Post #15 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From ktaylor [at] daac 2003-06-12 20:55 -------
FYI

utsname.h:#define _SYS_NMLN 257 /* 4.0 size of utsname elements.*/

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

bugzilla-daemon at mindrot

Jun 12, 2003, 3:20 PM

Post #16 of 16 (198 views)
Permalink

[Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled [In reply to]

http://bugzilla.mindrot.org/show_bug.cgi?id=585

------- Additional Comments From djm [at] mindrot 2003-06-13 09:20 -------
If you aren't already, you may want to try a CVS snapshot to see if the problem
has already been fixed there. Otherwise you will have to try malloc debugging -
the crash is definitely occurring inside malloc.

------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads