Mailing List Archive: Qmail: users

netqmail 1.06 + ucspi-tls problems (sslserver segfault)

Index | Next | Previous | View Threaded

bfaler at holdenlegal

Feb 13, 2009, 2:03 PM

Post #1 of 11 (3449 views)
Permalink

netqmail 1.06 + ucspi-tls problems (sslserver segfault)

Hey everyone,

I am running netqmail 1.06 on an Ubuntu 8.04 (x86) machine and I have
run into a problem that I can't seem to solve on my own.. I'm hoping
someone on the list may have run into a similar situation at some point..

I am attempting to get SSL/TLS support up and running using ucspi-ssl
0.70 and the ucspi-tls patch, and while mail seems to be flowing I am
getting segfaults similiar to the one below off of sslserver
(presumably?) at the end of every connection (both SMTP and POP3). If I
modify the smtp or pop daemon to run under tcpserver rather than
sslserver the problem goes away, but I would very much like to be able
to support SSL/TLS on this system when it goes live..

from syslog (the segfault in question):

sandbox kernel: [1107554.043727] sslserver[26877]: segfault at 5f4c5353
eip b7e1b34a esp bfd0c838 error 4

My qmail/supervise/qmail-smtp/run entry looks like this:

exec /usr/local/bin/softlimit -m 12000000 \
/usr/local/bin/sslserver -e -n -v -R -l "$LOCAL" -x
/etc/tcp.smtp.cdb -c "$MAXSMTPD" \
-u "$QMAILDUID" -g "$NOFILESGID" 0 smtp
/var/qmail/bin/qmail-smtpd 2>&1

Here is an excerpt from the smtpd log file:

@400000004995e1a119772464 sslserver: cafile 26845
@400000004995e1a11978800c sslserver: ccafile 26845
@400000004995e1a11978ed6c sslserver: cadir 26845 /usr/local/ssl/certs
@400000004995e1a1197952fc sslserver: cert 26845 /var/qmail/ssl/cert
@400000004995e1a11979bc74 sslserver: key 26845 /var/qmail/ssl/key
@400000004995e1a1197a2204 sslserver: param 26845 /var/qmail/ssl/dhparam 1024
@400000004995e1a1197b3f2c sslserver: status: 0/20
@400000004995e1cb331815f4 sslserver: status: 1/20
@400000004995e1cb332a618c sslserver: pid 26878 from 127.0.0.1
@400000004995e1cb33dc38e4 sslserver: ok 26878 sandbox.local:127.0.0.1:25
localhost:127.0.0.1::49464
@400000004995e1d12d30f264 sslserver: ssl 26877 accept
@400000004995e1d12d5cdc94 sslserver: end 26877 status 11
@400000004995e1d12d5e112c sslserver: status: 0/20

Has anyone run into a problem like this on a similiar? Any suggestions
how to go about tracking down what is causing this segfault?

Brad
bfaler [at] holdenlegal

Jason.Haar at trimble

Feb 13, 2009, 5:12 PM

Post #2 of 11 (3306 views)
Permalink

Re: netqmail 1.06 + ucspi-tls problems (sslserver segfault) [In reply to]

Brad Faler wrote:
> I am attempting to get SSL/TLS support up and running using ucspi-ssl
> 0.70 and the ucspi-tls patch, and while mail seems to be flowing I am
> getting segfaults similiar to the one below off of sslserver
> (presumably?) at the end of every connection (both SMTP and POP3).
Check your memory limits as set by softlimit/etc in your /service's...

SSL uses a tonne more memory than non-SSL code, so that could explain
the difference.

--
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

jms1 at jms1

Feb 17, 2009, 4:03 AM

Post #3 of 11 (3285 views)
Permalink

Re: netqmail 1.06 + ucspi-tls problems (sslserver segfault) [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2009-02-13, at 1703, Brad Faler wrote:
>
> I am attempting to get SSL/TLS support up and running using ucspi-ssl
> 0.70 and the ucspi-tls patch, and while mail seems to be flowing I am
> getting segfaults similiar to the one below off of sslserver
> (presumably?) at the end of every connection (both SMTP and POP3).
> If I
> modify the smtp or pop daemon to run under tcpserver rather than
> sslserver the problem goes away, but I would very much like to be able
> to support SSL/TLS on this system when it goes live..

i'm also using sslserver 0.70, and i'm not having any problems with
it. it allows users to connect to port 465, do their SSL negotiations,
and then start the SMTP conversation. i haven't had any problems with
it, and i've been using it for a few years now.

i'm also using a patch which adds support for TLS into qmail-smtpd.
the patch is the one many people have been using for years, most
recently updated by dr. hoffman, and included as part of my combined
patch. i've added a few configuration items (controlled by environment
variables) so the administrator can require, allow, or deny TLS for a
particular service (or client IP, if you change the variables in the
tcpserver access control file.)

setting up support for SSL, and setting up support for TLS, are two
totally different things. it sounds like you have combined the ideas
in your head, but you really need to consider each thing separately.
the SSL and STARTTLS mechanisms work totally differently, and are
handled by totally different programs.

> from syslog (the segfault in question):
>
> sandbox kernel: [1107554.043727] sslserver[26877]: segfault at
> 5f4c5353
> eip b7e1b34a esp bfd0c838 error 4

look at the value 5f4c5353. this is not a memory location one would
normally expect to see in use, at least not on any system for which
i've ever written and debugged code.

it IS, however, four bytes' worth of ASCII values, namely
"SSL_" (assuming you're using a little-endian processor, such as the
intel x86 series or one of its clones.)

a segfault happens when a process tries to access a memory address for
which the kernel hasn't mapped any physical memory. on most systems, a
process's memory usage would need to be on the order of 1.5GB in order
for such an address to actually have any memory mapped into it.

my guess is that either the patch you're using is broken, or there was
a failed hunk when you applied it, you didn't notice it (or ignored
it) and the resulting code (with most of the patch in there) just
happened to compile without any errors.

> @400000004995e1d12d5cdc94 sslserver: end 26877 status 11

the "status 11" tells you that it segfaulted. read "man 7 signal", i
suspect you will find that on your system, SIGSEGV is number 11.

- ----------------------------------------------------------------
| John M. Simpson --- KG4ZOW --- Programmer At Large |
| http://www.jms1.net/ <jms1 [at] jms1> |
- ----------------------------------------------------------------
| http://video.google.com/videoplay?docid=-1656880303867390173 |
- ----------------------------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAkmap3QACgkQj42MmpAUrRqiBACcCPHqn+JSfRFXCv36JpoW6/IF
o48AoNPKNYqfxlc/8NMfhs1us2isq5WX
=qVoM
-----END PGP SIGNATURE-----

bfaler at holdenlegal

Feb 17, 2009, 10:05 AM

Post #4 of 11 (3278 views)
Permalink

Re: netqmail 1.06 + ucspi-tls problems (sslserver segfault) [In reply to]

John Simpson wrote:
> setting up support for SSL, and setting up support for TLS, are two
> totally different things. it sounds like you have combined the ideas
> in your head, but you really need to consider each thing separately.
> the SSL and STARTTLS mechanisms work totally differently, and are
> handled by totally different programs.
I am aware that there is a significant difference between SSL and
STARTTLS, and currently on my sandbox machine I am only dealing with TLS
at this time. I am also using TLS patch for qmail-smtpd that you
mentioned.. but qmail-smtpd seems to perform like a champ. In fact
everything seems to work perfectly, except for thousands and thousands
of segfault errors showing up my syslog.

> my guess is that either the patch you're using is broken, or there was
> a failed hunk when you applied it, you didn't notice it (or ignored
> it) and the resulting code (with most of the patch in there) just
> happened to compile without any errors.
That was my initial thought as well, which is why I set up a 2nd system
(using the same OS) and redid everything to make sure that the patch
didn't have any failed hunks. The patch to ucspi-ssl I am using is
http://www.suspectclass.com/~sgifford/ucspi-tls/files/ucspi-ssl-0.70-ucspitls-0.1.patch.

It's also probably worth noting I have had all of these services running
on the same hardware using Debian 3.0 as the OS for a few years. I hit
the problem when I tried to update the machine to Ubuntu 8.04.

> the "status 11" tells you that it segfaulted. read "man 7 signal", i
> suspect you will find that on your system, SIGSEGV is number 11.
Interesting.. I never realized that.. You are of course correct.

Thanks for your thoughts!

Brad
bfaler [at] holdenlegal

sgifford at suspectclass

Feb 17, 2009, 10:57 AM

Post #5 of 11 (3281 views)
Permalink

Re: netqmail 1.06 + ucspi-tls problems (sslserver segfault) [In reply to]

Brad Faler <bfaler [at] holdenlegal> writes:

> John Simpson wrote:
>> setting up support for SSL, and setting up support for TLS, are two
>> totally different things. it sounds like you have combined the ideas
>> in your head, but you really need to consider each thing
>> separately. the SSL and STARTTLS mechanisms work totally
>> differently, and are handled by totally different programs.
> I am aware that there is a significant difference between SSL and
> STARTTLS, and currently on my sandbox machine I am only dealing with
> TLS at this time. I am also using TLS patch for qmail-smtpd that you
> mentioned.. but qmail-smtpd seems to perform like a champ. In fact
> everything seems to work perfectly, except for thousands and thousands
> of segfault errors showing up my syslog.

Can you reproduce the problem every time when connecting to your
server with STARTTLS?

Do you see the same problem if you use sslserver directly, without
STARTTLS?

You've tried inreasing the memory limit, right? Are all of your
certificates correct, in the right place, and with the right
permissions?

-----Scott.

bfaler at holdenlegal

Feb 17, 2009, 1:42 PM

Post #6 of 11 (3272 views)
Permalink

Re: netqmail 1.06 + ucspi-tls problems (sslserver segfault) [In reply to]

Scott Gifford wrote:
> Can you reproduce the problem every time when connecting to your
> server with STARTTLS?
>
> Do you see the same problem if you use sslserver directly, without
> STARTTLS?
>
I did some further testing this afternoon and actually it seems the
problem only occurs if the client does NOT use STARTTLS. If the client
connects to the server with TLS enabled everything works perfectly. If
the client does not use STARTTLS then sslserver segfaults after sending
the message (the message does still get delivered to the intended
recipient though).

> You've tried inreasing the memory limit, right? Are all of your
> certificates correct, in the right place, and with the right
> permissions?
>
I have tried raising the memory limit up to 22MB rather than 12MB with
no effect. The certificates were all created following the UCSPI-TLS on
netqmail HOWTO on your website. As far as I can see all the permissions
on them are correct (and I can't imagine things would work as they are
if certificates were bad or had the wrong permissions).

Brad
bfaler [at] holdenlegal

sgifford at suspectclass

Feb 17, 2009, 5:42 PM

Post #7 of 11 (3279 views)
Permalink

Re: netqmail 1.06 + ucspi-tls problems (sslserver segfault) [In reply to]

Brad Faler <bfaler [at] holdenlegal> writes:

> Scott Gifford wrote:
>> Can you reproduce the problem every time when connecting to your
>> server with STARTTLS?
>>
>> Do you see the same problem if you use sslserver directly, without
>> STARTTLS?
>>
> I did some further testing this afternoon and actually it seems the
> problem only occurs if the client does NOT use STARTTLS. If the client
> connects to the server with TLS enabled everything works perfectly. If
> the client does not use STARTTLS then sslserver segfaults after sending
> the message (the message does still get delivered to the intended
> recipient though).

Hrm. So if a client connects to port 25 and activates STARTTLS it
runs OK, but if a client connects to port 25 and does not active
STARTTLS it fails?

Does this happen even if they just connect and disconnect, or connect
and send QUIT? (you can use "openssl s_client -starttls smtp" to test
TLS from the commandline) Or do they have to send a message?

Did the patches to sslserver apply without any errors?

----Scott.

bfaler at holdenlegal

Feb 17, 2009, 8:25 PM

Post #8 of 11 (3266 views)
Permalink

Re: netqmail 1.06 + ucspi-tls problems (sslserver segfault) [In reply to]

Scott Gifford wrote:
> Hrm. So if a client connects to port 25 and activates STARTTLS it
> runs OK, but if a client connects to port 25 and does not active
> STARTTLS it fails?
>
Right. That seems to be what is going on.
> Does this happen even if they just connect and disconnect, or connect
> and send QUIT? (you can use "openssl s_client -starttls smtp" to test
> TLS from the commandline) Or do they have to send a message?
>
>
Yes, this happens regardless if they send a message or not, the segfault
seems to occur once QUIT is sent, which is why I imagine the messages
are going through even when sslserver segfaults.

> Did the patches to sslserver apply without any errors?
>
Yes they did. The system I am working with at the moment is a clean
install of netqmail 1.06 and ucspi-ssl along with the ucspi-tls patches
to qmail and ucspi-ssl. The original system I tried to get this working
on had quite a few other patches which required some patches to be
applied manually.. but in order to try to track the problem down I have
tried to keep things on this machine as simple as possible.

Brad
bfaler [at] holdenlegal

sgifford at suspectclass

Feb 17, 2009, 9:15 PM

Post #9 of 11 (3247 views)
Permalink

Re: netqmail 1.06 + ucspi-tls problems (sslserver segfault) [In reply to]

Brad Faler <bfaler [at] holdenlegal> writes:

> Scott Gifford wrote:
>> Hrm. So if a client connects to port 25 and activates STARTTLS it
>> runs OK, but if a client connects to port 25 and does not active
>> STARTTLS it fails?
>>
> Right. That seems to be what is going on.
>> Does this happen even if they just connect and disconnect, or connect
>> and send QUIT? (you can use "openssl s_client -starttls smtp" to test
>> TLS from the commandline) Or do they have to send a message?
>>
>>
> Yes, this happens regardless if they send a message or not, the
> segfault seems to occur once QUIT is sent, which is why I imagine the
> messages are going through even when sslserver segfaults.

Since you can reproduce it consistently, any chance of running it
under a debugger and seeing what's going on?

----Scott.

feh at fehcom

Feb 18, 2009, 12:13 AM

Post #10 of 11 (3269 views)
Permalink

Re: netqmail 1.06 + ucspi-tls problems (sslserver segfault) [In reply to]

Hi,

just for curiosity:

1. Download SPAMCONTROL 2.5 and install against qmail-1.03.

2. Patch ucspi-ssl (cleanly) with the provided patch (part of SPAMCONTROL's
delivery).

3. Bring up the system (you don't need to change any settings except
perhaps including UCSIPITLS="" in the qmail-smtpd environment).

Rational: SPAMCONTROL includes Scott's STARTTLS implementation (with some
minor changes).

Test your connection with and without STARTTLS invoked from a client.

Do you see any change regarding the error ?

Check qmail-smtpd logs: They are now much more verbose.

Send us your setting and the relevant pieces of the logs.

Which version of OpenSSL do use use ?

regards.
--eh.

--On Dienstag, 17. Februar 2009 21:25 -0700 Brad Faler
<bfaler [at] holdenlegal> wrote:

> Scott Gifford wrote:
>> Hrm. So if a client connects to port 25 and activates STARTTLS it
>> runs OK, but if a client connects to port 25 and does not active
>> STARTTLS it fails?
>>
> Right. That seems to be what is going on.
>> Does this happen even if they just connect and disconnect, or connect
>> and send QUIT? (you can use "openssl s_client -starttls smtp" to test
>> TLS from the commandline) Or do they have to send a message?
>>
>>
> Yes, this happens regardless if they send a message or not, the segfault
> seems to occur once QUIT is sent, which is why I imagine the messages are
> going through even when sslserver segfaults.
>
>
>> Did the patches to sslserver apply without any errors?
>>
> Yes they did. The system I am working with at the moment is a clean
> install of netqmail 1.06 and ucspi-ssl along with the ucspi-tls patches
> to qmail and ucspi-ssl. The original system I tried to get this working
> on had quite a few other patches which required some patches to be
> applied manually.. but in order to try to track the problem down I have
> tried to keep things on this machine as simple as possible.
>
> Brad
> bfaler [at] holdenlegal
>
>

jms1 at jms1

Feb 18, 2009, 12:52 AM

Post #11 of 11 (3270 views)
Permalink

Re: netqmail 1.06 + ucspi-tls problems (sslserver segfault) [In reply to]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2009-02-17, at 1305, Brad Faler wrote:
>
>> my guess is that either the patch you're using is broken, or there
>> was a failed hunk when you applied it, you didn't notice it (or
>> ignored it) and the resulting code (with most of the patch in
>> there) just happened to compile without any errors.
>
> That was my initial thought as well, which is why I set up a 2nd
> system (using the same OS) and redid everything to make sure that
> the patch didn't have any failed hunks. The patch to ucspi-ssl I am
> using is http://www.suspectclass.com/~sgifford/ucspi-tls/files/ucspi-ssl-0.70-ucspitls-0.1.patch
> .

i remember looking at this a while back... i haven't actually used it,
but my gut feeling is that it's more complex than it needs to be. i
just looked at the patch again, and while i do understand what it's
doing, i think it's un-necessarily complicated. the fact is that the
STARTTLS patch for qmail-smtpd, which puts the SSL code into qmail-
smtpd itself, has been working for myself and for thousands of others,
for several years.

it's an interesting idea, and i'm sure that when it's configured
correctly it does work (otherwise he wouldn't have released it) but
it's not for me. i already have a solution to the same problem.

i see that the author of that patch has joined the thread, i'll let
him take it from here. good luck to you.

- ----------------------------------------------------------------
| John M. Simpson --- KG4ZOW --- Programmer At Large |
| http://www.jms1.net/ <jms1 [at] jms1> |
- ----------------------------------------------------------------
| http://video.google.com/videoplay?docid=-1656880303867390173 |
- ----------------------------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAkmbzFIACgkQj42MmpAUrRp1JwCfW1Jo2MapGhSg8q5bltQSr9z6
RmcAoIDSxVr/2l71oi+YwmbepDRJMImV
=MvDt
-----END PGP SIGNATURE-----

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads