Mailing List Archive: Apache: Dev

Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12

1 2

View All

Index | Next | Previous | View Threaded

kamesh at collab

Oct 21, 2009, 6:33 AM

Post #1 of 36 (4191 views)
Permalink

Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12

Hi All,

We observe one strange error since exhibited in combination with
SVN(with bulk import having more than 20k files).

Original posting is at
http://subversion.tigris.org/ds/viewMessage.do?dsMessageId=2379671&dsForumId=462

The problem exists even in httpd-2.2.13 and httpd-2.2.14.

We get errors like the following

svn: PUT of '/svn/svntest/!svn/wrk/fca6bd35-b260-7942-8f52-bcf3dcdfd734/abc/trunk/publish/q/xyz.gz': SSL negotiation failed: SSL error:
parse tlsext (https://hostname <https://cu097.cubit.maa.collab.net>)

It happens only with windows client, server can be linux or win32.

I could manage to get the stack trace of apache child(in apache-2.2.13)
when this error occurs.

**

<stack trace of apache 2.2.13 when we get this tlsext parse error>
#0 ssl_filter_io_shutdown (filter_ctx=0xa07b910, c=0xa07b350, abortive=1)
at /home/kamesh/Download/httpd-2.2.13/modules/ssl/ssl_engine_io.c:976
#1 0x0038d5eb in ssl_io_filter_connect (filter_ctx=0xa07b910)
at /home/kamesh/Download/httpd-2.2.13/modules/ssl/ssl_engine_io.c:1146
#2 0x0038dc1d in ssl_io_filter_input (f=0xa08c898, bb=0xa0d2ac8, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0)
at /home/kamesh/Download/httpd-2.2.13/modules/ssl/ssl_engine_io.c:1336
#3 0x08086af9 in ap_get_brigade (next=0xa08c898, bb=0xa0d2ac8, mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0)
at /home/kamesh/Download/httpd-2.2.13/server/util_filter.c:489
#4 0x0806b274 in ap_rgetline_core (s=0xa0d1c78, n=8192, read=0xbf837c14, r=0xa0d1c60, fold=0, bb=0xa0d2ac8)
at /home/kamesh/Download/httpd-2.2.13/server/protocol.c:231
#5 0x0806b943 in read_request_line (r=0xa0d1c60, bb=0xa0d2ac8) at /home/kamesh/Download/httpd-2.2.13/server/protocol.c:596
#6 0x0806c299 in ap_read_request (conn=0xa07b350) at /home/kamesh/Download/httpd-2.2.13/server/protocol.c:891
#7 0x0808726e in ap_process_http_connection (c=0xa07b350)
at /home/kamesh/Download/httpd-2.2.13/modules/http/http_core.c:183
#8 0x08082c73 in ap_run_process_connection (c=0xa07b350) at /home/kamesh/Download/httpd-2.2.13/server/connection.c:43
#9 0x08083053 in ap_process_connection (c=0xa07b350, csd=0xa07b1b8)
at /home/kamesh/Download/httpd-2.2.13/server/connection.c:178
#10 0x080901df in child_main (child_num_arg=0) at /home/kamesh/Download/httpd-2.2.13/server/mpm/prefork/prefork.c:662
#11 0x080903ca in make_child (s=0x9f70fa0, slot=0) at /home/kamesh/Download/httpd-2.2.13/server/mpm/prefork/prefork.c:758
#12 0x08090424 in startup_children (number_to_start=1)
at /home/kamesh/Download/httpd-2.2.13/server/mpm/prefork/prefork.c:776
#13 0x080908c8 in ap_mpm_run (_pconf=0x9f6f0a8, plog=0x9f9d160, s=0x9f70fa0)
at /home/kamesh/Download/httpd-2.2.13/server/mpm/prefork/prefork.c:997
#14 0x08064bb8 in main (argc=3, argv=0xbf837fe4) at /home/kamesh/Download/httpd-2.2.13/server/main.c:740
</snip>

**

<snip from error log while this error happened last week>
[Sat Oct 10 20:41:18 2009] [debug] ssl_engine_io.c(1858): OpenSSL: read 5/5 bytes from BIO#8494dd0 [mem: 835bb00] (BIO dump follows)
[Sat Oct 10 20:41:18 2009] [debug] ssl_engine_io.c(1791): +-------------------------------------------------------------------------+
[Sat Oct 10 20:41:18 2009] [debug] ssl_engine_io.c(1830): | 0000: 15 03 01 00 02 ..... |
[Sat Oct 10 20:41:18 2009] [debug] ssl_engine_io.c(1836): +-------------------------------------------------------------------------+
[Sat Oct 10 20:41:18 2009] [debug] ssl_engine_io.c(1858): OpenSSL: read 2/2 bytes from BIO#8494dd0 [mem: 835bb05] (BIO dump follows)
[Sat Oct 10 20:41:18 2009] [debug] ssl_engine_io.c(1791): +-------------------------------------------------------------------------+
[Sat Oct 10 20:41:18 2009] [debug] ssl_engine_io.c(1830): | 0000: 02 32 .2 |
[Sat Oct 10 20:41:18 2009] [debug] ssl_engine_io.c(1836): +-------------------------------------------------------------------------+
[Sat Oct 10 20:41:18 2009] [debug] ssl_engine_kernel.c(1888): OpenSSL: Read: SSLv3 read client certificate A
[Sat Oct 10 20:41:18 2009] [debug] ssl_engine_kernel.c(1907): OpenSSL: Exit: failed in SSLv3 read client certificate A
[Sat Oct 10 20:41:18 2009] [info] [client IP] SSL library error 1 in handshake (server hostname:443)
[Sat Oct 10 20:41:18 2009] [info] SSL Library Error: 336151578 error:1409441A:SSL routines:SSL3_READ_BYTES:tlsv1 alert decode error
[Sat Oct 10 20:41:18 2009] [info] [client IP] Connection closed to child 5 with abortive shutdown (server hostname:443)
</snip>

I could not isolate this issue to openssl versions as it happens with
openssl-0.9.8k, openssl-0.9.8g, openssl-0.9.8-b

When I built the server against openssl-1.0.0-beta3, I could *not*
access svn at all using svn client while I could access the same via
browser.

Any clues?

With regards

Kamesh Jayachandran

fuankg at apache

Oct 21, 2009, 7:57 AM

Post #2 of 36 (4104 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Hi Kamesh,
nice to meet you here again!
Kamesh Jayachandran schrieb:
> I could not isolate this issue to openssl versions as it happens with
> openssl-0.9.8k, openssl-0.9.8g, openssl-0.9.8-b
>
> When I built the server against openssl-1.0.0-beta3, I could *not*
> access svn at all using svn client while I could access the same via
> browser.
>
> Any clues?
sounds all strange. I would say since we have SNI support since 2.2.12
that there is the problem, and from the bug report it seems that the OP
used already 2 SSL virtual hosts with same IP before 2.2.12 which was
neither supported feature nor it was working properly at all; so
probably his configuration is the problem?
On the other side the needed support in OpenSSL started with 0.9.8j
IIRC, and with 0.9.8k it started to be enabled by default. So I would
assume that builds with 0.9.8g and 0.9.8b are not affected ...
Also since you post that the problem is with the client - did you also
rebuild the client with newer OpenSSL 0.8.8k, or even with 1.0.0.b3?

G�nter.

kamesh at collab

Oct 21, 2009, 9:40 AM

Post #3 of 36 (4092 views)
Permalink

RE: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Hi Gunter,

Nice to meet you after a long time.

>sounds all strange. I would say since we have SNI support since 2.2.12
>that there is the problem, and from the bug report it seems that the OP
>used already 2 SSL virtual hosts with same IP before 2.2.12 which was
>neither supported feature nor it was working properly at all; so
>probably his configuration is the problem?

In my setup where this fails has only *one* SSL virtual host(_default_).

>On the other side the needed support in OpenSSL started with 0.9.8j
>IIRC, and with 0.9.8k it started to be enabled by default. So I would
>assume that builds with 0.9.8g and 0.9.8b are not affected ...

I need to double check it by myself(One of the internal tester was saying that this happens with openssl-0.9.8b).
I vaguely remember this happening with openssl-0.9.8g.

>Also since you post that the problem is with the client - did you also
>rebuild the client with newer OpenSSL 0.8.8k, or even with 1.0.0.b3?

Will experiment and get back.

With regards
Kamesh Jayachandran

httpd-dev.2009 at velox

Oct 21, 2009, 9:59 AM

Post #4 of 36 (4101 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Kamesh Jayachandran wrote:
> When I built the server against openssl-1.0.0-beta3, I could *not*
> access svn at all using svn client while I could access the same via
> browser.
>
> Any clues?

The TLS session ticket extension might be the culprit here (or more
precisely, OpenSSL's implementation of that extension). Can you try the
attached patch and see whether it makes a difference?

Kaspar

Attachments:

mod_ssl-disable_tls_tickets.diff (0.38 KB)

kamesh at collab

Oct 21, 2009, 11:39 AM

Post #5 of 36 (4092 views)
Permalink

RE: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Thanks Kaspar, will try that tomorrow(Right now away from my dev box) and let you know.

With regards
Kamesh Jayachandran

-----Original Message-----
From: Kaspar Brand [mailto:httpd-dev.2009 [at] velox]
Sent: Wed 10/21/2009 10:29 PM
To: dev [at] httpd
Subject: Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12

Kamesh Jayachandran wrote:
> When I built the server against openssl-1.0.0-beta3, I could *not*
> access svn at all using svn client while I could access the same via
> browser.
>
> Any clues?

The TLS session ticket extension might be the culprit here (or more
precisely, OpenSSL's implementation of that extension). Can you try the
attached patch and see whether it makes a difference?

Kaspar

kamesh at collab

Oct 22, 2009, 12:19 AM

Post #6 of 36 (4079 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

On 10/21/2009 10:29 PM, Kaspar Brand wrote:
> Kamesh Jayachandran wrote:
>
>> When I built the server against openssl-1.0.0-beta3, I could *not*
>> access svn at all using svn client while I could access the same via
>> browser.
>>
>> Any clues?
>>
> The TLS session ticket extension might be the culprit here (or more
> precisely, OpenSSL's implementation of that extension). Can you try the
> attached patch and see whether it makes a difference?
>

Hi Kaspar,

I tried your patch. It does *not* fix the issue.
One difference it makes is , triggers failure early at 20/30 files(PUT
requests) instead of 20k files earlier.

With regards
Kamesh Jayachandran

jorton at redhat

Oct 22, 2009, 1:39 AM

Post #7 of 36 (4080 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

On Thu, Oct 22, 2009 at 12:49:10PM +0530, Kamesh Jayachandran wrote:
> I tried your patch. It does *not* fix the issue.
> One difference it makes is , triggers failure early at 20/30 files(PUT
> requests) instead of 20k files earlier.

Can you get a packet dump/trace from the client side? Is there anything
between client and server which is intercepting the SSL traffic?
(physical/software firewall?) It would be good whether this problem is
due to the traffic becoming corrupted.

There seem to be two places in OpenSSL's ssl_parse_serverhello_tlsext()
which can send a "decode error" alert, if I am reading the code and
following the error handling correctly. It would be useful if you could
use a custom OpenSSL build with an fprintf(stderr, ... ) or similar
added before each of the "*al = SSL_AD_DECODE_ERROR;" lines in that
function (in ssl/t1_lib.c), if you're able to try that?

Regards, Joe

kamesh at collab

Oct 22, 2009, 4:34 AM

Post #8 of 36 (4072 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

> I need to double check it by myself(One of the internal tester was
> saying that this happens with openssl-0.9.8b).
> I vaguely remember this happening with openssl-0.9.8g.
>
>

Unfortunately I could not build/use openssl-0.9.8b and openssl-0.9.8e on
my box.

I get the following error for which I do not know *solution* right now,

[kamesh [at] kames openssl-0.9.8e]$ /home/kamesh/openssl-0.9.8e/bin/openssl
genrsa 1024 -out /tmp/server.key
Generating RSA private key, 1024 bit long modulus
................++++++
....++++++
e is 65537 (0x10001)
Illegal instruction

> >Also since you post that the problem is with the client - did you also
> >rebuild the client with newer OpenSSL 0.8.8k, or even with 1.0.0.b3?
>
> Will experiment and get back.
>

I do not have win32 build system to build svn clients against openssl
1.0.0.b3

With regards
Kamesh Jayachandran

shenson at oss-institute

Oct 22, 2009, 4:54 AM

Post #9 of 36 (4070 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Kamesh Jayachandran wrote:
>
>> I need to double check it by myself(One of the internal tester was
>> saying that this happens with openssl-0.9.8b).
>> I vaguely remember this happening with openssl-0.9.8g.
>>
>>
>
> Unfortunately I could not build/use openssl-0.9.8b and openssl-0.9.8e on
> my box.
>
> I get the following error for which I do not know *solution* right now,
>
>
> [kamesh [at] kames openssl-0.9.8e]$ /home/kamesh/openssl-0.9.8e/bin/openssl
> genrsa 1024 -out /tmp/server.key
> Generating RSA private key, 1024 bit long modulus
> ................++++++
> ....++++++
> e is 65537 (0x10001)
> Illegal instruction
>
>

That's due to the function pointer issues which gcc 4.2 and later doesn't like:
this was fixed in newer versions of OpenSSL.

Do you need TLS extensions on the client/server? If not try compiling OpenSSL
with no-tlsext.

Did you say what version of OpenSSL the failing client was using on Windows?

It is strange that this doesn't happen straight away.

Steve.
--
Dr Stephen N. Henson. Senior Technical/Cryptography Advisor,
Open Source Software Institute: www.oss-institute.org
OpenSSL Core team: www.openssl.org

kamesh at collab

Oct 22, 2009, 5:12 AM

Post #10 of 36 (4069 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

On 10/22/2009 05:24 PM, Dr Stephen Henson wrote:
> Kamesh Jayachandran wrote:
>
>>
>>> I need to double check it by myself(One of the internal tester was
>>> saying that this happens with openssl-0.9.8b).
>>> I vaguely remember this happening with openssl-0.9.8g.
>>>
>>>
>>>
>> Unfortunately I could not build/use openssl-0.9.8b and openssl-0.9.8e on
>> my box.
>>
>> I get the following error for which I do not know *solution* right now,
>>
>>
>> [kamesh [at] kames openssl-0.9.8e]$ /home/kamesh/openssl-0.9.8e/bin/openssl
>> genrsa 1024 -out /tmp/server.key
>> Generating RSA private key, 1024 bit long modulus
>> ................++++++
>> ....++++++
>> e is 65537 (0x10001)
>> Illegal instruction
>>
>>
>>
> That's due to the function pointer issues which gcc 4.2 and later doesn't like:
> this was fixed in newer versions of OpenSSL.
>
>

Is there any switch we can pass to gcc 4.2 to compile and make it work
properly.

> Do you need TLS extensions on the client/server? If not try compiling OpenSSL
> with no-tlsext.
>

May not be possible as *client* builds are not in our control.

I believe no-tlsext does *not* disable TLS functionality itself.

> Did you say what version of OpenSSL the failing client was using on Windows?
>
>

It happens with openssl-0.9.8j on client openssl-0.9.8k on server

With regards
Kamesh Jayachandran

shenson at oss-institute

Oct 22, 2009, 10:07 AM

Post #11 of 36 (4062 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Kamesh Jayachandran wrote:
> On 10/22/2009 05:24 PM, Dr Stephen Henson wrote:
>> That's due to the function pointer issues which gcc 4.2 and later
>> doesn't like:
>> this was fixed in newer versions of OpenSSL.
>>
>>
>
> Is there any switch we can pass to gcc 4.2 to compile and make it work
> properly.
>

No. If you really want to use 0.9.8b it needs an older version of gcc or you can
backport the fixes.

They are rather extensive but mainly contained in:

http://cvs.openssl.org/chngview?cn=16526

and

http://cvs.openssl.org/chngview?cn=16528

OpenSSL 0.9.8b doesn't use TLS extensions at all.

>> Do you need TLS extensions on the client/server? If not try compiling
>> OpenSSL
>> with no-tlsext.
>>
>
> May not be possible as *client* builds are not in our control.
>
> I believe no-tlsext does *not* disable TLS functionality itself.
>

The no-tlsext option disables TLS extension functionality. If that works on the
server side then an alternative workaround could be found.

>> Did you say what version of OpenSSL the failing client was using on
>> Windows?
>>
>>
>
> It happens with openssl-0.9.8j on client openssl-0.9.8k on server
>

Hmm... could be 0.9.8j sending bad data with invalid extension syntax under rare
circumstances.

A packet sniffer or logging the errant extensions received by OpenSSL could help
trace this further.

Steve.
--
Dr Stephen N. Henson. Senior Technical/Cryptography Advisor,
Open Source Software Institute: www.oss-institute.org
OpenSSL Core team: www.openssl.org

kamesh at collab

Oct 23, 2009, 6:22 AM

Post #12 of 36 (4040 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

On 10/22/2009 10:37 PM, Dr Stephen Henson wrote:
> Kamesh Jayachandran wrote:
>
>> On 10/22/2009 05:24 PM, Dr Stephen Henson wrote:
>>
>>> That's due to the function pointer issues which gcc 4.2 and later
>>> doesn't like:
>>> this was fixed in newer versions of OpenSSL.
>>>
>>>
>>>
>> Is there any switch we can pass to gcc 4.2 to compile and make it work
>> properly.
>>
>>
> No. If you really want to use 0.9.8b it needs an older version of gcc or you can
> backport the fixes.
>
> They are rather extensive but mainly contained in:
>
> http://cvs.openssl.org/chngview?cn=16526
>
> and
>
> http://cvs.openssl.org/chngview?cn=16528
>
> OpenSSL 0.9.8b doesn't use TLS extensions at all.
>
>
>>> Do you need TLS extensions on the client/server? If not try compiling
>>> OpenSSL
>>> with no-tlsext.
>>>
>>>
>> May not be possible as *client* builds are not in our control.
>>
>> I believe no-tlsext does *not* disable TLS functionality itself.
>>
>>
> The no-tlsext option disables TLS extension functionality. If that works on the
> server side then an alternative workaround could be found.
>
>

Yes, building 'openssl-0.9.8k' on the server side with no-tlsext works
around this issue irrespective of the client openssl.

>>> Did you say what version of OpenSSL the failing client was using on
>>> Windows?
>>>
>>>
>>>
>> It happens with openssl-0.9.8j on client openssl-0.9.8k on server
>>
>>
> Hmm... could be 0.9.8j sending bad data with invalid extension syntax under rare
> circumstances.
>
> A packet sniffer or logging the errant extensions received by OpenSSL could help
> trace this further.
>

Will post tcpdump once I set it up on win32.

Thanks.

With regards
Kamesh Jayachandran

kamesh at collab

Oct 23, 2009, 7:18 AM

Post #13 of 36 (4039 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

>>> Did you say what version of OpenSSL the failing client was using on
>>> Windows?
>>>
>>>
>>>
>> It happens with openssl-0.9.8j on client openssl-0.9.8k on server
>>
>>
> Hmm... could be 0.9.8j sending bad data with invalid extension syntax under rare
> circumstances.
>
> A packet sniffer or logging the errant extensions received by OpenSSL could help
> trace this further.
>
>

Find the tcpdump while this failure occurs at
http://www.livecipher.com/tlsext_dump/tlsext.dmp

Thanks

with regards
Kamesh Jayachandran

kamesh at collab

Oct 23, 2009, 7:23 AM

Post #14 of 36 (4033 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

On 10/22/2009 02:09 PM, Joe Orton wrote:
> On Thu, Oct 22, 2009 at 12:49:10PM +0530, Kamesh Jayachandran wrote:
>
>> I tried your patch. It does *not* fix the issue.
>> One difference it makes is , triggers failure early at 20/30 files(PUT
>> requests) instead of 20k files earlier.
>>
> Can you get a packet dump/trace from the client side? Is there anything
> between client and server which is intercepting the SSL traffic?
> (physical/software firewall?) It would be good whether this problem is
> due to the traffic becoming corrupted.
>

Find the tcpdump while this failure occurs at
http://www.livecipher.com/tlsext_dump/tlsext.dmp

I could not suspect the firewall as this occurs only with
httpd-2.2.12+openssl-with-tls-ext *not* with httpd-2.2.11 or
httpd-2.2.13+openssl-without-tls-ext.

Thanks

> There seem to be two places in OpenSSL's ssl_parse_serverhello_tlsext()
> which can send a "decode error" alert, if I am reading the code and
> following the error handling correctly. It would be useful if you could
> use a custom OpenSSL build with an fprintf(stderr, ... ) or similar
> added before each of the "*al = SSL_AD_DECODE_ERROR;" lines in that
> function (in ssl/t1_lib.c), if you're able to try that?
>
> Regards, Joe
>
>
>

Will try this next week as it involves building in win32 which I am not
used to.

With regards
Kamesh Jayachandran

httpd-dev.2009 at velox

Oct 23, 2009, 9:08 AM

Post #15 of 36 (4040 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Kamesh Jayachandran wrote:
> Find the tcpdump while this failure occurs at
> http://www.livecipher.com/tlsext_dump/tlsext.dmp

It seems that you used a URI with an IP address (https://10.2.1.97/...),
is that correct? This actually uncovers a - probably unrelated - bug in
the OpenSSL client (SNI extensions should never contain literal IPv4
addresses).

Could you retry the test and make sure that you use an FQDN in the URI
you specify for the client (through an entry in the hosts file or so)?

Kaspar

kamesh at collab

Oct 23, 2009, 11:44 AM

Post #16 of 36 (4027 views)
Permalink

RE: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

>It seems that you used a URI with an IP address (https://10.2.1.97/...),
>is that correct?

Yes.

>Could you retry the test and make sure that you use an FQDN in the URI
>you specify for the client (through an entry in the hosts file or so)?

Yes done, find the dump at http://www.livecipher.com/tlsext_dump/tlsext.dmp.2

Thanks
With regards
Kamesh Jayachandran

httpd-dev.2009 at velox

Oct 24, 2009, 1:58 AM

Post #17 of 36 (4017 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Kamesh Jayachandran wrote:
> Yes done, find the dump at http://www.livecipher.com/tlsext_dump/tlsext.dmp.2

Ok, thanks. So, for the sake of reference, your setup for this capture
was:

- (Windows) client with OpenSSL 0.9.8k, compiled with defaults
- server with OpenSSL 0.9.8j, compiled with defaults
- httpd 2.2.14, w/o the OP_NO_TICKET patch

Is that correct?

I have extracted the two Hello messages into a separate pcap file (attached).
It's the ServerHello in the second packet to which the client then immediately
replies to with SSL_AD_DECODE_ERROR. I'm also attaching Wireshark's
interpretation of the bytes in question, followed by their hex dumps.

As Joe observed in an earlier message, there are only two places in
t1_lib.c:ssl_parse_serverhello_tlsext() which set SSL_AD_DECODE_ERROR,
and it seems very likely that the following code is hit:

> if (!s->hit && tlsext_servername == 1)
> {
> if (s->tlsext_hostname)
> {
> if (s->session->tlsext_hostname == NULL)
> {
> s->session->tlsext_hostname = BUF_strdup(s->tlsext_hostname);
> if (!s->session->tlsext_hostname)
> {
> *al = SSL_AD_UNRECOGNIZED_NAME;
> return 0;
> }
> }
> else
> {
> *al = SSL_AD_DECODE_ERROR;
> return 0;
> }
> }
> }

I'll defer to the OpenSSL gurus for further analysis, but apparently
it has to do with the fact that the client reuses an SSL session
(s->session->tlsext_hostname is non-null), but this case is not
expected by ssl_parse_serverhello_tlsext(). It would also explain,
however, why the issue does not show up immediately, but only after
a couple of minutes (i.e., when a session is really being reused).

Kaspar

ClientHello: includes both an SNI and a SessionTicket extension
(i.e., client tries a stateless resume)

TLSv1 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.0 (0x0301)
Length: 316
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 312
Version: TLS 1.0 (0x0301)
Random
gmt_unix_time: Sep 1, 2009 18:24:07.000000000
random_bytes: DC61CBA8386F040540B164CA43C99C0A25618A34DC84B0CF...
Session ID Length: 0
Cipher Suites Length: 40
Cipher Suites (20 suites)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
Cipher Suite: TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016)
Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)
Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
Cipher Suite: TLS_RSA_WITH_IDEA_CBC_SHA (0x0007)
Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)
Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004)
Cipher Suite: TLS_DHE_RSA_WITH_DES_CBC_SHA (0x0015)
Cipher Suite: TLS_DHE_DSS_WITH_DES_CBC_SHA (0x0012)
Cipher Suite: TLS_RSA_WITH_DES_CBC_SHA (0x0009)
Cipher Suite: TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA (0x0014)
Cipher Suite: TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA (0x0011)
Cipher Suite: TLS_RSA_EXPORT_WITH_DES40_CBC_SHA (0x0008)
Cipher Suite: TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5 (0x0006)
Cipher Suite: TLS_RSA_EXPORT_WITH_RC4_40_MD5 (0x0003)
Compression Methods Length: 1
Compression Methods (1 method)
Compression Method: null (0)
Extensions Length: 231
Extension: server_name
Type: server_name (0x0000)
Length: 15
Data (15 bytes)
Extension: SessionTicket TLS
Type: SessionTicket TLS (0x0023)
Length: 208
Data (208 bytes)

0030 01 00 01 38 03 ...8.
0040 01 4a 9d 4a a7 dc 61 cb a8 38 6f 04 05 40 b1 64 .J.J..a..8o..@.d
0050 ca 43 c9 9c 0a 25 61 8a 34 dc 84 b0 cf e2 f2 0c .C...%a.4.......
0060 ff 00 00 28 00 39 00 38 00 35 00 16 00 13 00 0a ...(.9.8.5......
0070 00 33 00 32 00 2f 00 07 00 05 00 04 00 15 00 12 .3.2./..........
0080 00 09 00 14 00 11 00 08 00 06 00 03 01 00 00 e7 ................
0090 00 00 00 0f 00 0d 00 00 0a 6b 61 6d 65 73 68 2e .........kamesh.
00a0 63 6f 6d 00 23 00 d0 cf 41 ef 4a 9b 41 98 ca 3b com.#...A.J.A..;
00b0 11 01 5a 16 6d ed a0 86 56 79 24 5c a7 1e 3e ec ..Z.m...Vy$\..>.
00c0 47 27 d3 07 47 f7 17 04 3a 4e b3 64 bb de e5 ca G'..G...:N.d....
00d0 aa 48 1c 61 2c e4 80 df c0 f7 be 7e 81 b9 0c ad .H.a,......~....
00e0 03 8f f1 df 0b 89 80 d1 7a de 5c 21 af ff b0 94 ........z.\!....
00f0 67 83 b7 91 67 b8 b3 8b e7 af de 6e 53 37 05 89 g...g......nS7..
0100 e8 b8 a0 2f 39 ec 38 9f fa 35 7f 41 66 b1 57 fa .../9.8..5.Af.W.
0110 36 97 64 c9 39 22 9c f2 75 56 0d 4c b7 75 36 0f 6.d.9"..uV.L.u6.
0120 22 a2 5b 28 31 ff 26 26 5b 9f d9 ee 76 24 65 48 ".[.(1.&&[....v$eH
0130 1d ad 13 8e d3 5d 8b f9 09 ea e8 a4 6c 8f c2 ae .....]......l...
0140 73 67 e0 c4 a3 fd 13 cc 37 a9 20 eb 35 ba 66 86 sg......7. .5.f.
0150 3e 0e 5e af d8 6c df ba 8c ef 69 1a c1 fd 07 e9 >.^..l....i.....
0160 7a 44 04 b9 0f 70 90 b6 fe ff dc 49 f6 49 44 4c zD...p.....I.IDL
0170 a1 03 e9 5b 12 f6 30 ...[...0

-----

ServerHello: includes an acknowledgement for the SNI extension
(with an empty "extension_data" field, as specified in RFC 4366).

TLSv1 Record Layer: Handshake Protocol: Server Hello
Content Type: Handshake (22)
Version: TLS 1.0 (0x0301)
Length: 48
Handshake Protocol: Server Hello
Handshake Type: Server Hello (2)
Length: 44
Version: TLS 1.0 (0x0301)
Random
gmt_unix_time: Oct 23, 2009 18:26:28.000000000
random_bytes: 7BB7CEA14E031A61466D401D6633DD50D4B2513B845D2D0A...
Session ID Length: 0
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
Compression Method: null (0)
Extensions Length: 4
Extension: server_name
Type: server_name (0x0000)
Length: 0
Data (0 bytes)

0030 02 00 00 2c 03 ...,.
0040 01 4a e1 d9 34 7b b7 ce a1 4e 03 1a 61 46 6d 40 .J..4{...N..aFm@
0050 1d 66 33 dd 50 d4 b2 51 3b 84 5d 2d 0a 1f cd bb .f3.P..Q;.]-....
0060 f1 00 00 39 00 00 04 00 00 00 00 ...9.......

Attachments:

hello-messages.pcap (1.69 KB)

kamesh at collab

Oct 24, 2009, 4:15 AM

Post #18 of 36 (4009 views)
Permalink

RE: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

>Ok, thanks. So, for the sake of reference, your setup for this capture
>was:

>- (Windows) client with OpenSSL 0.9.8k, compiled with defaults
>- server with OpenSSL 0.9.8j, compiled with defaults
>- httpd 2.2.14, w/o the OP_NO_TICKET patch

>Is that correct?

No.
My setup is,

- win32 client with openssl 0.9.8j compiled with defaults.
- server with OpenSSL 0.9.8k, compiled with defaults
- httpd 2.2.12, w/o the OP_NO_TICKET patch

With regards
Kamesh Jayachandran

shenson at oss-institute

Oct 24, 2009, 6:13 AM

Post #19 of 36 (4008 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Kaspar Brand wrote:
>
> As Joe observed in an earlier message, there are only two places in
> t1_lib.c:ssl_parse_serverhello_tlsext() which set SSL_AD_DECODE_ERROR,
> and it seems very likely that the following code is hit:
>
>> if (!s->hit && tlsext_servername == 1)
>> {
>> if (s->tlsext_hostname)
>> {
>> if (s->session->tlsext_hostname == NULL)
>> {
>> s->session->tlsext_hostname = BUF_strdup(s->tlsext_hostname);
>> if (!s->session->tlsext_hostname)
>> {
>> *al = SSL_AD_UNRECOGNIZED_NAME;
>> return 0;
>> }
>> }
>> else
>> {
>> *al = SSL_AD_DECODE_ERROR;
>> return 0;
>> }
>> }
>> }
>
> I'll defer to the OpenSSL gurus for further analysis, but apparently
> it has to do with the fact that the client reuses an SSL session
> (s->session->tlsext_hostname is non-null), but this case is not
> expected by ssl_parse_serverhello_tlsext(). It would also explain,
> however, why the issue does not show up immediately, but only after
> a couple of minutes (i.e., when a session is really being reused).
>

OK that makes sense. I can now reproduce this with OpenSSL s_server and
s_client and stateless session resumption. A fix for this would have to be in
the client code, I'll look into that.

It doesn't occur when tickets are disabled so setting the SSL_OP_NO_TICKET
should be a workaround on the server as long as it is set in all relevant
SSL_CTX structures.

Steve.
--
Dr Stephen N. Henson. Senior Technical/Cryptography Advisor,
Open Source Software Institute: www.oss-institute.org
OpenSSL Core team: www.openssl.org

jorton at redhat

Oct 25, 2009, 4:21 AM

Post #20 of 36 (3986 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

On Fri, Oct 23, 2009 at 06:08:52PM +0200, Kaspar Brand wrote:
> Kamesh Jayachandran wrote:
> > Find the tcpdump while this failure occurs at
> > http://www.livecipher.com/tlsext_dump/tlsext.dmp
>
> It seems that you used a URI with an IP address (https://10.2.1.97/...),
> is that correct? This actually uncovers a - probably unrelated - bug in
> the OpenSSL client (SNI extensions should never contain literal IPv4
> addresses).

Good point - I've changed neon for future releases to only enable SNI if
the hostname is not a numeric IP address.

Regards, Joe

shenson at oss-institute

Oct 25, 2009, 6:55 AM

Post #21 of 36 (3981 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Dr Stephen Henson wrote:
> Kaspar Brand wrote:
>> As Joe observed in an earlier message, there are only two places in
>> t1_lib.c:ssl_parse_serverhello_tlsext() which set SSL_AD_DECODE_ERROR,
>> and it seems very likely that the following code is hit:
>>
>>> if (!s->hit && tlsext_servername == 1)
>>> {
>>> if (s->tlsext_hostname)
>>> {
>>> if (s->session->tlsext_hostname == NULL)
>>> {
>>> s->session->tlsext_hostname = BUF_strdup(s->tlsext_hostname);
>>> if (!s->session->tlsext_hostname)
>>> {
>>> *al = SSL_AD_UNRECOGNIZED_NAME;
>>> return 0;
>>> }
>>> }
>>> else
>>> {
>>> *al = SSL_AD_DECODE_ERROR;
>>> return 0;
>>> }
>>> }
>>> }
>
> OK that makes sense. I can now reproduce this with OpenSSL s_server and
> s_client and stateless session resumption. A fix for this would have to be in
> the client code, I'll look into that.
>

I've looked into this further now. The cause is an interaction between session
tickets and the server name extension.

When a stateless session resumption is attempted OpenSSL sends (as permitted by
RFC5077) an empty session ID. This has the side effect that it is not clear
whether the server has accepted the ticket until later in the handshake whereas
in a stateful resumption this is apparent after received the server hello by
comparing session IDs.

As a result the value of s->hit above cannot be trusted to check for a resumed
session.

I'm looking into the best way to fix this. For now changing the above to:

else if (!s->session->tlsext_tick)
{
*al = SSL_AD_DECODE_ERROR;
return 0;
}

client side is one way but it doesn't retain all the original logic.

Disabling tickets using SSL_OP_NO_TICKET server side SHOULD work too (does in my
tests) so I've no idea why that wouldn't in the OPs setup unless the patch
doesn't set it in all contexts. Try placing it right after any call to
SSL_CTX_new().

Steve.
--
Dr Stephen N. Henson. Senior Technical/Cryptography Advisor,
Open Source Software Institute: www.oss-institute.org
OpenSSL Core team: www.openssl.org

httpd-dev.2009 at velox

Oct 25, 2009, 8:51 AM

Post #22 of 36 (3969 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Dr Stephen Henson wrote:
> Disabling tickets using SSL_OP_NO_TICKET server side SHOULD work too (does in my
> tests) so I've no idea why that wouldn't in the OPs setup unless the patch
> doesn't set it in all contexts. Try placing it right after any call to
> SSL_CTX_new().

I'm still a bit puzzled as to why my previously posted patch does not
turn off TLS session tickets... there's only one place in mod_ssl where
a new context is created, and in my tests, SSL_OP_NO_TICKET was reliably
applied (i.e., I didn't see any session tickets on the wire). Maybe
there's another issue if tickets are turned off?

Kamesh, could you apply the attached patch, for diagnostic purposes (in
addition to mod_ssl-disable_tls_tickets.diff), and let us know what
"options=" values you see in your ErrorLog? Note that you don't have to
increase Apache's LogLevel, the options for any new SSL connection will
be logged with "warn" already. Also, it would be helpful to have another
capture (with mod_ssl patched like this) where the svn client still
fails with a "parse tlsext" error. Thanks.

Kaspar

Attachments:

mod_ssl-log_ssloptions.diff (0.51 KB)

httpd-dev.2009 at velox

Oct 25, 2009, 9:24 AM

Post #23 of 36 (3973 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Joe Orton wrote:
>> the OpenSSL client (SNI extensions should never contain literal IPv4
>> addresses).
>
> Good point - I've changed neon for future releases to only enable SNI if
> the hostname is not a numeric IP address.

This logic should go into OpenSSL, I think... I know that this is
httpd-dev (not openssl-dev), but since Steve is listening anyway:
something like the attached patch? For the client side (i.e. in
ssl3_ctrl()), depending on how schoolmasterish OpenSSL should be towards
its users, the check could also be moved further down / be rejected with
INVALID_SERVERNAME.

Kaspar

Attachments:

openssl-sni_noip.diff (3.25 KB)

kamesh at collab

Oct 25, 2009, 11:54 AM

Post #24 of 36 (3963 views)
Permalink

RE: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Hi Kaspar,

I am away from the test environment right now, Will get back in next 13 hours.

With regards
Kamesh Jayachandran

-----Original Message-----
From: Kaspar Brand [mailto:httpd-dev.2009 [at] velox]
Sent: Sun 10/25/2009 9:21 PM
To: dev [at] httpd
Subject: Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12

Dr Stephen Henson wrote:
> Disabling tickets using SSL_OP_NO_TICKET server side SHOULD work too (does in my
> tests) so I've no idea why that wouldn't in the OPs setup unless the patch
> doesn't set it in all contexts. Try placing it right after any call to
> SSL_CTX_new().

I'm still a bit puzzled as to why my previously posted patch does not
turn off TLS session tickets... there's only one place in mod_ssl where
a new context is created, and in my tests, SSL_OP_NO_TICKET was reliably
applied (i.e., I didn't see any session tickets on the wire). Maybe
there's another issue if tickets are turned off?

Kamesh, could you apply the attached patch, for diagnostic purposes (in
addition to mod_ssl-disable_tls_tickets.diff), and let us know what
"options=" values you see in your ErrorLog? Note that you don't have to
increase Apache's LogLevel, the options for any new SSL connection will
be logged with "warn" already. Also, it would be helpful to have another
capture (with mod_ssl patched like this) where the svn client still
fails with a "parse tlsext" error. Thanks.

Kaspar

peter.sylvester at edelweb

Oct 25, 2009, 2:15 PM

Post #25 of 36 (3961 views)
Permalink

Re: Strange error(parse tlsext bug) in mod_ssl since httpd-2.2.12 [In reply to]

Kaspar Brand wrote:
> Joe Orton wrote:
>
>>> the OpenSSL client (SNI extensions should never contain literal IPv4
>>> addresses).
>>>
>> Good point - I've changed neon for future releases to only enable SNI if
>> the hostname is not a numeric IP address.
>>
>
> This logic should go into OpenSSL, I think...
Making openssl "intelligent" like "you have requested some value that
I don't think is a valid hostname, so I will ignore you sni request"
is not exactly a nice thing. You must reject everything that is not
a DNS hostname. Looks ugly.

If you have just a "raw" IP address an application may probably
already know this case.

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads