Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Apache: Bugs

[Bug 46952] ssl renegotiation hangs with long ca list

 

 

Apache bugs RSS feed   Index | Next | Previous | View Threaded


bugzilla at apache

Jun 5, 2009, 5:20 AM

Post #1 of 10 (1086 views)
Permalink
[Bug 46952] ssl renegotiation hangs with long ca list

https://issues.apache.org/bugzilla/show_bug.cgi?id=46952





--- Comment #5 from Maarten Litmaath <litmaath [at] cern> 2009-06-05 05:20:07 PST ---

A ticket has been opened in the OpenSSL request tracker:

http://rt.openssl.org/Ticket/Display.html?id=1949

account: guest
password: guest

--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe [at] httpd
For additional commands, e-mail: bugs-help [at] httpd


bugzilla at apache

Jun 5, 2009, 5:59 AM

Post #2 of 10 (1035 views)
Permalink
[Bug 46952] ssl renegotiation hangs with long ca list [In reply to]

https://issues.apache.org/bugzilla/show_bug.cgi?id=46952


Maarten Litmaath <litmaath [at] cern> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |litmaath [at] cern




--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe [at] httpd
For additional commands, e-mail: bugs-help [at] httpd


bugzilla at apache

Jun 19, 2009, 3:15 AM

Post #3 of 10 (983 views)
Permalink
[Bug 46952] ssl renegotiation hangs with long ca list [In reply to]

https://issues.apache.org/bugzilla/show_bug.cgi?id=46952





--- Comment #6 from szamcsi <Akos.Frohner [at] cern> 2009-06-19 03:15:34 PST ---
Created an attachment (id=23831)
--> (https://issues.apache.org/bugzilla/attachment.cgi?id=23831)
test case for the bug

I've attached a test case for the bug, which generates test
certificate to trigger the problem.

The first point is that the certificate of the server has to be
bigger than OpenSSL's buffer size, which is 4kB by default. I have
padded the server certificate with comments.
One can play with this number by using the '--pads 123' option.

The second point is to create enough CA entries that the summary
of ServerHello, Certificate, ServerKeyExchange, CertificateRequest
and ServerHelloDone record sizes adds up over 12kB (4kB of the
buffer in OpenSSL and 8kB for the BIO of mod_ssl).
One can play with this number by using the '--cas 123' option.


Run the script and follow the instructions:
../test-certs
./httpd/server start
./httpd/client # which shall hang
./httpd/server stop

You can clean up/regernerate with different parameters:
../test-certs --cas 70 # works
../test-certs --cas 80 # hangs
../test-certs --cas 110 # hangs
../test-certs --cas 120 # works

I hope it clarifies the problem!

--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe [at] httpd
For additional commands, e-mail: bugs-help [at] httpd


bugzilla at apache

Jun 19, 2009, 3:33 AM

Post #4 of 10 (981 views)
Permalink
[Bug 46952] ssl renegotiation hangs with long ca list [In reply to]

https://issues.apache.org/bugzilla/show_bug.cgi?id=46952





--- Comment #7 from lassi.tuura [at] cern 2009-06-19 03:33:30 PST ---
For an additional mod_ssl / openssl data point, I confirm I have verified the
problem exists using the just-attached "test case for the bug" (or rather
version I tested yesterday) on Mac OS X 10.4.11 with:

$ openssl version
OpenSSL 0.9.8i 15 Sep 2008

$ xxx/bin/httpd -V
Server version: Apache/2.2.10 (Unix)
Server built: Dec 17 2008 00:20:21
Server's Module Magic Number: 20051115:18
Server loaded: APR 1.3.3, APR-Util 1.3.4
Compiled using: APR 1.3.3, APR-Util 1.3.4
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="xxx"
-D SUEXEC_BIN="xxx/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe [at] httpd
For additional commands, e-mail: bugs-help [at] httpd


bugzilla at apache

Jun 19, 2009, 4:36 AM

Post #5 of 10 (971 views)
Permalink
[Bug 46952] ssl renegotiation hangs with long ca list [In reply to]

https://issues.apache.org/bugzilla/show_bug.cgi?id=46952





--- Comment #8 from David Smith <David.Smith [at] cern> 2009-06-19 04:36:08 PST ---
Hello,

I have been making some investigation of the problem - and wanted to summarise
the current understanding of the details - which I'm fairly clear about. I've
also included a tentative fix, but as I explain there are some considerations -
the wider audience here will probably have some more ideas. (I'll write a
similar report on the openssl bug list, as there are also openssl issues
raised).

Apparently the problem (dialogue stall) arrises because of the two layers of
buffering between mod_ssl's filter_out BIO and the wbio buffer BIO (i.e. a
BIO_f_buffer()), which openssl adds to the stack of wbio BIOs for most of the
handshake process. In particular s3_srvr.c (in openssl) uses this check during
the flush state:

case SSL3_ST_SW_FLUSH:
/* number of bytes to be flushed */
num1=BIO_ctrl(s->wbio,BIO_CTRL_INFO,0,NULL);
if (num1 > 0)
{
s->rwstate=SSL_WRITING;
num1=BIO_flush(s->wbio);
if (num1 <= 0) { ret= -1; goto end; }
s->rwstate=SSL_NOTHING;
}

The BIO_CTRL_INFO call for the BIO_f_buffer (first in the list of s->wbio
stack) only checks the number of buffered outgoing bytes in that BIO; so if the
BIO_f_buffer has no bytes remaining to be written, but mod_ssl's filter_out BIO
does then BIO_flush is not called - bytes remain in the filter_out BIO and the
dialogue stalls.

e.g. observed sequence during a renegotiate sequence that stalls:

bio_filter_out_write receives 37 bytes (SSL renegotiate ciphers request)
37 bytes added to outctx->buffer: total 37 bytes

Flushed from s3_srvr.c: BIO_flush called on bio_filter out. outctx->buffer is 0
again.

openssl adds in the BIO_f_buffer BIO to the ssl wbio (via
ssl_init_wbio_buffer())

openssl server sends HELLO A: 101 bytes
Added to BIO_f_buffer: buffer total = 101 bytes (sending from BIO_f_buffer to
filter_out is done in crypto routine buffer_write(), bf_buff.c)

openssl sever sends SERVER CERT A: 4325 bytes
(buffer_write()) fills BIO_f_buffer with extra 3995 bytes: total = 4096
4096 bytes sent to bio_filter_out_write: outctx->buffer now 4096 bytes
BIO_f_buffer: 0 bytes buffered
Remainder bytes added to BIO_f_buffer: 330 bytes buffered

openssl server sends KEY EXCHANGE A: 565 bytes
BIO_f_buffer receives 565 bytes: total 895 bytes

openssl server sends CERT REQ A: 7797 bytes

BIO_f_buffer adds 3201 to buffer making total of 4096:
4096 bytes sent to bio_filter_out_write: All 8192 bytes flushed.
outctx->buffer is 0 again.
BIO_f_buffer: 0 bytes buffered
Remainder of 4596 sent to bio_filter_out_write(): Added to outctx->buffer for
total of 4596 buffed bytes.

openssl server passes through SSL3_ST_SW_FLUSH state and checks wbio (i.e.
BIO_f_buffer) with BIO_CTRL_INFO: Returns 0 as there is nothing buffered, and
so no call BIO_flush on wbio. Thus no flush call to mod_ssl's filter_out BIO.

Trial fix - a change in both openssl and mod_ssl:

--- s3_srvr.c- 2009-06-17 20:44:54.000000000 +0200
+++ s3_srvr.c 2009-06-17 17:20:33.000000000 +0200
@@ -413,7 +413,7 @@

case SSL3_ST_SW_FLUSH:
/* number of bytes to be flushed */
- num1=BIO_ctrl(s->wbio,BIO_CTRL_INFO,0,NULL);
+ num1=BIO_ctrl(s->wbio,BIO_CTRL_WPENDING,0,NULL);
if (num1 > 0)
{
s->rwstate=SSL_WRITING;


--- ssl_engine_io.c- 2009-06-17 20:48:19.000000000 +0200
+++ ssl_engine_io.c 2009-06-17 17:41:07.000000000 +0200
@@ -260,10 +260,10 @@
case BIO_CTRL_SET_CLOSE:
bio->shutdown = (int)num;
break;
- case BIO_CTRL_WPENDING:
+ case BIO_CTRL_PENDING:
ret = 0L;
break;
- case BIO_CTRL_PENDING:
+ case BIO_CTRL_WPENDING:
ret = (long)(outctx->blen + outctx->length);
break;
case BIO_CTRL_FLUSH:


the change in openssl is to modify the BIO_ctrl function used in the flush
check to BIO_CTRL_WPENDING: That call on the BIO_f_buffer returns either number
of buffer outgoing bytes (similar to BIO_CTRL_INFO) or if zero calls
BIO_CTRL_WPENDING on the next_bio (e.g. mod_ssl's filter_out BIO). In the there
is no BIO_f_buffer (e.g. when the SSL renegotiate ciphers request is sent)
s3_srvr will directly call BIO_CTRL_WPENDING on the filter_out BIO.
Unfortunately it looks like BIO_CTRL_PENDING and BIO_CTRL_WPENDING are reversed
in ssl_engine_io.c: so while I've seen the patch overall appears to work
without problem it would need to be applied to mod_ssl before or at the same
time as openssl. It also has the effect of changing the observed behavior for
any application providing their own BIO: WPENDING is called rather than
BIO_CTRL_INFO. Possibly the SSL3_ST_SW_FLUSH BIO_ctrl check in s3_srvr.c could
be explicitly made up of a BIO_CTRL_INFO call, and then a further WPENDNING
call only if zero bytes are returned.

Interestingly in the above example the stall would be avoided with only a
change in bio_filter_out_write(); to buffer bytes up to 8192, not 8191 bytes as
current check does: But this relies on the BIO_f_buffer size being 4096 bytes,
and there is still an apparent sequence of writes possible to leave the
BIO_f_buffer empty but the filter_out BIO full.

David

--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe [at] httpd
For additional commands, e-mail: bugs-help [at] httpd


bugzilla at apache

Jun 19, 2009, 5:01 AM

Post #6 of 10 (971 views)
Permalink
[Bug 46952] ssl renegotiation hangs with long ca list [In reply to]

https://issues.apache.org/bugzilla/show_bug.cgi?id=46952


David Smith <David.Smith [at] cern> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |David.Smith [at] cern




--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe [at] httpd
For additional commands, e-mail: bugs-help [at] httpd


bugzilla at apache

Jun 23, 2009, 1:36 AM

Post #7 of 10 (938 views)
Permalink
[Bug 46952] ssl renegotiation hangs with long ca list [In reply to]

https://issues.apache.org/bugzilla/show_bug.cgi?id=46952





--- Comment #9 from Joe Orton <jorton [at] redhat> 2009-06-23 01:36:18 PST ---
Superb analysis, thanks David. Your proposed change to bio_filter_out_ctrl()
makes sense to me.

--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe [at] httpd
For additional commands, e-mail: bugs-help [at] httpd


bugzilla at apache

Jun 23, 2009, 8:13 AM

Post #8 of 10 (938 views)
Permalink
[Bug 46952] ssl renegotiation hangs with long ca list [In reply to]

https://issues.apache.org/bugzilla/show_bug.cgi?id=46952





--- Comment #10 from David Smith <David.Smith [at] cern> 2009-06-23 08:13:43 PST ---
Hello Joe. We have an RT ticket (#1949) open with openssl for the same problem;
I've alerted them there is more detail and the test case here. I hope they will
consider the openssl/crypto suggestion and give some feedback.

--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe [at] httpd
For additional commands, e-mail: bugs-help [at] httpd


bugzilla at apache

Jun 23, 2009, 8:46 AM

Post #9 of 10 (922 views)
Permalink
[Bug 46952] ssl renegotiation hangs with long ca list [In reply to]

https://issues.apache.org/bugzilla/show_bug.cgi?id=46952





--- Comment #11 from Joe Orton <jorton [at] redhat> 2009-06-23 08:46:55 PST ---
I've committed your fix in r787722 ( https://svn.apache.org/viewcvs.cgi?view=rev&rev=787722 ).

It doesn't look like this will make any different to existing OpenSSL
implementations - I can't see anywhere significant in ssl/ which is calling
BIO_pending, BIO_wpending, or BIO_ctrl with _PENDING/_WPENDING similarly.

--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe [at] httpd
For additional commands, e-mail: bugs-help [at] httpd


bugzilla at apache

Jun 26, 2009, 7:24 AM

Post #10 of 10 (891 views)
Permalink
[Bug 46952] ssl renegotiation hangs with long ca list [In reply to]

https://issues.apache.org/bugzilla/show_bug.cgi?id=46952


Joe Orton <jorton [at] redhat> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED




--- Comment #12 from Joe Orton <jorton [at] redhat> 2009-06-26 07:24:21 PST ---
For reference: http://marc.info/?l=openssl-dev&m=124575128524176&w=2

I've committed a change to have mod_ssl flush pending output unconditionally in
r788715, which has been confirmed to fix this issue.

Thanks again for your work on this, David.

--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe [at] httpd
For additional commands, e-mail: bugs-help [at] httpd

Apache bugs RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.