Mailing List Archive: Apache: Users

Establishing a new connection sometimes takes long (30s).

Index | Next | Previous | View Threaded

alexg at computicket

May 13, 2008, 12:10 AM

Post #1 of 6 (202 views)
Permalink

Establishing a new connection sometimes takes long (30s).

Hi

We are experiencing an intermittent but reproducible problem, where establishing a new connection to Apache httpd sometimes takes up to 30 seconds. We are running 350 threads with +-20 threads in keepalive state (according to /server-stats) and nothing else connecting or executing in Apache at the time the new connection comes in. There _should_ be a thread available for a new connection and the socket accept _should_ be listening. What we are seeing is that there is a pause between when the network card receives the message and Apache gets the message. In the test we were using a Java HttpConnection but we can reproduce the problem with IE and Firefox as well. There is no firewall on the machine.

Can anybody please explain why this could be happening or how we can go about finding the delay.

Thank you
Alexander Grässer

-TRACE------------------------
A case in point (delay of 17 seconds), running Packetyzer 5.0.0 on the network card:

13:02:25.249190000 client -> server : SYN (62 bytes)
13:02:25.249224000 server -> client : SYN ACK (62 bytes)
13:02:25.251134000 client -> server : ACK (60 bytes)
13:02:25.251864000 client -> server : HTTP GET (268 bytes)
13:02:25.442069000 server -> client : ACK (54 bytes)
13:02:42.960316000 server -> client : HTTP DATA (1514 bytes)
13:02:42.960327000 server -> client : HTTP continuation (155 bytes)
13:02:42.963080000 client -> server : ACK (60 bytes)

and then access.log (using LogFormat "%h %l %u %t \"%r\" %>s %b (%T/%D) \"%{Referer}i\" \"%{User-Agent}i\"" combined) indicates an execution time of 31,250 milliseconds.

[client-ip] - - [12/May/2008:13:02:42 +0200] "GET /xxxxx/xxxx HTTP/1.1" 200 1310 (0/31250) "-" "Java(tm) 2 SDK, Standard Edition v1.5.0_10 Java/1.5.0_10"

-CONFIG-------------------------
The config of the system is as follows: (Although we see this on differently configured Win2k machines as well)
System:
Microsoft Windows 2000
5.00.2195
Service Pack 4

Computer:
Intel(R) Xeon(TM)
CPU 3.20GHz
AT/AT Compatible
2,096,432 KB RAM

Disk:
Device: PERC LD 0 PERCRAID SCSI 6.34 GB (6,810,531,328 bytes) with 842 MB free.
Driver: Microsoft disk.sys, Version 5.0.2183.1 / 5.00.2195.6655 (11/14/1999)

Network:
Card: Intel(R) PRO/1000 MT
Driver: Intel e1000nt5.sys, version 7.2.19.0002 (8/2/2004)
WinSock: 3.10
WinSock32: 5.00.2195.6603

Apache
Server Version: Apache/2.2.8 (Win32) mod_jk/1.2.26
Server Built: Jan 18 2008 00:37:19
Httpd.conf:
<IfModule mpm_winnt_module>
ThreadsPerChild 350
ThreadLimit 350
ListenBacklog 511
MaxRequestsPerChild 0
</IfModule>
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 120
HostnameLookups Off

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe[at]httpd.apache.org
" from the digest: users-digest-unsubscribe[at]httpd.apache.org
For additional commands, e-mail: users-help[at]httpd.apache.org

sunberg at gmail

May 13, 2008, 12:21 AM

Post #2 of 6 (195 views)
Permalink

Re: Establishing a new connection sometimes takes long (30s). [In reply to]

Hi

Its most likely to be an DNS issue. If your not using vhosts, try
accessing it by its IP-adress. If that helps, check your DNS. Dont
remember your reverse dns settings.

--
Lars

On Tue, May 13, 2008 at 9:10 AM, Alex Grässer <alexg[at]computicket.com> wrote:
> Hi
>
> We are experiencing an intermittent but reproducible problem, where establishing a new connection to Apache httpd sometimes takes up to 30 seconds. We are running 350 threads with +-20 threads in keepalive state (according to /server-stats) and nothing else connecting or executing in Apache at the time the new connection comes in. There _should_ be a thread available for a new connection and the socket accept _should_ be listening. What we are seeing is that there is a pause between when the network card receives the message and Apache gets the message. In the test we were using a Java HttpConnection but we can reproduce the problem with IE and Firefox as well. There is no firewall on the machine.
>
> Can anybody please explain why this could be happening or how we can go about finding the delay.
>
> Thank you
> Alexander Grässer
>
> -TRACE------------------------
> A case in point (delay of 17 seconds), running Packetyzer 5.0.0 on the network card:
>
> 13:02:25.249190000 client -> server : SYN (62 bytes)
> 13:02:25.249224000 server -> client : SYN ACK (62 bytes)
> 13:02:25.251134000 client -> server : ACK (60 bytes)
> 13:02:25.251864000 client -> server : HTTP GET (268 bytes)
> 13:02:25.442069000 server -> client : ACK (54 bytes)
> 13:02:42.960316000 server -> client : HTTP DATA (1514 bytes)
> 13:02:42.960327000 server -> client : HTTP continuation (155 bytes)
> 13:02:42.963080000 client -> server : ACK (60 bytes)
>
> and then access.log (using LogFormat "%h %l %u %t \"%r\" %>s %b (%T/%D) \"%{Referer}i\" \"%{User-Agent}i\"" combined) indicates an execution time of 31,250 milliseconds.
>
> [client-ip] - - [12/May/2008:13:02:42 +0200] "GET /xxxxx/xxxx HTTP/1.1" 200 1310 (0/31250) "-" "Java(tm) 2 SDK, Standard Edition v1.5.0_10 Java/1.5.0_10"
>
> -CONFIG-------------------------
> The config of the system is as follows: (Although we see this on differently configured Win2k machines as well)
> System:
> Microsoft Windows 2000
> 5.00.2195
> Service Pack 4
>
> Computer:
> Intel(R) Xeon(TM)
> CPU 3.20GHz
> AT/AT Compatible
> 2,096,432 KB RAM
>
> Disk:
> Device: PERC LD 0 PERCRAID SCSI 6.34 GB (6,810,531,328 bytes) with 842 MB free.
> Driver: Microsoft disk.sys, Version 5.0.2183.1 / 5.00.2195.6655 (11/14/1999)
>
> Network:
> Card: Intel(R) PRO/1000 MT
> Driver: Intel e1000nt5.sys, version 7.2.19.0002 (8/2/2004)
> WinSock: 3.10
> WinSock32: 5.00.2195.6603
>
> Apache
> Server Version: Apache/2.2.8 (Win32) mod_jk/1.2.26
> Server Built: Jan 18 2008 00:37:19
> Httpd.conf:
> <IfModule mpm_winnt_module>
> ThreadsPerChild 350
> ThreadLimit 350
> ListenBacklog 511
> MaxRequestsPerChild 0
> </IfModule>
> Timeout 300
> KeepAlive On
> MaxKeepAliveRequests 100
> KeepAliveTimeout 120
> HostnameLookups Off
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe[at]httpd.apache.org
> " from the digest: users-digest-unsubscribe[at]httpd.apache.org
> For additional commands, e-mail: users-help[at]httpd.apache.org
>
>

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe[at]httpd.apache.org
" from the digest: users-digest-unsubscribe[at]httpd.apache.org
For additional commands, e-mail: users-help[at]httpd.apache.org

alexg at computicket

May 13, 2008, 12:52 AM

Post #3 of 6 (195 views)
Permalink

RE: Establishing a new connection sometimes takes long (30s). [In reply to]

Lars

Why would this be a DNS issue? The problem happens on the SERVER and HostnameLookups is Off!
Which name could Apache possibly want to look up? The server name is stored in the hosts. file for good measure.
Also the VirtualHosts are IP based so there is no name to be looked up.

The problem seems to occur BEFORE Apache even receives the message.

Does Apache count (reverse) DNS lookups towards the execution time?

Thanks
Alex

-----Original Message-----
From: Lars [mailto:sunberg[at]gmail.com]
Sent: 13 May 2008 09:22 AM
To: users[at]httpd.apache.org
Subject: Re: [users[at]httpd] Establishing a new connection sometimes takes long (30s).

Hi

Its most likely to be an DNS issue. If your not using vhosts, try
accessing it by its IP-adress. If that helps, check your DNS. Dont
remember your reverse dns settings.

--
Lars

On Tue, May 13, 2008 at 9:10 AM, Alex Grässer <alexg[at]computicket.com> wrote:
> Hi
>
> We are experiencing an intermittent but reproducible problem, where establishing a new connection to Apache httpd sometimes takes up to 30 seconds. We are running 350 threads with +-20 threads in keepalive state (according to /server-stats) and nothing else connecting or executing in Apache at the time the new connection comes in. There _should_ be a thread available for a new connection and the socket accept _should_ be listening. What we are seeing is that there is a pause between when the network card receives the message and Apache gets the message. In the test we were using a Java HttpConnection but we can reproduce the problem with IE and Firefox as well. There is no firewall on the machine.
>
> Can anybody please explain why this could be happening or how we can go about finding the delay.
>
> Thank you
> Alexander Grässer
>
> -TRACE------------------------
> A case in point (delay of 17 seconds), running Packetyzer 5.0.0 on the network card:
>
> 13:02:25.249190000 client -> server : SYN (62 bytes)
> 13:02:25.249224000 server -> client : SYN ACK (62 bytes)
> 13:02:25.251134000 client -> server : ACK (60 bytes)
> 13:02:25.251864000 client -> server : HTTP GET (268 bytes)
> 13:02:25.442069000 server -> client : ACK (54 bytes)
> 13:02:42.960316000 server -> client : HTTP DATA (1514 bytes)
> 13:02:42.960327000 server -> client : HTTP continuation (155 bytes)
> 13:02:42.963080000 client -> server : ACK (60 bytes)
>
> and then access.log (using LogFormat "%h %l %u %t \"%r\" %>s %b (%T/%D) \"%{Referer}i\" \"%{User-Agent}i\"" combined) indicates an execution time of 31,250 milliseconds.
>
> [client-ip] - - [12/May/2008:13:02:42 +0200] "GET /xxxxx/xxxx HTTP/1.1" 200 1310 (0/31250) "-" "Java(tm) 2 SDK, Standard Edition v1.5.0_10 Java/1.5.0_10"
>
> -CONFIG-------------------------
> The config of the system is as follows: (Although we see this on differently configured Win2k machines as well)
> System:
> Microsoft Windows 2000
> 5.00.2195
> Service Pack 4
>
> Computer:
> Intel(R) Xeon(TM)
> CPU 3.20GHz
> AT/AT Compatible
> 2,096,432 KB RAM
>
> Disk:
> Device: PERC LD 0 PERCRAID SCSI 6.34 GB (6,810,531,328 bytes) with 842 MB free.
> Driver: Microsoft disk.sys, Version 5.0.2183.1 / 5.00.2195.6655 (11/14/1999)
>
> Network:
> Card: Intel(R) PRO/1000 MT
> Driver: Intel e1000nt5.sys, version 7.2.19.0002 (8/2/2004)
> WinSock: 3.10
> WinSock32: 5.00.2195.6603
>
> Apache
> Server Version: Apache/2.2.8 (Win32) mod_jk/1.2.26
> Server Built: Jan 18 2008 00:37:19
> Httpd.conf:
> <IfModule mpm_winnt_module>
> ThreadsPerChild 350
> ThreadLimit 350
> ListenBacklog 511
> MaxRequestsPerChild 0
> </IfModule>
> Timeout 300
> KeepAlive On
> MaxKeepAliveRequests 100
> KeepAliveTimeout 120
> HostnameLookups Off
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe[at]httpd.apache.org
> " from the digest: users-digest-unsubscribe[at]httpd.apache.org
> For additional commands, e-mail: users-help[at]httpd.apache.org
>
>

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe[at]httpd.apache.org
" from the digest: users-digest-unsubscribe[at]httpd.apache.org
For additional commands, e-mail: users-help[at]httpd.apache.org

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe[at]httpd.apache.org
" from the digest: users-digest-unsubscribe[at]httpd.apache.org
For additional commands, e-mail: users-help[at]httpd.apache.org

uhlar at fantomas

May 13, 2008, 3:04 AM

Post #4 of 6 (190 views)
Permalink

Re: Establishing a new connection sometimes takes long (30s). [In reply to]

On 13.05.08 09:52, Alex Grässer wrote:
> Why would this be a DNS issue? The problem happens on the SERVER

because the most common reason for this is DNS issue.

> and HostnameLookups is Off!

isn't it turned on somewhere?

> Also the VirtualHosts are IP based so there is no name to be looked up.

this is completely irelevant to DNS. The only think that matters here is,
that when there is host name in <VirtualHost> directive, it is being looked
up at start or reconfigure, but it does not affect request processing

> The problem seems to occur BEFORE Apache even receives the message.

another reason might be an IDENT being required...

> Does Apache count (reverse) DNS lookups towards the execution time?

I guess so.
--
Matus UHLAR - fantomas, uhlar[at]fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Spam is for losers who can't get business any other way.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe[at]httpd.apache.org
" from the digest: users-digest-unsubscribe[at]httpd.apache.org
For additional commands, e-mail: users-help[at]httpd.apache.org

alexg at computicket

May 13, 2008, 3:37 AM

Post #5 of 6 (190 views)
Permalink

RE: Establishing a new connection sometimes takes long (30s). [In reply to]

Thanks Matus

> > Why would this be a DNS issue? The problem happens on the SERVER

> because the most common reason for this is DNS issue.

Granted, but we have ruled that out by putting the names into the hosts. file. What is strange is, that this does not happen every time a new connection is made. When requesting the same page continuously, the connection gets disconnected after the 100th call. As it should because of the KeepAliveRequests setting. When the client reconnects, it sometimes is near instantaneous and sometimes takes up 30seconds. The times vary from 35ms to 30seconds.

> > and HostnameLookups is Off!

> isn't it turned on somewhere?

The HostnameLookups setting is global. I have not tried to put it into each VirtualHost. But if it were turned on would you not see the resolved host names in the access.log?

> > Also the VirtualHosts are IP based so there is no name to be looked up.

> this is completely irrelevant to DNS. The only think that matters here is,
> that when there is host name in <VirtualHost> directive, it is being looked
> up at start or reconfigure, but it does not affect request processing

> > The problem seems to occur BEFORE Apache even receives the message.

> another reason might be an IDENT being required...

This is a straightforward HTTP GET request. I don't think IDENT is a problem here.

> > Does Apache count (reverse) DNS lookups towards the execution time?

> I guess so.

Then you should see the time in the access.log, which we are not.

Thanks
Alex

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe[at]httpd.apache.org
" from the digest: users-digest-unsubscribe[at]httpd.apache.org
For additional commands, e-mail: users-help[at]httpd.apache.org

uhlar at fantomas

May 13, 2008, 2:18 PM

Post #6 of 6 (181 views)
Permalink

Re: Establishing a new connection sometimes takes long (30s). [In reply to]

> > > Why would this be a DNS issue? The problem happens on the SERVER

> > because the most common reason for this is DNS issue.

On 13.05.08 12:37, Alex Grässer wrote:
> Granted, but we have ruled that out by putting the names into the hosts.

using local caching DNS is usually much more effective. Aren't some IPs
missing in the hosts file? e.g. dnsmasq can read hosts file and provide its
content via DNS. But if some hosts are missing, it may sometimes delay the
resolution, and sometimes just cache negative answer (host not found).

> file. What is strange is, that this does not happen every time a new
> connection is made. When requesting the same page continuously, the
> connection gets disconnected after the 100th call. As it should because of
> the KeepAliveRequests setting. When the client reconnects, it sometimes is
> near instantaneous and sometimes takes up 30seconds. The times vary from
> 35ms to 30seconds.

> > > and HostnameLookups is Off!
>
> > isn't it turned on somewhere?
>
> The HostnameLookups setting is global. I have not tried to put it into
> each VirtualHost. But if it were turned on would you not see the resolved
> host names in the access.log?

Not, if they are not resolved, e/g/ if the resolving times out (which is
exactly what causes the delay in such cases)

> > another reason might be an IDENT being required...
>
> This is a straightforward HTTP GET request. I don't think IDENT is a
> problem here.

if you configured apache to get IDENT response from connecting user, the
server tries to get response by every connection made, which may sometimes
fail, sometimes not.

> > > Does Apache count (reverse) DNS lookups towards the execution time?
>
> > I guess so.
>
> Then you should see the time in the access.log, which we are not.

The lookups may be once fast, once slow... or "usually fast, sometimes slow"
which would explain your problem. I'm not telling you that it's the problem,
just explaining that it might be it.

--
Matus UHLAR - fantomas, uhlar[at]fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"To Boot or not to Boot, that's the question." [WD1270 Caviar]

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe[at]httpd.apache.org
" from the digest: users-digest-unsubscribe[at]httpd.apache.org
For additional commands, e-mail: users-help[at]httpd.apache.org

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com