Mailing List Archive: Linux Virtual Server: Users

Page cannot be displayed

Index | Next | Previous | View Threaded

mark.maiden at globoforce

Mar 23, 2007, 5:16 AM

Post #1 of 9 (600 views)
Permalink

Page cannot be displayed

Hi all,

We recently set up an Ultra Monkey load balancer with 2 real servers
and 95% of the time it seems to be working perfectly, but every now
and then our customers are getting "Page cannot be displayed" errors.
It happens at different stages on our websites and we can't seem to
reproduce the problem here. Out customers are very large Fortune 500
companies so we assume that their networking etc is top of the line,
and the fact that it is occurring with multiple customers we assume it
is our architecture. Our LB environment is as follows :

Ultra monkey box :
CentOS 4.4
ldirectord.cf :

# Global Directives
checktimeout=5
checkinterval=5
#fallback=127.0.0.1:80
autoreload=yes
#logfile="/var/log/ldirectord.log"
logfile="local0"
quiescent=no

# Virtual Server for HTTP
virtual=xxx.xxx.xxx.252:80
fallback=127.0.0.1:80
real=xxx.xxx.xxx.248:80 masq
real=xxx.xxx.xxx.249:80 masq
service=http
request="microsites/admin/UMPingServlet"
receive="Greetings"
scheduler=lc
persistent=2100
protocol=tcp
checktype=negotiate

# Virtual Service for HTTPS
virtual=xxx.xxx.xxx.252:443
fallback=127.0.0.1:443
real=xxx.xxx.xxx.248:443 masq
real=xxx.xxx.xxx.249:443 masq
service=https
request="microsites/admin/UMPingServlet"
receive="Greetings"
scheduler=lc
persistent=2100
protocol=tcp
checktype=negotiate

/etc/sysctl.conf :

# Controls IP packet forwarding
net.ipv4.ip_forward = 1
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core
filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1

Real servers :
SuSE 10.1
Gateway set to Ultra Monkey eth0 address
Apache 2.0.x

Questions :

1: I assume that because we are using masq(NAT) that we don't need to
worry about the noarp problem with DR or TUN?
2: Is there any ip tuning that we should do on the Ultra Monkey box as
not only is it acting as the load balancer but it is also a router
too?
3: Has anybody else seen this intermittent "Page cannot be displayed"
error with UM?

Thank you very much for your help in advance,

Mark.

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

jmack at wm7d

Mar 23, 2007, 5:51 AM

Post #2 of 9 (572 views)
Permalink

Re: Page cannot be displayed [In reply to]

On Fri, 23 Mar 2007, Mark Maiden wrote:

> 1: I assume that because we are using masq(NAT) that we don't need to
> worry about the noarp problem with DR or TUN?

correct

> 2: Is there any ip tuning that we should do on the Ultra Monkey box as
> not only is it acting as the load balancer but it is also a router
> too?

nope. Just use standard kernel settings.

> 3: Has anybody else seen this intermittent "Page cannot be displayed"
> error with UM?

Intermittent problems are hard to diagnose. You're going to
have to find out what is really going on. "Page cannot be
displayed" is not helpful. Are you getting icmp errors, tcp
errors, are they coming from the director, the realserver?

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!
_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

sithgunner at hotmail

Mar 23, 2007, 5:59 AM

Post #3 of 9 (573 views)
Permalink

Re: Page cannot be displayed [In reply to]

>>3: Has anybody else seen this intermittent "Page cannot be
>>displayed"
>>error with UM?
>
>Intermittent problems are hard to diagnose. You're going to have to
>find out what is really going on. "Page cannot be displayed" is not
>helpful. Are you getting icmp errors, tcp errors, are they coming
>from the director, the realserver?

Just a small suggestion, but if it is possible, let go of ultra monkey and
do a direct DNAT from the router to one of the real server to see if it is
really about the load balancer that's making the problem or not.

_________________________________________________________________
$B$3$s$J2sE><w;J20$_$?$3$H$J$$(B!?$B;R6!$N:"$KL4Cf$K$J$C$?2{$+$7%M%?$,2sE>Cf(B
http://livesushi.jp/

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

mark.maiden at globoforce

Mar 23, 2007, 7:49 AM

Post #4 of 9 (576 views)
Permalink

Re: Page cannot be displayed [In reply to]

Thank you for your response. I agree about the information, we're
finding it hard to get firm networking info from our customers. As
regards errors, can you give me a little indication as to how to get
detailed information on this? Here's my ifconfig which shows no errors
on the interface on the UM box :

eth0 Link encap:Ethernet HWaddr 00:15:C5:E1:10:8B
inet addr:xxx.xxx.xxx.251 Bcast:xxx.xxx.xxx.255
Mask:255.255.255.240
inet6 addr: fe80::215:c5ff:fee1:108b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5050416 errors:0 dropped:0 overruns:0 frame:0
TX packets:5207932 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2794012482 (2.6 GiB) TX bytes:2807683093 (2.6 GiB)
Interrupt:169

Mark Maiden
Systems Administrator
Globoforce, Ltd
6 Beckett Way Parkwest
Dublin 12
Ireland
t: +353 1 625 8812
f: +353 1 625 8880
e: markm[at]globoforce.com
www.globoforce.com

http://guidance.gospelcom.net/answer.htm

Joseph Mack NA3T wrote:
> On Fri, 23 Mar 2007, Mark Maiden wrote:
>
>> 1: I assume that because we are using masq(NAT) that we don't need to
>> worry about the noarp problem with DR or TUN?
>
> correct
>
>> 2: Is there any ip tuning that we should do on the Ultra Monkey box as
>> not only is it acting as the load balancer but it is also a router
>> too?
>
> nope. Just use standard kernel settings.
>
>> 3: Has anybody else seen this intermittent "Page cannot be displayed"
>> error with UM?
>
> Intermittent problems are hard to diagnose. You're going to have to find
> out what is really going on. "Page cannot be displayed" is not helpful.
> Are you getting icmp errors, tcp errors, are they coming from the
> director, the realserver?
>
> Joe
>

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

jmack at wm7d

Mar 23, 2007, 9:03 AM

Post #5 of 9 (571 views)
Permalink

Re: Page cannot be displayed [In reply to]

On Fri, 23 Mar 2007, Mark Maiden wrote:

> Hi all,
>
> We recently set up an Ultra Monkey load balancer with 2 real servers
> and 95% of the time it seems to be working perfectly, but every now
> and then our customers are getting "Page cannot be displayed" errors.
> It happens at different stages on our websites and we can't seem to
> reproduce the problem here.

Try the file which produces large packets. It's in the
LVS-Tun section of the HOWTO.

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!
_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

jmack at wm7d

Mar 23, 2007, 11:30 AM

Post #6 of 9 (566 views)
Permalink

Re: Page cannot be displayed [In reply to]

On Fri, 23 Mar 2007, Mark Maiden wrote:

> Thank you for your response. I agree about the information, we're finding it
> hard to get firm networking info from our customers. As regards errors, can
> you give me a little indication as to how to get detailed information on
> this?

tcpdump

the S/N will be higher if you can reproduce the error on
demand.

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!
_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

ratz at drugphish

Mar 29, 2007, 6:52 AM

Post #7 of 9 (549 views)
Permalink

Re: Page cannot be displayed [In reply to]

> We recently set up an Ultra Monkey load balancer with 2 real servers
> and 95% of the time it seems to be working perfectly, but every now
> and then our customers are getting "Page cannot be displayed" errors.

What's the average/peak request rate and size?

> It happens at different stages on our websites and we can't seem to
> reproduce the problem here. Out customers are very large Fortune 500
> companies so we assume that their networking etc is top of the line,
> and the fact that it is occurring with multiple customers we assume it
> is our architecture. Our LB environment is as follows :
>
> Ultra monkey box :
> CentOS 4.4
> ldirectord.cf :
>
> # Global Directives
> checktimeout=5
> checkinterval=5
> #fallback=127.0.0.1:80
> autoreload=yes
> #logfile="/var/log/ldirectord.log"
> logfile="local0"

Can you correlate any log messages from ldirectord with the 5% page
display problems? Since you seem to have a very high timeout value for
your persistency and no indication of expire_nodest_conn it's not easy
to pinpoint the problem. What kind of application is running behind the
services? Does the application logic span over both services within the
lifetime context? Does the fallback work?

> # Controls IP packet forwarding
> net.ipv4.ip_forward = 1
> # Controls source route verification
> net.ipv4.conf.default.rp_filter = 1
> # Do not accept source routing
> net.ipv4.conf.default.accept_source_route = 0
> # Controls the System Request debugging functionality of the kernel
> kernel.sysrq = 0

Besides the strange comment, enabling this can be helpful at times.

> 1: I assume that because we are using masq(NAT) that we don't need to
> worry about the noarp problem with DR or TUN?

Correct.

> 2: Is there any ip tuning that we should do on the Ultra Monkey box as
> not only is it acting as the load balancer but it is also a router
> too?

Only if you experience performance problems. So I'd like to ask back if
you've previously seen any indication of such problems in your log files
(including kernel log: dmesg -s 100000).

> 3: Has anybody else seen this intermittent "Page cannot be displayed"
> error with UM?

Sure, but there's tons of possibilities for this to happen. I can
envision that ldirectord takes one of the RS out and due to the high
service template timeout and the missing expire_nodest_conn setting and
probably other issues, client requests are still being forwarded to the
non-functional RS, which will definitely cause such a message to be
displayed on the client's browser.

For your own amusement, I've allowed myself to quote the KB241344
article from Microsoft:

http://support.microsoft.com/kb/241344/EN-US/

This is maybe a wonderful example of why Microsoft is so much more
successful than others: No mentioning of tcpdump/windump to their users
and of course it's always the fault of the user :).

Regards,
Roberto Nibali, ratz
--
echo
'[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc
_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

mark.maiden at globoforce

Apr 4, 2007, 2:27 AM

Post #8 of 9 (536 views)
Permalink

Re: Page cannot be displayed [In reply to]

Hi all,

Thank you for your thoughts on this. Turned out to be an apache setting
that we were missing! Doh! We added this to our SSL config and now we
have no more Page Cannot Be Displayed's. Thanks again.

SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown
downgrade-1.0 force-response-1.0

Mark Maiden
Systems Administrator
Globoforce, Ltd
6 Beckett Way Parkwest
Dublin 12
Ireland
t: +353 1 625 8812
f: +353 1 625 8880
e: markm[at]globoforce.com
www.globoforce.com

http://guidance.gospelcom.net/answer.htm

Marcelo Leal wrote:
> Page cannot be displayed on the real servers?
>
> -----Mensagem original-----
> De: Roberto Nibali
> Enviada em: 29/03/2007 10:52:50
> Para: mark.maiden[at]globoforce.com,
> Assunto: Re: Page cannot be displayed
>
> > We recently set up an Ultra Monkey load balancer with 2 real servers
> > and 95% of the time it seems to be working perfectly, but every now
> > and then our customers are getting "Page cannot be displayed" errors.
>
> What's the average/peak request rate and size?
>
> > It happens at different stages on our websites and we can't seem to
> > reproduce the problem here. Out customers are very large Fortune 500
> > companies so we assume that their networking etc is top of the line,
> > and the fact that it is occurring with multiple customers we assume it
> > is our architecture. Our LB environment is as follows :
> >
> > Ultra monkey box :
> > CentOS 4.4
> > ldirectord.cf :
> >
> > # Global Directives
> > checktimeout=5
> > checkinterval=5
> > #fallback=127.0.0.1:80
> > autoreload=yes
> > #logfile="/var/log/ldirectord.log"
> > logfile="local0"
>
> Can you correlate any log messages from ldirectord with the 5% page
> display problems? Since you seem to have a very high timeout value for
> your persistency and no indication of expire_nodest_conn it's not easy
> to pinpoint the problem. What kind of application is running behind the
> services? Does the application logic span over both services within the
> lifetime context? Does the fallback work?
>
> > # Controls IP packet forwarding
> > net.ipv4.ip_forward = 1
> > # Controls source route verification
> > net.ipv4.conf.default.rp_filter = 1
> > # Do not accept source routing
> > net.ipv4.conf.default.accept_source_route = 0
> > # Controls the System Request debugging functionality of the kernel
> > kernel.sysrq = 0
>
> Besides the strange comment, enabling this can be helpful at times.
>
> > 1: I assume that because we are using masq(NAT) that we don't need to
> > worry about the noarp problem with DR or TUN?
>
> Correct.
>
> > 2: Is there any ip tuning that we should do on the Ultra Monkey box as
> > not only is it acting as the load balancer but it is also a router
> > too?
>
> Only if you experience performance problems. So I'd like to ask back if
> you've previously seen any indication of such problems in your log files
> (including kernel log: dmesg -s 100000).
>
> > 3: Has anybody else seen this intermittent "Page cannot be displayed"
> > error with UM?
>
> Sure, but there's tons of possibilities for this to happen. I can
> envision that ldirectord takes one of the RS out and due to the high
> service template timeout and the missing expire_nodest_conn setting and
> probably other issues, client requests are still being forwarded to the
> non-functional RS, which will definitely cause such a message to be
> displayed on the client's browser.
>
> For your own amusement, I've allowed myself to quote the KB241344
> article from Microsoft:
>
> http://support.microsoft.com/kb/241344/EN-US/
>
> This is maybe a wonderful example of why Microsoft is so much more
> successful than others: No mentioning of tcpdump/windump to their users
> and of course it's always the fault of the user :).
>
> Regards,
> Roberto Nibali, ratz
> --
> echo
> '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
> Send requests to lvs-users-request[at]LinuxVirtualServer.org
> or go to http://www.in-addr.de/mailman/listinfo/lvs-users

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

jmack at wm7d

Apr 4, 2007, 4:32 AM

Post #9 of 9 (526 views)
Permalink

Re: Page cannot be displayed [In reply to]

On Wed, 4 Apr 2007, Mark Maiden wrote:

> Hi all,
>
> Thank you for your thoughts on this. Turned out to be an apache setting that
> we were missing! Doh! We added this to our SSL config and now we have no more
> Page Cannot Be Displayed's. Thanks again.

incase someone else falls over this one,

o what was the SSL setting?

o any idea why the users intermittantly were getting the
error, rather than every time?

o why did it only occur in an LVS, but not with a single
server (if you know)?

Thanks Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!
_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://www.in-addr.de/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com