Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Linux Virtual Server: Users

[lvs-users] Load Balancer Not Listening on Specified Ports

 

 

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded


moj0rising at aim

Oct 23, 2009, 3:25 PM

Post #1 of 7 (366 views)
Permalink
[lvs-users] Load Balancer Not Listening on Specified Ports

Hello! I am having some trouble with a test set up of LVS.

I am trying to balance traffic between two web servers listening on port 80.

The web servers are up and running. I can download web pages
successfully from them from my desktop as well as directly from the
load balancer with netcat or similar.

When I try to access the web servers from my desktop via the load
balancer virtual IP, the connection times out. I get replies when
pining that IP and it shows up assigned to eth0:1 When I probe port
8080 on that same virtual IP from my desktop, it looks like no
services listening on that port. netstat on the load balancer shows no
services are indeed listening on that port. iptables is stopped. The
configuration has been checked by me and a co-worker several times and
it looks right as far as we can tell. It defintely looks like LVS
should be listening on 8080. The contents of my lvs.cf file are pasted
below this message.

/varlog/messages says (public IP changed for privacy):
Oct 23 14:15:40 omsbuild nanny[25773]: READ to 65.39.129.xxx:80 timed out
Oct 23 14:15:47 omsbuild nanny[25772]: READ to 192.168.18.29:80 timed out
Oct 23 14:15:52 omsbuild nanny[25773]: READ to 65.39.129.xxx:80 timed out
Oct 23 14:15:59 omsbuild nanny[25772]: READ to 192.168.18.29:80 timed out

I am also using piranha-gui but we tried starting nanny processes from
the command line (for the heck of it) and got the same results. When
we first checked, there were "extra" nanny processes listening on
different ports (from when I was testing different configurations on
the GUI -- they never stopped after I changed the configuration, for
some reason). It almost looks like the error messages from
/var/log/messages and the other symptoms I am seeing are from two
different problems but I'm not sure.

Can you tell me why no services are listening on the ports they should
be listening on? I've been over this quite a few times, checked the
documentation, searched Google and still haven't seen much that I was
able to use to resolve the issue. There was one interesting line at
http://www.austintek.com/LVS/LVS-HOWTO/mini-HOWTO/LVS-mini-HOWTO.html#doesnt_work,
under item 7.6 -- "ipvsadm (on the director) has not added the service
to the forwarding table (seen in the output of ipvsadm)." I'm not sure
if this applies to me, if so, how can I get thes service in the
forwarding table?


Thanks very much for any help you can provide.


Mike



serial_no = 74
primary = 192.168.3.28
service = lvs
backup = 0.0.0.0
heartbeat = 1
heartbeat_port = 539
keepalive = 6
deadtime = 18
network = direct
debug_level = NONE
virtual test1 {
active = 1
address = 192.168.0.69 eth0:1
vip_nmask = 255.255.248.0
port = 8080
expect = "HTTP"
use_regex = 0
load_monitor = none
scheduler = lc
protocol = tcp
timeout = 6
reentry = 15
quiesce_server = 0
server Speedy {
address = 192.168.18.29
active = 1
port = 80
weight = 1
}
server test1 {
address = 65.39.129.xxx
active = 1
port = 80
weight = 1
}

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


graeme at graemef

Oct 24, 2009, 3:42 AM

Post #2 of 7 (344 views)
Permalink
Re: [lvs-users] Load Balancer Not Listening on Specified Ports [In reply to]

Hi

On Fri, 2009-10-23 at 15:25 -0700, mojorising wrote:
> When I try to access the web servers from my desktop via the load
> balancer virtual IP, the connection times out. I get replies when
> pining that IP and it shows up assigned to eth0:1 When I probe port
> 8080 on that same virtual IP from my desktop, it looks like no
> services listening on that port. netstat on the load balancer shows no
> services are indeed listening on that port. iptables is stopped. The
> configuration has been checked by me and a co-worker several times and
> it looks right as far as we can tell. It defintely looks like LVS
> should be listening on 8080.

LVS doesn't listen. It's a router, not an application, so it has no
"open ports" in the sense that a TCP listener would.

> /varlog/messages says (public IP changed for privacy):
> Oct 23 14:15:40 omsbuild nanny[25773]: READ to 65.39.129.xxx:80 timed out
> Oct 23 14:15:47 omsbuild nanny[25772]: READ to 192.168.18.29:80 timed out
> Oct 23 14:15:52 omsbuild nanny[25773]: READ to 65.39.129.xxx:80 timed out
> Oct 23 14:15:59 omsbuild nanny[25772]: READ to 192.168.18.29:80 timed out

OK - firstly, Piranha has its own mailing list over at RedHat and is not
well known here, but let's have a look anyway...

> I am also using piranha-gui but we tried starting nanny processes from
> the command line (for the heck of it) and got the same results. When
> we first checked, there were "extra" nanny processes listening on
> different ports (from when I was testing different configurations on
> the GUI -- they never stopped after I changed the configuration, for
> some reason). It almost looks like the error messages from
> /var/log/messages and the other symptoms I am seeing are from two
> different problems but I'm not sure.

nanny is a "client" application - it doesn't listen. The processes you
saw were probably waiting for responses from your realservers.

> Can you tell me why no services are listening on the ports they should
> be listening on?

See my first comment.

So let's check the config:

> serial_no = 74
> primary = 192.168.3.28
> service = lvs
> backup = 0.0.0.0
> heartbeat = 1
> heartbeat_port = 539
> keepalive = 6
> deadtime = 18
> network = direct
> debug_level = NONE
> virtual test1 {
> active = 1
> address = 192.168.0.69 eth0:1
> vip_nmask = 255.255.248.0
> port = 8080
> expect = "HTTP"
> use_regex = 0
> load_monitor = none
> scheduler = lc
> protocol = tcp
> timeout = 6
> reentry = 15
> quiesce_server = 0
> server Speedy {
> address = 192.168.18.29
> active = 1
> port = 80
> weight = 1
> }
> server test1 {
> address = 65.39.129.xxx
> active = 1
> port = 80
> weight = 1
> }

So - your director has a management IP of 192.168.3.28, the VIP is
192.168.0.69:8080 and the realservers are 192.168.18.29:80 and
65.39.129.xxx:80, right? Using -DR (I assume that's what "network =
direct" means) this all makes sense.

However - if nanny is complaining that reads are timing out, then the
director can't connect to the realservers on port 80. That's pretty
fundamental.

Can you telnet to port 80 on the realservers from the director?

Graeme


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


moj0rising at aim

Oct 26, 2009, 11:18 AM

Post #3 of 7 (335 views)
Permalink
Re: [lvs-users] Load Balancer Not Listening on Specified Ports [In reply to]

Graeme,

Thanks a lot for your response. It helps me understand a bit more
about how LVS works.

So, as I mentioned in my first post, "The web servers are up and
running. I can download web pages successfully from them from my
desktop as well as directly from the
load balancer with netcat or similar." I just checked again and I can
download web pages from the real servers on port 80 with netcat or
telnet from the LVS machine. So we're all good there. Now I'm
wondering why nanny might have issues talking to the real servers on
80.

As for the bits about the configuration, yes. You are right. I just
verified the IP addresses in the configuration and the ones I am
testing with via netcat and telnet are the same.

I think the below items you mentioned are particularly beneficial for
me to understand how things work and what might be the matter:

> LVS doesn't listen. It's a router, not an application, so it has no
> "open ports" in the sense that a TCP listener would.

> nanny is a "client" application - it doesn't listen. The processes you
> saw were probably waiting for responses from your realservers.

With respect to the former, if nothing is listening on the load
balancer, maybe I am trying to use it the wrong way. On other load
balancers I have used, such as pen or BigIP, the load balancer is
actually listening and waiting for TCP connections, which it then
forwards on to the "balanced" nodes (real servers) on specified ports.
Basically, as far as PCs browsing the balanced sites know, that load
balancer *is* the web server. Does that make sense? So when I go to
test those load balancers to see if they work, I check to make sure
they are listening on the approriate port and try to connect to them
in the same way I would connect to a real server directly. Am I
supposed to be doing things differently with LVS?


Thanks again,

Mike



On 24/10/2009, Graeme Fowler <graeme[at]graemef.net> wrote:
> Hi
>
> On Fri, 2009-10-23 at 15:25 -0700, mojorising wrote:
>> When I try to access the web servers from my desktop via the load
>> balancer virtual IP, the connection times out. I get replies when
>> pining that IP and it shows up assigned to eth0:1 When I probe port
>> 8080 on that same virtual IP from my desktop, it looks like no
>> services listening on that port. netstat on the load balancer shows no
>> services are indeed listening on that port. iptables is stopped. The
>> configuration has been checked by me and a co-worker several times and
>> it looks right as far as we can tell. It defintely looks like LVS
>> should be listening on 8080.
>
> LVS doesn't listen. It's a router, not an application, so it has no
> "open ports" in the sense that a TCP listener would.
>
>> /varlog/messages says (public IP changed for privacy):
>> Oct 23 14:15:40 omsbuild nanny[25773]: READ to 65.39.129.xxx:80 timed out
>> Oct 23 14:15:47 omsbuild nanny[25772]: READ to 192.168.18.29:80 timed out
>> Oct 23 14:15:52 omsbuild nanny[25773]: READ to 65.39.129.xxx:80 timed out
>> Oct 23 14:15:59 omsbuild nanny[25772]: READ to 192.168.18.29:80 timed out
>
> OK - firstly, Piranha has its own mailing list over at RedHat and is not
> well known here, but let's have a look anyway...
>
>> I am also using piranha-gui but we tried starting nanny processes from
>> the command line (for the heck of it) and got the same results. When
>> we first checked, there were "extra" nanny processes listening on
>> different ports (from when I was testing different configurations on
>> the GUI -- they never stopped after I changed the configuration, for
>> some reason). It almost looks like the error messages from
>> /var/log/messages and the other symptoms I am seeing are from two
>> different problems but I'm not sure.
>
> nanny is a "client" application - it doesn't listen. The processes you
> saw were probably waiting for responses from your realservers.
>
>> Can you tell me why no services are listening on the ports they should
>> be listening on?
>
> See my first comment.
>
> So let's check the config:
>
>> serial_no = 74
>> primary = 192.168.3.28
>> service = lvs
>> backup = 0.0.0.0
>> heartbeat = 1
>> heartbeat_port = 539
>> keepalive = 6
>> deadtime = 18
>> network = direct
>> debug_level = NONE
>> virtual test1 {
>> active = 1
>> address = 192.168.0.69 eth0:1
>> vip_nmask = 255.255.248.0
>> port = 8080
>> expect = "HTTP"
>> use_regex = 0
>> load_monitor = none
>> scheduler = lc
>> protocol = tcp
>> timeout = 6
>> reentry = 15
>> quiesce_server = 0
>> server Speedy {
>> address = 192.168.18.29
>> active = 1
>> port = 80
>> weight = 1
>> }
>> server test1 {
>> address = 65.39.129.xxx
>> active = 1
>> port = 80
>> weight = 1
>> }
>
> So - your director has a management IP of 192.168.3.28, the VIP is
> 192.168.0.69:8080 and the realservers are 192.168.18.29:80 and
> 65.39.129.xxx:80, right? Using -DR (I assume that's what "network =
> direct" means) this all makes sense.
>
> However - if nanny is complaining that reads are timing out, then the
> director can't connect to the realservers on port 80. That's pretty
> fundamental.
>
> Can you telnet to port 80 on the realservers from the director?
>
> Graeme
>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
> Send requests to lvs-users-request[at]LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


graeme at graemef

Oct 26, 2009, 12:09 PM

Post #4 of 7 (335 views)
Permalink
Re: [lvs-users] Load Balancer Not Listening on Specified Ports [In reply to]

On Mon, 2009-10-26 at 11:18 -0700, mojorising wrote:
> Thanks a lot for your response. It helps me understand a bit more
> about how LVS works.

Grand :)

> So, as I mentioned in my first post, "The web servers are up and
> running. I can download web pages successfully from them from my
> desktop as well as directly from the
> load balancer with netcat or similar." I just checked again and I can
> download web pages from the real servers on port 80 with netcat or
> telnet from the LVS machine. So we're all good there. Now I'm
> wondering why nanny might have issues talking to the real servers on
> 80.

Only you can tell that one I'm afraid. Have you checked the logs which
nanny produces?

> As for the bits about the configuration, yes. You are right. I just
> verified the IP addresses in the configuration and the ones I am
> testing with via netcat and telnet are the same.

Good.

> With respect to the former, if nothing is listening on the load
> balancer, maybe I am trying to use it the wrong way. On other load
> balancers I have used, such as pen or BigIP, the load balancer is
> actually listening and waiting for TCP connections, which it then
> forwards on to the "balanced" nodes (real servers) on specified ports.
> Basically, as far as PCs browsing the balanced sites know, that load
> balancer *is* the web server. Does that make sense? So when I go to
> test those load balancers to see if they work, I check to make sure
> they are listening on the approriate port and try to connect to them
> in the same way I would connect to a real server directly. Am I
> supposed to be doing things differently with LVS?

The two devices/systems you mention can work in mixed modes;
"directing" (load balancing at L4) or "proxy" (load balancing at L7). L7
apps must first terminate the TCP session themselves before sending
making a further L7 connection on to the server.

LVS works at L4 - TCP or UDP. It is not application aware, and it only
ever routes packets based on some rule it's configured with. That's it,
in a nutshell.

To the client, as you state, the load balancer *is* the webserver as
they connect to an IP address (the VIP) and replies come back from that.
At the TCP or UDP level, the client application is unaware of the magic
going on at the other end of the connection.

To check how LVS is configured, you can either rely on nanny to tell you
(how? I don't know) or use the ipvsadm command line tool:

ipvsadm -L # displays LVS table
ipvsadm -L --stats # displays usage stats
ipvsadm -L --rate # displays rate statistics
[the -n switch can be used for numerical output]

It's worth remembering at this point that nanny is an abstraction layer
away from what's really going on behind the scenes; it monitors the
servers and puts them into/takes them out of the server "pool" as
necessary. It doesn't really do *any* load balancing itself.

Graeme


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


moj0rising at aim

Nov 2, 2009, 1:19 PM

Post #5 of 7 (288 views)
Permalink
Re: [lvs-users] Load Balancer Not Listening on Specified Ports [In reply to]

Okay. I've finally had time to come back to this. I've looked over
your latest message and now think I'm kinda stuck.

I checked /var/log/messages and see those same nanny messages like this:

Nov 2 11:59:20 omsbuild nanny[25772]: READ to 192.168.18.29:80 timed out

but I don't see anything more from nanny on my system. Is there
another place on the server I can get nanny log information?

The ipvsadm commands are helpful to know, yet there isn't much for
them to report right now:

[root[at]omsbuild ~]# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP ganassi.radius.co:webcache lc

[root[at]omsbuild ~]# ipvsadm -L --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
TCP ganassi.radius.co:webcache 0 0 0 0 0


Can you think of anything else I can do to find more clues on what
might be the matter?


Thank you!

Mike

On 26/10/2009, Graeme Fowler <graeme[at]graemef.net> wrote:
> On Mon, 2009-10-26 at 11:18 -0700, mojorising wrote:
>> Thanks a lot for your response. It helps me understand a bit more
>> about how LVS works.
>
> Grand :)
>
>> So, as I mentioned in my first post, "The web servers are up and
>> running. I can download web pages successfully from them from my
>> desktop as well as directly from the
>> load balancer with netcat or similar." I just checked again and I can
>> download web pages from the real servers on port 80 with netcat or
>> telnet from the LVS machine. So we're all good there. Now I'm
>> wondering why nanny might have issues talking to the real servers on
>> 80.
>
> Only you can tell that one I'm afraid. Have you checked the logs which
> nanny produces?
>
>> As for the bits about the configuration, yes. You are right. I just
>> verified the IP addresses in the configuration and the ones I am
>> testing with via netcat and telnet are the same.
>
> Good.
>
>> With respect to the former, if nothing is listening on the load
>> balancer, maybe I am trying to use it the wrong way. On other load
>> balancers I have used, such as pen or BigIP, the load balancer is
>> actually listening and waiting for TCP connections, which it then
>> forwards on to the "balanced" nodes (real servers) on specified ports.
>> Basically, as far as PCs browsing the balanced sites know, that load
>> balancer *is* the web server. Does that make sense? So when I go to
>> test those load balancers to see if they work, I check to make sure
>> they are listening on the approriate port and try to connect to them
>> in the same way I would connect to a real server directly. Am I
>> supposed to be doing things differently with LVS?
>
> The two devices/systems you mention can work in mixed modes;
> "directing" (load balancing at L4) or "proxy" (load balancing at L7). L7
> apps must first terminate the TCP session themselves before sending
> making a further L7 connection on to the server.
>
> LVS works at L4 - TCP or UDP. It is not application aware, and it only
> ever routes packets based on some rule it's configured with. That's it,
> in a nutshell.
>
> To the client, as you state, the load balancer *is* the webserver as
> they connect to an IP address (the VIP) and replies come back from that.
> At the TCP or UDP level, the client application is unaware of the magic
> going on at the other end of the connection.
>
> To check how LVS is configured, you can either rely on nanny to tell you
> (how? I don't know) or use the ipvsadm command line tool:
>
> ipvsadm -L # displays LVS table
> ipvsadm -L --stats # displays usage stats
> ipvsadm -L --rate # displays rate statistics
> [the -n switch can be used for numerical output]
>
> It's worth remembering at this point that nanny is an abstraction layer
> away from what's really going on behind the scenes; it monitors the
> servers and puts them into/takes them out of the server "pool" as
> necessary. It doesn't really do *any* load balancing itself.
>
> Graeme
>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
> Send requests to lvs-users-request[at]LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


graeme at graemef

Nov 2, 2009, 1:35 PM

Post #6 of 7 (289 views)
Permalink
Re: [lvs-users] Load Balancer Not Listening on Specified Ports [In reply to]

On Mon, 2009-11-02 at 13:19 -0800, mojorising wrote:
> Okay. I've finally had time to come back to this. I've looked over
> your latest message and now think I'm kinda stuck.

Uh-oh.

> I checked /var/log/messages and see those same nanny messages like this:
>
> Nov 2 11:59:20 omsbuild nanny[25772]: READ to 192.168.18.29:80 timed out
>
> but I don't see anything more from nanny on my system. Is there
> another place on the server I can get nanny log information?

No idea. Nanny is RedHat's monitoring widget, and is supported by them
as part of the Piranha package.

> The ipvsadm commands are helpful to know, yet there isn't much for
> them to report right now:

Well... if nanny cannout connect to the HTTP server on your realservers,
it won't work.

That said I notice that the virtual service is on the webcache port...
are the realservers webcache servers or HTTP servers? If you change
from :80 to :3128, does the behaviour change?

My next suggestion would be to try ldirectord and heartbeat together, or
keepalived on its own - they're better understood here.

Graeme


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


moj0rising at aim

Nov 2, 2009, 4:44 PM

Post #7 of 7 (287 views)
Permalink
Re: [lvs-users] Load Balancer Not Listening on Specified Ports [In reply to]

Thanks again, Graeme.

I changed the port to 3128. Still no dice. :|

I just posted a message about this to the Piranha list to see what
those cats know. I'll post back here with the results (in case it's
useful to others in the future). If that doesn't work out, I'll go for
that next suggestion you proposed, which I'm quite sure will do the
trick.


Mike

On 02/11/2009, Graeme Fowler <graeme[at]graemef.net> wrote:
> On Mon, 2009-11-02 at 13:19 -0800, mojorising wrote:
>> Okay. I've finally had time to come back to this. I've looked over
>> your latest message and now think I'm kinda stuck.
>
> Uh-oh.
>
>> I checked /var/log/messages and see those same nanny messages like this:
>>
>> Nov 2 11:59:20 omsbuild nanny[25772]: READ to 192.168.18.29:80 timed out
>>
>> but I don't see anything more from nanny on my system. Is there
>> another place on the server I can get nanny log information?
>
> No idea. Nanny is RedHat's monitoring widget, and is supported by them
> as part of the Piranha package.
>
>> The ipvsadm commands are helpful to know, yet there isn't much for
>> them to report right now:
>
> Well... if nanny cannout connect to the HTTP server on your realservers,
> it won't work.
>
> That said I notice that the virtual service is on the webcache port...
> are the realservers webcache servers or HTTP servers? If you change
> from :80 to :3128, does the behaviour change?
>
> My next suggestion would be to try ldirectord and heartbeat together, or
> keepalived on its own - they're better understood here.
>
> Graeme
>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
> Send requests to lvs-users-request[at]LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users[at]LinuxVirtualServer.org
Send requests to lvs-users-request[at]LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact lists@gossamer-threads.com
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.