Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Linux Virtual Server: Users

[lvs-users] LVS-DR + 2 pools in 2 networks = hair pulling

 

 

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded


tom at calixo

Apr 19, 2012, 2:49 AM

Post #1 of 3 (497 views)
Permalink
[lvs-users] LVS-DR + 2 pools in 2 networks = hair pulling

Hey guys,

I would like to have to following flow with A and B being two different networks (one public and one private range) on the same load balancer.

InternetClient-->VIPA-->RIPA1orRIPA2-->VIPB-->RIPB1orRIPB2
^........replyA........'^........replyB.......'

*Description*
A client hits a serverA in pool A via VIPA which serverA with its RIP (RIPA1 or RIPA2) will hits VIPB and will receive an answer from a server in pool B. Everything works until RIPA1 or RIPA2 tries to connect to VIPB. The packets arrives from RIPA1 or RIPA2 on the load balancer (lb1) then nothing, it's like the packet disappears.

lb1 is both networks A _and_ B and only use one gateway, gateway from A.
lb1 eth0=123.2.2.20/24 - gw is 123.2.2.1
lb1 eth1= 10.1.1.10/24 - no gw

A servers are only in network A and use their respective gateway for this network
VIPA=123.2.2.21/24 (on eth0 so gw is 123.2.2.1)
RIPA1=123.2.2.22
RIPA2=123.2.2.23

B servers are only in network B and use their respective gateway for this network
VIPB=10.1.1.11/24 (on eth1 so gw is still 123.2.2.1)
RIPB1=10.1.1.12
RIPB2=10.1.1.13

*Screenshots*

lb1# ipvsadm -Ln
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 123.2.2.21:443 rr
-> 123.2.2.22:443 Route 1 0 0
-> 123.2.2.23:443 Route 1 0 0
TCP 10.1.1.11:80 rr
-> 10.1.1.12:80 Route 1 0 0
-> 10.1.1.13:80 Route 1 0 0

lb1# netstat -nr
Destination Gateway Genmask Flags MSS Window irtt Iface
123.2.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
0.0.0.0 123.2.2.1 0.0.0.0 UG 0 0 0 eth0

lb1# tcpdump -i eth1 ip dst 10.1.1.11 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
19:18:50.969769 IP 123.2.2.23.60877 > 10.1.1.11.http: Flags [S], seq 1331246417, win 5840, options [mss 1460,nop,nop,TS val 1603281160 ecr 0,nop,wscale 7], length 0

lb1# arp -a
RIPA1(123.2.2.22) at 00:18:51:28:aa:d3 [ether] on eth0
RIPB1 (10.1.1.12) at 00:18:51:5f:cd:11 [ether] on eth1
RIPA2 (123.2.2.23) at 00:18:51:e0:c6:e3 [ether] on eth0
gwA (123.200.162.193) at 00:18:19:9e:cf:ef [ether] on eth0
RIPB2 (10.1.1.13) at 00:18:51:9f:88:bd [ether] on eth1

*What I see*
Everything is load balanced properly in pool A 123.2.2.0/24 from ANY networks.
Everything is load balanced properly in pool B 10.1.1.0/24 from this network ONLY. So any client in 10.1.1.0/24 will be load balanced and will hit RIPB1 or RIPB2.
But when lets say RIPA1 (123.2.2.22) sends a packet to VIPB (10.1.1.11). I see the packet coming in via eth1 on the loadbalancer lb1 and then nothing. No ARP rewrite nothing. Weirdly enough, desperate, I changed the gw on lb1 to be 10.1.1.1, gateway of B. So it's not 123.2.2.1 anymore. And guess what then it works but I lose a working pool A. It's like I cannot have both. So why should I have a gateway to make it work ? The packet is coming on the right interface (eth1) therefore the good network, so an ARP rewrite should happen and bob's your uncle... except it's not ;)

*What I CANNOT see*
A packet coming on one of the B server (RIPB1 or RIPB2)

*Questions*
1. So is it possible to use one load balancer to load balance 2 differents network __AND__ let the servers in the pools to hit each others. Or is it wrong by design ?
2. Where is the packet going on the load balancer ? How can I track it ? (tcpdump is not enough or I am doing it wrong):

*Software*
- Centos 6.2 - kernel: 2.6.32-220.4.2.el6.x86_64
- keepalived
- VMs on VMWare with VMXNET3 NICs

Thanks for reading so far. Any kind of hints will be greatly appreciated. I am really curious to understand why this happens.

Cheers,
Tom

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


david at davidcoulson

Apr 19, 2012, 4:02 AM

Post #2 of 3 (478 views)
Permalink
Re: [lvs-users] LVS-DR + 2 pools in 2 networks = hair pulling [In reply to]

The tcpdump below indicates a packet going to the 10.1.1 network, not
the packet from 123.2.2, at least from an Ethernet perspective. You
should be able to have tcpdump log src/dst MAC addresses also, which is
helpful with LVS-DR.

Have you tried disabling reverse path filtering?

echo 0 > /proc/sys/net/conf/eth0/rp_filter

Probably should just do it for both interfaces. Not sure if tit will
help, but since when you change your default gw it works, it might be
worth a shot.

On 4/19/12 5:49 AM, Thomas wrote:
> Hey guys,
>
> I would like to have to following flow with A and B being two different networks (one public and one private range) on the same load balancer.
>
> InternetClient-->VIPA-->RIPA1orRIPA2-->VIPB-->RIPB1orRIPB2
> ^........replyA........'^........replyB.......'
>
> *Description*
> A client hits a serverA in pool A via VIPA which serverA with its RIP (RIPA1 or RIPA2) will hits VIPB and will receive an answer from a server in pool B. Everything works until RIPA1 or RIPA2 tries to connect to VIPB. The packets arrives from RIPA1 or RIPA2 on the load balancer (lb1) then nothing, it's like the packet disappears.
>
> lb1 is both networks A _and_ B and only use one gateway, gateway from A.
> lb1 eth0=123.2.2.20/24 - gw is 123.2.2.1
> lb1 eth1= 10.1.1.10/24 - no gw
>
> A servers are only in network A and use their respective gateway for this network
> VIPA=123.2.2.21/24 (on eth0 so gw is 123.2.2.1)
> RIPA1=123.2.2.22
> RIPA2=123.2.2.23
>
> B servers are only in network B and use their respective gateway for this network
> VIPB=10.1.1.11/24 (on eth1 so gw is still 123.2.2.1)
> RIPB1=10.1.1.12
> RIPB2=10.1.1.13
>
> *Screenshots*
>
> lb1# ipvsadm -Ln
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 123.2.2.21:443 rr
> -> 123.2.2.22:443 Route 1 0 0
> -> 123.2.2.23:443 Route 1 0 0
> TCP 10.1.1.11:80 rr
> -> 10.1.1.12:80 Route 1 0 0
> -> 10.1.1.13:80 Route 1 0 0
>
> lb1# netstat -nr
> Destination Gateway Genmask Flags MSS Window irtt Iface
> 123.2.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
> 10.1.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
> 0.0.0.0 123.2.2.1 0.0.0.0 UG 0 0 0 eth0
>
> lb1# tcpdump -i eth1 ip dst 10.1.1.11 -n
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
> 19:18:50.969769 IP 123.2.2.23.60877> 10.1.1.11.http: Flags [S], seq 1331246417, win 5840, options [mss 1460,nop,nop,TS val 1603281160 ecr 0,nop,wscale 7], length 0
>
> lb1# arp -a
> RIPA1(123.2.2.22) at 00:18:51:28:aa:d3 [ether] on eth0
> RIPB1 (10.1.1.12) at 00:18:51:5f:cd:11 [ether] on eth1
> RIPA2 (123.2.2.23) at 00:18:51:e0:c6:e3 [ether] on eth0
> gwA (123.200.162.193) at 00:18:19:9e:cf:ef [ether] on eth0
> RIPB2 (10.1.1.13) at 00:18:51:9f:88:bd [ether] on eth1
>
> *What I see*
> Everything is load balanced properly in pool A 123.2.2.0/24 from ANY networks.
> Everything is load balanced properly in pool B 10.1.1.0/24 from this network ONLY. So any client in 10.1.1.0/24 will be load balanced and will hit RIPB1 or RIPB2.
> But when lets say RIPA1 (123.2.2.22) sends a packet to VIPB (10.1.1.11). I see the packet coming in via eth1 on the loadbalancer lb1 and then nothing. No ARP rewrite nothing. Weirdly enough, desperate, I changed the gw on lb1 to be 10.1.1.1, gateway of B. So it's not 123.2.2.1 anymore. And guess what then it works but I lose a working pool A. It's like I cannot have both. So why should I have a gateway to make it work ? The packet is coming on the right interface (eth1) therefore the good network, so an ARP rewrite should happen and bob's your uncle... except it's not ;)
>
> *What I CANNOT see*
> A packet coming on one of the B server (RIPB1 or RIPB2)
>
> *Questions*
> 1. So is it possible to use one load balancer to load balance 2 differents network __AND__ let the servers in the pools to hit each others. Or is it wrong by design ?
> 2. Where is the packet going on the load balancer ? How can I track it ? (tcpdump is not enough or I am doing it wrong):
>
> *Software*
> - Centos 6.2 - kernel: 2.6.32-220.4.2.el6.x86_64
> - keepalived
> - VMs on VMWare with VMXNET3 NICs
>
> Thanks for reading so far. Any kind of hints will be greatly appreciated. I am really curious to understand why this happens.
>
> Cheers,
> Tom
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


tg01 at calixo

Apr 28, 2012, 11:55 AM

Post #3 of 3 (435 views)
Permalink
Re: [lvs-users] LVS-DR + 2 pools in 2 networks = hair pulling [In reply to]

Re,

> Have you tried disabling reverse path filtering?
> echo 0 > /proc/sys/net/conf/eth0/rp_filter

Spot on! This was the issue.

I set up rp_filter (I didn't know this parameter before your answer) to
be 'loose'. Indeed in my case the packet was silently dropped because a
packet coming from 123.2.2 was coming on eth1 being the 10.1.1 network.
And because eth0 was 123.2.2, the kernel thought "Wait a minute a packet
from 123.2.2 should be seen on eth0, not eth1! This does not look right,
I am going to, silently, drop it !". Being "loose", echo 2 >
/proc/sys/net/conf/eth1/rp_filter allows any networks on the server to
be allowed to arrive on eth1. So eth0's network is ok to be seen on
eth1. Disabling rp_filter (echo 0) allows anything, which would work as
well but the loose option (echo 2) is more conservative without being
too much of a nazi :)

Anyway, thanks a lot David for pointing that out !

Cheers,
Tom


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.