Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Linux Virtual Server: Users

[lvs-users] lvs masq response package not getting picked up

 

 

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded


kl at vsen

Aug 13, 2012, 4:20 AM

Post #1 of 8 (536 views)
Permalink
[lvs-users] lvs masq response package not getting picked up

Hi,

I've setup a server, with public IPs (currently only one though) on one
interface, and an internal ip on another interface - where the backend
webservers are hosted.

My problem is that I see the request package (going to X.X.X.167 port
80) go through to the backend fine (the Syn package) and I see the
backend (which has the LVS server as default gw) response when I tcpdump
on the internal interface of the LVS server.

The response however, is newer relayed on the LVS servers external
interface :(

I hope you can help me debug, or suggest things I could do, to debug.

My setup is as follows:

I've setup ipvs in kernel, using ldirectord - ipvsadm -ln shows:
TCP X.X.X.167:80 wrr persistent 900
-> Y.Y.Y.105:80 Masq 2 0 0
TCP X.X.X.167:443 wrr persistent 900
-> Y.Y.Y.105:443 Masq 2 0 0

with X.X.X.167 being my public ip, and Y.Y.Y.105 being the internal one.

Iptables is enabled and output is like this:
[root [at] lb ~]# iptables -L -n
Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
RELATED,ESTABLISHED
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW
tcp dpt:22
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
ACCEPT tcp -- B.B.B.162 0.0.0.0/0 state NEW tcp
dpt:8080
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with
icmp-host-prohibited

Chain FORWARD (policy DROP)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW
tcp dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW
tcp dpt:443
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
RELATED,ESTABLISHED
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with
icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

[root [at] lb ~]# iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

--
Regards,
Klavs Klavsen, GSEC - kl [at] vsen - http://www.vsen.dk - Tlf. 61281200

"Those who do not understand Unix are condemned to reinvent it, poorly."
--Henry Spencer


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


kl at vsen

Aug 13, 2012, 4:26 AM

Post #2 of 8 (511 views)
Permalink
Re: [lvs-users] lvs masq response package not getting picked up [In reply to]

Y.Y.Y.105 is not the IP of the LVS server internal interface ofcourse -
it's the ip of the backendserver.

Klavs Klavsen said the following on 08/13/2012 01:20 PM:
> Hi,
>
> I've setup a server, with public IPs (currently only one though) on one
> interface, and an internal ip on another interface - where the backend
> webservers are hosted.
>
> My problem is that I see the request package (going to X.X.X.167 port
> 80) go through to the backend fine (the Syn package) and I see the
> backend (which has the LVS server as default gw) response when I tcpdump
> on the internal interface of the LVS server.
>
> The response however, is newer relayed on the LVS servers external
> interface :(
>
> I hope you can help me debug, or suggest things I could do, to debug.
>
> My setup is as follows:
>
> I've setup ipvs in kernel, using ldirectord - ipvsadm -ln shows:
> TCP X.X.X.167:80 wrr persistent 900
> -> Y.Y.Y.105:80 Masq 2 0 0
> TCP X.X.X.167:443 wrr persistent 900
> -> Y.Y.Y.105:443 Masq 2 0 0
>
> with X.X.X.167 being my public ip, and Y.Y.Y.105 being the internal one.
>
> Iptables is enabled and output is like this:
> [root [at] lb ~]# iptables -L -n
> Chain INPUT (policy DROP)
> target prot opt source destination
> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
> RELATED,ESTABLISHED
> ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW
> tcp dpt:22
> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
> ACCEPT tcp -- B.B.B.162 0.0.0.0/0 state NEW tcp
> dpt:8080
> REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with
> icmp-host-prohibited
>
> Chain FORWARD (policy DROP)
> target prot opt source destination
> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW
> tcp dpt:80
> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW
> tcp dpt:443
> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
> RELATED,ESTABLISHED
> REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with
> icmp-host-prohibited
>
> Chain OUTPUT (policy ACCEPT)
> target prot opt source destination
>
> [root [at] lb ~]# iptables -L -n -t nat
> Chain PREROUTING (policy ACCEPT)
> target prot opt source destination
>
> Chain POSTROUTING (policy ACCEPT)
> target prot opt source destination
>
> Chain OUTPUT (policy ACCEPT)
> target prot opt source destination
>


--
Regards,
Klavs Klavsen, GSEC - kl [at] vsen - http://www.vsen.dk - Tlf. 61281200

"Those who do not understand Unix are condemned to reinvent it, poorly."
--Henry Spencer


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


kl at vsen

Aug 13, 2012, 4:44 AM

Post #3 of 8 (512 views)
Permalink
Re: [lvs-users] lvs masq response package not getting picked up [In reply to]

I enabled logging in iptables for both INPUT and FORWARD, and got this
in logs:
IN=eth2 OUT=eth2 SRC=Y.Y.Y.105 DST=MyInternetClientIP LEN=60 TOS=0x00
PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=80 DPT=51617 WINDOW=5792 RES=0x00
ACK SYN URGP=0

The odd thing, is that eth2 is internal and eth3 is internet. That
package should have been destined for eth3 (seeing as that's where it
came from originally) - but perhaps that some LVS detail?

Klavs Klavsen said the following on 08/13/2012 01:20 PM:
> Hi,
>
> I've setup a server, with public IPs (currently only one though) on one
> interface, and an internal ip on another interface - where the backend
> webservers are hosted.
>
> My problem is that I see the request package (going to X.X.X.167 port
> 80) go through to the backend fine (the Syn package) and I see the
> backend (which has the LVS server as default gw) response when I tcpdump
> on the internal interface of the LVS server.
>
> The response however, is newer relayed on the LVS servers external
> interface :(
>
> I hope you can help me debug, or suggest things I could do, to debug.
>
> My setup is as follows:
>
> I've setup ipvs in kernel, using ldirectord - ipvsadm -ln shows:
> TCP X.X.X.167:80 wrr persistent 900
> -> Y.Y.Y.105:80 Masq 2 0 0
> TCP X.X.X.167:443 wrr persistent 900
> -> Y.Y.Y.105:443 Masq 2 0 0
>
> with X.X.X.167 being my public ip, and Y.Y.Y.105 being the internal one.
>
> Iptables is enabled and output is like this:
> [root [at] lb ~]# iptables -L -n
> Chain INPUT (policy DROP)
> target prot opt source destination
> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
> RELATED,ESTABLISHED
> ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW
> tcp dpt:22
> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
> ACCEPT tcp -- B.B.B.162 0.0.0.0/0 state NEW tcp
> dpt:8080
> REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with
> icmp-host-prohibited
>
> Chain FORWARD (policy DROP)
> target prot opt source destination
> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW
> tcp dpt:80
> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW
> tcp dpt:443
> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
> RELATED,ESTABLISHED
> REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with
> icmp-host-prohibited
>
> Chain OUTPUT (policy ACCEPT)
> target prot opt source destination
>
> [root [at] lb ~]# iptables -L -n -t nat
> Chain PREROUTING (policy ACCEPT)
> target prot opt source destination
>
> Chain POSTROUTING (policy ACCEPT)
> target prot opt source destination
>
> Chain OUTPUT (policy ACCEPT)
> target prot opt source destination
>


--
Regards,
Klavs Klavsen, GSEC - kl [at] vsen - http://www.vsen.dk - Tlf. 61281200

"Those who do not understand Unix are condemned to reinvent it, poorly."
--Henry Spencer


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


graeme at graemef

Aug 13, 2012, 4:46 AM

Post #4 of 8 (509 views)
Permalink
Re: [lvs-users] lvs masq response package not getting picked up [In reply to]

On Mon, 2012-08-13 at 13:20 +0200, Klavs Klavsen wrote:
> Chain FORWARD (policy DROP)
> target prot opt source destination
> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW
> tcp dpt:80
> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW
> tcp dpt:443
> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
> RELATED,ESTABLISHED
> REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with
> icmp-host-prohibited

I'm not 100% sure, but it looks like this is your problem. Remove those
rules and see what happens.

* I say "not sure" because I'm not sure whether the incoming packets
will traverse the FORWARD chain or be hoiked past it by ipvs.

Graeme


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


kl at vsen

Aug 13, 2012, 4:54 AM

Post #5 of 8 (509 views)
Permalink
Re: [lvs-users] lvs masq response package not getting picked up [In reply to]

Appearently they are going through FORWARD - with the source IP of the
backend - instead of the sourceIP of the VIP - that the client actually
accessed.

Also - for some reason there's no state - so I had to allow ALL packages
with source-port of 80 or 443 in the FORWARD chain.

Not exactly great for a secure setup :(

Graeme Fowler said the following on 08/13/2012 01:46 PM:
> On Mon, 2012-08-13 at 13:20 +0200, Klavs Klavsen wrote:
>> Chain FORWARD (policy DROP)
>> target prot opt source destination
>> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW
>> tcp dpt:80
>> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW
>> tcp dpt:443
>> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
>> RELATED,ESTABLISHED
>> REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with
>> icmp-host-prohibited
> I'm not 100% sure, but it looks like this is your problem. Remove those
> rules and see what happens.
>
> * I say "not sure" because I'm not sure whether the incoming packets
> will traverse the FORWARD chain or be hoiked past it by ipvs.
>
> Graeme
>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users


--
Regards,
Klavs Klavsen, GSEC - kl [at] vsen - http://www.vsen.dk - Tlf. 61281200

"Those who do not understand Unix are condemned to reinvent it, poorly."
--Henry Spencer


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


graeme at graemef

Aug 13, 2012, 5:11 AM

Post #6 of 8 (514 views)
Permalink
Re: [lvs-users] lvs masq response package not getting picked up [In reply to]

On Mon, 2012-08-13 at 13:54 +0200, Klavs Klavsen wrote:
> Appearently they are going through FORWARD - with the source IP of the
> backend - instead of the sourceIP of the VIP - that the client actually
> accessed.

You're using LVS-NAT. The only place the VIP is present in the usual
usage of this is in the external (client-facing) interface of the
director.

> Also - for some reason there's no state - so I had to allow ALL packages
> with source-port of 80 or 443 in the FORWARD chain.

ipvs works in tandem with netfilter (is part of it nowadays,
effectively), so state is recorded in the usual way in the conntrack
tables. If yours isn't, then you may be using an old enough kernel that
this doesn't happen or you don't have the appropriate netfilter modules
loaded.

Graeme


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


kl at vsen

Aug 13, 2012, 6:05 AM

Post #7 of 8 (512 views)
Permalink
Re: [lvs-users] lvs masq response package not getting picked up [In reply to]

Graeme Fowler said the following on 08/13/2012 02:11 PM:
[CUT]
> You're using LVS-NAT. The only place the VIP is present in the usual
> usage of this is in the external (client-facing) interface of the
> director.
Ok. thank you for clarifying. So the external Ip would be in the OUTPUT
chain, and I could filter more specificly there (unless I get state
working - which would be preferable :)

>> Also - for some reason there's no state - so I had to allow ALL packages
>> with source-port of 80 or 443 in the FORWARD chain.
> ipvs works in tandem with netfilter (is part of it nowadays,
> effectively), so state is recorded in the usual way in the conntrack
> tables. If yours isn't, then you may be using an old enough kernel that
> this doesn't happen or you don't have the appropriate netfilter modules
> loaded.

It's CentOS 6 - 2.6.32-220.el6.x86_64

Is that too old?

These modules are loaded:
nf_conntrack_ipv4 9506 4
nf_defrag_ipv4 1483 1 nf_conntrack_ipv4
nf_conntrack_ipv6 8748 2
nf_defrag_ipv6 12182 1 nf_conntrack_ipv6
nf_conntrack 79453 3 nf_conntrack_ipv4,nf_conntrack_ipv6,xt_state
ipv6 322029 38
ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6

Thank you for your help.

--
Regards,
Klavs Klavsen, GSEC - kl [at] vsen - http://www.vsen.dk - Tlf. 61281200

"Those who do not understand Unix are condemned to reinvent it, poorly."
--Henry Spencer


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


kl at vsen

Aug 13, 2012, 6:31 AM

Post #8 of 8 (510 views)
Permalink
Re: [lvs-users] lvs masq response package not getting picked up [In reply to]

Klavs Klavsen said the following on 08/13/2012 03:05 PM:
> Ok. thank you for clarifying. So the external Ip would be in the
> OUTPUT chain, and I could filter more specificly there (unless I get
> state working - which would be preferable :)

I believe I just confused how iptables vs. ipchains worked - sorry for
being daft :)

--
Regards,
Klavs Klavsen, GSEC - kl [at] vsen - http://www.vsen.dk - Tlf. 61281200

"Those who do not understand Unix are condemned to reinvent it, poorly."
--Henry Spencer


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.