Mailing List Archive: Linux Virtual Server: Users

[lvs-users] LVS/TUN Help with FWMARK

Index | Next | Previous | View Threaded

bodycare_5 at live

Jun 27, 2012, 8:13 PM

Post #1 of 4 (398 views)
Permalink

[lvs-users] LVS/TUN Help with FWMARK

Dear LVS Users,

First of all, thank you for the excellent product

I am a bit stuck. I would apreciate some of you experts have a look at this.

I have 2 machines (they are in different datacenters):

############# SETUP ############################################
Both machines use:
RHEL6 x64 2.6.32-131.0.15.el6.x86_64
iptables v1.4.7
IP Virtual Server version 1.2.1 (size=4096)

MACHINE 1
1.1.1.1 (eth0)
1.1.1.3 secondary ip (ip addr add 1.1.1.3 dev eth0)

MACHINE 2
2.2.2.2 (eth0)
2.2.2.3 secondary ip (ip addr add 2.2.2.3 dev eth0)

############# PURPOSE ###########################################
I am trying to have ip 1.1.1.3 accessible on MACHINE 2 (in its simplest explanation)

############# COMMANDS ##########################################
On MACHINE 1, I run:

iptables -F
iptables -F -t mangle
iptables -A INPUT -p tcp -j ACCEPT
iptables -A INPUT -p udp -j ACCEPT
iptables -t mangle -A PREROUTING -i eth0 -p tcp -d 1.1.1.3 -m state ! --state ESTABLISHED -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -i eth0 -p udp -d 1.1.1.3 -m state ! --state ESTABLISHED -j MARK --set-mark 1
ipvsadm -C
ipvsadm -A -f 1 -s rr
ipvsadm -a -f 1 -r 2.2.2.2 -i

On MACHINE 2, I run:

service iptables stop
modprobe ipip
ip addr add 1.1.1.3/32 dev tunl0
ip link set tunl0 up arp off
nc -l -v -D 23

############# LOGS ################################################
On MACHINE 1:

tcpdump -lnnn host 2.2.2.2
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
05:41:35.367295 IP 1.1.1.1 > 2.2.2.2: IP cl.ie.nt.IP.53178 > 1.1.1.3.23: Flags [S], seq 64858347, win 8192, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
05:41:38.369261 IP 1.1.1.1 > 2.2.2.2: IP cl.ie.nt.IP.53178 > 1.1.1.3.23: Flags [S], seq 64858347, win 8192, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
05:41:44.374032 IP 1.1.1.1 > 2.2.2.2: IP cl.ie.nt.IP.53178 > 1.1.1.3.23: Flags [S], seq 64858347, win 8192, options [mss 1400,nop,nop,sackOK], length 0 (ipip-proto-4)

ipvsadm -L -n --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
FWM 1 5 13 0 760 0
-> 2.2.2.2:0 5 13 0 760 0

On MACHINE 2:

tcpdump -lnnn -i tunl0 host cl.ie.nt.IP
listening on tunl0, link-type RAW (Raw IP), capture size 65535 bytes
05:32:15.694994 IP cl.ie.nt.IP.52809 > 1.1.1.3.23: Flags [S], seq 4020678079, win 8192, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0
05:32:18.703968 IP cl.ie.nt.IP.52809 > 1.1.1.3.23: Flags [S], seq 4020678079, win 8192, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0
05:32:24.710999 IP cl.ie.nt.IP.52809 > 1.1.1.3.23: Flags [S], seq 4020678079, win 8192, options [mss 1400,nop,nop,sackOK], length 0

tcpdump -lnnn -i eth0 host 1.1.1.1
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
05:29:48.719022 IP 1.1.1.1 > 2.2.2.2: IP cl.ie.nt.IP.52711 > 1.1.1.3.23: Flags [S], seq 2945894575, win 8192, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
05:29:51.730887 IP 1.1.1.1 > 2.2.2.2: IP cl.ie.nt.IP.52711 > 1.1.1.3.23: Flags [S], seq 2945894575, win 8192, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
05:29:57.737162 IP 1.1.1.1 > 2.2.2.2: IP cl.ie.nt.IP.52711 > 1.1.1.3.23: Flags [S], seq 2945894575, win 8192, options [mss 1400,nop,nop,sackOK], length 0 (ipip-proto-4)
######################################################################

No matter what I do, I am not able to connect to netcat on MACHINE2 when I try: telnet 1.1.1.3 <enter>

I don't think DC is blocking packets. Doing "traceroute -s 1.1.1.3 some.other.ip" showed that "some.other.ip" indeed saw connections coming from 1.1.1.3

I would be eternally grateful if someone can help.

Jenny

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

ja at ssi

Jun 27, 2012, 11:56 PM

Post #2 of 4 (392 views)
Permalink

Re: [lvs-users] LVS/TUN Help with FWMARK [In reply to]

Hello,

On Thu, 28 Jun 2012, Jenny Lee wrote:

> On MACHINE 2, I run:
>
> service iptables stop
> modprobe ipip
> ip addr add 1.1.1.3/32 dev tunl0

Can this help?

cat /proc/sys/net/ipv4/conf/tunl0/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter

> ip link set tunl0 up arp off
> nc -l -v -D 23
>
> ############# LOGS ################################################
> On MACHINE 1:
>
> tcpdump -lnnn host 2.2.2.2
> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
> 05:41:35.367295 IP 1.1.1.1 > 2.2.2.2: IP cl.ie.nt.IP.53178 > 1.1.1.3.23: Flags [S], seq 64858347, win 8192, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
> 05:41:38.369261 IP 1.1.1.1 > 2.2.2.2: IP cl.ie.nt.IP.53178 > 1.1.1.3.23: Flags [S], seq 64858347, win 8192, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
> 05:41:44.374032 IP 1.1.1.1 > 2.2.2.2: IP cl.ie.nt.IP.53178 > 1.1.1.3.23: Flags [S], seq 64858347, win 8192, options [mss 1400,nop,nop,sackOK], length 0 (ipip-proto-4)
>
> ipvsadm -L -n --stats
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
> -> RemoteAddress:Port
> FWM 1 5 13 0 760 0
> -> 2.2.2.2:0 5 13 0 760 0
>
>
> On MACHINE 2:
>
> tcpdump -lnnn -i tunl0 host cl.ie.nt.IP
> listening on tunl0, link-type RAW (Raw IP), capture size 65535 bytes
> 05:32:15.694994 IP cl.ie.nt.IP.52809 > 1.1.1.3.23: Flags [S], seq 4020678079, win 8192, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0
> 05:32:18.703968 IP cl.ie.nt.IP.52809 > 1.1.1.3.23: Flags [S], seq 4020678079, win 8192, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0
> 05:32:24.710999 IP cl.ie.nt.IP.52809 > 1.1.1.3.23: Flags [S], seq 4020678079, win 8192, options [mss 1400,nop,nop,sackOK], length 0
>
> tcpdump -lnnn -i eth0 host 1.1.1.1
> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
> 05:29:48.719022 IP 1.1.1.1 > 2.2.2.2: IP cl.ie.nt.IP.52711 > 1.1.1.3.23: Flags [S], seq 2945894575, win 8192, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
> 05:29:51.730887 IP 1.1.1.1 > 2.2.2.2: IP cl.ie.nt.IP.52711 > 1.1.1.3.23: Flags [S], seq 2945894575, win 8192, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0 (ipip-proto-4)
> 05:29:57.737162 IP 1.1.1.1 > 2.2.2.2: IP cl.ie.nt.IP.52711 > 1.1.1.3.23: Flags [S], seq 2945894575, win 8192, options [mss 1400,nop,nop,sackOK], length 0 (ipip-proto-4)
> ######################################################################
>
> No matter what I do, I am not able to connect to netcat on MACHINE2 when I try: telnet 1.1.1.3 <enter>
>
> I don't think DC is blocking packets. Doing "traceroute -s 1.1.1.3 some.other.ip" showed that "some.other.ip" indeed saw connections coming from 1.1.1.3
>
> I would be eternally grateful if someone can help.
>
> Jenny

Regards

--
Julian Anastasov <ja [at] ssi>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

brouer at redhat

Jun 28, 2012, 2:46 AM

Post #3 of 4 (392 views)
Permalink

Re: [lvs-users] LVS/TUN Help with FWMARK [In reply to]

On Thu, 2012-06-28 at 09:56 +0300, Julian Anastasov wrote:
> On Thu, 28 Jun 2012, Jenny Lee wrote:
>
> > On MACHINE 2, I run:
> >
> > service iptables stop
> > modprobe ipip
> > ip addr add 1.1.1.3/32 dev tunl0
>
> Can this help?
>
> cat /proc/sys/net/ipv4/conf/tunl0/rp_filter
> echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter

Hi Jenny

The problem is most likely Reverse Path Filtering, like Julian is
pointing out.

Please provide output of command:
grep . /proc/sys/net/ipv4/conf/*/rp_filter

You also have to make sure the the "all" rp_filter = 0, by running:

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

For IPVS TUN mode I recommend that you edit your /etc/sysctl.conf and
add:
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0

You mentioned your were using different datacenters. Do notice that the
datacenter might also have implemented Reverse Path Filtering on their
border routers, which will cause your responses to be dropped... but
this should at least get you a step further.

--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Sr. Network Kernel Developer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

bodycare_5 at live

Jun 28, 2012, 5:17 AM

Post #4 of 4 (386 views)
Permalink

Re: [lvs-users] LVS/TUN Help with FWMARK [In reply to]

----------------------------------------
> Subject: Re: [lvs-users] LVS/TUN Help with FWMARK
> From: brouer [at] redhat
> To: lvs-users [at] linuxvirtualserver
> CC: bodycare_5 [at] live
> Date: Thu, 28 Jun 2012 11:46:22 +0200
>
> On Thu, 2012-06-28 at 09:56 +0300, Julian Anastasov wrote:
> > On Thu, 28 Jun 2012, Jenny Lee wrote:
> >
> > > On MACHINE 2, I run:
> > >
> > > service iptables stop
> > > modprobe ipip
> > > ip addr add 1.1.1.3/32 dev tunl0
> >
> > Can this help?
> >
> > cat /proc/sys/net/ipv4/conf/tunl0/rp_filter
> > echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
>
> Hi Jenny
>
> The problem is most likely Reverse Path Filtering, like Julian is
> pointing out.
>
> Please provide output of command:
> grep . /proc/sys/net/ipv4/conf/*/rp_filter
>
> You also have to make sure the the "all" rp_filter = 0, by running:
>
> echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
>
>
> For IPVS TUN mode I recommend that you edit your /etc/sysctl.conf and
> add:
> net.ipv4.conf.default.rp_filter = 0
> net.ipv4.conf.all.rp_filter = 0
>

Jesus! I spent 2 entire days days on this.

This fixed it for me on RS1 (they were all 1's except "all"): for i in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 0 > $i; done

Now I can access telnet on RS1 when I am connecting to an IP on MACHINE1 (VIP).

Thank you so much Jullian and Jesper.

Correspondence works both ways (if I type something on nc on RS1, client sees it), so I am assuming DC is not doing RP filtering (Hurricane Electric).

Now a little bit constructive criticism ... Site and FAQS and everything has too much clutter. Some effort must be done to remove all references to a 2.0.36 kernel and ipchains. It is hard to understand what is what (I read entire austintek FAQS which were full of outdated information and served no purpose except to confuse the matters). For example, I know i saw rp filtering references, but i could not find them again.

Thanks again... soooo much.

Jenny

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads