Mailing List Archive: Linux Virtual Server: Users

[lvs-users] lvs-tun MTU and fragmentation

Index | Next | Previous | View Threaded

sameer.garg at gmail

Aug 31, 2007, 5:33 AM

Post #1 of 10 (902 views)
Permalink

[lvs-users] lvs-tun MTU and fragmentation

Hi All,

In my quest for setting up LVS-Tun we encountered another problem.
MTU and fragmentation is a known problem. First some back ground
information.

1. We can't use LVS-DR because our machines are all over the place.
(Thank the ISP)
2. The setup comprises of 2 machines. 1 director and 1 Real Server.
3. We are using RHEL 4.

The setup works when the packet size is small. When the packet size
increase (because of cookies) we experience timeouts.

I have gone through the MTU fragmentation section of the Howto but
haven't been able to solve the problem. We changed the MTU value on
the director, RS and tunl0 individually and combined but that didn't
help either.

Am pasting the tcpdump output on the director.

17:09:49.348719 122.162.81.169.50120 > 198.65.11.8.http: S [tcp sum
ok] 2024274182:2024274182(0) win 65535 <mss 1452,nop,wscale
1,nop,nop,timestamp 51880267 0,sackOK,eol> (DF) (ttl 51, id 43796, len
64)
17:09:49.645469 122.162.81.169.50120 > 198.65.11.8.http: . [tcp sum
ok] ack 2909078 win 33120 <nop,nop,timestamp 51880565 881526975> (DF)
(ttl 51, id 43809, len 52)
17:09:49.670865 122.162.81.169.50120 > 198.65.11.8.http: .
0:1440(1440) ack 1 win 33120 <nop,nop,timestamp 51880565 881526975>
(DF) (ttl 51, id 43810, len 1492)
17:09:49.670892 198.65.11.8 > 122.162.81.169: icmp: 198.65.11.8
unreachable - need to frag (mtu 1480) [tos 0xc0] (ttl 64, id 64006,
len 576)
17:09:49.670867 122.162.81.169.50120 > 198.65.11.8.http: P [tcp sum
ok] 1440:1446(6) ack 1 win 33120 <nop,nop,timestamp 51880565
881526975> (DF) (ttl 51, id 43811, len 58)
17:09:50.770837 122.162.81.169.50120 > 198.65.11.8198.65.11.8.http: .
0:1440(1440) ack 1 win 33120 <nop,nop,timestamp 51881662 881527298>
(DF) (ttl 51, id 43856, len 1492)
17:09:50.770859 198.65.11.8 > 122.162.81.169: icmp: 198.65.11.8
unreachable - need to frag (mtu 1480) [tos 0xc0] (ttl 64, id 64007,
len 576)
17:09:52.758639 122.162.81.169.50120 > 198.65.11.8.http: .
0:1440(1440) ack 1 win 33120 <nop,nop,timestamp 51883656 881527298>
(DF) (ttl 51, id 43938, len 1492)
17:09:52.758660 198.65.11.8 > 122.162.81.169: icmp: 198.65.11.8
unreachable - need to frag (mtu 1480) [tos 0xc0] (ttl 64, id 64008,
len 576)
17:09:56.548754 122.162.81.169.50120 > 198.65.11.8.http: .
0:1440(1440) ack 1 win 33120 <nop,nop,timestamp 51887444 881527298>
(DF) (ttl 51, id 44078, len 1492)
17:09:56.548781 198.65.11.8 > 122.162.81.169: icmp: 198.65.11.8
unreachable - need to frag (mtu 1480) [tos 0xc0] (ttl 64, id 64009,
len 576)

If needed I can post the configuration file of ldirectord.

Thanks.

Sameer

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

graeme at graemef

Aug 31, 2007, 6:19 AM

Post #2 of 10 (870 views)
Permalink

Re: [lvs-users] lvs-tun MTU and fragmentation [In reply to]

On Fri, 2007-08-31 at 18:03 +0530, Sameer Garg wrote:
> In my quest for setting up LVS-Tun we encountered another problem.
> MTU and fragmentation is a known problem. First some back ground
> information.
>
> 1. We can't use LVS-DR because our machines are all over the place.
> (Thank the ISP)
> 2. The setup comprises of 2 machines. 1 director and 1 Real Server.
> 3. We are using RHEL 4.

What you have to remember is as follows:

1. The initial packet from the client to the realserver - the SYN packet
- sets the parameters for the connection.

2. With TUN, the realserver is responding directly to the client.

3. That means if the tun0 interface(s) have an MTU of (for example)
1400, then you need to make the *realserver's VIP interface* have an MTU
of 1400. How you do that varies with OS, but for a Linux server you'd do
the following on the realserver:

iptables -A OUTPUT -s VIRTUAL-IP -p tcp -m tcp --tcp-flags SYN,RST,ACK
SYN,ACK -j TCPMSS --set-mss 1440

...which is in the HOWTO :)

That then means the realserver will respond to the initial ACK with the
MSS set for that connection; the client will see that MSS and *should*
then adjust its' following packet sizes.

You should leave the director and tun0 interfaces to do what they want,
as I recall.

Graeme

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

sameer.garg at gmail

Aug 31, 2007, 6:34 AM

Post #3 of 10 (870 views)
Permalink

Re: [lvs-users] lvs-tun MTU and fragmentation [In reply to]

Yeah I tried that iptables rule on the director and that didn;t work.
I still get

Tcpdump output:
18:59:30.870598 IP 198.64.15.26 >
ABTS-NCR-Dynamic-169.81.162.122.airtelbroadband.in: icmp 556:
198.64.15.26 unreachable - need to frag (mtu 1380)

Iptable rule:
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
TCPMSS tcp -- 198.64.15.26 anywhere tcp
flags:SYN,RST,ACK/SYN,ACK TCPMSS set 1440

-Sameer

On 8/31/07, Graeme Fowler <graeme [at] graemef> wrote:
> On Fri, 2007-08-31 at 18:03 +0530, Sameer Garg wrote:
> > In my quest for setting up LVS-Tun we encountered another problem.
> > MTU and fragmentation is a known problem. First some back ground
> > information.
> >
> > 1. We can't use LVS-DR because our machines are all over the place.
> > (Thank the ISP)
> > 2. The setup comprises of 2 machines. 1 director and 1 Real Server.
> > 3. We are using RHEL 4.
>
> What you have to remember is as follows:
>
> 1. The initial packet from the client to the realserver - the SYN packet
> - sets the parameters for the connection.
>
> 2. With TUN, the realserver is responding directly to the client.
>
> 3. That means if the tun0 interface(s) have an MTU of (for example)
> 1400, then you need to make the *realserver's VIP interface* have an MTU
> of 1400. How you do that varies with OS, but for a Linux server you'd do
> the following on the realserver:
>
> iptables -A OUTPUT -s VIRTUAL-IP -p tcp -m tcp --tcp-flags SYN,RST,ACK
> SYN,ACK -j TCPMSS --set-mss 1440
>
> ...which is in the HOWTO :)
>
> That then means the realserver will respond to the initial ACK with the
> MSS set for that connection; the client will see that MSS and *should*
> then adjust its' following packet sizes.
>
> You should leave the director and tun0 interfaces to do what they want,
> as I recall.
>
> Graeme
>
>
>
>
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

graeme at graemef

Aug 31, 2007, 7:01 AM

Post #4 of 10 (875 views)
Permalink

Re: [lvs-users] lvs-tun MTU and fragmentation [In reply to]

On Fri, 2007-08-31 at 19:04 +0530, Sameer Garg wrote:
> Yeah I tried that iptables rule on the director and that didn;t work.

As I explained, it needs to go on the realserver. That's the device
talking to the client.

Graeme

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

sameer.garg at gmail

Aug 31, 2007, 9:24 AM

Post #5 of 10 (878 views)
Permalink

Re: [lvs-users] lvs-tun MTU and fragmentation [In reply to]

sorry for the confusion. i tried it on the real server and not the director.

sameer

On 8/31/07, Graeme Fowler <graeme [at] graemef> wrote:
> On Fri, 2007-08-31 at 19:04 +0530, Sameer Garg wrote:
> > Yeah I tried that iptables rule on the director and that didn;t work.
>
> As I explained, it needs to go on the realserver. That's the device
> talking to the client.
>
> Graeme
>
>
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

cz at neospire

Aug 31, 2007, 11:40 AM

Post #6 of 10 (877 views)
Permalink

Re: [lvs-users] lvs-tun MTU and fragmentation [In reply to]

Graeme Fowler wrote (at Fri, Aug 31, 2007 at 02:19:49PM +0100):
> 3. That means if the tun0 interface(s) have an MTU of (for example)
> 1400, then you need to make the *realserver's VIP interface* have an MTU
> of 1400. How you do that varies with OS, but for a Linux server you'd do
> the following on the realserver:
>
> iptables -A OUTPUT -s VIRTUAL-IP -p tcp -m tcp --tcp-flags SYN,RST,ACK
> SYN,ACK -j TCPMSS --set-mss 1440
>
> ...which is in the HOWTO :)
>
> That then means the realserver will respond to the initial ACK with the
> MSS set for that connection; the client will see that MSS and *should*
> then adjust its' following packet sizes.
>
> You should leave the director and tun0 interfaces to do what they want,
> as I recall.

But, for some reason that I cannot remember, I have switched off of
this iptables method in favor of using some advanced routing to take
care of the MSS setting. I wish I would have shared with the group
when I started it, because I can't remember why I'm doing it this way now.
Still on the real servers, I use routing like so:

This assumes the VIP is in a class C network

ip route flush table 42
ip route add table 42 to VIP_NETWORK/24 dev eth0 advmss 1440
ip route add table 42 to default via VIP_NETWORK_GATEWAY advmss 1440
ip rule add from VIP table 42 priority 42
ip route flush cache

So, for example, say VIP is 10.2.2.38
VIP_NETWORK is 10.2.2.0
VIP_NETWORK_GATEWAY is 10.2.2.1 (probably)

ip route flush table 42
ip route add table 42 to 10.2.2.0/24 dev eth0 advmss 1440
ip route add table 42 to default via 10.2.2.1 advmss 1440
ip rule add from VIP table 42 priority 42
ip route flush cache

The number 42 is just a number I chose when I started this.

--
--------------------------------------
Casey Zacek
Network Services
NeoSpire, Inc.
1807 Ross Ave., Ste. 300
Dallas, TX 75201
www.neospire.net -- Managed Hosting Solutions
P. 214-468-0768
F. 214-720-1836
--------------------------------------

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Aug 31, 2007, 12:47 PM

Post #7 of 10 (871 views)
Permalink

Re: [lvs-users] lvs-tun MTU and fragmentation [In reply to]

On Fri, 31 Aug 2007, Casey Zacek wrote:

> But, for some reason that I cannot remember,

Ratz (quoted somewhere in the HOWTO) said that the MTU
should be set for the route and not for the device, since
not all routes/packet types to/from a device need an altered
MTU.

> I have switched off of
> this iptables method in favor of using some advanced routing to take
> care of the MSS setting. I wish I would have shared with the group
> when I started it, because I can't remember why I'm doing it this way now.
> Still on the real servers, I use routing like so:
>
> This assumes the VIP is in a class C network
>
> ip route flush table 42
> ip route add table 42 to VIP_NETWORK/24 dev eth0 advmss 1440

I've never had a good grasp on the /32 network size for the
VIP. Do you know why you need /24 (other than you need to
get the packet to the router)? What happens if you use /32?

> ip route add table 42 to default via VIP_NETWORK_GATEWAY advmss 1440
> ip rule add from VIP table 42 priority 42
> ip route flush cache
>
> So, for example, say VIP is 10.2.2.38
> VIP_NETWORK is 10.2.2.0
> VIP_NETWORK_GATEWAY is 10.2.2.1 (probably)
>
> ip route flush table 42
> ip route add table 42 to 10.2.2.0/24 dev eth0 advmss 1440
> ip route add table 42 to default via 10.2.2.1 advmss 1440
> ip rule add from VIP table 42 priority 42
> ip route flush cache
>
> The number 42 is just a number I chose when I started this.

It's the answer to everything :-)

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

cz at neospire

Aug 31, 2007, 1:01 PM

Post #8 of 10 (876 views)
Permalink

Re: [lvs-users] lvs-tun MTU and fragmentation [In reply to]

Joseph Mack NA3T wrote (at Fri, Aug 31, 2007 at 12:47:45PM -0700):
> On Fri, 31 Aug 2007, Casey Zacek wrote:
> > ip route add table 42 to VIP_NETWORK/24 dev eth0 advmss 1440
>
> I've never had a good grasp on the /32 network size for the
> VIP. Do you know why you need /24 (other than you need to
> get the packet to the router)? What happens if you use /32?

For the life of me, I cannot remember.

--
--------------------------------------
Casey Zacek
Network Services
NeoSpire, Inc.
1807 Ross Ave., Ste. 300
Dallas, TX 75201
www.neospire.net -- Managed Hosting Solutions
P. 214-468-0768
F. 214-720-1836
--------------------------------------

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

sameer.garg at gmail

Sep 3, 2007, 1:09 AM

Post #9 of 10 (857 views)
Permalink

Re: [lvs-users] lvs-tun MTU and fragmentation [In reply to]

> So, for example, say VIP is 10.2.2.38
> VIP_NETWORK is 10.2.2.0
> VIP_NETWORK_GATEWAY is 10.2.2.1 (probably)
>
> ip route flush table 42
> ip route add table 42 to 10.2.2.0/24 dev eth0 advmss 1440
> ip route add table 42 to default via 10.2.2.1 advmss 1440
> ip rule add from VIP table 42 priority 42
> ip route flush cache

Based on the instructions above and my interpretation I tried the
following without success.

tunl0/ VIP --- 10.2.2.38
REAL_SERVER_NET --- 10.0.1.0/26
REAL_SERVER_GW --- 10.0.1.1
REAL_SERVER_IP --- 10.0.1.2

This is the proposed setup.
LVS Real Server
------------------------
-----------------------------
| LVS | | IP
10.0.0.2/26 |
| VIP=10.2.2.38 | -------------------------> | tunl0 10.2.2.28/32 |
------------------------ | GW
10.0.1.1 |

------------------------------
ip route flush table 42
ip route add table 42 to REAL_SERVER_NET dev eth0 advmss 1420
ip route add table 42 to default via REAL_SERVER_GW advmss 1420
ip rule add from VIP table 42 priority 42
ip route flush cache

Also tried this on the Real Server without any luck.

iptables -A OUTPUT -s VIP -p tcp -m tcp --tcp-flags SYN,RST,ACK
SYN,ACK -j TCPMSS --set-mss 1440

Thanks.
Sameer

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

sameer.garg at gmail

Sep 6, 2007, 6:43 AM

Post #10 of 10 (852 views)
Permalink

Re: [lvs-users] lvs-tun MTU and fragmentation [In reply to]

By trial and error I was able to find a work around this:

On the director I did the following

# ip route add REAL_SERVER_IP via DIRECTOR_GATEWAY dev eth0 advmss 1400

On the Real Server
# ip route change default via REAL_SERVER_GATEWAY dev eth0 advmss 1400

I am still not sure why I need to make the change on the director
because technically during the three way handshake the real server
should tell the client about MSS being 1400.I have tried it without
making the changes on the director but it doesn't work.

Sameer

On 9/3/07, Sameer Garg <sameer.garg [at] gmail> wrote:
> > So, for example, say VIP is 10.2.2.38
> > VIP_NETWORK is 10.2.2.0
> > VIP_NETWORK_GATEWAY is 10.2.2.1 (probably)
> >
> > ip route flush table 42
> > ip route add table 42 to 10.2.2.0/24 dev eth0 advmss 1440
> > ip route add table 42 to default via 10.2.2.1 advmss 1440
> > ip rule add from VIP table 42 priority 42
> > ip route flush cache
>
>
> Based on the instructions above and my interpretation I tried the
> following without success.
>
>
> tunl0/ VIP --- 10.2.2.38
> REAL_SERVER_NET --- 10.0.1.0/26
> REAL_SERVER_GW --- 10.0.1.1
> REAL_SERVER_IP --- 10.0.1.2
>
>
> This is the proposed setup.
> LVS Real Server
> ------------------------
> -----------------------------
> | LVS | | IP
> 10.0.0.2/26 |
> | VIP=10.2.2.38 | -------------------------> | tunl0 10.2.2.28/32 |
> ------------------------ | GW
> 10.0.1.1 |
>
> ------------------------------
> ip route flush table 42
> ip route add table 42 to REAL_SERVER_NET dev eth0 advmss 1420
> ip route add table 42 to default via REAL_SERVER_GW advmss 1420
> ip rule add from VIP table 42 priority 42
> ip route flush cache
>
>
> Also tried this on the Real Server without any luck.
>
> iptables -A OUTPUT -s VIP -p tcp -m tcp --tcp-flags SYN,RST,ACK
> SYN,ACK -j TCPMSS --set-mss 1440
>
>
> Thanks.
> Sameer
>

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads