Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Linux Virtual Server: Users

[lvs-users] LVS problem with unreachable - need to frag

 

 

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded


khapare77 at gmail

Mar 8, 2012, 7:28 AM

Post #1 of 9 (1349 views)
Permalink
[lvs-users] LVS problem with unreachable - need to frag

Hello all,

I am running pulse on virtual machine for webserver, it seems working
well, but there is one problem with networking I think. It seems there
is a problem with lvs server is not fragmenting larger than 1500
packets size. I did the quick tcpdump and I get following info.

my problem is when user is uploading files it takes for ever and never
completes. I am thinking lvs (pulse) is not not fragmenting the larger
packet size than 1500 hence it never goes to backend server.

16:25:22.543563 00:1f:6d:cf:4e:49 > 52:54:00:b5:30:3c, ethertype IPv4
(0x0800), length 2974: 10.128.1.5.60715 > 10.192.6.1.https: Flags [.],
seq 7705:10625, ack 146, win 16388, length 2920
16:25:22.543590 52:54:00:b5:30:3c > 00:1f:6d:cf:4e:49, ethertype IPv4
(0x0800), length 590: 10.192.6.1 > 10.128.1.5: ICMP 130.208.165.177
unreachable - need to frag (mtu 1500), length 556

I tested following:

1. IP forward is enabled.
2. lro,gro are off on eth0
ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off
3. IPVS version is v1.2.1
4. Backend server has arptables and set to mangaled to right interface
5. Firewall is allowed on both lvs and backend server.
6. http and https are open for all.

Any help in this would be great.

K
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


misch at schwartzkopff

Mar 8, 2012, 9:33 AM

Post #2 of 9 (1306 views)
Permalink
Re: [lvs-users] LVS problem with unreachable - need to frag [In reply to]

> Hello all,
>
> I am running pulse on virtual machine for webserver, it seems working
> well, but there is one problem with networking I think. It seems there
> is a problem with lvs server is not fragmenting larger than 1500
> packets size. I did the quick tcpdump and I get following info.
>
> my problem is when user is uploading files it takes for ever and never
> completes. I am thinking lvs (pulse) is not not fragmenting the larger
> packet size than 1500 hence it never goes to backend server.
>
> 16:25:22.543563 00:1f:6d:cf:4e:49 > 52:54:00:b5:30:3c, ethertype IPv4
> (0x0800), length 2974: 10.128.1.5.60715 > 10.192.6.1.https: Flags [.],
> seq 7705:10625, ack 146, win 16388, length 2920
> 16:25:22.543590 52:54:00:b5:30:3c > 00:1f:6d:cf:4e:49, ethertype IPv4
> (0x0800), length 590: 10.192.6.1 > 10.128.1.5: ICMP 130.208.165.177
> unreachable - need to frag (mtu 1500), length 556
>
> I tested following:
>
> 1. IP forward is enabled.
> 2. lro,gro are off on eth0
> ethtool -k eth0
> Offload parameters for eth0:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: on
> udp-fragmentation-offload: on
> generic-segmentation-offload: on
> generic-receive-offload: off
> large-receive-offload: off
> 3. IPVS version is v1.2.1
> 4. Backend server has arptables and set to mangaled to right interface
> 5. Firewall is allowed on both lvs and backend server.
> 6. http and https are open for all.
>
> Any help in this would be great.

It seems that the LVS cannot send out the packet since the MTU is too large
for the 1500 Byte of the interface. Please don't be confused by the length of
2974 byte that tcpdump shows. tcpdump just sees the packet on the wrong point
on the chain.

Be sure the Path MTU discovery works or clear the DF bit of the packet
entering the load balancer.

Greetings,


--
Dr. Michael Schwartzkopff
Guardinistr. 63
81375 München

Tel: (0163) 172 50 98
Fax: (089) 620 304 13
Attachments: signature.asc (0.19 KB)


khapare77 at gmail

Mar 8, 2012, 11:05 AM

Post #3 of 9 (1309 views)
Permalink
Re: [lvs-users] LVS problem with unreachable - need to frag [In reply to]

Thanks for your quick response.

On Thu, Mar 8, 2012 at 5:33 PM, Michael Schwartzkopff <
misch [at] schwartzkopff> wrote:

> > Hello all,
> >
> > I am running pulse on virtual machine for webserver, it seems working
> > well, but there is one problem with networking I think. It seems there
> > is a problem with lvs server is not fragmenting larger than 1500
> > packets size. I did the quick tcpdump and I get following info.
> >
> > my problem is when user is uploading files it takes for ever and never
> > completes. I am thinking lvs (pulse) is not not fragmenting the larger
> > packet size than 1500 hence it never goes to backend server.
> >
> > 16:25:22.543563 00:1f:6d:cf:4e:49 > 52:54:00:b5:30:3c, ethertype IPv4
> > (0x0800), length 2974: 10.128.1.5.60715 > 10.192.6.1.https: Flags [.],
> > seq 7705:10625, ack 146, win 16388, length 2920
> > 16:25:22.543590 52:54:00:b5:30:3c > 00:1f:6d:cf:4e:49, ethertype IPv4
> > (0x0800), length 590: 10.192.6.1 > 10.128.1.5: ICMP 130.208.165.177
> > unreachable - need to frag (mtu 1500), length 556
> >
> > I tested following:
> >
> > 1. IP forward is enabled.
> > 2. lro,gro are off on eth0
> > ethtool -k eth0
> > Offload parameters for eth0:
> > rx-checksumming: on
> > tx-checksumming: on
> > scatter-gather: on
> > tcp-segmentation-offload: on
> > udp-fragmentation-offload: on
> > generic-segmentation-offload: on
> > generic-receive-offload: off
> > large-receive-offload: off
> > 3. IPVS version is v1.2.1
> > 4. Backend server has arptables and set to mangaled to right interface
> > 5. Firewall is allowed on both lvs and backend server.
> > 6. http and https are open for all.
> >
> > Any help in this would be great.
>
> It seems that the LVS cannot send out the packet since the MTU is too large
> for the 1500 Byte of the interface. Please don't be confused by the length
> of
> 2974 byte that tcpdump shows. tcpdump just sees the packet on the wrong
> point
> on the chain.
>
> I see, how tcpdump reference length 2974 byte ? I was trying to understand
this length and concluded it is something problem with lvs.


> Be sure the Path MTU discovery works or clear the DF bit of the packet
> entering the load balancer.
>
> how do I check this, or is there a way to tell lvs to if the packet size
is larger then force to defrag ? Perhaps this should be handed in linux
kernel ? I am using redhat 6.2

> Greetings,
>
>
> --
> Dr. Michael Schwartzkopff
> Guardinistr. 63
> 81375 München
>
> Tel: (0163) 172 50 98
> Fax: (089) 620 304 13
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


khapare77 at gmail

Mar 9, 2012, 8:30 AM

Post #4 of 9 (1288 views)
Permalink
Re: [lvs-users] LVS problem with unreachable - need to frag [In reply to]

could it be due to i install on virtual server which mean network interface
is shared - is that could be a bottleneck ? I am not sure how to debug this
further.

K

On Thu, Mar 8, 2012 at 7:05 PM, Khapare Joshi <khapare77 [at] gmail> wrote:

> Thanks for your quick response.
>
> On Thu, Mar 8, 2012 at 5:33 PM, Michael Schwartzkopff <
> misch [at] schwartzkopff> wrote:
>
>> > Hello all,
>> >
>> > I am running pulse on virtual machine for webserver, it seems working
>> > well, but there is one problem with networking I think. It seems there
>> > is a problem with lvs server is not fragmenting larger than 1500
>> > packets size. I did the quick tcpdump and I get following info.
>> >
>> > my problem is when user is uploading files it takes for ever and never
>> > completes. I am thinking lvs (pulse) is not not fragmenting the larger
>> > packet size than 1500 hence it never goes to backend server.
>> >
>> > 16:25:22.543563 00:1f:6d:cf:4e:49 > 52:54:00:b5:30:3c, ethertype IPv4
>> > (0x0800), length 2974: 10.128.1.5.60715 > 10.192.6.1.https: Flags [.],
>> > seq 7705:10625, ack 146, win 16388, length 2920
>> > 16:25:22.543590 52:54:00:b5:30:3c > 00:1f:6d:cf:4e:49, ethertype IPv4
>> > (0x0800), length 590: 10.192.6.1 > 10.128.1.5: ICMP 130.208.165.177
>> > unreachable - need to frag (mtu 1500), length 556
>> >
>> > I tested following:
>> >
>> > 1. IP forward is enabled.
>> > 2. lro,gro are off on eth0
>> > ethtool -k eth0
>> > Offload parameters for eth0:
>> > rx-checksumming: on
>> > tx-checksumming: on
>> > scatter-gather: on
>> > tcp-segmentation-offload: on
>> > udp-fragmentation-offload: on
>> > generic-segmentation-offload: on
>> > generic-receive-offload: off
>> > large-receive-offload: off
>> > 3. IPVS version is v1.2.1
>> > 4. Backend server has arptables and set to mangaled to right interface
>> > 5. Firewall is allowed on both lvs and backend server.
>> > 6. http and https are open for all.
>> >
>> > Any help in this would be great.
>>
>> It seems that the LVS cannot send out the packet since the MTU is too
>> large
>> for the 1500 Byte of the interface. Please don't be confused by the
>> length of
>> 2974 byte that tcpdump shows. tcpdump just sees the packet on the wrong
>> point
>> on the chain.
>>
>> I see, how tcpdump reference length 2974 byte ? I was trying to
> understand this length and concluded it is something problem with lvs.
>
>
>> Be sure the Path MTU discovery works or clear the DF bit of the packet
>> entering the load balancer.
>>
>> how do I check this, or is there a way to tell lvs to if the packet size
> is larger then force to defrag ? Perhaps this should be handed in linux
> kernel ? I am using redhat 6.2
>
>> Greetings,
>>
>>
>> --
>> Dr. Michael Schwartzkopff
>> Guardinistr. 63
>> 81375 München
>>
>> Tel: (0163) 172 50 98
>> Fax: (089) 620 304 13
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
>> Send requests to lvs-users-request [at] LinuxVirtualServer
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>
>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


khapare77 at gmail

Mar 9, 2012, 9:07 AM

Post #5 of 9 (1291 views)
Permalink
Re: [lvs-users] LVS problem with unreachable - need to frag [In reply to]

On Thu, Mar 8, 2012 at 5:33 PM, Michael Schwartzkopff <
misch [at] schwartzkopff> wrote:

> > Hello all,
> >
> > I am running pulse on virtual machine for webserver, it seems working
> > well, but there is one problem with networking I think. It seems there
> > is a problem with lvs server is not fragmenting larger than 1500
> > packets size. I did the quick tcpdump and I get following info.
> >
> > my problem is when user is uploading files it takes for ever and never
> > completes. I am thinking lvs (pulse) is not not fragmenting the larger
> > packet size than 1500 hence it never goes to backend server.
> >
> > 16:25:22.543563 00:1f:6d:cf:4e:49 > 52:54:00:b5:30:3c, ethertype IPv4
> > (0x0800), length 2974: 10.128.1.5.60715 > 10.192.6.1.https: Flags [.],
> > seq 7705:10625, ack 146, win 16388, length 2920
> > 16:25:22.543590 52:54:00:b5:30:3c > 00:1f:6d:cf:4e:49, ethertype IPv4
> > (0x0800), length 590: 10.192.6.1 > 10.128.1.5: ICMP 130.208.165.177
> > unreachable - need to frag (mtu 1500), length 556
> >
> > I tested following:
> >
> > 1. IP forward is enabled.
> > 2. lro,gro are off on eth0
> > ethtool -k eth0
> > Offload parameters for eth0:
> > rx-checksumming: on
> > tx-checksumming: on
> > scatter-gather: on
> > tcp-segmentation-offload: on
> > udp-fragmentation-offload: on
> > generic-segmentation-offload: on
> > generic-receive-offload: off
> > large-receive-offload: off
> > 3. IPVS version is v1.2.1
> > 4. Backend server has arptables and set to mangaled to right interface
> > 5. Firewall is allowed on both lvs and backend server.
> > 6. http and https are open for all.
> >
> > Any help in this would be great.
>
> It seems that the LVS cannot send out the packet since the MTU is too large
> for the 1500 Byte of the interface. Please don't be confused by the length
> of
> 2974 byte that tcpdump shows. tcpdump just sees the packet on the wrong
> point
> on the chain.
>
> Be sure the Path MTU discovery works or clear the DF bit of the packet
> entering the load balancer.
>
> As you pointed out to clear the DF bit of the packet entering the load
balancer - how do i do that ? or which is the right method ?


> Greetings,
>
>
> --
> Dr. Michael Schwartzkopff
> Guardinistr. 63
> 81375 München
>
> Tel: (0163) 172 50 98
> Fax: (089) 620 304 13
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
> Send requests to lvs-users-request [at] LinuxVirtualServer
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Mar 11, 2012, 11:00 PM

Post #6 of 9 (1272 views)
Permalink
Re: [lvs-users] LVS problem with unreachable - need to frag [In reply to]

On Fri, Mar 09, 2012 at 04:30:14PM +0000, Khapare Joshi wrote:
> could it be due to i install on virtual server which mean network interface
> is shared - is that could be a bottleneck ? I am not sure how to debug this
> further.

Hi,

this may be related to IPVS not being able to handle LRO or
IPVS in older kernels not being able to handle GRO.

Please try turning off both LRO and GRO on the interface
that receives the packet(s). This can be done using ethtool.

e.g.

ethtool -K eth0 lro off
ethtool -K eth0 gre off

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


khapare77 at gmail

Mar 12, 2012, 2:09 AM

Post #7 of 9 (1285 views)
Permalink
Re: [lvs-users] LVS problem with unreachable - need to frag [In reply to]

Hi simon,

I have already check those parameter. and those are already off.
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off

i am not sure what is causing this issue :!

On Mon, Mar 12, 2012 at 6:00 AM, Simon Horman <horms [at] verge> wrote:

> On Fri, Mar 09, 2012 at 04:30:14PM +0000, Khapare Joshi wrote:
> > could it be due to i install on virtual server which mean network
> interface
> > is shared - is that could be a bottleneck ? I am not sure how to debug
> this
> > further.
>
> Hi,
>
> this may be related to IPVS not being able to handle LRO or
> IPVS in older kernels not being able to handle GRO.
>
> Please try turning off both LRO and GRO on the interface
> that receives the packet(s). This can be done using ethtool.
>
> e.g.
>
> ethtool -K eth0 lro off
> ethtool -K eth0 gre off
>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


ja at ssi

Mar 12, 2012, 2:54 AM

Post #8 of 9 (1274 views)
Permalink
Re: [lvs-users] LVS problem with unreachable - need to frag [In reply to]

Hello,

On Thu, 8 Mar 2012, Khapare Joshi wrote:

> Hello all,
>
> I am running pulse on virtual machine for webserver, it seems working
> well, but there is one problem with networking I think. It seems there
> is a problem with lvs server is not fragmenting larger than 1500
> packets size. I did the quick tcpdump and I get following info.
>
> my problem is when user is uploading files it takes for ever and never
> completes. I am thinking lvs (pulse) is not not fragmenting the larger
> packet size than 1500 hence it never goes to backend server.
>
> 16:25:22.543563 00:1f:6d:cf:4e:49 > 52:54:00:b5:30:3c, ethertype IPv4
> (0x0800), length 2974: 10.128.1.5.60715 > 10.192.6.1.https: Flags [.],
> seq 7705:10625, ack 146, win 16388, length 2920
> 16:25:22.543590 52:54:00:b5:30:3c > 00:1f:6d:cf:4e:49, ethertype IPv4
> (0x0800), length 590: 10.192.6.1 > 10.128.1.5: ICMP 130.208.165.177
> unreachable - need to frag (mtu 1500), length 556

Where is this tcpdump created? Please, provide the
following information:

- kernel version of LVS box

- kernel version of test client box if running Linux

- 'tcpdump -lnnnn -vvv -s0 host <client_ip>' running both
on client and on incoming LVS interface, 10 packets around
the first ICMP FRAG NEEDED message.

Is it happening with any client (real and test client),
check if test client accepts the ICMP error (firewall?) and that
the client actually changes its length in following TCP packets
not to exceed the provided limit with the ICMP message.

> I tested following:
>
> 1. IP forward is enabled.
> 2. lro,gro are off on eth0
> ethtool -k eth0
> Offload parameters for eth0:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: on
> udp-fragmentation-offload: on
> generic-segmentation-offload: on
> generic-receive-offload: off
> large-receive-offload: off
> 3. IPVS version is v1.2.1
> 4. Backend server has arptables and set to mangaled to right interface
> 5. Firewall is allowed on both lvs and backend server.
> 6. http and https are open for all.
>
> Any help in this would be great.
>
> K

Regards

--
Julian Anastasov <ja [at] ssi>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


misch at schwartzkopff

Mar 12, 2012, 2:58 AM

Post #9 of 9 (1274 views)
Permalink
Re: [lvs-users] LVS problem with unreachable - need to frag [In reply to]

> Hi simon,
>
> I have already check those parameter. and those are already off.
> Offload parameters for eth0:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: on
> udp-fragmentation-offload: on
> generic-segmentation-offload: on
> generic-receive-offload: off
> large-receive-offload: off
>
> i am not sure what is causing this issue :!

The packet comming in is too large and says: "do not fragment me". Due to some
misconfiguration your loadbalancer cannot send the ICMP "need to frag but DF
bit set" back to the client. So MTU path discovery does not work and the
connection cannot be established.

You have two options:

- Clear the DF bit on the incomming interface of the loadbalancer. This can be
done within the netfilter frame. I'd have to lookup
the details, but I'd suggest the prerouting mangle table.

- Make the path MTU discovery work, i.e. let the loadbalancer send ICMP need
to frag packets.


--
Dr. Michael Schwartzkopff
Guardinistr. 63
81375 München

Tel: (0163) 172 50 98 50
Fax: (089) 620 304 13
Attachments: signature.asc (0.19 KB)

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.