Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Linux Virtual Server: Users

[lvs-users] LVS and Broadcom bug

 

 

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded


aleksey at bb

Nov 20, 2009, 6:56 AM

Post #1 of 15 (2606 views)
Permalink
[lvs-users] LVS and Broadcom bug

Hello!

I have HP ProLiant BL460c G6 servers with Broadcom Corporation NetXtreme II
BCM57711E 10-Gigabit PCIe controller. After enabling Virtual Server in
kernel and placing server to the test environment with just 10Mbit/s traffic
I see a lot of error messages in logs.

I try different kernel versions 2.6.27.39, 2.6.31.2, 2.6.31.6 but every time
I see the same error:

With 2.6.31.x kernel:

Nov 19 23:35:30 bsrvm8-1 kernel: ------------[ cut here ]------------
Nov 19 23:35:30 bsrvm8-1 kernel: WARNING: at net/core/dev.c:1563
skb_gso_segment+0x10f/0x260()
Nov 19 23:35:30 bsrvm8-1 kernel: Hardware name: ProLiant BL460c G6
Nov 19 23:35:30 bsrvm8-1 kernel: bnx2x: caps=(0x198829, 0x0) len=719
data_len=119 ip_summed=1Pid: 0, comm: swapper Tainted: G W
2.6.31.2-lv
s #1
Nov 19 23:35:30 bsrvm8-1 kernel: Call Trace:
Nov 19 23:35:30 bsrvm8-1 kernel: <IRQ> [<ffffffff812c1b09>] ?
skb_gso_segment+0x10f/0x260
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff810352e1>]
warn_slowpath_common+0x72/0x8a
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8103536e>]
warn_slowpath_fmt+0x64/0x66
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81255c2e>] ?
bnx2x_release_phy_lock+0x28/0x2d
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81261e51>] ?
bnx2x_get_drvinfo+0xea/0x121
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c1b09>]
skb_gso_segment+0x10f/0x260
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c1e04>]
dev_hard_start_xmit+0x1aa/0x2bc
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d0126>]
__qdisc_run+0xeb/0x1ef
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c4a08>]
dev_queue_xmit+0x1cd/0x2d9
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e8549>] ?
ip_finish_output+0x0/0x264
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e876d>]
ip_finish_output+0x224/0x264
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e8adb>] ip_output+0xa9/0xad
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812dc676>]
ip_vs_dr_xmit+0x137/0x16f
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d762f>] ip_vs_in+0x241/0x305
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e2b63>] ?
ip_route_input+0xc84/0xdf3
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d41f3>] nf_iterate+0x43/0x80
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4a26>] ?
ip_local_deliver_finish+0x0/0x176
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d4291>]
nf_hook_slow+0x61/0xc3
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4a26>] ?
ip_local_deliver_finish+0x0/0x176
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4fe5>]
ip_local_deliver+0x63/0x7e
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4a0c>]
ip_rcv_finish+0x308/0x322
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4f4d>] ip_rcv+0x2bd/0x2f2
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c128e>]
netif_receive_skb+0x253/0x273
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81257a5b>]
bnx2x_rx_int+0xbbc/0x1504
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8102d223>] ?
enqueue_task_fair+0xcb/0xd8
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8117fdc2>] ?
cpumask_next_and+0x2b/0x3d
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8102e884>] ?
find_busiest_group+0x22d/0x799
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81265d17>] bnx2x_poll+0xd3/0x1bf
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81030ac1>] ?
run_rebalance_domains+0x183/0x481
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c3f10>]
net_rx_action+0x8b/0x146
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81266d56>] ?
bnx2x_msix_fp_int+0x139/0x14b
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81039abf>]
__do_softirq+0xa2/0x13d
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100bfec>]
call_softirq+0x1c/0x28
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100dba8>] do_softirq+0x42/0x88
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81039a1b>] irq_exit+0x3f/0x41
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100d227>] do_IRQ+0xa3/0xba
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100b813>]
ret_from_intr+0x0/0x11
Nov 19 23:35:30 bsrvm8-1 kernel: <EOI> [<ffffffff810122aa>] ?
mwait_idle+0x67/0x72
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81009bde>] ?
enter_idle+0x20/0x22
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81009f6d>] ? cpu_idle+0x4e/0x6c
Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81338fb1>] ?
start_secondary+0x1a8/0x1ac
Nov 19 23:35:30 bsrvm8-1 kernel: ---[ end trace 5563ae7e045b479e ]---

With 2.6.27.39 kernel:

Nov 19 22:09:11 bsrvm7-1 kernel: ------------[ cut here ]------------
Nov 19 22:09:11 bsrvm7-1 kernel: WARNING: at net/core/dev.c:1505
skb_gso_segment+0x88/0x1ae()
Nov 19 22:09:11 bsrvm7-1 kernel: Pid: 0, comm: swapper Tainted: G W
2.6.27.39-lvs #1
Nov 19 22:09:11 bsrvm7-1 kernel:
Nov 19 22:09:11 bsrvm7-1 kernel: Call Trace:
Nov 19 22:09:11 bsrvm7-1 kernel: <IRQ> [<ffffffff8022f518>]
warn_on_slowpath+0x4c/0x72
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff8022974b>]
__enqueue_rt_entity+0xe6/0x15a
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff8036eb45>]
swiotlb_map_single_attrs+0x32/0xa7
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff8021ef80>]
swiotlb_map_single_phys+0x0/0x14
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff803551e7>]
elv_next_request+0x183/0x193
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff80226c96>]
activate_task+0x28/0x30
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff80227970>]
resched_task+0x2d/0x74
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff803d5aa9>]
do_cciss_request+0x415/0x427
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff804722cb>]
skb_gso_segment+0x88/0x1ae
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff80472571>]
dev_hard_start_xmit+0x180/0x23a
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff8047f733>]
__qdisc_run+0xb5/0x1c1
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff80474a55>]
dev_queue_xmit+0x303/0x416
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff804b8241>]
ip_vs_dr_xmit+0x118/0x14e
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff804b4805>] ip_vs_in+0x1e1/0x289
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff8048386d>] nf_iterate+0x41/0x7d
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff80488aca>]
ip_local_deliver_finish+0x0/0x16a
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff80483906>]
nf_hook_slow+0x5d/0xbe
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff80488aca>]
ip_local_deliver_finish+0x0/0x16a
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff80488ff2>]
ip_local_deliver+0x5f/0x79
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff80488ab0>]
ip_rcv_finish+0x2d4/0x2ee
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff8046d787>]
__alloc_skb+0x71/0x11c
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff804077d3>]
bnx2x_rx_int+0xb62/0x145f
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff80226df0>] source_load+0x2a/0x4f
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff8041134e>] bnx2x_poll+0xd3/0x211
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff804dcd86>]
_spin_lock_irq+0xd/0xf
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff80473f27>]
net_rx_action+0x77/0x147
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff8023382a>]
__do_softirq+0x65/0xdb
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff8020cbcc>]
call_softirq+0x1c/0x28
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff8020eb90>] do_softirq+0x3c/0x81
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff8020ecba>] do_IRQ+0xb9/0xd7
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff8020b95e>]
ret_from_intr+0x0/0x29
Nov 19 22:09:11 bsrvm7-1 kernel: <EOI> [<ffffffff80212404>]
mwait_idle+0x41/0x4d
Nov 19 22:09:11 bsrvm7-1 kernel: [<ffffffff8020a05e>] cpu_idle+0x45/0x63
Nov 19 22:09:11 bsrvm7-1 kernel:
Nov 19 22:09:11 bsrvm7-1 kernel: ---[ end trace 64c43c810c59d16c ]---

Regards, Aleksey


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


jmack at wm7d

Nov 20, 2009, 2:21 PM

Post #2 of 15 (2542 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

"Aleksey Chudov" wrote

> I have HP ProLiant BL460c G6 servers with Broadcom
> Corporation NetXtreme II BCM57711E 10-Gigabit PCIe
> controller. After enabling Virtual Server in kernel and
> placing server to the test environment with just 10Mbit/s
> traffic I see a lot of error messages in logs.

Don't have any information about your situation, but other
people on this list don't seem to have much success with
Broadcom NICs either. I have no idea what the problem is

Joe
--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


aleksey at bb

Nov 20, 2009, 3:13 PM

Post #3 of 15 (2539 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

Thank you for answer!

As far as I understand the bug is with LVS + Broadcom Corporation NetXtreme
II BCM57711E 10-Gigabit only. May be because of relatively new hardware. I
can reproduce it with different kernel versions and different physical
servers.

For a quick solution I replace one server in enclosure with previous model
HP ProLiant BL460c G5 Broadcom Corporation with NetXtreme II BCM5708S
Gigabit Ethernet by just replacing hard disks and error is gone.

As problem is only with LVS enabled Kernel I think that it is LVS specific
bug and should be fixed.

Regards, Aleksey


-----Original Message-----
From: lvs-users-bounces [at] linuxvirtualserver
[mailto:lvs-users-bounces [at] linuxvirtualserver] On Behalf Of Joseph Mack
NA3T
Sent: Saturday, November 21, 2009 12:21 AM
To: lvs-users [at] linuxvirtualserver
Subject: Re: [lvs-users] LVS and Broadcom bug

"Aleksey Chudov" wrote

> I have HP ProLiant BL460c G6 servers with Broadcom
> Corporation NetXtreme II BCM57711E 10-Gigabit PCIe
> controller. After enabling Virtual Server in kernel and
> placing server to the test environment with just 10Mbit/s
> traffic I see a lot of error messages in logs.

Don't have any information about your situation, but other
people on this list don't seem to have much success with
Broadcom NICs either. I have no idea what the problem is

Joe
--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


graeme at graemef

Nov 20, 2009, 3:49 PM

Post #4 of 15 (2542 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

On Sat, 2009-11-21 at 01:13 +0200, Aleksey Chudov wrote:
> As problem is only with LVS enabled Kernel I think that it is LVS specific
> bug and should be fixed.

Apologies for arguing:

As the problem is only with an LVS enabled kernel using the Broadcom 10
Gb PCI driver, and one of those codebases is far newer than the other,
one of them needs fixing.

I would surmise that since the bug has only surfaced recently, and only
wirh Broadcom cards, that the bug is within the driver for those cards.

As I don't have one, and I would hazard a guess that no LVS developers
do, tracking that one down is going to be really rather hard. In any
case, the -devel list is the place for it rather than this one.

Graeme


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


aleksey at bb

Nov 20, 2009, 4:30 PM

Post #5 of 15 (2542 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

Thank you, Graeme.

I'll send this messages to lvs-devel list.

Regards, Aleksey


-----Original Message-----
From: lvs-users-bounces [at] linuxvirtualserver
[mailto:lvs-users-bounces [at] linuxvirtualserver] On Behalf Of Graeme Fowler
Sent: Saturday, November 21, 2009 1:50 AM
To: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] LVS and Broadcom bug

On Sat, 2009-11-21 at 01:13 +0200, Aleksey Chudov wrote:
> As problem is only with LVS enabled Kernel I think that it is LVS specific
> bug and should be fixed.

Apologies for arguing:

As the problem is only with an LVS enabled kernel using the Broadcom 10
Gb PCI driver, and one of those codebases is far newer than the other,
one of them needs fixing.

I would surmise that since the bug has only surfaced recently, and only
wirh Broadcom cards, that the bug is within the driver for those cards.

As I don't have one, and I would hazard a guess that no LVS developers
do, tracking that one down is going to be really rather hard. In any
case, the -devel list is the place for it rather than this one.

Graeme


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


ja at ssi

Nov 20, 2009, 4:41 PM

Post #6 of 15 (2543 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

Hello,

On Fri, 20 Nov 2009, Aleksey Chudov wrote:

> Hello!
>
> I have HP ProLiant BL460c G6 servers with Broadcom Corporation NetXtreme II
> BCM57711E 10-Gigabit PCIe controller. After enabling Virtual Server in
> kernel and placing server to the test environment with just 10Mbit/s traffic
> I see a lot of error messages in logs.
>
> I try different kernel versions 2.6.27.39, 2.6.31.2, 2.6.31.6 but every time
> I see the same error:
>
> With 2.6.31.x kernel:
>
> Nov 19 23:35:30 bsrvm8-1 kernel: ------------[ cut here ]------------
> Nov 19 23:35:30 bsrvm8-1 kernel: WARNING: at net/core/dev.c:1563
> skb_gso_segment+0x10f/0x260()
> Nov 19 23:35:30 bsrvm8-1 kernel: Hardware name: ProLiant BL460c G6
> Nov 19 23:35:30 bsrvm8-1 kernel: bnx2x: caps=(0x198829, 0x0) len=719
> data_len=119 ip_summed=1Pid: 0, comm: swapper Tainted: G W

caps 0x198829 has 0x8000 (NETIF_F_LRO). It seems LRO is
not disabled for your devices, may be it should happen when
you enable the IP forwarding for the concerned devices. It
seems you didn't enabled IP forwarding. Can you confirm it?
Also, may be LRO can be disabled with ethtool.

The problem exists because IPVS does not
disable LRO, it must be done under RTNL and IPVS never runs
in this context. And LRO is not supported for forwarding:

http://marc.info/?l=linux-netdev&m=121389887114416&w=2

IPVS does not call ip_forward for DR method, that
is why you do not need forwarding and the LRO warning
does not occur before hitting the GSO code. ip_forward
just drops LRO packets:

if (skb_warn_if_lro(skb))
goto drop;

> 2.6.31.2-lv
> s #1
> Nov 19 23:35:30 bsrvm8-1 kernel: Call Trace:
> Nov 19 23:35:30 bsrvm8-1 kernel: <IRQ> [<ffffffff812c1b09>] ?
> skb_gso_segment+0x10f/0x260
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff810352e1>]
> warn_slowpath_common+0x72/0x8a
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8103536e>]
> warn_slowpath_fmt+0x64/0x66
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81255c2e>] ?
> bnx2x_release_phy_lock+0x28/0x2d
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81261e51>] ?
> bnx2x_get_drvinfo+0xea/0x121
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c1b09>]
> skb_gso_segment+0x10f/0x260
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c1e04>]
> dev_hard_start_xmit+0x1aa/0x2bc
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d0126>]
> __qdisc_run+0xeb/0x1ef
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c4a08>]
> dev_queue_xmit+0x1cd/0x2d9
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e8549>] ?
> ip_finish_output+0x0/0x264
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e876d>]
> ip_finish_output+0x224/0x264
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e8adb>] ip_output+0xa9/0xad
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812dc676>]
> ip_vs_dr_xmit+0x137/0x16f
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d762f>] ip_vs_in+0x241/0x305
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e2b63>] ?
> ip_route_input+0xc84/0xdf3
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d41f3>] nf_iterate+0x43/0x80
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4a26>] ?
> ip_local_deliver_finish+0x0/0x176
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d4291>]
> nf_hook_slow+0x61/0xc3
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4a26>] ?
> ip_local_deliver_finish+0x0/0x176
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4fe5>]
> ip_local_deliver+0x63/0x7e
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4a0c>]
> ip_rcv_finish+0x308/0x322
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4f4d>] ip_rcv+0x2bd/0x2f2
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c128e>]
> netif_receive_skb+0x253/0x273
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81257a5b>]
> bnx2x_rx_int+0xbbc/0x1504
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8102d223>] ?
> enqueue_task_fair+0xcb/0xd8
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8117fdc2>] ?
> cpumask_next_and+0x2b/0x3d
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8102e884>] ?
> find_busiest_group+0x22d/0x799
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81265d17>] bnx2x_poll+0xd3/0x1bf
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81030ac1>] ?
> run_rebalance_domains+0x183/0x481
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c3f10>]
> net_rx_action+0x8b/0x146
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81266d56>] ?
> bnx2x_msix_fp_int+0x139/0x14b
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81039abf>]
> __do_softirq+0xa2/0x13d
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100bfec>]
> call_softirq+0x1c/0x28
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100dba8>] do_softirq+0x42/0x88
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81039a1b>] irq_exit+0x3f/0x41
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100d227>] do_IRQ+0xa3/0xba
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100b813>]
> ret_from_intr+0x0/0x11
> Nov 19 23:35:30 bsrvm8-1 kernel: <EOI> [<ffffffff810122aa>] ?
> mwait_idle+0x67/0x72
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81009bde>] ?
> enter_idle+0x20/0x22
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81009f6d>] ? cpu_idle+0x4e/0x6c
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81338fb1>] ?
> start_secondary+0x1a8/0x1ac
> Nov 19 23:35:30 bsrvm8-1 kernel: ---[ end trace 5563ae7e045b479e ]---

Regards

--
Julian Anastasov <ja [at] ssi>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


aleksey at bb

Nov 21, 2009, 1:12 AM

Post #7 of 15 (2530 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

Hello,

ip_forward is disabled by default and I doesn't enable it for my DR setup.

I seems my version of ethtool 6 doesn't know anything about LRO.
How could I see is LRO enabled or disabled?


Regards, Aleksey


-----Original Message-----
From: lvs-users-bounces [at] linuxvirtualserver
[mailto:lvs-users-bounces [at] linuxvirtualserver] On Behalf Of Julian
Anastasov
Sent: Saturday, November 21, 2009 2:42 AM
To: Aleksey Chudov
Cc: 'LinuxVirtualServer.org users mailing list.'
Subject: Re: [lvs-users] LVS and Broadcom bug


Hello,

On Fri, 20 Nov 2009, Aleksey Chudov wrote:

> Hello!
>
> I have HP ProLiant BL460c G6 servers with Broadcom Corporation NetXtreme
II
> BCM57711E 10-Gigabit PCIe controller. After enabling Virtual Server in
> kernel and placing server to the test environment with just 10Mbit/s
traffic
> I see a lot of error messages in logs.
>
> I try different kernel versions 2.6.27.39, 2.6.31.2, 2.6.31.6 but every
time
> I see the same error:
>
> With 2.6.31.x kernel:
>
> Nov 19 23:35:30 bsrvm8-1 kernel: ------------[ cut here ]------------
> Nov 19 23:35:30 bsrvm8-1 kernel: WARNING: at net/core/dev.c:1563
> skb_gso_segment+0x10f/0x260()
> Nov 19 23:35:30 bsrvm8-1 kernel: Hardware name: ProLiant BL460c G6
> Nov 19 23:35:30 bsrvm8-1 kernel: bnx2x: caps=(0x198829, 0x0) len=719
> data_len=119 ip_summed=1Pid: 0, comm: swapper Tainted: G W

caps 0x198829 has 0x8000 (NETIF_F_LRO). It seems LRO is
not disabled for your devices, may be it should happen when
you enable the IP forwarding for the concerned devices. It
seems you didn't enabled IP forwarding. Can you confirm it?
Also, may be LRO can be disabled with ethtool.

The problem exists because IPVS does not
disable LRO, it must be done under RTNL and IPVS never runs
in this context. And LRO is not supported for forwarding:

http://marc.info/?l=linux-netdev&m=121389887114416&w=2

IPVS does not call ip_forward for DR method, that
is why you do not need forwarding and the LRO warning
does not occur before hitting the GSO code. ip_forward
just drops LRO packets:

if (skb_warn_if_lro(skb))
goto drop;

> 2.6.31.2-lv
> s #1
> Nov 19 23:35:30 bsrvm8-1 kernel: Call Trace:
> Nov 19 23:35:30 bsrvm8-1 kernel: <IRQ> [<ffffffff812c1b09>] ?
> skb_gso_segment+0x10f/0x260
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff810352e1>]
> warn_slowpath_common+0x72/0x8a
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8103536e>]
> warn_slowpath_fmt+0x64/0x66
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81255c2e>] ?
> bnx2x_release_phy_lock+0x28/0x2d
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81261e51>] ?
> bnx2x_get_drvinfo+0xea/0x121
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c1b09>]
> skb_gso_segment+0x10f/0x260
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c1e04>]
> dev_hard_start_xmit+0x1aa/0x2bc
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d0126>]
> __qdisc_run+0xeb/0x1ef
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c4a08>]
> dev_queue_xmit+0x1cd/0x2d9
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e8549>] ?
> ip_finish_output+0x0/0x264
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e876d>]
> ip_finish_output+0x224/0x264
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e8adb>] ip_output+0xa9/0xad
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812dc676>]
> ip_vs_dr_xmit+0x137/0x16f
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d762f>]
ip_vs_in+0x241/0x305
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e2b63>] ?
> ip_route_input+0xc84/0xdf3
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d41f3>]
nf_iterate+0x43/0x80
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4a26>] ?
> ip_local_deliver_finish+0x0/0x176
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d4291>]
> nf_hook_slow+0x61/0xc3
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4a26>] ?
> ip_local_deliver_finish+0x0/0x176
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4fe5>]
> ip_local_deliver+0x63/0x7e
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4a0c>]
> ip_rcv_finish+0x308/0x322
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4f4d>] ip_rcv+0x2bd/0x2f2
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c128e>]
> netif_receive_skb+0x253/0x273
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81257a5b>]
> bnx2x_rx_int+0xbbc/0x1504
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8102d223>] ?
> enqueue_task_fair+0xcb/0xd8
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8117fdc2>] ?
> cpumask_next_and+0x2b/0x3d
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8102e884>] ?
> find_busiest_group+0x22d/0x799
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81265d17>]
bnx2x_poll+0xd3/0x1bf
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81030ac1>] ?
> run_rebalance_domains+0x183/0x481
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c3f10>]
> net_rx_action+0x8b/0x146
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81266d56>] ?
> bnx2x_msix_fp_int+0x139/0x14b
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81039abf>]
> __do_softirq+0xa2/0x13d
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100bfec>]
> call_softirq+0x1c/0x28
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100dba8>]
do_softirq+0x42/0x88
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81039a1b>] irq_exit+0x3f/0x41
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100d227>] do_IRQ+0xa3/0xba
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100b813>]
> ret_from_intr+0x0/0x11
> Nov 19 23:35:30 bsrvm8-1 kernel: <EOI> [<ffffffff810122aa>] ?
> mwait_idle+0x67/0x72
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81009bde>] ?
> enter_idle+0x20/0x22
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81009f6d>] ?
cpu_idle+0x4e/0x6c
> Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81338fb1>] ?
> start_secondary+0x1a8/0x1ac
> Nov 19 23:35:30 bsrvm8-1 kernel: ---[ end trace 5563ae7e045b479e ]---

Regards

--
Julian Anastasov <ja [at] ssi>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


ja at ssi

Nov 21, 2009, 1:42 AM

Post #8 of 15 (2534 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

Hello,

On Sat, 21 Nov 2009, Aleksey Chudov wrote:

> Hello,
>
> ip_forward is disabled by default and I doesn't enable it for my DR setup.
>
> I seems my version of ethtool 6 doesn't know anything about LRO.
> How could I see is LRO enabled or disabled?

According to drivers/net/bnx2x_main.c bnx2x_init_bp()
can alter LRO:

/* Set TPA flags */
if (disable_tpa) {
bp->flags &= ~TPA_ENABLE_FLAG;
bp->dev->features &= ~NETIF_F_LRO;
} else {
bp->flags |= TPA_ENABLE_FLAG;
bp->dev->features |= NETIF_F_LRO;
}

You probably can disable it as follows:

modprobe bnx2x disable_tpa=1

or in /etc/modprobe.conf:
options bnx2x disable_tpa=1

For checking the current state you can look at the
module parameters for disable_tpa:

cat /sys/module/<MODULE>/parameters/...

Regards

--
Julian Anastasov <ja [at] ssi>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


aleksey at bb

Nov 24, 2009, 2:57 AM

Post #9 of 15 (2469 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

Hello!

In my case bnx2x device driver are compiled into kernel.
So I disable LRO with kernel parameter "bnx2x.disable_tpa=1" and after
reboot error message is gone.
The problem is fixed. Thank you for your suggestion.


Regards, Aleksey



> According to drivers/net/bnx2x_main.c bnx2x_init_bp()
>can alter LRO:
>
> /* Set TPA flags */
> if (disable_tpa) {
> bp->flags &= ~TPA_ENABLE_FLAG;
> bp->dev->features &= ~NETIF_F_LRO;
> } else {
> bp->flags |= TPA_ENABLE_FLAG;
> bp->dev->features |= NETIF_F_LRO;
> }
>
> You probably can disable it as follows:
>
>modprobe bnx2x disable_tpa=1


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Nov 29, 2009, 11:03 PM

Post #10 of 15 (2326 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

On Sat, Nov 21, 2009 at 02:41:49AM +0200, Julian Anastasov wrote:
>
> Hello,
>
> On Fri, 20 Nov 2009, Aleksey Chudov wrote:
>
> > Hello!
> >
> > I have HP ProLiant BL460c G6 servers with Broadcom Corporation NetXtreme II
> > BCM57711E 10-Gigabit PCIe controller. After enabling Virtual Server in
> > kernel and placing server to the test environment with just 10Mbit/s traffic
> > I see a lot of error messages in logs.
> >
> > I try different kernel versions 2.6.27.39, 2.6.31.2, 2.6.31.6 but every time
> > I see the same error:
> >
> > With 2.6.31.x kernel:
> >
> > Nov 19 23:35:30 bsrvm8-1 kernel: ------------[ cut here ]------------
> > Nov 19 23:35:30 bsrvm8-1 kernel: WARNING: at net/core/dev.c:1563
> > skb_gso_segment+0x10f/0x260()
> > Nov 19 23:35:30 bsrvm8-1 kernel: Hardware name: ProLiant BL460c G6
> > Nov 19 23:35:30 bsrvm8-1 kernel: bnx2x: caps=(0x198829, 0x0) len=719
> > data_len=119 ip_summed=1Pid: 0, comm: swapper Tainted: G W
>
> caps 0x198829 has 0x8000 (NETIF_F_LRO). It seems LRO is
> not disabled for your devices, may be it should happen when
> you enable the IP forwarding for the concerned devices. It
> seems you didn't enabled IP forwarding. Can you confirm it?
> Also, may be LRO can be disabled with ethtool.
>
> The problem exists because IPVS does not
> disable LRO, it must be done under RTNL and IPVS never runs
> in this context. And LRO is not supported for forwarding:
>
> http://marc.info/?l=linux-netdev&m=121389887114416&w=2
>
> IPVS does not call ip_forward for DR method, that
> is why you do not need forwarding and the LRO warning
> does not occur before hitting the GSO code. ip_forward
> just drops LRO packets:
>
> if (skb_warn_if_lro(skb))
> goto drop;

Hi Julian,

do you have any thoughts on how the code might be improved
to handle this case a bit better?

Perhaps something along the lines of the
code for LRO in ip_forward?


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


ja at ssi

Dec 5, 2009, 3:30 AM

Post #11 of 15 (2230 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

Hello,

On Mon, 30 Nov 2009, Simon Horman wrote:

> > The problem exists because IPVS does not
> > disable LRO, it must be done under RTNL and IPVS never runs
> > in this context. And LRO is not supported for forwarding:
> >
> > http://marc.info/?l=linux-netdev&m=121389887114416&w=2
> >
> > IPVS does not call ip_forward for DR method, that
> > is why you do not need forwarding and the LRO warning
> > does not occur before hitting the GSO code. ip_forward
> > just drops LRO packets:
> >
> > if (skb_warn_if_lro(skb))
> > goto drop;
>
> Hi Julian,
>
> do you have any thoughts on how the code might be improved
> to handle this case a bit better?
>
> Perhaps something along the lines of the
> code for LRO in ip_forward?

If you want to disable LRO in IPVS
net/ipv4/devinet.c:inet_forward_change() is an example what
should be done in process context if you want to disable
LRO for all existing devices. Then call skb_warn_if_lro
near or in IP_VS_XMIT and also before calling ip_local_out().
May be LRO can be disabled when the first virtual or
may be real service is added to allow LRO to work if IPVS
is just compiled.

Regards

--
Julian Anastasov <ja [at] ssi>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


jean-sebastien.frerot at gameloft

Oct 26, 2010, 8:33 AM

Post #12 of 15 (1170 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

> Hello,
>
> On Mon, 30 Nov 2009, Simon Horman wrote:
>
> >/ > The problem exists because IPVS does not/
> >/ > disable LRO, it must be done under RTNL and IPVS never runs/
> >/ > in this context. And LRO is not supported for forwarding:/
> >/ > /
> >/ > http://marc.info/?l=linux-netdev&m=121389887114416&w=2 <http://marc.info/?l=linux-netdev&m=121389887114416&w=2>/
> >/ > /
> >/ > IPVS does not call ip_forward for DR method, that/
> >/ > is why you do not need forwarding and the LRO warning/
> >/ > does not occur before hitting the GSO code. ip_forward/
> >/ > just drops LRO packets:/
> >/ > /
> >/ > if (skb_warn_if_lro(skb))/
> >/ > goto drop;/
> >/ /
> >/ Hi Julian,/
> >/ /
> >/ do you have any thoughts on how the code might be improved/
> >/ to handle this case a bit better?/
> >/ /
> >/ Perhaps something along the lines of the/
> >/ code for LRO in ip_forward?/
>
> If you want to disable LRO in IPVS
> net/ipv4/devinet.c:inet_forward_change() is an example what
> should be done in process context if you want to disable
> LRO for all existing devices. Then call skb_warn_if_lro
> near or in IP_VS_XMIT and also before calling ip_local_out().
> May be LRO can be disabled when the first virtual or
> may be real service is added to allow LRO to work if IPVS
> is just compiled.
>
> Regards
>
> --
> Julian Anastasov <ja [at] xxxxx>
>
Hi,
Do you guys know if there is any plan to fix this in ipvs soon ? We
have this exact problem when using ipvs and 2 different network cards
(intel and broadcom).

Thx.

Jean-Sébastien Frerot
--


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


chchen at pdx

Oct 26, 2010, 9:52 AM

Post #13 of 15 (1169 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

Quoting Jean-Sébastien Frerot <jean-sebastien.frerot [at] gameloft>:

>> Hello,
>>
>> On Mon, 30 Nov 2009, Simon Horman wrote:
>>
>> >/ > The problem exists because IPVS does not/
>> >/ > disable LRO, it must be done under RTNL and IPVS never runs/
>> >/ > in this context. And LRO is not supported for forwarding:/
>> >/ > /
>> >/ > http://marc.info/?l=linux-netdev&m=121389887114416&w=2
>> <http://marc.info/?l=linux-netdev&m=121389887114416&w=2>/
>> >/ > /
>> >/ > IPVS does not call ip_forward for DR method, that/
>> >/ > is why you do not need forwarding and the LRO warning/
>> >/ > does not occur before hitting the GSO code. ip_forward/
>> >/ > just drops LRO packets:/
>> >/ > /
>> >/ > if (skb_warn_if_lro(skb))/
>> >/ > goto drop;/
>> >/ /
>> >/ Hi Julian,/
>> >/ /
>> >/ do you have any thoughts on how the code might be improved/
>> >/ to handle this case a bit better?/
>> >/ /
>> >/ Perhaps something along the lines of the/
>> >/ code for LRO in ip_forward?/
>>
>> If you want to disable LRO in IPVS
>> net/ipv4/devinet.c:inet_forward_change() is an example what
>> should be done in process context if you want to disable
>> LRO for all existing devices. Then call skb_warn_if_lro
>> near or in IP_VS_XMIT and also before calling ip_local_out().
>> May be LRO can be disabled when the first virtual or
>> may be real service is added to allow LRO to work if IPVS
>> is just compiled.
>>
>> Regards
>>
>> --
>> Julian Anastasov <ja [at] xxxxx>
>>
> Hi,
> Do you guys know if there is any plan to fix this in ipvs soon ? We
> have this exact problem when using ipvs and 2 different network cards
> (intel and broadcom).


Could this be breaking SSL over LVS-DR? I've been seeing a problem
where SSL handshakes fail intermittently with certain clients (Windows
7, particularly), and using LVS-NAT seems to fix it.

cc



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Oct 26, 2010, 2:33 PM

Post #14 of 15 (1172 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

On Tue, Oct 26, 2010 at 09:52:00AM -0700, Chris Chen wrote:
> Quoting Jean-Sébastien Frerot <jean-sebastien.frerot [at] gameloft>:
>
> >> Hello,
> >>
> >> On Mon, 30 Nov 2009, Simon Horman wrote:
> >>
> >> >/ > The problem exists because IPVS does not/
> >> >/ > disable LRO, it must be done under RTNL and IPVS never runs/
> >> >/ > in this context. And LRO is not supported for forwarding:/
> >> >/ > /
> >> >/ > http://marc.info/?l=linux-netdev&m=121389887114416&w=2
> >> <http://marc.info/?l=linux-netdev&m=121389887114416&w=2>/
> >> >/ > /
> >> >/ > IPVS does not call ip_forward for DR method, that/
> >> >/ > is why you do not need forwarding and the LRO warning/
> >> >/ > does not occur before hitting the GSO code. ip_forward/
> >> >/ > just drops LRO packets:/
> >> >/ > /
> >> >/ > if (skb_warn_if_lro(skb))/
> >> >/ > goto drop;/
> >> >/ /
> >> >/ Hi Julian,/
> >> >/ /
> >> >/ do you have any thoughts on how the code might be improved/
> >> >/ to handle this case a bit better?/
> >> >/ /
> >> >/ Perhaps something along the lines of the/
> >> >/ code for LRO in ip_forward?/
> >>
> >> If you want to disable LRO in IPVS
> >> net/ipv4/devinet.c:inet_forward_change() is an example what
> >> should be done in process context if you want to disable
> >> LRO for all existing devices. Then call skb_warn_if_lro
> >> near or in IP_VS_XMIT and also before calling ip_local_out().
> >> May be LRO can be disabled when the first virtual or
> >> may be real service is added to allow LRO to work if IPVS
> >> is just compiled.
> >>
> >> Regards
> >>
> >> --
> >> Julian Anastasov <ja [at] xxxxx>
> >>
> > Hi,
> > Do you guys know if there is any plan to fix this in ipvs soon ? We
> > have this exact problem when using ipvs and 2 different network cards
> > (intel and broadcom).

Sorry, no I don't believe that there is a fix available.
I will try and rectify that situation. In the mean time
I believe that a work-around is to disable LRO on any
interface that receives packets for fowarding (or LVS)
using ethtool.

> Could this be breaking SSL over LVS-DR? I've been seeing a problem
> where SSL handshakes fail intermittently with certain clients (Windows
> 7, particularly), and using LVS-NAT seems to fix it.

The most recent manifestation of this problem that I have observed is that
the linux-director will request fragmentation of packets that are greater
than MTU size which LRO has actually assembled from packets which are MTU
length or less.

That manifestation can be observed as ICMP need to frag messages on the wire.
The kernel in question is 2.5.35. I believe the behaviour should
be the same in newer kernels. I am less sure about older kernels.

I'm not sure if there are other manifestations of this problem.


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Oct 26, 2010, 2:40 PM

Post #15 of 15 (1181 views)
Permalink
Re: [lvs-users] LVS and Broadcom bug [In reply to]

On Wed, Oct 27, 2010 at 06:33:54AM +0900, Simon Horman wrote:
> On Tue, Oct 26, 2010 at 09:52:00AM -0700, Chris Chen wrote:
> > Quoting Jean-Sébastien Frerot <jean-sebastien.frerot [at] gameloft>:
> >
> > >> Hello,
> > >>
> > >> On Mon, 30 Nov 2009, Simon Horman wrote:
> > >>
> > >> >/ > The problem exists because IPVS does not/
> > >> >/ > disable LRO, it must be done under RTNL and IPVS never runs/
> > >> >/ > in this context. And LRO is not supported for forwarding:/
> > >> >/ > /
> > >> >/ > http://marc.info/?l=linux-netdev&m=121389887114416&w=2
> > >> <http://marc.info/?l=linux-netdev&m=121389887114416&w=2>/
> > >> >/ > /
> > >> >/ > IPVS does not call ip_forward for DR method, that/
> > >> >/ > is why you do not need forwarding and the LRO warning/
> > >> >/ > does not occur before hitting the GSO code. ip_forward/
> > >> >/ > just drops LRO packets:/
> > >> >/ > /
> > >> >/ > if (skb_warn_if_lro(skb))/
> > >> >/ > goto drop;/
> > >> >/ /
> > >> >/ Hi Julian,/
> > >> >/ /
> > >> >/ do you have any thoughts on how the code might be improved/
> > >> >/ to handle this case a bit better?/
> > >> >/ /
> > >> >/ Perhaps something along the lines of the/
> > >> >/ code for LRO in ip_forward?/
> > >>
> > >> If you want to disable LRO in IPVS
> > >> net/ipv4/devinet.c:inet_forward_change() is an example what
> > >> should be done in process context if you want to disable
> > >> LRO for all existing devices. Then call skb_warn_if_lro
> > >> near or in IP_VS_XMIT and also before calling ip_local_out().
> > >> May be LRO can be disabled when the first virtual or
> > >> may be real service is added to allow LRO to work if IPVS
> > >> is just compiled.
> > >>
> > >> Regards
> > >>
> > >> --
> > >> Julian Anastasov <ja [at] xxxxx>
> > >>
> > > Hi,
> > > Do you guys know if there is any plan to fix this in ipvs soon ? We
> > > have this exact problem when using ipvs and 2 different network cards
> > > (intel and broadcom).
>
> Sorry, no I don't believe that there is a fix available.
> I will try and rectify that situation. In the mean time
> I believe that a work-around is to disable LRO on any
> interface that receives packets for fowarding (or LVS)
> using ethtool.

Correction, the work-around for the problem that I have seen
is to disable GRO.

> > Could this be breaking SSL over LVS-DR? I've been seeing a problem
> > where SSL handshakes fail intermittently with certain clients (Windows
> > 7, particularly), and using LVS-NAT seems to fix it.
>
> The most recent manifestation of this problem that I have observed is that
> the linux-director will request fragmentation of packets that are greater
> than MTU size which LRO has actually assembled from packets which are MTU
> length or less.
>
> That manifestation can be observed as ICMP need to frag messages on the wire.
> The kernel in question is 2.5.35. I believe the behaviour should
> be the same in newer kernels. I am less sure about older kernels.
>
> I'm not sure if there are other manifestations of this problem.
>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.