Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Linux Virtual Server: Users

[lvs-users] "connection refused" when persistence enabled

  

 
First page Previous page 1 2 Next page Last page  View All Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded


dvdm at truteq

Nov 17, 2009, 10:33 AM

Post #1 of 32 (3655 views)
Permalink
[lvs-users] "connection refused" when persistence enabled
Hi,

Been using this setup on at least 4 different installations without this
issue... we have multiple virtual services and use iptables MARK to tag
the packets for each virtual service.

My problem is that when I enable/configure persistence on IPVS the
client gets "connection refused". The same config *without* persistence
works fine.


System:
=======
Kernel: 2.6.29.6-smp (vanilla from Slackware 13.0)
ipvsadm v1.25 2008/5/15 (compiled with popt and IPVS v1.2.1)
iptables v1.4.3.2


iptables:
=========
iptables -L -n -t mangle
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
MARK tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080 MARK xset 0x1/0xffffffff
MARK tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9201 MARK xset 0x1/0xffffffff
MARK udp -- 0.0.0.0/0 0.0.0.0/0 udp dpts:9200:9201 MARK xset
0x2/0xffffffff


ipvsadm:
========
ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
FWM 1 lc
-> 192.168.22.11:0 Masq 100 0 0
FWM 2 lc
-> 192.168.22.11:0 Masq 100 0 0


IPVS debug when persistence is *ON*:
=====================================
kernel: IPVS: lookup/in TCP 160.124.109.65:43100->192.168.1.1:8080 not hit
kernel: IPVS: lookup/out TCP 160.124.109.65:43100->192.168.1.1:8080 not hit
kernel: IPVS: lookup service: fwm 1 TCP 192.168.1.1:8080 hit
kernel: IPVS: p-schedule: src 160.124.109.65:43100 dest 192.168.1.1:8080
mnet 160.124.109.65
kernel: IPVS: template lookup/in IP 160.124.109.65:0->0.0.0.0:0 not hit



IPVS debug when persistence is *OFF*:
=====================================
kernel: IPVS: lookup/in TCP 160.124.109.65:43098->192.168.1.1:8080 not hit
kernel: IPVS: lookup/out TCP 160.124.109.65:43098->192.168.1.1:8080 not hit
kernel: IPVS: lookup service: fwm 1 TCP 192.168.1.1:8080 hit
kernel: IPVS: ip_vs_rr_schedule(): Scheduling...
kernel: IPVS: RR: server 192.168.22.11:0 activeconns 0 refcnt 3 weight 100
kernel: IPVS: Bind-dest TCP c:160.124.109.65:43098 v:192.168.1.1:8080
d:192.168.22.11:8080 fwd:M s:0 conn->flags:100 conn->refcnt:1
dest->refcnt:4
kernel: IPVS: Schedule fwd:M c:160.124.109.65:43098 v:192.168.1.1:8080
d:192.168.22.11:8080 conn->flags:140 conn->refcnt:2
kernel: IPVS: TCP input [S...] 192.168.22.11:8080->160.124.109.65:43098
state: NONE->SYN_RECV conn->refcnt:2
kernel: Enter: ip_vs_nat_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 359
kernel: IPVS: After DNAT: TCP 160.124.109.65:43098->192.168.22.11:8080
kernel: Leave: ip_vs_nat_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 411
kernel: IPVS: lookup/out TCP 192.168.22.11:8080->160.124.109.65:43098 hit
kernel: IPVS: After SNAT: TCP 192.168.1.1:8080->160.124.109.65:43098
kernel: IPVS: lookup/in TCP 160.124.109.65:43098->192.168.1.1:8080 hit
kernel: IPVS: TCP input [..A.] 192.168.22.11:8080->160.124.109.65:43098
state: SYN_RECV->ESTABLISHED conn->refcnt:2
kernel: Enter: ip_vs_nat_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 359
kernel: IPVS: After DNAT: TCP 160.124.109.65:43098->192.168.22.11:8080
kernel: Leave: ip_vs_nat_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 411





--
__Deon_______________________________________________
TruTeq Wireless (Pty) Ltd. Tel: +27 12 667 1530
http://www.truteq.co.za Fax: +27 12 667 1531
Timezone: SAST GMT+2
Copyright&Legal: http://truteq.co.za/legal_notice.pdf

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Nov 27, 2009, 4:54 AM

Post #2 of 32 (3539 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
On Tue, Nov 17, 2009 at 08:33:52PM +0200, Deon van der Merwe wrote:
> Hi,
>
> Been using this setup on at least 4 different installations without this
> issue... we have multiple virtual services and use iptables MARK to tag
> the packets for each virtual service.
>
> My problem is that when I enable/configure persistence on IPVS the
> client gets "connection refused". The same config *without* persistence
> works fine.

Hi Deon,

that looks very odd. I've been able to reproduce the problem here
with 2.6.30. But its getting a bit late in the day for me to debug it.
I'll try and get a chance to do so shortly if no one else gets
there first.


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Nov 27, 2009, 4:50 PM

Post #3 of 32 (3525 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
On Fri, Nov 27, 2009 at 11:54:41PM +1100, Simon Horman wrote:
> On Tue, Nov 17, 2009 at 08:33:52PM +0200, Deon van der Merwe wrote:
> > Hi,
> >
> > Been using this setup on at least 4 different installations without this
> > issue... we have multiple virtual services and use iptables MARK to tag
> > the packets for each virtual service.
> >
> > My problem is that when I enable/configure persistence on IPVS the
> > client gets "connection refused". The same config *without* persistence
> > works fine.
>
> Hi Deon,
>
> that looks very odd. I've been able to reproduce the problem here
> with 2.6.30. But its getting a bit late in the day for me to debug it.
> I'll try and get a chance to do so shortly if no one else gets
> there first.

Hi Deon,

could you try the following patch?

----------------------------------------------------------------------

ipvs: fwmark services don't have ports

Make sure that the port for fwmark services is always zero, it seems that
in some cases ipvsadm provides bogus port values for fwmark services. As
observed by Deon van der Merwe, this manifests as a clients not being able
to connect to the virtual service if persistence is set.

Cc: Deon van der Merwe <dvdm [at] truteq>
Signed-off-by: Simon Horman <horms [at] verge>

diff --git a/net/netfilter/ipvs/ip_vs_ctl.c b/net/netfilter/ipvs/ip_vs_ctl.c
index e01061f..9a4ccd4 100644
--- a/net/netfilter/ipvs/ip_vs_ctl.c
+++ b/net/netfilter/ipvs/ip_vs_ctl.c
@@ -1188,8 +1188,9 @@ ip_vs_add_service(struct ip_vs_service_user_kern *u,
svc->af = u->af;
svc->protocol = u->protocol;
ip_vs_addr_copy(svc->af, &svc->addr, &u->addr);
- svc->port = u->port;
svc->fwmark = u->fwmark;
+ if (!svc->fwmark)
+ svc->port = u->port;
svc->flags = u->flags;
svc->timeout = u->timeout * HZ;
svc->netmask = u->netmask;

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


ja at ssi

Nov 29, 2009, 2:16 AM

Post #4 of 32 (3523 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
Hello,

On Sat, 28 Nov 2009, Simon Horman wrote:

> On Fri, Nov 27, 2009 at 11:54:41PM +1100, Simon Horman wrote:
> > On Tue, Nov 17, 2009 at 08:33:52PM +0200, Deon van der Merwe wrote:
> > > Hi,
> > >
> > > Been using this setup on at least 4 different installations without this
> > > issue... we have multiple virtual services and use iptables MARK to tag
> > > the packets for each virtual service.
> > >
> > > My problem is that when I enable/configure persistence on IPVS the
> > > client gets "connection refused". The same config *without* persistence
> > > works fine.
> >
> > Hi Deon,
> >
> > that looks very odd. I've been able to reproduce the problem here
> > with 2.6.30. But its getting a bit late in the day for me to debug it.
> > I'll try and get a chance to do so shortly if no one else gets
> > there first.

Hm, your change in 2.6.30 should fix this problem:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=be8be9eccbf2d908a7e56b3f7a71105cd88da06b

looking at

kernel: IPVS: template lookup/in IP 160.124.109.65:0->0.0.0.0:0 not hit

daddr=0.0.0.0 shows that fwmark is not provided at the
right place, so the user just needs to upgrade or to apply
the patch from May, 2009:

http://marc.info/?t=124050022700001&r=1&w=2

> Hi Deon,
>
> could you try the following patch?
>
> ----------------------------------------------------------------------
>
> ipvs: fwmark services don't have ports
>
> Make sure that the port for fwmark services is always zero, it seems that
> in some cases ipvsadm provides bogus port values for fwmark services. As
> observed by Deon van der Merwe, this manifests as a clients not being able
> to connect to the virtual service if persistence is set.
>
> Cc: Deon van der Merwe <dvdm [at] truteq>
> Signed-off-by: Simon Horman <horms [at] verge>
>
> diff --git a/net/netfilter/ipvs/ip_vs_ctl.c b/net/netfilter/ipvs/ip_vs_ctl.c
> index e01061f..9a4ccd4 100644
> --- a/net/netfilter/ipvs/ip_vs_ctl.c
> +++ b/net/netfilter/ipvs/ip_vs_ctl.c
> @@ -1188,8 +1188,9 @@ ip_vs_add_service(struct ip_vs_service_user_kern *u,
> svc->af = u->af;
> svc->protocol = u->protocol;
> ip_vs_addr_copy(svc->af, &svc->addr, &u->addr);
> - svc->port = u->port;
> svc->fwmark = u->fwmark;
> + if (!svc->fwmark)
> + svc->port = u->port;
> svc->flags = u->flags;
> svc->timeout = u->timeout * HZ;
> svc->netmask = u->netmask;

Regards

--
Julian Anastasov <ja [at] ssi>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Nov 29, 2009, 4:04 PM

Post #5 of 32 (3505 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
On Sun, Nov 29, 2009 at 12:16:22PM +0200, Julian Anastasov wrote:
>
> Hello,
>
> On Sat, 28 Nov 2009, Simon Horman wrote:
>
> > On Fri, Nov 27, 2009 at 11:54:41PM +1100, Simon Horman wrote:
> > > On Tue, Nov 17, 2009 at 08:33:52PM +0200, Deon van der Merwe wrote:
> > > > Hi,
> > > >
> > > > Been using this setup on at least 4 different installations without this
> > > > issue... we have multiple virtual services and use iptables MARK to tag
> > > > the packets for each virtual service.
> > > >
> > > > My problem is that when I enable/configure persistence on IPVS the
> > > > client gets "connection refused". The same config *without* persistence
> > > > works fine.
> > >
> > > Hi Deon,
> > >
> > > that looks very odd. I've been able to reproduce the problem here
> > > with 2.6.30. But its getting a bit late in the day for me to debug it.
> > > I'll try and get a chance to do so shortly if no one else gets
> > > there first.
>
> Hm, your change in 2.6.30 should fix this problem:
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=be8be9eccbf2d908a7e56b3f7a71105cd88da06b
>
> looking at
>
> kernel: IPVS: template lookup/in IP 160.124.109.65:0->0.0.0.0:0 not hit
>
> daddr=0.0.0.0 shows that fwmark is not provided at the
> right place, so the user just needs to upgrade or to apply
> the patch from May, 2009:
>
> http://marc.info/?t=124050022700001&r=1&w=2

Hi Julian,

I've been able to reproduce the problem that Deon reported
with 2.6.30, which includes the change above, so I think
that it is a separate problem.

That said, Deon also needs to change above, "ipvs: Fix IPv4 FWMARK virtual
services", as he is using more than one fwmark.


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


dvdm at truteq

Dec 1, 2009, 11:08 AM

Post #6 of 32 (3483 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
Hi Simon,

Simon Horman wrote:
> On Fri, Nov 27, 2009 at 11:54:41PM +1100, Simon Horman wrote:
>> On Tue, Nov 17, 2009 at 08:33:52PM +0200, Deon van der Merwe wrote:
>>> Hi,
>>>
>>> Been using this setup on at least 4 different installations without this
>>> issue... we have multiple virtual services and use iptables MARK to tag
>>> the packets for each virtual service.
>>>
>>> My problem is that when I enable/configure persistence on IPVS the
>>> client gets "connection refused". The same config *without* persistence
>>> works fine.
>> Hi Deon,
>>
>> that looks very odd. I've been able to reproduce the problem here
>> with 2.6.30. But its getting a bit late in the day for me to debug it.
>> I'll try and get a chance to do so shortly if no one else gets
>> there first.
>
> Hi Deon,
>
> could you try the following patch?
>

Thanks allot for the update. I have updated my server toe 2.6.31.6 and
then application the supplied patch. Will now test and confirm...


--
__Deon_______________________________________________
TruTeq Wireless (Pty) Ltd. Tel: +27 12 667 1530
http://www.truteq.co.za Fax: +27 12 667 1531
Timezone: SAST GMT+2
Copyright&Legal: http://truteq.co.za/legal_notice.pdf

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


dvdm at truteq

Dec 3, 2009, 10:28 AM

Post #7 of 32 (3461 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
Hi Simon,

Simon Horman wrote:
> could you try the following patch?
>

Applied the patch to 2.6.31.6 and everything is running smoothly with
persistence enabled. Thanks allot!


--
__Deon_______________________________________________
TruTeq Wireless (Pty) Ltd. Tel: +27 12 667 1530
http://www.truteq.co.za Fax: +27 12 667 1531
Timezone: SAST GMT+2
Copyright&Legal: http://truteq.co.za/legal_notice.pdf

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Dec 3, 2009, 1:46 PM

Post #8 of 32 (3464 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
On Thu, Dec 03, 2009 at 08:28:56PM +0200, Deon van der Merwe wrote:
> Hi Simon,
>
> Simon Horman wrote:
> > could you try the following patch?
> >
>
> Applied the patch to 2.6.31.6 and everything is running smoothly with
> persistence enabled. Thanks allot!

Great, thanks for the feedback.


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


ja at ssi

Dec 7, 2009, 12:37 AM

Post #9 of 32 (3401 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
Hello,

On Mon, 30 Nov 2009, Simon Horman wrote:

> > looking at
> >
> > kernel: IPVS: template lookup/in IP 160.124.109.65:0->0.0.0.0:0 not hit
> >
> > daddr=0.0.0.0 shows that fwmark is not provided at the
> > right place, so the user just needs to upgrade or to apply
> > the patch from May, 2009:
> >
> > http://marc.info/?t=124050022700001&r=1&w=2
>
> Hi Julian,
>
> I've been able to reproduce the problem that Deon reported
> with 2.6.30, which includes the change above, so I think
> that it is a separate problem.

It is still not clear to me whether Deon needed just 2.6.30
or your latest change was needed. I don't see how ipvsadm 1.25
can provide virtual port without -t/-u options. What is your
ipvsadm command line that reproduces this svc port problem in
2.6.30? May be you have both -f and -t/-u ?

> That said, Deon also needs to change above, "ipvs: Fix IPv4 FWMARK virtual
> services", as he is using more than one fwmark.

Regards

--
Julian Anastasov <ja [at] ssi>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Dec 7, 2009, 12:45 AM

Post #10 of 32 (3396 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
On Mon, Dec 07, 2009 at 10:37:32AM +0200, Julian Anastasov wrote:
>
> Hello,
>
> On Mon, 30 Nov 2009, Simon Horman wrote:
>
> > > looking at
> > >
> > > kernel: IPVS: template lookup/in IP 160.124.109.65:0->0.0.0.0:0 not hit
> > >
> > > daddr=0.0.0.0 shows that fwmark is not provided at the
> > > right place, so the user just needs to upgrade or to apply
> > > the patch from May, 2009:
> > >
> > > http://marc.info/?t=124050022700001&r=1&w=2
> >
> > Hi Julian,
> >
> > I've been able to reproduce the problem that Deon reported
> > with 2.6.30, which includes the change above, so I think
> > that it is a separate problem.
>
> It is still not clear to me whether Deon needed just 2.6.30
> or your latest change was needed. I don't see how ipvsadm 1.25
> can provide virtual port without -t/-u options. What is your
> ipvsadm command line that reproduces this svc port problem in
> 2.6.30? May be you have both -f and -t/-u ?

>From memory, I simply tried something like:

ipvsadm -A -t 1
ipvsadm -a -t 1 -r 10.0.0.1:80

and then later

ipvsadm -A -t 1
ipvsadm -a -t 1 -r 10.0.0.1:0

I'll try and retest this.

> > That said, Deon also needs to change above, "ipvs: Fix IPv4 FWMARK virtual
> > services", as he is using more than one fwmark.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Dec 7, 2009, 5:15 PM

Post #11 of 32 (3394 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
On Mon, Dec 07, 2009 at 07:45:22PM +1100, Simon Horman wrote:
> On Mon, Dec 07, 2009 at 10:37:32AM +0200, Julian Anastasov wrote:
> >
> > Hello,
> >
> > On Mon, 30 Nov 2009, Simon Horman wrote:
> >
> > > > looking at
> > > >
> > > > kernel: IPVS: template lookup/in IP 160.124.109.65:0->0.0.0.0:0 not hit
> > > >
> > > > daddr=0.0.0.0 shows that fwmark is not provided at the
> > > > right place, so the user just needs to upgrade or to apply
> > > > the patch from May, 2009:
> > > >
> > > > http://marc.info/?t=124050022700001&r=1&w=2
> > >
> > > Hi Julian,
> > >
> > > I've been able to reproduce the problem that Deon reported
> > > with 2.6.30, which includes the change above, so I think
> > > that it is a separate problem.
> >
> > It is still not clear to me whether Deon needed just 2.6.30
> > or your latest change was needed. I don't see how ipvsadm 1.25
> > can provide virtual port without -t/-u options. What is your
> > ipvsadm command line that reproduces this svc port problem in
> > 2.6.30? May be you have both -f and -t/-u ?

Hi Julian,

here are the results of some tests that I ran today.
They seem to indicate that this problem is real.

2.6.32
======

Test 1
------

# iptables -t mangle -F
# iptables -t mangle -A PREROUTING -p tcp --dport 10000 -j MARK --set-mark 1
# ipvsadm -C
# ipvsadm -A -f 1
# ipvsadm -a -f 1 -r localhost:10000

$ telnet 172.17.60.197 1000
Trying 172.17.60.197...
Ok

Test 2
------

# iptables -t mangle -F
# iptables -t mangle -A PREROUTING -p tcp --dport 10000 -j MARK --set-mark 1
# ipvsadm -C
# ipvsadm -A -f 1 -p 600
# ipvsadm -a -f 1 -r localhost:10000

telnet 172.17.60.197 10000
Trying 172.17.60.197...
telnet: Unable to connect to remote host: Connection refused

2.6.32 + the following debug patch
==================================

--- linux-2.6.orig/net/netfilter/ipvs/ip_vs_core.c 2009-12-08 09:57:41.000000000 +0900
+++ linux-2.6/net/netfilter/ipvs/ip_vs_core.c 2009-12-08 10:02:35.000000000 +0900
@@ -295,8 +295,11 @@ ip_vs_sched_persist(struct ip_vs_service
* If it is not persistent port zero, return NULL,
* otherwise create a connection template.
*/
- if (svc->port)
+ if (svc->port) {
+ IP_VS_DBG(6, "p-schedule: non-zero svc port: "
+ "0x%x\n", svc->port);
return NULL;
+ }

dest = svc->scheduler->schedule(svc, skb);
if (dest == NULL) {

Test 1
------

# iptables -t mangle -F
# iptables -t mangle -A PREROUTING -p tcp --dport 10000 -j MARK --set-mark 1
# ipvsadm -C
# ipvsadm -A -f 1
# ipvsadm -a -f 1 -r localhost:10000

$ telnet 172.17.60.197 1000
Trying 172.17.60.197...
Ok

Test 2
------

# iptables -t mangle -F
# iptables -t mangle -A PREROUTING -p tcp --dport 10000 -j MARK --set-mark 1
# ipvsadm -C
# ipvsadm -A -f 1 -p 600
# ipvsadm -a -f 1 -r localhost:10000

telnet 172.17.60.197 10000
Trying 172.17.60.197...
telnet: Unable to connect to remote host: Connection refused

In dmsg:
[ 1134.971085] IPVS: p-schedule: src 172.17.60.192:37337 dest 172.17.60.197:10000 mnet 172.17.60.192
[ 1134.971089] IPVS: p-schedule: non-zero svc port: 0x8801


2.6.32 + my proposed fix
========================

Test 1
------

# iptables -t mangle -F
# iptables -t mangle -A PREROUTING -p tcp --dport 10000 -j MARK --set-mark 1
# ipvsadm -C
# ipvsadm -A -f 1
# ipvsadm -a -f 1 -r localhost:10000

$ telnet 172.17.60.197 1000
Trying 172.17.60.197...
Ok

Test 2
------

# iptables -t mangle -F
# iptables -t mangle -A PREROUTING -p tcp --dport 10000 -j MARK --set-mark 1
# ipvsadm -C
# ipvsadm -A -f 1 -p 600
# ipvsadm -a -f 1 -r localhost:10000

telnet 172.17.60.197 10000
Trying 172.17.60.197...
Ok


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


ja at ssi

Dec 8, 2009, 1:06 AM

Post #12 of 32 (3385 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
Hello,

On Tue, 8 Dec 2009, Simon Horman wrote:

> Hi Julian,
>
> here are the results of some tests that I ran today.
> They seem to indicate that this problem is real.
>
> - if (svc->port)
> + if (svc->port) {
> + IP_VS_DBG(6, "p-schedule: non-zero svc port: "
> + "0x%x\n", svc->port);
> return NULL;
> + }
>
> dest = svc->scheduler->schedule(svc, skb);
> if (dest == NULL) {

I think, the problem is at another place. It seems
you are using netlink. May be you should fix ip_vs_genl_find_service
to memset usvc before parsing the service. Check other such
places such as ip_vs_genl_set_cmd. Or may be memset should be in
ip_vs_genl_parse_service. I don't have much time today but
if you walk the genl functions may be other such problems can popup.
Better create fix for these problems and I think your problem
will disappear.

Regards

--
Julian Anastasov <ja [at] ssi>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Dec 10, 2009, 9:32 PM

Post #13 of 32 (3301 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
On Tue, Dec 08, 2009 at 11:06:30AM +0200, Julian Anastasov wrote:
>
> Hello,
>
> On Tue, 8 Dec 2009, Simon Horman wrote:
>
> > Hi Julian,
> >
> > here are the results of some tests that I ran today.
> > They seem to indicate that this problem is real.
> >
> > - if (svc->port)
> > + if (svc->port) {
> > + IP_VS_DBG(6, "p-schedule: non-zero svc port: "
> > + "0x%x\n", svc->port);
> > return NULL;
> > + }
> >
> > dest = svc->scheduler->schedule(svc, skb);
> > if (dest == NULL) {
>
> I think, the problem is at another place. It seems
> you are using netlink. May be you should fix ip_vs_genl_find_service
> to memset usvc before parsing the service. Check other such
> places such as ip_vs_genl_set_cmd. Or may be memset should be in
> ip_vs_genl_parse_service. I don't have much time today but
> if you walk the genl functions may be other such problems can popup.
> Better create fix for these problems and I think your problem
> will disappear.

Hi Julian,

I took a look at your suggestion and zeroing usvc in
ip_vs_genl_parse_service() also seems to does seem to resolve the problem
that was reported. And inspecting the code it looks like it should
be sufficient to resolve any similar problems that may be lurking.

How do you feel about the following?

----------------------------------------------------------------------

ipvs: zero usvc

Make sure that any otherwise uninitialised fields of usvc are zero.

This has been obvserved to cause a problem whereby the port of
fwmark services may end up as a non-zero value which causes
scheduling of a destination server to fail for persisitent services.

As observed by Deon van der Merwe <dvdm [at] truteq>.
This fix suggested by Julian Anastasov <ja [at] ssi>.

Cc: Deon van der Merwe <dvdm [at] truteq>
Cc: Julian Anastasov <ja [at] ssi>
Signed-off-by: Simon Horman <horms [at] verge>

Index: linux-2.6/net/netfilter/ipvs/ip_vs_ctl.c
===================================================================
--- linux-2.6.orig/net/netfilter/ipvs/ip_vs_ctl.c 2009-12-11 14:22:23.000000000 +0900
+++ linux-2.6/net/netfilter/ipvs/ip_vs_ctl.c 2009-12-11 14:25:03.000000000 +0900
@@ -2714,6 +2714,8 @@ static int ip_vs_genl_parse_service(stru
if (!(nla_af && (nla_fwmark || (nla_port && nla_protocol && nla_addr))))
return -EINVAL;

+ memset(usvc, 0, sizeof(*usvc));
+
usvc->af = nla_get_u16(nla_af);
#ifdef CONFIG_IP_VS_IPV6
if (usvc->af != AF_INET && usvc->af != AF_INET6)

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


ja at ssi

Dec 11, 2009, 1:01 AM

Post #14 of 32 (3299 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
Hello,

On Fri, 11 Dec 2009, Simon Horman wrote:

> Hi Julian,
>
> I took a look at your suggestion and zeroing usvc in
> ip_vs_genl_parse_service() also seems to does seem to resolve the problem
> that was reported. And inspecting the code it looks like it should
> be sufficient to resolve any similar problems that may be lurking.
>
> How do you feel about the following?

Looks good. May be you can do the same for ip_vs_genl_parse_dest,
just to be safe, if new parameter is added but netlink does not support
it, we do not want random data from stack.

> ----------------------------------------------------------------------
>
> ipvs: zero usvc
>
> Make sure that any otherwise uninitialised fields of usvc are zero.
>
> This has been obvserved to cause a problem whereby the port of
> fwmark services may end up as a non-zero value which causes
> scheduling of a destination server to fail for persisitent services.
>
> As observed by Deon van der Merwe <dvdm [at] truteq>.
> This fix suggested by Julian Anastasov <ja [at] ssi>.
>
> Cc: Deon van der Merwe <dvdm [at] truteq>
> Cc: Julian Anastasov <ja [at] ssi>
> Signed-off-by: Simon Horman <horms [at] verge>


Acked-by: Julian Anastasov <ja [at] ssi>


> Index: linux-2.6/net/netfilter/ipvs/ip_vs_ctl.c
> ===================================================================
> --- linux-2.6.orig/net/netfilter/ipvs/ip_vs_ctl.c 2009-12-11 14:22:23.000000000 +0900
> +++ linux-2.6/net/netfilter/ipvs/ip_vs_ctl.c 2009-12-11 14:25:03.000000000 +0900
> @@ -2714,6 +2714,8 @@ static int ip_vs_genl_parse_service(stru
> if (!(nla_af && (nla_fwmark || (nla_port && nla_protocol && nla_addr))))
> return -EINVAL;
>
> + memset(usvc, 0, sizeof(*usvc));
> +
> usvc->af = nla_get_u16(nla_af);
> #ifdef CONFIG_IP_VS_IPV6
> if (usvc->af != AF_INET && usvc->af != AF_INET6)

Regards

--
Julian Anastasov <ja [at] ssi>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Dec 13, 2009, 6:14 PM

Post #15 of 32 (3230 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
On Fri, Dec 11, 2009 at 11:01:47AM +0200, Julian Anastasov wrote:
>
> Hello,
>
> On Fri, 11 Dec 2009, Simon Horman wrote:
>
> > Hi Julian,
> >
> > I took a look at your suggestion and zeroing usvc in
> > ip_vs_genl_parse_service() also seems to does seem to resolve the problem
> > that was reported. And inspecting the code it looks like it should
> > be sufficient to resolve any similar problems that may be lurking.
> >
> > How do you feel about the following?
>
> Looks good. May be you can do the same for ip_vs_genl_parse_dest,
> just to be safe, if new parameter is added but netlink does not support
> it, we do not want random data from stack.

Good idea. How about this?

----------------------------------------------------------------------

ipvs: zero usvc and udest

Make sure that any otherwise uninitialised fields of usvc are zero.

This has been obvserved to cause a problem whereby the port of
fwmark services may end up as a non-zero value which causes
scheduling of a destination server to fail for persisitent services.

As observed by Deon van der Merwe <dvdm [at] truteq>.
This fix suggested by Julian Anastasov <ja [at] ssi>.

For good measure also zero udest.

Cc: Deon van der Merwe <dvdm [at] truteq>
Cc: Julian Anastasov <ja [at] ssi>
Signed-off-by: Simon Horman <horms [at] verge>

Index: linux-2.6/net/netfilter/ipvs/ip_vs_ctl.c
===================================================================
--- linux-2.6.orig/net/netfilter/ipvs/ip_vs_ctl.c 2009-12-11 14:22:23.000000000 +0900
+++ linux-2.6/net/netfilter/ipvs/ip_vs_ctl.c 2009-12-14 11:05:50.000000000 +0900
@@ -2714,6 +2714,8 @@ static int ip_vs_genl_parse_service(stru
if (!(nla_af && (nla_fwmark || (nla_port && nla_protocol && nla_addr))))
return -EINVAL;

+ memset(usvc, 0, sizeof(*usvc));
+
usvc->af = nla_get_u16(nla_af);
#ifdef CONFIG_IP_VS_IPV6
if (usvc->af != AF_INET && usvc->af != AF_INET6)
@@ -2901,6 +2903,8 @@ static int ip_vs_genl_parse_dest(struct
if (!(nla_addr && nla_port))
return -EINVAL;

+ memset(udest, 0, sizeof(*udest));
+
nla_memcpy(&udest->addr, nla_addr, sizeof(udest->addr));
udest->port = nla_get_u16(nla_port);


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


aleksey at bb

Dec 14, 2009, 1:16 AM

Post #16 of 32 (3220 views)
Permalink
Re: [lvs-users] ip_vs_send_async error [In reply to]
Hello!

Finally we have place our LVS in production. We doesn't find any errors in
test scheme. But there is problem in production environment. We are using
connection sync and every second above 100 error messages comes to
/var/log/messages on LVS Master server.

Dec 14 12:06:42 lvs-master kernel: IPVS: ip_vs_send_async error
Dec 14 12:06:42 lvs-master kernel: IPVS: ip_vs_send_async error
Dec 14 12:06:42 lvs-master kernel: IPVS: ip_vs_send_async error
Dec 14 12:06:42 lvs-master kernel: IPVS: ip_vs_send_async error

I try to start connection sync daemon different ways:

1)
/sbin/ipvsadm --start-daemon=master --mcast-interface=eth0 --syncid=1
/sbin/ipvsadm --start-daemon=backup --mcast-interface=eth0 --syncid=1

2)
/sbin/ipvsadm --start-daemon=master
/sbin/ipvsadm --start-daemon=backup

3)
/sbin/ipvsadm --start-daemon=master --mcast-interface=eth1
/sbin/ipvsadm --start-daemon=backup --mcast-interface=eth1

But every time get the same errors. I can see connections from lvs master on
lvs backup so connection sync is work.

My kernel version is 2.6.31.2.
My ipvsadm version is ipvsadm-1.24.


Regards, Aleksey


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Dec 14, 2009, 2:24 AM

Post #17 of 32 (3215 views)
Permalink
Re: [lvs-users] ip_vs_send_async error [In reply to]
On Mon, Dec 14, 2009 at 11:16:44AM +0200, Aleksey Chudov wrote:
> Hello!
>
> Finally we have place our LVS in production. We doesn't find any errors in
> test scheme. But there is problem in production environment. We are using
> connection sync and every second above 100 error messages comes to
> /var/log/messages on LVS Master server.
>
> Dec 14 12:06:42 lvs-master kernel: IPVS: ip_vs_send_async error
> Dec 14 12:06:42 lvs-master kernel: IPVS: ip_vs_send_async error
> Dec 14 12:06:42 lvs-master kernel: IPVS: ip_vs_send_async error
> Dec 14 12:06:42 lvs-master kernel: IPVS: ip_vs_send_async error
>
> I try to start connection sync daemon different ways:
>
> 1)
> /sbin/ipvsadm --start-daemon=master --mcast-interface=eth0 --syncid=1
> /sbin/ipvsadm --start-daemon=backup --mcast-interface=eth0 --syncid=1
>
> 2)
> /sbin/ipvsadm --start-daemon=master
> /sbin/ipvsadm --start-daemon=backup
>
> 3)
> /sbin/ipvsadm --start-daemon=master --mcast-interface=eth1
> /sbin/ipvsadm --start-daemon=backup --mcast-interface=eth1
>
> But every time get the same errors. I can see connections from lvs master on
> lvs backup so connection sync is work.
>
> My kernel version is 2.6.31.2.
> My ipvsadm version is ipvsadm-1.24.

At a wild guess I'd say that there is some issue with the
network configuration on your production machine such
that the kernel can't send packets to the multicast group
224.0.0.81. Perhaps a missing route?


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


aleksey at bb

Dec 14, 2009, 2:46 AM

Post #18 of 32 (3208 views)
Permalink
Re: [lvs-users] ip_vs_send_async error [In reply to]
Hello!

All test was on the same machine. There is no problems with only few
connections while testing. Now we have above 8000 conn/ps. But it is only
half of planned rate. I can see connections from lvs master on lvs backup so
connection sync is work. May be not all connections is synchronized, but I
don't know how to test this.

Regards, Aleksey


-----Original Message-----
From: lvs-users-bounces [at] linuxvirtualserver
[mailto:lvs-users-bounces [at] linuxvirtualserver] On Behalf Of Simon Horman
Sent: Monday, December 14, 2009 12:25 PM
To: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] ip_vs_send_async error

On Mon, Dec 14, 2009 at 11:16:44AM +0200, Aleksey Chudov wrote:
> Hello!
>
> Finally we have place our LVS in production. We doesn't find any errors in
> test scheme. But there is problem in production environment. We are using
> connection sync and every second above 100 error messages comes to
> /var/log/messages on LVS Master server.
>
> Dec 14 12:06:42 lvs-master kernel: IPVS: ip_vs_send_async error
> Dec 14 12:06:42 lvs-master kernel: IPVS: ip_vs_send_async error
> Dec 14 12:06:42 lvs-master kernel: IPVS: ip_vs_send_async error
> Dec 14 12:06:42 lvs-master kernel: IPVS: ip_vs_send_async error
>
> I try to start connection sync daemon different ways:
>
> 1)
> /sbin/ipvsadm --start-daemon=master --mcast-interface=eth0 --syncid=1
> /sbin/ipvsadm --start-daemon=backup --mcast-interface=eth0 --syncid=1
>
> 2)
> /sbin/ipvsadm --start-daemon=master
> /sbin/ipvsadm --start-daemon=backup
>
> 3)
> /sbin/ipvsadm --start-daemon=master --mcast-interface=eth1
> /sbin/ipvsadm --start-daemon=backup --mcast-interface=eth1
>
> But every time get the same errors. I can see connections from lvs master
on
> lvs backup so connection sync is work.
>
> My kernel version is 2.6.31.2.
> My ipvsadm version is ipvsadm-1.24.

At a wild guess I'd say that there is some issue with the
network configuration on your production machine such
that the kernel can't send packets to the multicast group
224.0.0.81. Perhaps a missing route?


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


siim at p6drad-teel

Dec 14, 2009, 3:10 AM

Post #19 of 32 (3215 views)
Permalink
Re: [lvs-users] ip_vs_send_async error [In reply to]
Hi

Aleksey Chudov wrote:
> All test was on the same machine. There is no problems with only few
> connections while testing. Now we have above 8000 conn/ps. But it is only
> half of planned rate. I can see connections from lvs master on lvs backup so
> connection sync is work. May be not all connections is synchronized, but I
> don't know how to test this.

The kernel only wakes connsync thread up every second to send sync
messages in one batch. When the UDP send buffer cant fit all that data
any more, you will get those errors on sync. To get around it, make
your buffers bigger.

net.core.{w,r}mem_{max,default} are the right sysctls IIRC. Also be
sure to increase the receiver side buffer, otherways you will drop the
packets on the other end and that is more difficult to spot (Udp
InErrors in netstat -s can hint about it, for example).

Alternatively, you may want to build your kernel with the following
patch (sorry about the broken wrapping, starts the sync operation 10
times as often):

--- a/net/ipv4/ipvs/ip_vs_sync.c 2008-02-11 05:51:11.000000000
+0000
+++ b/net/ipv4/ipvs/ip_vs_sync.c 2008-10-06 12:17:07.883882381
+0000
@@ -679,7 +686,7 @@
if (stop_master_sync)
break;

- msleep_interruptible(1000);
+ msleep_interruptible(100);
}

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


ja at ssi

Dec 14, 2009, 12:51 PM

Post #20 of 32 (3197 views)
Permalink
Re: [lvs-users] "connection refused" when persistence enabled [In reply to]
Hello,

On Mon, 14 Dec 2009, Simon Horman wrote:

> > Looks good. May be you can do the same for ip_vs_genl_parse_dest,
> > just to be safe, if new parameter is added but netlink does not support
> > it, we do not want random data from stack.
>
> Good idea. How about this?

Very good, thanks!

> ----------------------------------------------------------------------
>
> ipvs: zero usvc and udest
>
> Make sure that any otherwise uninitialised fields of usvc are zero.
>
> This has been obvserved to cause a problem whereby the port of
> fwmark services may end up as a non-zero value which causes
> scheduling of a destination server to fail for persisitent services.
>
> As observed by Deon van der Merwe <dvdm [at] truteq>.
> This fix suggested by Julian Anastasov <ja [at] ssi>.
>
> For good measure also zero udest.
>
> Cc: Deon van der Merwe <dvdm [at] truteq>
> Cc: Julian Anastasov <ja [at] ssi>
> Signed-off-by: Simon Horman <horms [at] verge>


Acked-by: Julian Anastasov <ja [at] ssi>


> Index: linux-2.6/net/netfilter/ipvs/ip_vs_ctl.c
> ===================================================================
> --- linux-2.6.orig/net/netfilter/ipvs/ip_vs_ctl.c 2009-12-11 14:22:23.000000000 +0900
> +++ linux-2.6/net/netfilter/ipvs/ip_vs_ctl.c 2009-12-14 11:05:50.000000000 +0900
> @@ -2714,6 +2714,8 @@ static int ip_vs_genl_parse_service(stru
> if (!(nla_af && (nla_fwmark || (nla_port && nla_protocol && nla_addr))))
> return -EINVAL;
>
> + memset(usvc, 0, sizeof(*usvc));
> +
> usvc->af = nla_get_u16(nla_af);
> #ifdef CONFIG_IP_VS_IPV6
> if (usvc->af != AF_INET && usvc->af != AF_INET6)
> @@ -2901,6 +2903,8 @@ static int ip_vs_genl_parse_dest(struct
> if (!(nla_addr && nla_port))
> return -EINVAL;
>
> + memset(udest, 0, sizeof(*udest));
> +
> nla_memcpy(&udest->addr, nla_addr, sizeof(udest->addr));
> udest->port = nla_get_u16(nla_port);

Regards

--
Julian Anastasov <ja [at] ssi>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


aleksey at bb

Dec 14, 2009, 4:07 PM

Post #21 of 32 (3186 views)
Permalink
Re: [lvs-users] ip_vs_send_async error [In reply to]
Hello!

I have the following parameters set on LVS Master:
net.core.wmem_max = 16777216
net.core.wmem_default = 118784
net.core.rmem_max = 16777216
net.core.rmem_default = 118784

# netstat -s
Udp:
12669936 packets received
398141 packets to unknown port received.
0 packet receive errors
47229187 packets sent
RcvbufErrors: 0
SndbufErrors: 29490577


How to know correct buffer parameters for 16k conn/ps?
Is it possible to fix connection sync daemon messages without recompiling
the krenel?


Regards, Aleksey



-----Original Message-----

Hi

Aleksey Chudov wrote:
> All test was on the same machine. There is no problems with only few
> connections while testing. Now we have above 8000 conn/ps. But it is only
> half of planned rate. I can see connections from lvs master on lvs backup
so
> connection sync is work. May be not all connections is synchronized, but I
> don't know how to test this.

The kernel only wakes connsync thread up every second to send sync
messages in one batch. When the UDP send buffer cant fit all that data
any more, you will get those errors on sync. To get around it, make
your buffers bigger.

net.core.{w,r}mem_{max,default} are the right sysctls IIRC. Also be
sure to increase the receiver side buffer, otherways you will drop the
packets on the other end and that is more difficult to spot (Udp
InErrors in netstat -s can hint about it, for example).

Alternatively, you may want to build your kernel with the following
patch (sorry about the broken wrapping, starts the sync operation 10
times as often):

--- a/net/ipv4/ipvs/ip_vs_sync.c 2008-02-11 05:51:11.000000000
+0000
+++ b/net/ipv4/ipvs/ip_vs_sync.c 2008-10-06 12:17:07.883882381
+0000
@@ -679,7 +686,7 @@
if (stop_master_sync)
break;

- msleep_interruptible(1000);
+ msleep_interruptible(100);
}

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Dec 14, 2009, 11:57 PM

Post #22 of 32 (3161 views)
Permalink
Re: [lvs-users] ip_vs_send_async error [In reply to]
On Tue, Dec 15, 2009 at 02:07:12AM +0200, Aleksey Chudov wrote:
> Hello!
>
> I have the following parameters set on LVS Master:
> net.core.wmem_max = 16777216
> net.core.wmem_default = 118784
> net.core.rmem_max = 16777216
> net.core.rmem_default = 118784
>
> # netstat -s
> Udp:
> 12669936 packets received
> 398141 packets to unknown port received.
> 0 packet receive errors
> 47229187 packets sent
> RcvbufErrors: 0
> SndbufErrors: 29490577
>
>
> How to know correct buffer parameters for 16k conn/ps?
> Is it possible to fix connection sync daemon messages without recompiling
> the krenel?

I wonder if the kernel should be fixed to a) use a lower pause between
batches of sends (its currently HZ = 1s) or b) use a dynamic pause depending
on many packets are being handled.


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


aleksey at bb

Dec 15, 2009, 2:18 AM

Post #23 of 32 (3149 views)
Permalink
Re: [lvs-users] ip_vs_send_async error [In reply to]
Do you think the size of UDP buffer is enough?
Patch for lower pause was in previous message but how to use a dynamic
pause?


Regards, Aleksey


-----Original Message-----


On Tue, Dec 15, 2009 at 02:07:12AM +0200, Aleksey Chudov wrote:
> Hello!
>
> I have the following parameters set on LVS Master:
> net.core.wmem_max = 16777216
> net.core.wmem_default = 118784
> net.core.rmem_max = 16777216
> net.core.rmem_default = 118784
>
> # netstat -s
> Udp:
> 12669936 packets received
> 398141 packets to unknown port received.
> 0 packet receive errors
> 47229187 packets sent
> RcvbufErrors: 0
> SndbufErrors: 29490577
>
>
> How to know correct buffer parameters for 16k conn/ps?
> Is it possible to fix connection sync daemon messages without recompiling
> the krenel?

I wonder if the kernel should be fixed to a) use a lower pause between
batches of sends (its currently HZ = 1s) or b) use a dynamic pause depending
on many packets are being handled.


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


horms at verge

Dec 15, 2009, 3:37 AM

Post #24 of 32 (3148 views)
Permalink
Re: [lvs-users] ip_vs_send_async error [In reply to]
On Tue, Dec 15, 2009 at 12:18:04PM +0200, Aleksey Chudov wrote:
> Do you think the size of UDP buffer is enough?

I was hoping for some feedback on you for that.
I suspect it is enough.

> Patch for lower pause was in previous message but how to use a dynamic
> pause?

I was thinking of a different patch, that I haven't written yet,
and I don't think anyone else has either. I was just thinking aloud.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users


aleksey at bb

Dec 15, 2009, 3:52 AM

Post #25 of 32 (3150 views)
Permalink
Re: [lvs-users] ip_vs_send_async error [In reply to]
Thank you!

I try to use lower pause.


Regards, Aleksey


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

First page Previous page 1 2 Next page Last page  View All Linux Virtual Server users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.