Mailing List Archive: Linux Virtual Server: Users

[lvs-users] LVS DR dropping the FIN packets

Index | Next | Previous | View Threaded

thomas at pedoussaut

Apr 22, 2008, 7:02 AM

Post #1 of 7 (992 views)
Permalink

[lvs-users] LVS DR dropping the FIN packets

I came across a very strange problem.

For one of my dozen of services ( a straight TCP connection), the
TCP-FIN packets that are arriving on the load balancer are never passed
to the real server.

I activated the logs of iptable and could see the FIN packets being dropped.
No idea why the FIN are dropped and not the other ones. I obviously have
the --state ESTABLISHED,RELATED -j ACCEPT in my iptable rules.

I had a quick look at /proc/net/ip_conntrack before, during and after
the connection but nothing specific to that connection seems to be
inserted (the module is loaded and other traffic gets tracked).

Google doesn't really help. Someone had a similar problem last year but
was never publicly solved.

If someone had the same issue and managed to solve it, I'd be glad to
hear from you.

--
Thomas

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Apr 22, 2008, 7:56 AM

Post #2 of 7 (916 views)
Permalink

Re: [lvs-users] LVS DR dropping the FIN packets [In reply to]

On Tue, 22 Apr 2008, Thomas Pedoussaut wrote:

> For one of my dozen of services ( a straight TCP connection), the
> TCP-FIN packets that are arriving on the load balancer are never passed
> to the real server.

looking in

http://marc.info/?l=linux-virtual-server&w=2

for "LVS DR FIN"

I find some postings by Siim_Poder with the same problem
(he's using LVS-NAT). He didn't really have a good idea what
was going on, but assumed that it was a flaky connection to
the client (I know the FIN packet has got as far as the
director, and you'd wonder why it couldn't get the next
step to the realserver). He increased his timeouts. He
hasn't come back saying that he still has the problem. Maybe
he's living with it, or maybe it's solved. I can't tell.

Another post from Andreas Lundqvist

http://marc.info/?l=linux-virtual-server&m=116254182228697&w=2

showed that the problem only occured with encrypted
services. He fixed the problem by changing the packet size

Let us know what you find.

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

thomas at pedoussaut

Apr 22, 2008, 9:32 AM

Post #3 of 7 (919 views)
Permalink

Re: [lvs-users] LVS DR dropping the FIN packets [In reply to]

Joseph Mack NA3T wrote:
> On Tue, 22 Apr 2008, Thomas Pedoussaut wrote:
>
>
>> For one of my dozen of services ( a straight TCP connection), the
>> TCP-FIN packets that are arriving on the load balancer are never passed
>> to the real server.
>>
>
> looking in
>
> http://marc.info/?l=linux-virtual-server&w=2
>
> for "LVS DR FIN"
>
> I find some postings by Siim_Poder with the same problem
> (he's using LVS-NAT). He didn't really have a good idea what
> was going on, but assumed that it was a flaky connection to
> the client (I know the FIN packet has got as far as the
> director, and you'd wonder why it couldn't get the next
> step to the realserver). He increased his timeouts. He
> hasn't come back saying that he still has the problem. Maybe
> he's living with it, or maybe it's solved. I can't tell.
>
> Another post from Andreas Lundqvist
>
> http://marc.info/?l=linux-virtual-server&m=116254182228697&w=2
>
> showed that the problem only occured with encrypted
> services. He fixed the problem by changing the packet size
>
> Let us know what you find.
>
It even happen when I close the client connection within seconds of
creation, so I don't think timeouts are involved.
My issue is that the application in backend doesn't deal with timeouts,
so never initiate the closing of the connection.

My best guess at the moment, as stated earlier is a problem with DR and
the state machine of netfilter.
I mailed Patrik Karén who had the same problem on the netfilter mailing
list but never came back.

--
Thomas

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Apr 22, 2008, 9:55 AM

Post #4 of 7 (913 views)
Permalink

Re: [lvs-users] LVS DR dropping the FIN packets [In reply to]

On Tue, 22 Apr 2008, Thomas Pedoussaut wrote:

> It even happen when I close the client connection within
> seconds of creation, so I don't think timeouts are
> involved. My issue is that the application in backend
> doesn't deal with timeouts, so never initiate the closing
> of the connection.

don't know what the problem is. There are other people with
the same/similar symptoms. You said in your earlier posting
that it only happened with one of the services. What is it?
Is there anything different about it? Do you have iptables
rules running on the realservers/director (if so can you
turn them off for testing).

> My best guess at the moment, as stated earlier is a problem with DR and
> the state machine of netfilter.

netfilter and ipvs() do not get on well together. You may
have to remove the netfilter tracking

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

thomas at pedoussaut

Apr 25, 2008, 4:33 AM

Post #5 of 7 (899 views)
Permalink

Re: [lvs-users] LVS DR dropping the FIN packets [In reply to]

Thomas Pedoussaut wrote:
> I came across a very strange problem.
>
I'm replying here to have a track in archive in case someone else has
the same issue.
> For one of my dozen of services ( a straight TCP connection), the
> TCP-FIN packets that are arriving on the load balancer are never passed
> to the real server.
>
> I activated the logs of iptable and could see the FIN packets being dropped.
> No idea why the FIN are dropped and not the other ones. I obviously have
> the --state ESTABLISHED,RELATED -j ACCEPT in my iptable rules.
>
Basically, all packets (SYN and non-SYN) are allowed by the "--state
NEW" iptables but not by the ESTABLISHED,RELATED, because the director
never sees the replies from the real server and so never creates a
conntrack for that connection.
When a FIN packet arrives, it is not validated as a --state NEW, because
it's flag FIN is activated and so, that particular packet is dropped.

> I had a quick look at /proc/net/ip_conntrack before, during and after
> the connection but nothing specific to that connection seems to be
> inserted (the module is loaded and other traffic gets tracked).
>
So the solution is to change the iptables rule from
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport $VPORT
-j ACCEPT
to
-A RH-Firewall-1-INPUT -m tcp -p tcp --dport $VPORT -j ACCEPT
> Google doesn't really help. Someone had a similar problem last year but
> was never publicly solved.
>
And now I hope this post will get crawled and indexed.

--
Thomas Pedoussaut
http://www.synerginetworking.com/blog/

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Apr 25, 2008, 5:04 AM

Post #6 of 7 (900 views)
Permalink

Re: [lvs-users] LVS DR dropping the FIN packets [In reply to]

On Fri, 25 Apr 2008, Thomas Pedoussaut wrote:

> So the solution is to change the iptables rule from
> -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport $VPORT
> -j ACCEPT
> to
> -A RH-Firewall-1-INPUT -m tcp -p tcp --dport $VPORT -j ACCEPT

thanks. In retrospect (of course) it's obvious. You can't do
stateful inspection of packets with LVS-DR

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

May 7, 2008, 5:19 AM

Post #7 of 7 (825 views)
Permalink

Re: [lvs-users] LVS DR dropping the FIN packets [In reply to]

On Fri, 25 Apr 2008, Thomas Pedoussaut wrote:

> Thomas Pedoussaut wrote:
>> I came across a very strange problem.

do Julian's nfct patches address your problem?

http://www.ssi.bg/~ja/nfct/

Joe

--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads