Mailing List Archive: Linux Virtual Server: Users

[lvs-users] CentOS 4.7 (2.6.9-based) -- LVS-NAT return packets leaving via wrong interface

Index | Next | Previous | View Threaded

charles at dyfis

Mar 11, 2009, 5:14 PM

Post #1 of 6 (1200 views)
Permalink

[lvs-users] CentOS 4.7 (2.6.9-based) -- LVS-NAT return packets leaving via wrong interface

Howdy!

I have a two-interface configuration on my director, where each
interface is on a different subnet -- an internal interface with the
realservers, and an external one with the VIPs. Using LVS-NAT, SYN
packets are correctly routed by the director to an appropriate
realserver and ACKs are appropriately routed back to the director from
the realclient (via the default gateway) -- but when the director emits
the demasqueraded ACK to be sent to the client, it does so on the
internal interface rather than the external one, and the router between
the two (which I don't control) is disinclined to forward it.

I've tried to work around this using source routing, as follows:

# ip rule show
0: from all lookup local
32764: from <INTERNAL_NET> lookup int
32765: from <EXTERNAL_NET> lookup ext
32766: from all lookup main
32767: from all lookup default
# ip route show table ext
<EXTERNAL_NET> dev eth1 scope link
default via <EXTERNAL_GW> dev eth1
# ip route show table int
<INTERNAL_NET> dev eth0 scope link
default via <INTERNAL_GW> dev eth0

As the demasqueraded packets have a source address on <EXTERNAL_NET>, I
would expect them to leave on eth1 via <EXTERNAL_GW>. However, this does
not happen -- the demasqueraded packet attempts to leave via eth0, and
thus never reaches its destination.

Any hints?

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

jmack at wm7d

Mar 11, 2009, 6:27 PM

Post #2 of 6 (1153 views)
Permalink

Re: [lvs-users] CentOS 4.7 (2.6.9-based) -- LVS-NAT return packets leaving via wrong interface [In reply to]

On Wed, 11 Mar 2009, Charles Duffy wrote:

> Howdy!
>
> I have a two-interface configuration on my director, where each
> interface is on a different subnet -- an internal interface with the
> realservers, and an external one with the VIPs. Using LVS-NAT, SYN
> packets are correctly routed by the director to an appropriate
> realserver and ACKs are appropriately routed back to the director from
> the realclient (via the default gateway) -- but when the director emits
> the demasqueraded ACK to be sent to the client, it does so on the
> internal interface rather than the external one,

This is supposed to work.

Things to look for would be

o you have an after market enhanced version of LVS. Use a
standard kernel not a centos kernel

o you have iptables rules running.

> I've tried to work around this using source routing, as follows:

this is not the solution

Joe
--
Joseph Mack NA3T EME(B,D), FM05lw North Carolina
jmack (at) wm7d (dot) net - azimuthal equidistant map
generator at http://www.wm7d.net/azproj.shtml
Homepage http://www.austintek.com/ It's GNU/Linux!

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

graeme at graemef

Mar 12, 2009, 3:57 AM

Post #3 of 6 (1133 views)
Permalink

Re: [lvs-users] CentOS 4.7 (2.6.9-based) -- LVS-NAT return packets leaving via wrong interface [In reply to]

On Wed, 2009-03-11 at 19:14 -0500, Charles Duffy wrote:
> # ip rule show
> 0: from all lookup local
> 32764: from <INTERNAL_NET> lookup int
> 32765: from <EXTERNAL_NET> lookup ext
> 32766: from all lookup main
> 32767: from all lookup default
> # ip route show table ext
> <EXTERNAL_NET> dev eth1 scope link
> default via <EXTERNAL_GW> dev eth1
> # ip route show table int
> <INTERNAL_NET> dev eth0 scope link
> default via <INTERNAL_GW> dev eth0

Simple fix: have a single default route. You should only very rarely
have two defaults. If you make sure your box has a single default route
via EXTERNAL_GW then your problem will resolve itself.

Networking 101 :)

Graeme

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

charles at dyfis

Mar 12, 2009, 8:24 AM

Post #4 of 6 (1130 views)
Permalink

Re: [lvs-users] CentOS 4.7 (2.6.9-based) -- LVS-NAT return packets leaving via wrong interface [In reply to]

Joseph Mack NA3T wrote:
> Things to look for would be
>
> o you have an after market enhanced version of LVS. Use a
> standard kernel not a centos kernel

This was it; moving to a freshly compiled 2.6.28.7 resolved the issue.

I'll probably be looking into whether RHEL5 exhibits this issue at some
point, and submitting a ticket to Red Hat if so.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

charles at dyfis

Mar 12, 2009, 10:20 AM

Post #5 of 6 (1128 views)
Permalink

Re: [lvs-users] CentOS 4.7 (2.6.9-based) -- LVS-NAT return packets leaving via wrong interface [In reply to]

Graeme Fowler wrote:
> Simple fix: have a single default route. You should only very rarely
> have two defaults. If you make sure your box has a single default route
> via EXTERNAL_GW then your problem will resolve itself.

Except that my box isn't allowed to talk to anything via EXTERNAL_GW;
packets routed out through it, except those coming from the VIPs, are
silently dropped. I'm only allowed to talk to the outside world when
going through <INTERNAL_GW>.

Finding out why, or getting this changed, would mean putting in a work
order with the firewall guys and waiting a few weeks.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Charles_Duffy at messageone

Mar 25, 2009, 2:57 PM

Post #6 of 6 (1009 views)
Permalink

Re: [lvs-users] CentOS 4.7 (2.6.9-based) -- LVS-NAT return packets leaving via wrong interface [In reply to]

Charles Duffy wrote:
> Joseph Mack NA3T wrote:
>> Things to look for would be
>>
>> o you have an after market enhanced version of LVS. Use a
>> standard kernel not a centos kernel
>
> This was it; moving to a freshly compiled 2.6.28.7 resolved the issue.

Actually, this was incorrect: Moving to a freshly compiled 2.6.28.7
resolved the issue *only when combined with the source routing
configuration previously described*.

Without source routing used to shepherd demasqueraded packets to eth1
(with a different, appropriate gateway), they still were leaving eth0
and being discarded by the local router.

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users [at] LinuxVirtualServer
Send requests to lvs-users-request [at] LinuxVirtualServer
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads