Mailing List Archive: nsp: foundry

Asymmetrical routing on ADX

Index | Next | Previous | View Threaded

drew.weaver at thenap

May 31, 2012, 2:24 PM

Post #1 of 9 (535 views)
Permalink

Asymmetrical routing on ADX

Hi,

I have recently experienced a problem where performance to a VIP is terrible when the ADX is uplinked to two separate routers running VRRP. TAC suggested that it is because source-nat replies were coming back on a different physical interface than the requests went out on.

In my config I have ports 1 and 3 assigned to the same VLAN with a virtual ethernet attached. If both of the physical ports are assigned to the same VLAN/VE then why would the ADX care which VLAN members the replies return on? That seems to defeat the purpose of having virtual ethernet or L3 VLAN style functionality.

There has to be a work around for this, does anyone know what it is?

Sent from my Samsung Galaxy Tab

dschout at high5

May 31, 2012, 3:59 PM

Post #2 of 9 (518 views)
Permalink

Re: Asymmetrical routing on ADX [In reply to]

Since the session/ip/port information does not change, I very much doubt it had anything to do with the way source-nat replies are coming back.

I expect it had all to do with the much more logical impact of the various security features and packetprocessing/forwarding logic.

Are the replies coming back from at least the same MAC address?

Looking at your setup I'm quite sure they are not.
MAC info is also included in the sessioncache/state-tables.

For fast processing (as the ADX) in its core is a switch it would be most efficient to return packets belonging to a particular session to the MAC address the packets were received from...
Or for security reasons you normally would want packets from a particular session to keep coming in on the same interface from the same MAC address... as anti-spoofing solution.

There is a not-so well documented command to set this to IP based rather than MAC based if I remember correctly.

I can't find it for the moment, but will look into it tomorrow.

I have to add that asym-routing in general is bad and should be avoided.
Why is this happening in your network? And is there a way to avoid it?

Greetings,

Diederik

Sent from my iPhone

On 31 mei 2012, at 23:24, Drew Weaver <drew.weaver [at] thenap> wrote:

> Hi,
>
> I have recently experienced a problem where performance to a VIP is terrible when the ADX is uplinked to two separate routers running VRRP. TAC suggested that it is because source-nat replies were coming back on a different physical interface than the requests went out on.
>
> In my config I have ports 1 and 3 assigned to the same VLAN with a virtual ethernet attached. If both of the physical ports are assigned to the same VLAN/VE then why would the ADX care which VLAN members the replies return on? That seems to defeat the purpose of having virtual ethernet or L3 VLAN style functionality.
>
> There has to be a work around for this, does anyone know what it is?
>
>
>
>
>
>
> Sent from my Samsung Galaxy Tab
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp [at] puck
> http://puck.nether.net/mailman/listinfo/foundry-nsp

dschout at high5

Jun 1, 2012, 8:02 AM

Post #3 of 9 (516 views)
Permalink

Re: Asymmetrical routing on ADX [In reply to]

Hello,

Try the command:

server identify-server-by-ip

This command identifies reverse SLB traffic based on Source IP not just Source MAC.

Greetings,

Diederik

On 1 Jun 2012, at 12:59 , Diederik Schouten wrote:

> Since the session/ip/port information does not change, I very much doubt it had anything to do with the way source-nat replies are coming back.
>
> I expect it had all to do with the much more logical impact of the various security features and packetprocessing/forwarding logic.
>
> Are the replies coming back from at least the same MAC address?
>
> Looking at your setup I'm quite sure they are not.
> MAC info is also included in the sessioncache/state-tables.
>
> For fast processing (as the ADX) in its core is a switch it would be most efficient to return packets belonging to a particular session to the MAC address the packets were received from...
> Or for security reasons you normally would want packets from a particular session to keep coming in on the same interface from the same MAC address... as anti-spoofing solution.
>
> There is a not-so well documented command to set this to IP based rather than MAC based if I remember correctly.
>
> I can't find it for the moment, but will look into it tomorrow.
>
>
> I have to add that asym-routing in general is bad and should be avoided.
> Why is this happening in your network? And is there a way to avoid it?
>
>
> Greetings,
>
> Diederik
>
> Sent from my iPhone
>
> On 31 mei 2012, at 23:24, Drew Weaver <drew.weaver [at] thenap> wrote:
>
>> Hi,
>>
>> I have recently experienced a problem where performance to a VIP is terrible when the ADX is uplinked to two separate routers running VRRP. TAC suggested that it is because source-nat replies were coming back on a different physical interface than the requests went out on.
>>
>> In my config I have ports 1 and 3 assigned to the same VLAN with a virtual ethernet attached. If both of the physical ports are assigned to the same VLAN/VE then why would the ADX care which VLAN members the replies return on? That seems to defeat the purpose of having virtual ethernet or L3 VLAN style functionality.
>>
>> There has to be a work around for this, does anyone know what it is?
>>
>>
>>
>>
>>
>>
>> Sent from my Samsung Galaxy Tab
>> _______________________________________________
>> foundry-nsp mailing list
>> foundry-nsp [at] puck
>> http://puck.nether.net/mailman/listinfo/foundry-nsp
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp [at] puck
> http://puck.nether.net/mailman/listinfo/foundry-nsp

drew.weaver at thenap

Jun 1, 2012, 9:25 AM

Post #4 of 9 (517 views)
Permalink

Re: Asymmetrical routing on ADX [In reply to]

Why is asymmetrical routing bad if you have a complete mesh?

-Drew

From: Diederik Schouten [mailto:dschout [at] high5]
Sent: Thursday, May 31, 2012 7:00 PM
To: Drew Weaver
Cc: foundry-nsp [at] puck
Subject: Re: [f-nsp] Asymmetrical routing on ADX

Since the session/ip/port information does not change, I very much doubt it had anything to do with the way source-nat replies are coming back.

I expect it had all to do with the much more logical impact of the various security features and packetprocessing/forwarding logic.

Are the replies coming back from at least the same MAC address?

Looking at your setup I'm quite sure they are not.
MAC info is also included in the sessioncache/state-tables.

For fast processing (as the ADX) in its core is a switch it would be most efficient to return packets belonging to a particular session to the MAC address the packets were received from...
Or for security reasons you normally would want packets from a particular session to keep coming in on the same interface from the same MAC address... as anti-spoofing solution.

There is a not-so well documented command to set this to IP based rather than MAC based if I remember correctly.

I can't find it for the moment, but will look into it tomorrow.

I have to add that asym-routing in general is bad and should be avoided.
Why is this happening in your network? And is there a way to avoid it?

Greetings,

Diederik

Sent from my iPhone

On 31 mei 2012, at 23:24, Drew Weaver <drew.weaver [at] thenap<mailto:drew.weaver [at] thenap>> wrote:
Hi,

I have recently experienced a problem where performance to a VIP is terrible when the ADX is uplinked to two separate routers running VRRP. TAC suggested that it is because source-nat replies were coming back on a different physical interface than the requests went out on.

In my config I have ports 1 and 3 assigned to the same VLAN with a virtual ethernet attached. If both of the physical ports are assigned to the same VLAN/VE then why would the ADX care which VLAN members the replies return on? That seems to defeat the purpose of having virtual ethernet or L3 VLAN style functionality.

There has to be a work around for this, does anyone know what it is?

Sent from my Samsung Galaxy Tab
_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck<mailto:foundry-nsp [at] puck>
http://puck.nether.net/mailman/listinfo/foundry-nsp

jeffm at iglou

Jun 1, 2012, 12:07 PM

Post #5 of 9 (516 views)
Permalink

Re: Asymmetrical routing on ADX [In reply to]

On Fri, June 1, 2012 12:25, Drew Weaver wrote:
> Why is asymmetrical routing bad if you have a complete mesh?

It isn't (even without a complete mesh).

Asymmetric routing is commonplace and normal, if you need a chokepoint to
apply some sort of network policy (load balancing, firewalling, etc.) you
need to make sure that you make it a chokepoint both coming and going, and
if gear can't handle the next-hop (or even the interface that it uses) may
be different for transmitted and received traffic, then bug reports need
to be filed.

--
Jeff

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp

drew.weaver at thenap

Jun 2, 2012, 7:16 AM

Post #6 of 9 (515 views)
Permalink

Re: Asymmetrical routing on ADX [In reply to]

That's exactly my stance on this issue.

It's especially annoying considering that the two physical ports in question are attached to a VLAN/Virtual Ethernet, I don't understand why it matters which physical interface in the VLAN handles the traffic destined for the VE..

To me that is really broken.

Thanks,
-Drew

-----Original Message-----
From: foundry-nsp-bounces [at] puck [mailto:foundry-nsp-bounces [at] puck] On Behalf Of jeffm [at] iglou
Sent: Friday, June 01, 2012 3:08 PM
To: foundry-nsp [at] puck
Subject: Re: [f-nsp] Asymmetrical routing on ADX

On Fri, June 1, 2012 12:25, Drew Weaver wrote:
> Why is asymmetrical routing bad if you have a complete mesh?

It isn't (even without a complete mesh).

Asymmetric routing is commonplace and normal, if you need a chokepoint to apply some sort of network policy (load balancing, firewalling, etc.) you need to make sure that you make it a chokepoint both coming and going, and if gear can't handle the next-hop (or even the interface that it uses) may be different for transmitted and received traffic, then bug reports need to be filed.

--
Jeff

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp

drew.weaver at thenap

Jun 3, 2012, 5:27 AM

Post #7 of 9 (509 views)
Permalink

Re: Asymmetrical routing on ADX [In reply to]

I hate to reply to myself but I noticed that even though the switch is running router code and it's IP addresses are all handled via the VE when I do show ARP it is still seeing the arp on the physical ports rather than the VLAN.

This is pretty unfortunate.

Thanks,
-Drew

-----Original Message-----
From: foundry-nsp-bounces [at] puck [mailto:foundry-nsp-bounces [at] puck] On Behalf Of Drew Weaver
Sent: Saturday, June 02, 2012 10:16 AM
To: 'jeffm [at] iglou'; foundry-nsp [at] puck
Subject: Re: [f-nsp] Asymmetrical routing on ADX

That's exactly my stance on this issue.

It's especially annoying considering that the two physical ports in question are attached to a VLAN/Virtual Ethernet, I don't understand why it matters which physical interface in the VLAN handles the traffic destined for the VE..

To me that is really broken.

Thanks,
-Drew

-----Original Message-----
From: foundry-nsp-bounces [at] puck [mailto:foundry-nsp-bounces [at] puck] On Behalf Of jeffm [at] iglou
Sent: Friday, June 01, 2012 3:08 PM
To: foundry-nsp [at] puck
Subject: Re: [f-nsp] Asymmetrical routing on ADX

On Fri, June 1, 2012 12:25, Drew Weaver wrote:
> Why is asymmetrical routing bad if you have a complete mesh?

It isn't (even without a complete mesh).

Asymmetric routing is commonplace and normal, if you need a chokepoint to apply some sort of network policy (load balancing, firewalling, etc.) you need to make sure that you make it a chokepoint both coming and going, and if gear can't handle the next-hop (or even the interface that it uses) may be different for transmitted and received traffic, then bug reports need to be filed.

--
Jeff

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck
http://puck.nether.net/mailman/listinfo/foundry-nsp

mkallen at gmail

Jun 3, 2012, 9:38 AM

Post #8 of 9 (509 views)
Permalink

Re: Asymmetrical routing on ADX [In reply to]

Drew, this is actually expected behavior, for pretty much every L3
Switch/Router I have seen. Arp resolves ip address to mac address, and mac
address (normally) is associated with a single port (per vlan) at any one
point in time. If you have Mac addresses moving back and forth, that is
generally a bad thing in a switched environment. Traffic has the
possibility of being black holed, as it will be forwarded only to the port
where the destination mac address is at the instant the switch is ready to
forward.

If you are looking to duplicate the traffic across multiple ports, you need
to do a static arp entry, and possibly a static mac entry as well. This
will duplicate all packets to that ip/mac combo, and send it to multiple
ports. I have generally only seen this in clustering scenarios (like
Microsoft Load Balancing), not in VRRP. In normal VRRP, there is still a
singular owner of that VRRP VIP at any one point in time, one router that
will answer Arp's, thus it should only be learned on one port, not both.
If you are seeing it on both, it sounds like there may be a problem there.

WIthout seeing the configs, or more info, those are my thoughts, hope it
helps.

Mike

On Sun, Jun 3, 2012 at 5:27 AM, Drew Weaver <drew.weaver [at] thenap> wrote:

> I hate to reply to myself but I noticed that even though the switch is
> running router code and it's IP addresses are all handled via the VE when I
> do show ARP it is still seeing the arp on the physical ports rather than
> the VLAN.
>
> This is pretty unfortunate.
>
> Thanks,
> -Drew
>
>
> -----Original Message-----
> From: foundry-nsp-bounces [at] puck [mailto:
> foundry-nsp-bounces [at] puck] On Behalf Of Drew Weaver
> Sent: Saturday, June 02, 2012 10:16 AM
> To: 'jeffm [at] iglou'; foundry-nsp [at] puck
> Subject: Re: [f-nsp] Asymmetrical routing on ADX
>
> That's exactly my stance on this issue.
>
> It's especially annoying considering that the two physical ports in
> question are attached to a VLAN/Virtual Ethernet, I don't understand why it
> matters which physical interface in the VLAN handles the traffic destined
> for the VE..
>
> To me that is really broken.
>
> Thanks,
> -Drew
>
>
> -----Original Message-----
> From: foundry-nsp-bounces [at] puck [mailto:
> foundry-nsp-bounces [at] puck] On Behalf Of jeffm [at] iglou
> Sent: Friday, June 01, 2012 3:08 PM
> To: foundry-nsp [at] puck
> Subject: Re: [f-nsp] Asymmetrical routing on ADX
>
> On Fri, June 1, 2012 12:25, Drew Weaver wrote:
> > Why is asymmetrical routing bad if you have a complete mesh?
>
> It isn't (even without a complete mesh).
>
> Asymmetric routing is commonplace and normal, if you need a chokepoint to
> apply some sort of network policy (load balancing, firewalling, etc.) you
> need to make sure that you make it a chokepoint both coming and going, and
> if gear can't handle the next-hop (or even the interface that it uses) may
> be different for transmitted and received traffic, then bug reports need to
> be filed.
>
>
> --
> Jeff
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp [at] puck
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp [at] puck
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp [at] puck
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>

drew.weaver at thenap

Jun 4, 2012, 5:18 PM

Post #9 of 9 (505 views)
Permalink

Re: Asymmetrical routing on ADX [In reply to]

Yes, on traffic leaving the load balancer, all of that traffic will go out via the current master in the VRRP group, but the responses from both the client and the remote server could come back via either physical port.

Thanks,
-Drew

From: Mike Allen [mailto:mkallen [at] gmail]
Sent: Sunday, June 03, 2012 12:38 PM
To: Drew Weaver
Cc: jeffm [at] iglou; foundry-nsp [at] puck
Subject: Re: [f-nsp] Asymmetrical routing on ADX

Drew, this is actually expected behavior, for pretty much every L3 Switch/Router I have seen. Arp resolves ip address to mac address, and mac address (normally) is associated with a single port (per vlan) at any one point in time. If you have Mac addresses moving back and forth, that is generally a bad thing in a switched environment. Traffic has the possibility of being black holed, as it will be forwarded only to the port where the destination mac address is at the instant the switch is ready to forward.

If you are looking to duplicate the traffic across multiple ports, you need to do a static arp entry, and possibly a static mac entry as well. This will duplicate all packets to that ip/mac combo, and send it to multiple ports. I have generally only seen this in clustering scenarios (like Microsoft Load Balancing), not in VRRP. In normal VRRP, there is still a singular owner of that VRRP VIP at any one point in time, one router that will answer Arp's, thus it should only be learned on one port, not both. If you are seeing it on both, it sounds like there may be a problem there.

WIthout seeing the configs, or more info, those are my thoughts, hope it helps.

Mike
On Sun, Jun 3, 2012 at 5:27 AM, Drew Weaver <drew.weaver [at] thenap<mailto:drew.weaver [at] thenap>> wrote:
I hate to reply to myself but I noticed that even though the switch is running router code and it's IP addresses are all handled via the VE when I do show ARP it is still seeing the arp on the physical ports rather than the VLAN.

This is pretty unfortunate.

Thanks,
-Drew

-----Original Message-----
From: foundry-nsp-bounces [at] puck<mailto:foundry-nsp-bounces [at] puck> [mailto:foundry-nsp-bounces [at] puck<mailto:foundry-nsp-bounces [at] puck>] On Behalf Of Drew Weaver
Sent: Saturday, June 02, 2012 10:16 AM
To: 'jeffm [at] iglou<mailto:jeffm [at] iglou>'; foundry-nsp [at] puck<mailto:foundry-nsp [at] puck>
Subject: Re: [f-nsp] Asymmetrical routing on ADX

That's exactly my stance on this issue.

It's especially annoying considering that the two physical ports in question are attached to a VLAN/Virtual Ethernet, I don't understand why it matters which physical interface in the VLAN handles the traffic destined for the VE..

To me that is really broken.

Thanks,
-Drew

-----Original Message-----
From: foundry-nsp-bounces [at] puck<mailto:foundry-nsp-bounces [at] puck> [mailto:foundry-nsp-bounces [at] puck<mailto:foundry-nsp-bounces [at] puck>] On Behalf Of jeffm [at] iglou<mailto:jeffm [at] iglou>
Sent: Friday, June 01, 2012 3:08 PM
To: foundry-nsp [at] puck<mailto:foundry-nsp [at] puck>
Subject: Re: [f-nsp] Asymmetrical routing on ADX

On Fri, June 1, 2012 12:25, Drew Weaver wrote:
> Why is asymmetrical routing bad if you have a complete mesh?

It isn't (even without a complete mesh).

Asymmetric routing is commonplace and normal, if you need a chokepoint to apply some sort of network policy (load balancing, firewalling, etc.) you need to make sure that you make it a chokepoint both coming and going, and if gear can't handle the next-hop (or even the interface that it uses) may be different for transmitted and received traffic, then bug reports need to be filed.

--
Jeff

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck<mailto:foundry-nsp [at] puck>
http://puck.nether.net/mailman/listinfo/foundry-nsp

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck<mailto:foundry-nsp [at] puck>
http://puck.nether.net/mailman/listinfo/foundry-nsp

_______________________________________________
foundry-nsp mailing list
foundry-nsp [at] puck<mailto:foundry-nsp [at] puck>
http://puck.nether.net/mailman/listinfo/foundry-nsp

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads