Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Cisco: NSP

6500 - What determines whether certain traffic is punted or not?

 

 

Cisco nsp RSS feed   Index | Next | Previous | View Threaded


drew.weaver at thenap

Nov 24, 2009, 8:33 AM

Post #1 of 9 (1452 views)
Permalink
6500 - What determines whether certain traffic is punted or not?

Howdy,

I've been having some issues with queue drops/CLI sluggishness on a 6500 and I wanted to check what kind of volume of traffic I was getting punted to the RP.

I made a span session and began checking out the traffic with tethereal.

It seems like a huge (30,000) or so packets every few seconds of just UDP traffic is being punted.

The system is a Sup720-3BXL.

Does anyone know how to determine what kind of traffic should be punted to the RP and more importantly why this UDP traffic is hitting the RP?

It almost looks like p2p traffic, but I also see other types of traffic, tcp 445, DNS, port 80, etc.

thanks,
-Drew

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


ler762 at gmail

Nov 24, 2009, 10:00 AM

Post #2 of 9 (1397 views)
Permalink
Re: 6500 - What determines whether certain traffic is punted or not? [In reply to]

Hi Drew,

On Tue, Nov 24, 2009 at 11:33 AM, Drew Weaver wrote:

> Howdy,
>
> I've been having some issues with queue drops/CLI sluggishness on a 6500
> and I wanted to check what kind of volume of traffic I was getting punted to
> the RP.
>
> I made a span session and began checking out the traffic with tethereal.
>

How did you make the span session? I think a regular span session gets you
everything - not just punted packets.

I haven't actually tried this, but here's the notes I have for setting up a
span session to see punted packets:
------------------

Here are the instructions to setup inband span (which monitors traffic sent
to the MSFC):

Router#monitor session 1 source interface fa 3/3 !--- Use any interface that
is administratively shut down.
Router#monitor session 1 destination interface fa 3/2

Now, go to the SP console. Here is an example:

Router#remote login switch
Router-sp#test monitor add 1 rp-inband rx <--- check the syntax as it varies
from one IOS to the next so use ?

Verify monitor session:

Router-sp#test monitor show session 1
Ingress Source Ports: 3/3 15/1
Egress Source Ports: 3/3
Ingress Source Vlans: <empty>
Egress Source Vlans: <empty>
Filter Vlans: <empty>
Destination Ports: 3/2

Go back to the RP and verify the monitor session as well:

Router#show monitor
Session 1
---------
Type : Local Session
Source Ports :
Both : Fa3/3
Destination Ports : Fa3/2
SP console:
Router-sp#test monitor session 1 show
Ingress Source Ports: 3/3 15/1
Egress Source Ports: 3/3
Ingress Source Vlans: <empty>
Egress Source Vlans: <empty>
Filter Vlans: <empty>
Destination Ports: 3/2

To remove the inband span from sp do
test monitor session 1 del
and from the rp do
no mon sess all
-------------------------------

Regards,
Lee



> It seems like a huge (30,000) or so packets every few seconds of just UDP
> traffic is being punted.
>
> The system is a Sup720-3BXL.
>
> Does anyone know how to determine what kind of traffic should be punted to
> the RP and more importantly why this UDP traffic is hitting the RP?
>
> It almost looks like p2p traffic, but I also see other types of traffic,
> tcp 445, DNS, port 80, etc.


> thanks,
> -Drew
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp [at] puck
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


drew.weaver at thenap

Nov 24, 2009, 10:03 AM

Post #3 of 9 (1407 views)
Permalink
Re: 6500 - What determines whether certain traffic is punted or not? [In reply to]

Hi,

Yeah I followed the exact same instructions you posted when creating the RP span session.

Source Port-VLAN Info
---------------------
Ingress Source Ports: 4/23 15/1
Egress Source Ports : 4/23
Ingress Source Vlans: <null>
Egress Source Vlans : <null>
Ingress Filter Vlans : <null>
Egress Filter Vlans : <null>
Exclude Filter Vlans : <empty>
Exclude Alt Filter Vlans : <empty>
Ingress Filter Vlan Count: 0
Egress Filter Vlan Count : 0
Exclude Filter Vlan Count: 0
Exclude Alt Vlan Count : 0

Destination ports: 4/24

Thanks,
-Drew

From: Lee [mailto:ler762 [at] gmail]
Sent: Tuesday, November 24, 2009 1:00 PM
To: Drew Weaver
Cc: Cisco-nsp
Subject: Re: [c-nsp] 6500 - What determines whether certain traffic is punted or not?

Hi Drew,
On Tue, Nov 24, 2009 at 11:33 AM, Drew Weaver wrote:
Howdy,

I've been having some issues with queue drops/CLI sluggishness on a 6500 and I wanted to check what kind of volume of traffic I was getting punted to the RP.

I made a span session and began checking out the traffic with tethereal.

How did you make the span session? I think a regular span session gets you everything - not just punted packets.

I haven't actually tried this, but here's the notes I have for setting up a span session to see punted packets:
------------------

Here are the instructions to setup inband span (which monitors traffic sent to the MSFC):

Router#monitor session 1 source interface fa 3/3 !--- Use any interface that is administratively shut down.
Router#monitor session 1 destination interface fa 3/2

Now, go to the SP console. Here is an example:

Router#remote login switch
Router-sp#test monitor add 1 rp-inband rx <--- check the syntax as it varies from one IOS to the next so use ?

Verify monitor session:

Router-sp#test monitor show session 1
Ingress Source Ports: 3/3 15/1
Egress Source Ports: 3/3
Ingress Source Vlans: <empty>
Egress Source Vlans: <empty>
Filter Vlans: <empty>
Destination Ports: 3/2

Go back to the RP and verify the monitor session as well:

Router#show monitor
Session 1
---------
Type : Local Session
Source Ports :
Both : Fa3/3
Destination Ports : Fa3/2
SP console:
Router-sp#test monitor session 1 show
Ingress Source Ports: 3/3 15/1
Egress Source Ports: 3/3
Ingress Source Vlans: <empty>
Egress Source Vlans: <empty>
Filter Vlans: <empty>
Destination Ports: 3/2

To remove the inband span from sp do
test monitor session 1 del
and from the rp do
no mon sess all
-------------------------------

Regards,
Lee


It seems like a huge (30,000) or so packets every few seconds of just UDP traffic is being punted.

The system is a Sup720-3BXL.

Does anyone know how to determine what kind of traffic should be punted to the RP and more importantly why this UDP traffic is hitting the RP?

It almost looks like p2p traffic, but I also see other types of traffic, tcp 445, DNS, port 80, etc.

thanks,
-Drew

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck<mailto:cisco-nsp [at] puck>
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


ler762 at gmail

Nov 24, 2009, 10:14 AM

Post #4 of 9 (1394 views)
Permalink
Re: 6500 - What determines whether certain traffic is punted or not? [In reply to]

On Tue, Nov 24, 2009 at 1:03 PM, Drew Weaver <drew.weaver [at] thenap> wrote:

> Hi,
>
>
>
> Yeah I followed the exact same instructions you posted when creating the RP
> span session.
>

Well.. it was worth a shot :)
Have you seen any syslog messages about a fib or tcam table overflow?
Someone else will have to chime in with the show commands to see hardware
resource utilization - I'm not at work & don't remember what they are.
Sorry..

Lee
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


drew.weaver at thenap

Nov 24, 2009, 10:20 AM

Post #5 of 9 (1394 views)
Permalink
Re: 6500 - What determines whether certain traffic is punted or not? [In reply to]

Hi Lee,

I believe you're referring to show 'platform hardware capacity' and nothing looks extremely out of the ordinary.

-Drew

From: Lee [mailto:ler762 [at] gmail]
Sent: Tuesday, November 24, 2009 1:14 PM
To: Drew Weaver
Cc: Cisco-nsp
Subject: Re: [c-nsp] 6500 - What determines whether certain traffic is punted or not?


On Tue, Nov 24, 2009 at 1:03 PM, Drew Weaver <drew.weaver [at] thenap<mailto:drew.weaver [at] thenap>> wrote:
Hi,

Yeah I followed the exact same instructions you posted when creating the RP span session.

Well.. it was worth a shot :)
Have you seen any syslog messages about a fib or tcam table overflow? Someone else will have to chime in with the show commands to see hardware resource utilization - I'm not at work & don't remember what they are. Sorry..

Lee


_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


mhuff at ox

Nov 24, 2009, 10:41 AM

Post #6 of 9 (1395 views)
Permalink
Re: 6500 - What determines whether certain traffic is punted or not? [In reply to]

Are you using first-hop redundancy like hsrp, glbp, vrrp? This can cause asymmetrical MAC based FIB timeouts which leads to unicast flooding. I didn't think these were RP switched, but it could be.

If so, what is your setting for "mac-address-table aging-time" ? We have ours set > fib timeout...so:


mac-address-table aging-time 14400


----
Matthew Huff       | One Manhattanville Rd
OTA Management LLC | Purchase, NY 10577
http://www.ox.com  | Phone: 914-460-4039
aim: matthewbhuff  | Fax:   914-460-4139


-----Original Message-----
From: cisco-nsp-bounces [at] puck [mailto:cisco-nsp-bounces [at] puck] On Behalf Of Drew Weaver
Sent: Tuesday, November 24, 2009 1:21 PM
To: 'Lee'
Cc: Cisco-nsp
Subject: Re: [c-nsp] 6500 - What determines whether certain traffic is punted or not?

Hi Lee,

I believe you're referring to show 'platform hardware capacity' and nothing looks extremely out of the ordinary.

-Drew

From: Lee [mailto:ler762 [at] gmail]
Sent: Tuesday, November 24, 2009 1:14 PM
To: Drew Weaver
Cc: Cisco-nsp
Subject: Re: [c-nsp] 6500 - What determines whether certain traffic is punted or not?


On Tue, Nov 24, 2009 at 1:03 PM, Drew Weaver <drew.weaver [at] thenap<mailto:drew.weaver [at] thenap>> wrote:
Hi,

Yeah I followed the exact same instructions you posted when creating the RP span session.

Well.. it was worth a shot :)
Have you seen any syslog messages about a fib or tcam table overflow? Someone else will have to chime in with the show commands to see hardware resource utilization - I'm not at work & don't remember what they are. Sorry..

Lee


_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


drew.weaver at thenap

Nov 24, 2009, 10:44 AM

Post #7 of 9 (1398 views)
Permalink
Re: 6500 - What determines whether certain traffic is punted or not? [In reply to]

Hi,

No HSRP, VRRP or GLBP on this box.

#sh mac-address-table aging-time
Vlan Aging Time
---- ----------
Global 300
no vlan age other than global age configured

Routed MAC aging time: 300 seconds

This is on our core, though so there are no hosts connected here.

-Drew


-----Original Message-----
From: Matthew Huff [mailto:mhuff [at] ox]
Sent: Tuesday, November 24, 2009 1:41 PM
To: Drew Weaver; 'Lee'
Cc: Cisco-nsp
Subject: RE: [c-nsp] 6500 - What determines whether certain traffic is punted or not?

Are you using first-hop redundancy like hsrp, glbp, vrrp? This can cause asymmetrical MAC based FIB timeouts which leads to unicast flooding. I didn't think these were RP switched, but it could be.

If so, what is your setting for "mac-address-table aging-time" ? We have ours set > fib timeout...so:


mac-address-table aging-time 14400


----
Matthew Huff       | One Manhattanville Rd
OTA Management LLC | Purchase, NY 10577
http://www.ox.com  | Phone: 914-460-4039
aim: matthewbhuff  | Fax:   914-460-4139


-----Original Message-----
From: cisco-nsp-bounces [at] puck [mailto:cisco-nsp-bounces [at] puck] On Behalf Of Drew Weaver
Sent: Tuesday, November 24, 2009 1:21 PM
To: 'Lee'
Cc: Cisco-nsp
Subject: Re: [c-nsp] 6500 - What determines whether certain traffic is punted or not?

Hi Lee,

I believe you're referring to show 'platform hardware capacity' and nothing looks extremely out of the ordinary.

-Drew

From: Lee [mailto:ler762 [at] gmail]
Sent: Tuesday, November 24, 2009 1:14 PM
To: Drew Weaver
Cc: Cisco-nsp
Subject: Re: [c-nsp] 6500 - What determines whether certain traffic is punted or not?


On Tue, Nov 24, 2009 at 1:03 PM, Drew Weaver <drew.weaver [at] thenap<mailto:drew.weaver [at] thenap>> wrote:
Hi,

Yeah I followed the exact same instructions you posted when creating the RP span session.

Well.. it was worth a shot :)
Have you seen any syslog messages about a fib or tcam table overflow? Someone else will have to chime in with the show commands to see hardware resource utilization - I'm not at work & don't remember what they are. Sorry..

Lee


_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


justin at justinshore

Nov 24, 2009, 11:19 AM

Post #8 of 9 (1395 views)
Permalink
Re: 6500 - What determines whether certain traffic is punted or not? [In reply to]

Drew Weaver wrote:
> Hi,
>
> No HSRP, VRRP or GLBP on this box.
>
> #sh mac-address-table aging-time
> Vlan Aging Time
> ---- ----------
> Global 300
> no vlan age other than global age configured
>
> Routed MAC aging time: 300 seconds
>
> This is on our core, though so there are no hosts connected here.

Well, I guess the next step would be to identify the ingress and egress
interfaces that for these example packets and dive into the interface
config to see if something on the interface is causing the punting. Can
you sanitize it and post it? I once saw a situation with netflow on an
interface causing all packets ingressing or egressing that interface to
get punted. Something in NF got screwed up. Removing it and reapplying
it to the interface fixed the problem.

Sometimes things just break.(tm)

Justin


_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


gert at greenie

Nov 24, 2009, 12:50 PM

Post #9 of 9 (1386 views)
Permalink
Re: 6500 - What determines whether certain traffic is punted or not? [In reply to]

Hi,

On Tue, Nov 24, 2009 at 01:14:18PM -0500, Lee wrote:
> Well.. it was worth a shot :)
> Have you seen any syslog messages about a fib or tcam table overflow?
> Someone else will have to chime in with the show commands to see hardware
> resource utilization - I'm not at work & don't remember what they are.
> Sorry..

"show mls cef exception status"

If you see anything shown as "TRUE" there, reload... (no way to recover).

gert
--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert [at] greenie
fax: +49-89-35655025 gert [at] net

Cisco nsp RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.