Mailing List Archive: Cisco: NSP

uRPF bug on C6k SXI1?

Index | Next | Previous | View Threaded

peter at rathlev

Nov 10, 2009, 4:22 AM

Post #1 of 4 (367 views)
Permalink

uRPF bug on C6k SXI1?

Hi,

I've discovered what seems to be a bug on C6k at least in SXI1. I
haven't been able to find anything about it in the bug toolkit. It might
be related to CSCsk65860 though.

If I configure a SVI in a VRF and add "ip verify source reachable-via
any" and afterwards enable "ip verify source reachable-via any
allow-default" the switch seems to drop a lot of traffic, something like
every 12th packet.

If I remove the "ip verify"-command and then add the version with
"allow-default" directly, I have no problems. Without uRPF there's no
problem either. Only when first entering the command without
"allow-default" and then adding "allow-default" does the problem appear.

Have anybody seen anything like this? Would anybody know how to debug
this?

When the problem appears, the "show ip interface VlanX" aren't showing
any uRPF drops:

R1#sh ip int vlan 901
Vlan901 is up, line protocol is up
[...]
IP verify source reachable-via ANY, allow default
0 verification drops
0 suppressed verification drops
IP multicast multilayer switching is disabled
R2#

Sending traffic out of this interface gives the errors:

R2#ping vrf RM03313 10.100.28.1 so 10.100.141.2 re 100

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.100.28.1, timeout is 2 seconds:
Packet sent with a source address of 10.100.141.2
!!!!!!!!!!!.!!!!!!!!!!!.!!!!!!!!!!!.!!!!!!!!!!!.!!!!!!!!!!!.!!!!!!!!!!
!!.!!!!!!!!!!!.!!!!!!!!!!!.!!!
Success rate is 92 percent (92/100), round-trip min/avg/max = 1/1/4 ms
R2#

When removing/re-adding the uRPF command the forwarding works fine:
R2#ping vrf RM03313 10.100.28.1 so 10.100.141.2 re 100

Type escape sequence to abort.
Sending 100, 100-byte ICMP Echos to 10.100.28.1, timeout is 2 seconds:
Packet sent with a source address of 10.100.141.2
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (100/100), round-trip min/avg/max = 1/1/4 ms
R2#

We're glad we found a fix, but maybe others have been pulling out hair
over this one. :-)

--
Peter

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

p.mayers at imperial

Nov 10, 2009, 5:23 AM

Post #2 of 4 (342 views)
Permalink

Re: uRPF bug on C6k SXI1? [In reply to]

Peter Rathlev wrote:
> Hi,
>
> I've discovered what seems to be a bug on C6k at least in SXI1. I
> haven't been able to find anything about it in the bug toolkit. It might
> be related to CSCsk65860 though.
>
> If I configure a SVI in a VRF and add "ip verify source reachable-via
> any" and afterwards enable "ip verify source reachable-via any
> allow-default" the switch seems to drop a lot of traffic, something like
> every 12th packet.

Do you have CoPP or MLS rate limiters? Is the traffic being CPU punted
(use a SPAN session to find out) and this rate-limiting what's causing
the drops?

If so, it could be a hardware/tcam programming error; we've seen a few
of these in obscure cases on SXI, and I've not found a reliable way to
clear them. Does a "shut" / "no shut" of the SVI fix the problem? Or the
various "clear" commands (e.g. "clear cef" etc.)

>
> If I remove the "ip verify"-command and then add the version with
> "allow-default" directly, I have no problems. Without uRPF there's no
> problem either. Only when first entering the command without
> "allow-default" and then adding "allow-default" does the problem appear.

We haven't seen that, but have seen other issues where (apparently) CEF
entries are programmed incorrectly resulting in traffic being CPU punted
and having to pass through CoPP, and thus being very lossy.

See e.g.

http://www.gossamer-threads.com/lists/cisco/nsp/112984
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

peter at rathlev

Nov 10, 2009, 1:54 PM

Post #3 of 4 (329 views)
Permalink

Re: uRPF bug on C6k SXI1? [In reply to]

Hi Phil,

Thanks for the input.

On Tue, 2009-11-10 at 13:23 +0000, Phil Mayers wrote:
> Do you have CoPP or MLS rate limiters? Is the traffic being CPU punted
> (use a SPAN session to find out) and this rate-limiting what's causing
> the drops?

No CoPP or rate-limiters configured, only defaults. Is there any way to
see counters for the rate-limiters? The "show

> If so, it could be a hardware/tcam programming error; we've seen a few
> of these in obscure cases on SXI, and I've not found a reliable way to
> clear them. Does a "shut" / "no shut" of the SVI fix the problem? Or
> the various "clear" commands (e.g. "clear cef" etc.)

Well, I tried shutting/unshutting the SVI, and now I can't seem to
recreate the problem. :-(

> > If I remove the "ip verify"-command and then add the version with
> > "allow-default" directly, I have no problems. Without uRPF there's
> > no problem either. Only when first entering the command without
> > "allow-default" and then adding "allow-default" does the problem
> > appear.
>
> We haven't seen that, but have seen other issues where (apparently)
> CEF entries are programmed incorrectly resulting in traffic being CPU
> punted and having to pass through CoPP, and thus being very lossy.

I would really like to have looked more into this, but with the problem
gone, I'm stuck: If it would happen again, is there any way to check
what the rate-limiters/CoPP drops via some counters?

--
Regards,
Peter

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

p.mayers at imperial

Nov 11, 2009, 8:31 AM

Post #4 of 4 (320 views)
Permalink

Re: uRPF bug on C6k SXI1? [In reply to]

Peter Rathlev wrote:
> Hi Phil,
>
> Thanks for the input.
>
> On Tue, 2009-11-10 at 13:23 +0000, Phil Mayers wrote:
>> Do you have CoPP or MLS rate limiters? Is the traffic being CPU punted
>> (use a SPAN session to find out) and this rate-limiting what's causing
>> the drops?
>
> No CoPP or rate-limiters configured, only defaults. Is there any way to
> see counters for the rate-limiters? The "show
>
>> If so, it could be a hardware/tcam programming error; we've seen a few
>> of these in obscure cases on SXI, and I've not found a reliable way to
>> clear them. Does a "shut" / "no shut" of the SVI fix the problem? Or
>> the various "clear" commands (e.g. "clear cef" etc.)
>
> Well, I tried shutting/unshutting the SVI, and now I can't seem to
> recreate the problem. :-(

Yep, that sounds familiar. We've seen the problem with dodgy CEF
prefixes "suddenly" go away when SVIs are shut/no shut. Someone
suggested the next-hop MTU getting set wrong in the hardware and causing
CPU punts, and that this can happen when SVIs come up/down very
occasionally :o(

>
>>> If I remove the "ip verify"-command and then add the version with
>>> "allow-default" directly, I have no problems. Without uRPF there's
>>> no problem either. Only when first entering the command without
>>> "allow-default" and then adding "allow-default" does the problem
>>> appear.
>> We haven't seen that, but have seen other issues where (apparently)
>> CEF entries are programmed incorrectly resulting in traffic being CPU
>> punted and having to pass through CoPP, and thus being very lossy.
>
> I would really like to have looked more into this, but with the problem
> gone, I'm stuck: If it would happen again, is there any way to check
> what the rate-limiters/CoPP drops via some counters?

Well, CoPP drop can be see with:

sh policy-map control-plane

...but if you haven't got it setup, you'll see nothing.

sh mls rate-limit

...shows the current config for MLS rate limiters, but again if you've
not got it setup then the defaults are some pretty conservative
multicast punts and nothing else IIRC.

Hmm.
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads