Mailing List Archive: nsp: ipv6

enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity.

Index | Next | Previous | View Threaded

jtrotz at gmail

May 31, 2012, 5:46 PM

Post #1 of 11 (1017 views)
Permalink

enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity.

We have a pair of Cat6500s with sup720XL supervisors running 12.2(33)SXJ1
IOS that are configured as enterprise Edge routers. These routers have a
VRF for Internet "outside" connections and the global table is used for
"Inside" connections. The outside VRF has full BGP routing tables with 406K
V4 routes.

Cisco ASAs are connected on ports associated each routing table. Named mode
EIGRP is used for IGP both inside and in the external VRF to communicate
with our 2 Internet Hub routers (also 6500s, dual stacked, working fine).

The problem we have is that when we try to configure IPV6 on the edge
routers we loose the ability to pass IPv4 traffic as soon as the "ipv6
unicast routing" command is entered - no IPV6 addresses have been applied
at that point, no interfaces enabled for IPv6 and no ipv6 routing
configured either.

Cisco TAC seems to be stumped. The logs don't show anything, Eigrp & BGP
neighbors stay up and the CPU rises to 99% utilization.

The odd thing is that when we apply the "ipv6 unicast routing" command to
the other router it works fine. The only thing we could think of thats
different between the two is the "broken" router's ASA firewall is in
active state and the "good" router's ASA is in standby. We plan to fail
them over to see if thats really the difference.

I have tried duplicating the problem in the lab without luck so far.

Has anyone had a similar problem or have any suggestions?

jon at bovre

May 31, 2012, 10:14 PM

Post #2 of 11 (965 views)
Permalink

Re: enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity. [In reply to]

Could it be anything related to 'mls cef maximum-routes'?

Jon

Sent from my iPad

On 1. juni 2012, at 02:46, Jim Trotz <jtrotz [at] gmail> wrote:

> We have a pair of Cat6500s with sup720XL supervisors running 12.2(33)SXJ1 IOS that are configured as enterprise Edge routers. These routers have a VRF for Internet "outside" connections and the global table is used for "Inside" connections. The outside VRF has full BGP routing tables with 406K V4 routes.
>
> Cisco ASAs are connected on ports associated each routing table. Named mode EIGRP is used for IGP both inside and in the external VRF to communicate with our 2 Internet Hub routers (also 6500s, dual stacked, working fine).
>
> The problem we have is that when we try to configure IPV6 on the edge routers we loose the ability to pass IPv4 traffic as soon as the "ipv6 unicast routing" command is entered - no IPV6 addresses have been applied at that point, no interfaces enabled for IPv6 and no ipv6 routing configured either.
>
> Cisco TAC seems to be stumped. The logs don't show anything, Eigrp & BGP neighbors stay up and the CPU rises to 99% utilization.
>
> The odd thing is that when we apply the "ipv6 unicast routing" command to the other router it works fine. The only thing we could think of thats different between the two is the "broken" router's ASA firewall is in active state and the "good" router's ASA is in standby. We plan to fail them over to see if thats really the difference.
>
> I have tried duplicating the problem in the lab without luck so far.
>
> Has anyone had a similar problem or have any suggestions?

mohacsi at niif

Jun 1, 2012, 12:57 AM

Post #3 of 11 (966 views)
Permalink

Re: enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity. [In reply to]

On Fri, 1 Jun 2012, Jon Harald B?vre wrote:

> Could it be anything related to 'mls cef maximum-routes'?

I suspect similar things. Probably TCAM space exhausted, or limited
someway, and non TCAM handled packets sent to RP....
Do you collect netflow on your router? Try disabling netflow collection.
Or look at mls counters if they above the configured limits....
Best Regards,
Janos Mohacsi

>
> Jon
>
>
> Sent from my iPad
>
> On 1. juni 2012, at 02:46, Jim Trotz <jtrotz [at] gmail> wrote:
>
>> We have a pair of Cat6500s with sup720XL supervisors running 12.2(33)SXJ1 IOS that are configured as enterprise Edge routers. These routers have a VRF for Internet "outside" connections and the global table is used for "Inside" connections. The outside VRF has full BGP routing tables with 406K V4 routes.
>>
>> Cisco ASAs are connected on ports associated each routing table. Named mode EIGRP is used for IGP both inside and in the external VRF to communicate with our 2 Internet Hub routers (also 6500s, dual stacked, working fine).
>>
>> The problem we have is that when we try to configure IPV6 on the edge routers we loose the ability to pass IPv4 traffic as soon as the "ipv6 unicast routing" command is entered - no IPV6 addresses have been applied at that point, no interfaces enabled for IPv6 and no ipv6 routing configured either.
>>
>> Cisco TAC seems to be stumped. The logs don't show anything, Eigrp & BGP neighbors stay up and the CPU rises to 99% utilization.
>>
>> The odd thing is that when we apply the "ipv6 unicast routing" command to the other router it works fine. The only thing we could think of thats different between the two is the "broken" router's ASA firewall is in active state and the "good" router's ASA is in standby. We plan to fail them over to see if thats really the difference.
>>
>> I have tried duplicating the problem in the lab without luck so far.
>>
>> Has anyone had a similar problem or have any suggestions?

>

evyncke at cisco

Jun 1, 2012, 3:41 AM

Post #4 of 11 (968 views)
Permalink

RE: enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity. [In reply to]

By any chance, can you check whether the 'broken' router receives a lot
of IPv6 traffic already? Or if there are some 'multicast' storm between
this 'broken' router and another device?

From: ipv6-ops-bounces+evyncke=cisco.com [at] lists
[mailto:ipv6-ops-bounces+evyncke=cisco.com [at] lists] On Behalf
Of Jim Trotz
Sent: vendredi 1 juin 2012 02:47
To: ipv6-ops [at] lists
Subject: enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity.

We have a pair of Cat6500s with sup720XL supervisors running
12.2(33)SXJ1 IOS that are configured as enterprise Edge routers. These
routers have a VRF for Internet "outside" connections and the global
table is used for "Inside" connections. The outside VRF has full BGP
routing tables with 406K V4 routes.

Cisco ASAs are connected on ports associated each routing table. Named
mode EIGRP is used for IGP both inside and in the external VRF to
communicate with our 2 Internet Hub routers (also 6500s, dual stacked,
working fine).

The problem we have is that when we try to configure IPV6 on the edge
routers we loose the ability to pass IPv4 traffic as soon as the "ipv6
unicast routing" command is entered - no IPV6 addresses have been
applied at that point, no interfaces enabled for IPv6 and no ipv6
routing configured either.

Cisco TAC seems to be stumped. The logs don't show anything, Eigrp & BGP
neighbors stay up and the CPU rises to 99% utilization.

The odd thing is that when we apply the "ipv6 unicast routing" command
to the other router it works fine. The only thing we could think of
thats different between the two is the "broken" router's ASA firewall is
in active state and the "good" router's ASA is in standby. We plan to
fail them over to see if thats really the difference.

I have tried duplicating the problem in the lab without luck so far.

Has anyone had a similar problem or have any suggestions?

jared at puck

Jun 1, 2012, 4:22 AM

Post #5 of 11 (968 views)
Permalink

Re: enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity. [In reply to]

Make sure ipv6 redirects are off on each interface. Cisco hides that default and can cause this trouble.

Saw a significant issue in the past related to this.

Jared Mauch

On Jun 1, 2012, at 6:41 AM, "Eric Vyncke (evyncke)" <evyncke [at] cisco> wrote:

> By any chance, can you check whether the ‘broken’ router receives a lot of IPv6 traffic already? Or if there are some ‘multicast’ storm between this ‘broken’ router and another device?
>
> From: ipv6-ops-bounces+evyncke=cisco.com [at] lists [mailto:ipv6-ops-bounces+evyncke=cisco.com [at] lists] On Behalf Of Jim Trotz
> Sent: vendredi 1 juin 2012 02:47
> To: ipv6-ops [at] lists
> Subject: enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity.
>
> We have a pair of Cat6500s with sup720XL supervisors running 12.2(33)SXJ1 IOS that are configured as enterprise Edge routers. These routers have a VRF for Internet "outside" connections and the global table is used for "Inside" connections. The outside VRF has full BGP routing tables with 406K V4 routes.
>
> Cisco ASAs are connected on ports associated each routing table. Named mode EIGRP is used for IGP both inside and in the external VRF to communicate with our 2 Internet Hub routers (also 6500s, dual stacked, working fine).
>
> The problem we have is that when we try to configure IPV6 on the edge routers we loose the ability to pass IPv4 traffic as soon as the "ipv6 unicast routing" command is entered - no IPV6 addresses have been applied at that point, no interfaces enabled for IPv6 and no ipv6 routing configured either.
>
> Cisco TAC seems to be stumped. The logs don't show anything, Eigrp & BGP neighbors stay up and the CPU rises to 99% utilization.
>
> The odd thing is that when we apply the "ipv6 unicast routing" command to the other router it works fine. The only thing we could think of thats different between the two is the "broken" router's ASA firewall is in active state and the "good" router's ASA is in standby. We plan to fail them over to see if thats really the difference.
>
> I have tried duplicating the problem in the lab without luck so far.
>
> Has anyone had a similar problem or have any suggestions?

tom at ninjabadger

Jun 3, 2012, 4:51 AM

Post #6 of 11 (966 views)
Permalink

Re: enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity. [In reply to]

On 01/06/12 08:57, Mohacsi Janos wrote:
> I suspect similar things. Probably TCAM space exhausted, or limited
> someway, and non TCAM handled packets sent to RP....
> Do you collect netflow on your router? Try disabling netflow collection.
> Or look at mls counters if they above the configured limits....
> Best Regards,
> Janos Mohacsi

This.

I've also heard some suggestions recently that storing the global table
in a VRF uses 2x the space than it would without. I cannot speak from
personal experience, but you may wish to spend some time researching
and/or experimenting without that 'feature', also.

But even then, it's only going to come down to how the TCAM space is
partitioned...

Tom

liviu.pislaru at rcs-rds

Jun 3, 2012, 5:18 AM

Post #7 of 11 (959 views)
Permalink

Re: enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity. [In reply to]

> I've also heard some suggestions recently that storing the global
> table in a VRF uses 2x the space than it would without. I cannot speak
> from personal experience, but you may wish to spend some time
> researching and/or experimenting without that 'feature', also.
>
> But even then, it's only going to come down to how the TCAM space is
> partitioned...
>
> Tom
>

this is true Tom.
you may check it with:
# sh mls cef summary

then compare with:
# sh mls cef maximum-routes

and maybe adjust with:
config)# mls cef maximum-routes ipv6 ...
config)# mls cef maximum-routes ip-multicast ...
etc.

if this is suitable for you (in case of vrf lite for example) you may
use only one label for VRF with this command:
config)# mpls label mode all-vrfs protocol bgp-vpnv4 per-vrf

reducing ~ 400.000 entries (MPLS routes) from TCAM (for every prefix in
BGP table).

--
liviu.

jtrotz at gmail

Jun 7, 2012, 3:44 PM

Post #8 of 11 (926 views)
Permalink

RE: enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity. [In reply to]

By way of an update on the problem:

We have been able to recreate the problem in a network lab environment.

The router stops forwarding IPv4 traffic (ICMP still flows) whenever two
IPv6 configuration statements are entered. Traffic resumes after some time
(2-10 minutes).

Used separately, everything works fine. The commands are: 1) "ipv6
unicast-routing" and 2) adding "ipv6 address-family" to a VRF definition.

There is nothing in the router log indicating a problem, EIGRP and BGP
neighbors still remain connected (if BFD is not enabled), however the BGP
neighbors do fall off if the traffic is blocked for more than a few
minutes. Telnet sessions to the router still work. In the "live" routers
the CPU goes very high - this seems BGP related, but goes away.

We tried different versions of code and applying the commands in in
different sequences - same result.

Cisco TAC still has no suggestions.

I tried all of the CLI commands list members suggested (thanks for the
feedback!). It doesn't appear to be related to CEF or TCAM, the XL versions
of the DFCs all have 1GB of ram. The TCAM is configured for 512K IPV4
routes and 256K IPV6 routes. We currently have 408K IPV4 routes in our
tables.

We are planning on replacing these routers at some point - any suggestions?
We are a semi-small service provider; servicing Internet and enterprise
traffic for several hospital and University systems in the
Maryland/Washington DC area. I was considering the Cisco ASR9000 line - we
have mostly 10Gbs links to ISPs and the metro Ethernet WAN. We need good
support for IPv6, MPLs and 100Gbs links - but not a super huge box like a
CRS, Juniper or Brocade?

lists at hojmark

Jun 7, 2012, 11:38 PM

Post #9 of 11 (922 views)
Permalink

Re: enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity. [In reply to]

On Thu, 7 Jun 2012 18:44:41 -0400, you wrote:

> The commands are: 1) "ipv6 unicast-routing" and 2) adding "ipv6
> address-family" to a VRF definition.

You do realize that you need to do 'mls ipv6 vrf' to route v6 in VRFs?

http://www.cisco.com/en/US/docs/ios/ipv6/command/reference/ipv6_09.html#wp2372609

-A

p.mayers at imperial

Jun 8, 2012, 1:57 AM

Post #10 of 11 (926 views)
Permalink

Re: enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity. [In reply to]

On 06/07/2012 11:44 PM, Jim Trotz wrote:
> By way of an update on the problem:
>
> We have been able to recreate the problem in a network lab environment.
>
> The router stops forwarding IPv4 traffic (ICMP still flows) whenever two
> IPv6 configuration statements are entered. Traffic resumes after some
> time (2-10 minutes).

This is all very odd.

This should not be happening on this platform, if what you've said is
true - the box should be capable of forwarding IPv6 in VRFs, and if
you're neither overflowing the TCAM of hammering the CPU with packets,
then it is hard to imagine what is causing it.

It is particularly bizarre that ICMP continues to work; the 6500 doesn't
forward ICMP any differently.

Have you:

1. Used SPAN to look at what it hitting the CPU during an outage, as
you enable it?

2. Used "sh proc cpu sorted 1m" to confirm where the CPU is being
spent during an outage?

3. Ensured you have not enabled IPv6 uRPF (which is done in software
on this platform) or any ACLs with logging statements?

4. Ensured you don't have a routing loop?

TAC should be able to help you find the problem here; "no suggestions"
is codeword for "I'm a dumb and lazy TAC engineer". Ask for your case to
be escalated; hassle your account manager with the SR number. You should
be aiming to get an engineer from backbone TAC on the phone with you,
and let him screen-control a shell onto the box whilst you reproduce the
outage, and then he should start poking at the innards of the box
(probably with ELAM or similar).

If you can share your config (sanitised) on- or off-list, I can take a
look; although we don't run full tables or EIGRP, we do run IPv6 in VRFs
on this platform with no problems.

Cheers,
Phil

jared at puck

Jun 23, 2012, 3:14 PM

Post #11 of 11 (787 views)
Permalink

Re: Enabling IPv6 on Cisco 6500 breaks IPv4 Internet connectivity. [In reply to]

Is this sup2t or 720?

Jared Mauch

On Jun 23, 2012, at 5:50 PM, Jim Trotz <jtrotz [at] gmail> wrote:

> Final update:
>
> After much testing in the lab and working with Cisco TAC (almost no help),
> I have reached a conclusion about the problem - its a hardware limitation.
>
> Enabling IPV6 routing on a 6500 (with XL cards) and a full Internet routing
> table in a VRF exceeds the limits of SP processing, The SP goes to 99%
> utilization reconfiguring something but eventually recovers. In the lab
> this took almost 5 minutes! In real life with many 10Gb interfaces active
> - who knows!!
>
> The problem is that the router still passes enough traffic that EIGRP and BGP stay
> up, but all user traffic is "black hole'd" due to the 1-10kbs effective throughput.
>
> It looks like this may be a one time event, but neither Cisco TAC or the BU
> could say for sure this wouldn't happen again under some kind of BGP flap
> of VRF reconfig.
>
> Our TCAM limit is 512K ipV4 routes now and we have 409K routes today.
>
> We will probably resort to filtering down the BGP learned routes to 100-200K and
> then default for everything else to our Internet routers and then go shopping
> for a new router.
>
> The problem isn't noticeable until we have more than about 250K routes.
>
> There was no interest in redesigning the network to not use VRFs for the
> Internet table.
>
> Once IPV6 is enabled and all is stable we will probably go shopping for new routers.
>
> Thanks again for everyone's suggestions, it helped us figure out the root
> cause.

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads