Mailing List Archive: Cisco: NSP

debugging stack corruption

Index | Next | Previous | View Threaded

billf at mu

Aug 18, 2008, 1:10 PM

Post #1 of 5 (597 views)
Permalink

debugging stack corruption

anyone see anything like this. i assume only a reload will fix this:

rtr1#sh proc cpu | e 0.0
CPU utilization for five seconds: 33%/8%; one minute: 37%; five minutes:
35%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
3 528125122320274973 22 23.35% 20.79% 20.97% 0 Exec
70 3616544001417549298 255 0.15% 0.11% 0.12% 0 IP Input
115 4851843096833738 0 0.15% 0.14% 0.15% 0 HQF Shaper Backg
rtr1#

nobody else is logged on, little to no amount of traffic is running
through the aux/cons ports, but this is interesting:

rtr1#show stacks
Minimum process stacks:
Free/Size Name
5676/6000 CDP BLOB
8640/9000 EM ED RF
11052/12000 Router Init
8676/9000 cdp init process
8348/12000 Init
5304/6000 RADIUS INITCONFIG
3616/6000 BGP Open
2264/3000 Rom Random Update Process
5616/6000 URPF stats
5316/6000 BGP Accepter
9248/12000 Exec
7176/12000 SSH Process
4264/6000 TFTP Read Process
4204/6000 MSDP Open
34540/36000 TCP Command
5236/7200 TTY Daemon
8496/9000 IP-EIGRP Router
3360/6000
d^\ytd^[^P^Ld^\zTd^[`Dd^[I$d^\^[Td^[T^Dd^\y^Dd^\^P<d^[@Td^[p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[e\d^[NTd^[)$d^[pLd^[|^\d^\
,d^[mdd^\^Nld^\
dd^[ 4d^[Q<d^[^V^\d^\1dd^[1<d^[O4d^[|Dd^\^P<d^[^Ydd^[e\d^[)$d^[NTd^[
4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
,d^[^R^Dd^[^Fld^[\d^[b^Td^[^LDd^\^P^Dd^[^B4d^[^NLd^[^Y,d^[^Kdd^\
^\d^\^CDd^[s^Td^[^A^\d^[U,d^[j,d^[~^Dd^\^QDd^[Jtd^[~Ld^[|^Td^[,Dd^^\d^[rld^[R|d^[{Dd^[
\d^[^Add^[^Q\d^[^QLd^[
ld^[ttd^[zdd^\,$d^[^Vdd^[)4d^\34d^[wLd^[m,d^[^Z|d^[\,d^[g|d^[y|d^[^D<d^[x$d^[^^Dd^[
ld^[^Bld^[RDd^[ptd^[^Q$d^[v4d^\^Ptd^[^VTd^[7$d^\1td^[P$d^[uTd^[^VTd^[zdd^[7$d^[z,d^[z^\d^[ytd^[@Td^[<^Dd^\,$d^\+Dd^\,4d^[^D<d^\`^Dd^[^VTd^[k4d^[P^Td^[a$d^[$d^[^V^\d^[^Utd^[mdd^[^Ytd^[|^Ld^[^L^Ld^\^ALd^[#^Dd^[e\d^[f^Dd^\^FTd^[^Pld^[^B|d^[n^\d^[d4d^[H|d^[^Rtd^[^N^Td^Td^[^Td^[{,d^[+dd^[`Td^[.^Dd^[s\d^[^ETd^[^Z^Ld^[
$d^[YTd^\^L^Dd^[1^Dd^[^O^\d^[^PDd^[^L^\d^\
dd^[
Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[Q<d^[U^\d^[~td^[l$d^[8<d^[6^Ld^[^F4d^[^Odd^\^O$d^[^K<d^[^N<d^[^K^Dd^[^W4d^[_,d^[p^Dd^[+^\d^[N,d^[$Td^[~^\d^[eLd^[NTd^[
,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
<d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[.w^Ld^[.bLd^[.L\d^[.
Dd^[.dld^[..<d^[.Lld^\ td^\4d^\ld^^Td^\d^\ <d^\
^Dd^Ld^$d^[,d^[dd^[^\d^[Td^\
6856/9000
d^\^[Td^[T^Dd^\y^Dd^\^P<d^[@Td^[p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[e\d^[NTd^[)$d^[pLd^[|^\d^\
,d^[mdd^\^Nld^\
dd^[ 4d^[Q<d^[^V^\d^\1dd^[1<d^[O4d^[|Dd^\^P<d^[^Ydd^[e\d^[)$d^[NTd^[
4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
Minimum process stacks:
Free/Size Name
,d^[^R^Dd^[^Fld^[\d^[b^Td^[^LDd^\^P^Dd^[^B4d^[^NLd^[^Y,d^[^Kdd^\
^\d^\^CDd^[s^Td^[^A^\d^[U,d^[j,d^[~^Dd^\^QDd^[Jtd^[~Ld^[|^Td^[,Dd^^\d^[rld^[R|d^[{Dd^[
\d^[^Add^[^Q\d^[^QLd^[
ld^[ttd^[zdd^\,$d^[^Vdd^[)4d^\34d^[wLd^[m,d^[^Z|d^[\,d^[g|d^[y|d^[^D<d^[x$d^[^^Dd^[
ld^[^Bld^[RDd^[ptd^[^Q$d^[v4d^\^Ptd^[^VTd^[7$d^\1td^[P$d^[uTd^[^VTd^[zdd^[7$d^[z,d^[z^\d^[ytd^[@Td^[<^Dd^\,$d^\+Dd^\,4d^[^D<d^\`^Dd^[^VTd^[k4d^[P^Td^[a$d^[$d^[^V^\d^[^Utd^[mdd^[^Ytd^[|^Ld^[^L^Ld^\^ALd^[#^Dd^[e\d^[f^Dd^\^FTd^[^Pld^[^B|d^[n^\d^[d4d^[H|d^[^Rtd^[^N^Td^Td^[^Td^[{,d^[+dd^[`Td^[.^Dd^[s\d^[^ETd^[^Z^Ld^[
$d^[YTd^\^L^Dd^[1^Dd^[^O^\d^[^PDd^[^L^\d^\
dd^[
Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[Q<d^[U^\d^[~td^[l$d^[8<d^[6^Ld^[^F4d^[^Odd^\^O$d^[^K<d^[^N<d^[^K^Dd^[^W4d^[_,d^[p^Dd^[+^\d^[N,d^[$Td^[~^\d^[eLd^[NTd^[
,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
<d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[.w^Ld^[.bLd^[.L\d^[.
Dd^[.dld^[..<d^[.Lld^\ td^\4d^\ld^^Td^\d^\ <d^\
^Dd^Ld^$d^[.,d^[.dd^[.^\d^[.Td^\
10468/12000 HSRP (Standby)

Interrupt level stacks:
Level Called Unused/Size Name
1 2648551315 6280/9000 Network interfaces
2 0 9000/9000 DMA/Timer Interrupt
3 185107 7472/9000 PA Management Int Handler
4 1715750501 8444/9000 Console Uart
5 0 9000/9000 OIR/Error Interrupt
7 3207930022 8532/9000 NMI Interrupt Handler

Spurious interrupts: 233
rtr1#

and on a different router:

rtr1.chi#sh stacks
Minimum process stacks:
Free/Size Name
[....]
3500/6000
7160/9000 5,<$/jDSw_h 5,< 5,< 5,< 5,< 5,< d(X d(X 5,< 5,< 5,< 5,<
5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< d'X 5,<
5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
5,< 5,<V7;@ 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^BT 5,<
5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
5,< 5,< 5,< 5,< 5,< 5,<Q6mT 5,< 5,< 5,< 5,< 5,< 5,< 5,<
5,<#^Qz|#^Qy|#^Qy| 5,<#^Qx|#^Qx| 5,<%Dtx%Dtx%Dtx%Dtx%Dsx%Dsx%Dsx%Dsx 5,<
5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
5,< 5,<%Dsx 5,< 5,< 5,<%Drx 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
5,<#^Qw|#^Qw|#^Qv| 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
5,<S3^R@ 5,< 5,< 5,<Vo^BT d'X d&X d&X d%X 5,< 5,< 5,<&S<<!A`t 5,< 5,<
5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<#W:x#W9x
5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^AT-4,4 5,<PN^P^T 5,<
5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
5,< 5,<Vo^ATVo
5316/6000 BGP Accepter
10176/12000 Exec

although that router doesn't display the same CPU symptoms.

first router is running:
Cisco IOS Software, 7301 Software (C7301-K91P-M), Version 12.2(31)SB11, RELEASE SOFTWARE (fc3)
ROM: System Bootstrap, Version 12.3(4r)T4, RELEASE SOFTWARE (fc1)
BOOTLDR: 7301 Software (C7301-BOOT-M), Version 12.3(26), RELEASE SOFTWARE (fc2)

second router is running:
Cisco IOS Software, 7301 Software (C7301-K91P-M), Version 12.2(31)SB12,
RELEASE SOFTWARE (fc3)

-- bill

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

gtb at slac

Aug 18, 2008, 3:51 PM

Post #2 of 5 (571 views)
Permalink

Re: debugging stack corruption [In reply to]

> anyone see anything like this. i assume only a reload will fix this:

Nothing exactly like this, but I have a number of
crash files from SB11/12 on a 7200 with memory
corruption (Block overrun/redzone corruption).
Unfortunately the 7200 (a non-VXR) cannot be on
maintenance (EOS/EOL), so I cannot open a TAC case
(and no existing bugid seemed relevant).

That is what I would recommend to you (open a
TAC case).

Gary
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

rodunn at cisco

Aug 19, 2008, 7:41 AM

Post #3 of 5 (539 views)
Permalink

Re: debugging stack corruption [In reply to]

How are you getting this output?

If you ssh/telnet to it and run the command do you get th esame output?

That's not stack corruption to me.

Rodney

On Mon, Aug 18, 2008 at 01:10:44PM -0700, bill fumerola wrote:
>
> anyone see anything like this. i assume only a reload will fix this:
>
> rtr1#sh proc cpu | e 0.0
> CPU utilization for five seconds: 33%/8%; one minute: 37%; five minutes:
> 35%
> PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
> 3 528125122320274973 22 23.35% 20.79% 20.97% 0 Exec
> 70 3616544001417549298 255 0.15% 0.11% 0.12% 0 IP Input
> 115 4851843096833738 0 0.15% 0.14% 0.15% 0 HQF Shaper Backg
> rtr1#
>
> nobody else is logged on, little to no amount of traffic is running
> through the aux/cons ports, but this is interesting:
>
> rtr1#show stacks
> Minimum process stacks:
> Free/Size Name
> 5676/6000 CDP BLOB
> 8640/9000 EM ED RF
> 11052/12000 Router Init
> 8676/9000 cdp init process
> 8348/12000 Init
> 5304/6000 RADIUS INITCONFIG
> 3616/6000 BGP Open
> 2264/3000 Rom Random Update Process
> 5616/6000 URPF stats
> 5316/6000 BGP Accepter
> 9248/12000 Exec
> 7176/12000 SSH Process
> 4264/6000 TFTP Read Process
> 4204/6000 MSDP Open
> 34540/36000 TCP Command
> 5236/7200 TTY Daemon
> 8496/9000 IP-EIGRP Router
> 3360/6000
> d^\ytd^[.^P^Ld^\zTd^[.`Dd^[.I$d^\^[.Td^[.T^Dd^\y^Dd^\^P<d^[.@Td^[.p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[.e\d^[.NTd^[.)$d^[.pLd^[.|^\d^\
> ,d^[mdd^\^Nld^\
> dd^[ 4d^[Q<d^[^V^\d^\1dd^[1<d^[O4d^[|Dd^\^P<d^[^Ydd^[e\d^[)$d^[NTd^[
> 4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
> ,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
> ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
> ,d^[.^R^Dd^[.^Fld^[.\d^[.b^Td^[.^LDd^\^P^Dd^[.^B4d^[.^NLd^[.^Y,d^[.^Kdd^\
> ^\d^\^CDd^[.s^Td^[.^A^\d^[.U,d^[.j,d^[.~^Dd^\^QDd^[.Jtd^[.~Ld^[.|^Td^[.,Dd^^\d^[.rld^[.R|d^[.{Dd^[.
> \d^[.^Add^[.^Q\d^[.^QLd^[.
> ld^[.ttd^[.zdd^\,$d^[.^Vdd^[.)4d^\34d^[.wLd^[.m,d^[.^Z|d^[.\,d^[.g|d^[.y|d^[.^D<d^[.x$d^[.^^Dd^[.
> ld^[.^Bld^[.RDd^[.ptd^[.^Q$d^[.v4d^\^Ptd^[.^VTd^[.7$d^\1td^[.P$d^[.uTd^[.^VTd^[.zdd^[.7$d^[.z,d^[.z^\d^[.ytd^[.@Td^[.<^Dd^\,$d^\+Dd^\,4d^[.^D<d^\`^Dd^[.^VTd^[.k4d^[.P^Td^[.a$d^[.$d^[.^V^\d^[.^Utd^[.mdd^[.^Ytd^[.|^Ld^[.^L^Ld^\^ALd^[.#^Dd^[.e\d^[.f^Dd^\^FTd^[.^Pld^[.^B|d^[.n^\d^[.d4d^[.H|d^[.^Rtd^[.^N^Td^Td^[.^Td^[.{,d^[.+dd^[.`Td^[..^Dd^[.s\d^[.^ETd^[.^Z^Ld^[.
> $d^[YTd^\^L^Dd^[1^Dd^[^O^\d^[^PDd^[^L^\d^\
> dd^[
> Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
> 4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
> 4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[Q<d^[U^\d^[~td^[l$d^[8<d^[6^Ld^[^F4d^[^Odd^\^O$d^[^K<d^[^N<d^[^K^Dd^[^W4d^[_,d^[p^Dd^[+^\d^[N,d^[$Td^[~^\d^[eLd^[NTd^[
> ,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
> ,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
> <d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[.w^Ld^[.bLd^[.L\d^[.
> Dd^[.dld^[..<d^[.Lld^\ td^\4d^\ld^^Td^\d^\ <d^\
> ^Dd^Ld^$d^[.,d^[.dd^[.^\d^[.Td^\
> 6856/9000
> d^\^[.Td^[.T^Dd^\y^Dd^\^P<d^[.@Td^[.p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[.e\d^[.NTd^[.)$d^[.pLd^[.|^\d^\
> ,d^[mdd^\^Nld^\
> dd^[ 4d^[Q<d^[^V^\d^\1dd^[1<d^[O4d^[|Dd^\^P<d^[^Ydd^[e\d^[)$d^[NTd^[
> 4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
> ,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
> ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
> Minimum process stacks:
> Free/Size Name
> ,d^[.^R^Dd^[.^Fld^[.\d^[.b^Td^[.^LDd^\^P^Dd^[.^B4d^[.^NLd^[.^Y,d^[.^Kdd^\
> ^\d^\^CDd^[.s^Td^[.^A^\d^[.U,d^[.j,d^[.~^Dd^\^QDd^[.Jtd^[.~Ld^[.|^Td^[.,Dd^^\d^[.rld^[.R|d^[.{Dd^[.
> \d^[.^Add^[.^Q\d^[.^QLd^[.
> ld^[.ttd^[.zdd^\,$d^[.^Vdd^[.)4d^\34d^[.wLd^[.m,d^[.^Z|d^[.\,d^[.g|d^[.y|d^[.^D<d^[.x$d^[.^^Dd^[.
> ld^[.^Bld^[.RDd^[.ptd^[.^Q$d^[.v4d^\^Ptd^[.^VTd^[.7$d^\1td^[.P$d^[.uTd^[.^VTd^[.zdd^[.7$d^[.z,d^[.z^\d^[.ytd^[.@Td^[.<^Dd^\,$d^\+Dd^\,4d^[.^D<d^\`^Dd^[.^VTd^[.k4d^[.P^Td^[.a$d^[.$d^[.^V^\d^[.^Utd^[.mdd^[.^Ytd^[.|^Ld^[.^L^Ld^\^ALd^[.#^Dd^[.e\d^[.f^Dd^\^FTd^[.^Pld^[.^B|d^[.n^\d^[.d4d^[.H|d^[.^Rtd^[.^N^Td^Td^[.^Td^[.{,d^[.+dd^[.`Td^[..^Dd^[.s\d^[.^ETd^[.^Z^Ld^[.
> $d^[YTd^\^L^Dd^[1^Dd^[^O^\d^[^PDd^[^L^\d^\
> dd^[
> Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
> 4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
> 4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[Q<d^[U^\d^[~td^[l$d^[8<d^[6^Ld^[^F4d^[^Odd^\^O$d^[^K<d^[^N<d^[^K^Dd^[^W4d^[_,d^[p^Dd^[+^\d^[N,d^[$Td^[~^\d^[eLd^[NTd^[
> ,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
> ,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
> <d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[.w^Ld^[.bLd^[.L\d^[.
> Dd^[.dld^[..<d^[.Lld^\ td^\4d^\ld^^Td^\d^\ <d^\
> ^Dd^Ld^$d^[.,d^[.dd^[.^\d^[.Td^\
> 10468/12000 HSRP (Standby)
>
> Interrupt level stacks:
> Level Called Unused/Size Name
> 1 2648551315 6280/9000 Network interfaces
> 2 0 9000/9000 DMA/Timer Interrupt
> 3 185107 7472/9000 PA Management Int Handler
> 4 1715750501 8444/9000 Console Uart
> 5 0 9000/9000 OIR/Error Interrupt
> 7 3207930022 8532/9000 NMI Interrupt Handler
>
> Spurious interrupts: 233
> rtr1#
>
> and on a different router:
>
> rtr1.chi#sh stacks
> Minimum process stacks:
> Free/Size Name
> [....]
> 3500/6000
> 7160/9000 5,<$/jDSw_h 5,< 5,< 5,< 5,< 5,< d(X d(X 5,< 5,< 5,< 5,<
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< d'X 5,<
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,< 5,<V7;@ 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^BT 5,<
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,< 5,< 5,< 5,< 5,< 5,<Q6mT 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,<#^Qz|#^Qy|#^Qy| 5,<#^Qx|#^Qx| 5,<%Dtx%Dtx%Dtx%Dtx%Dsx%Dsx%Dsx%Dsx 5,<
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,< 5,<%Dsx 5,< 5,< 5,<%Drx 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,<#^Qw|#^Qw|#^Qv| 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,<S3^R@ 5,< 5,< 5,<Vo^BT d'X d&X d&X d%X 5,< 5,< 5,<&S<<!A`t 5,< 5,<
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<#W:x#W9x
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^AT-4,4 5,<PN^P^T 5,<
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,< 5,<Vo^ATVo
> 5316/6000 BGP Accepter
> 10176/12000 Exec
>
> although that router doesn't display the same CPU symptoms.
>
> first router is running:
> Cisco IOS Software, 7301 Software (C7301-K91P-M), Version 12.2(31)SB11, RELEASE SOFTWARE (fc3)
> ROM: System Bootstrap, Version 12.3(4r)T4, RELEASE SOFTWARE (fc1)
> BOOTLDR: 7301 Software (C7301-BOOT-M), Version 12.3(26), RELEASE SOFTWARE (fc2)
>
> second router is running:
> Cisco IOS Software, 7301 Software (C7301-K91P-M), Version 12.2(31)SB12,
> RELEASE SOFTWARE (fc3)
>
>
> -- bill
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp [at] puck
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

billf at mu

Aug 19, 2008, 11:49 AM

Post #4 of 5 (545 views)
Permalink

Re: debugging stack corruption [In reply to]

On Tue, Aug 19, 2008 at 10:41:05AM -0400, Rodney Dunn wrote:
> How are you getting this output?

ssh rtr1
en
sh stacks

> If you ssh/telnet to it and run the command do you get th esame output?

it is not signal noise (serial spew, ip corruption, etc).

> That's not stack corruption to me.

i'll try and profile the exec process, but i'm not so good w/ profiling
and tracing w/o at least symbols.

there is also the matter of the 30% solid EXEC process. however, the
switch that device is attached to (both in network and by serial via
rtr1:aux<>sw1:cons) is exhibiting the same behavior. it could be a feedback
loop on the serial connection, but i've tried turning all of that down
and still no relief. the jump occurred to both at the same time.

it could just be corruption in the display, but the CPU spike is what
made me investigate in the first place.

-- bill

> > rtr1#show stacks
> > Minimum process stacks:
> > Free/Size Name
[...]
> > 3360/6000
> > d^\ytd^[.^P^Ld^\zTd^[.`Dd^[.I$d^\^[.Td^[.T^Dd^\y^Dd^\^P<d^[.@Td^[.p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[.e\d^[.NTd^[.)$d^[.pLd^[.|^\d^\
> > ,d^[.mdd^\^Nld^\
> > dd^[. 4d^[Q<d^[^V^\d^\1dd^[1<d^[O4d^[|Dd^\^P<d^[^Ydd^[e\d^[)$d^[NTd^[
> > 4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
> > ,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
> > ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
> > ,d^[.^R^Dd^[.^Fld^[.\d^[.b^Td^[.^LDd^\^P^Dd^[.^B4d^[.^NLd^[.^Y,d^[.^Kdd^\
> > ^\d^\^CDd^[.s^Td^[.^A^\d^[.U,d^[.j,d^[.~^Dd^\^QDd^[.Jtd^[.~Ld^[.|^Td^[.,Dd^^\d^[.rld^[.R|d^[.{Dd^[.
> > \d^[.^Add^[.^Q\d^[.^QLd^[.
> > ld^[.ttd^[.zdd^\,$d^[.^Vdd^[.)4d^\34d^[.wLd^[.m,d^[.^Z|d^[.\,d^[.g|d^[.y|d^[.^D<d^[.x$d^[.^^Dd^[.
> > ld^[.^Bld^[.RDd^[.ptd^[.^Q$d^[.v4d^\^Ptd^[.^VTd^[.7$d^\1td^[.P$d^[.uTd^[.^VTd^[.zdd^[.7$d^[.z,d^[.z^\d^[.ytd^[.@Td^[.<^Dd^\,$d^\+Dd^\,4d^[.^D<d^\`^Dd^[.^VTd^[.k4d^[.P^Td^[.a$d^[.$d^[.^V^\d^[.^Utd^[.mdd^[.^Ytd^[.|^Ld^[.^L^Ld^\^ALd^[.#^Dd^[.e\d^[.f^Dd^\^FTd^[.^Pld^[.^B|d^[.n^\d^[.d4d^[.H|d^[.^Rtd^[.^N^Td^Td^[.^Td^[.{,d^[.+dd^[.`Td^[..^Dd^[.s\d^[.^ETd^[.^Z^Ld^[.
> > $d^[.YTd^\^L^Dd^[.1^Dd^[.^O^\d^[.^PDd^[.^L^\d^\
> > dd^[
> > Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
> > 4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
> > 4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[Q<d^[U^\d^[~td^[l$d^[8<d^[6^Ld^[^F4d^[^Odd^\^O$d^[^K<d^[^N<d^[^K^Dd^[^W4d^[_,d^[p^Dd^[+^\d^[N,d^[$Td^[~^\d^[eLd^[NTd^[
> > ,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
> > ,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
> > <d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[.w^Ld^[.bLd^[.L\d^[.
> > Dd^[.dld^[..<d^[.Lld^\ td^\4d^\ld^^Td^\d^\ <d^\
> > ^Dd^Ld^$d^[.,d^[.dd^[.^\d^[.Td^\
> > 6856/9000
> > d^\^[.Td^[.T^Dd^\y^Dd^\^P<d^[.@Td^[.p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[.e\d^[.NTd^[.)$d^[.pLd^[.|^\d^\
> > ,d^[.mdd^\^Nld^\
> > dd^[. 4d^[Q<d^[^V^\d^\1dd^[1<d^[O4d^[|Dd^\^P<d^[^Ydd^[e\d^[)$d^[NTd^[
> > 4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
> > ,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
> > ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
> > Minimum process stacks:
> > Free/Size Name
> > ,d^[.^R^Dd^[.^Fld^[.\d^[.b^Td^[.^LDd^\^P^Dd^[.^B4d^[.^NLd^[.^Y,d^[.^Kdd^\
> > ^\d^\^CDd^[.s^Td^[.^A^\d^[.U,d^[.j,d^[.~^Dd^\^QDd^[.Jtd^[.~Ld^[.|^Td^[.,Dd^^\d^[.rld^[.R|d^[.{Dd^[.
> > \d^[.^Add^[.^Q\d^[.^QLd^[.
> > ld^[.ttd^[.zdd^\,$d^[.^Vdd^[.)4d^\34d^[.wLd^[.m,d^[.^Z|d^[.\,d^[.g|d^[.y|d^[.^D<d^[.x$d^[.^^Dd^[.
> > ld^[.^Bld^[.RDd^[.ptd^[.^Q$d^[.v4d^\^Ptd^[.^VTd^[.7$d^\1td^[.P$d^[.uTd^[.^VTd^[.zdd^[.7$d^[.z,d^[.z^\d^[.ytd^[.@Td^[.<^Dd^\,$d^\+Dd^\,4d^[.^D<d^\`^Dd^[.^VTd^[.k4d^[.P^Td^[.a$d^[.$d^[.^V^\d^[.^Utd^[.mdd^[.^Ytd^[.|^Ld^[.^L^Ld^\^ALd^[.#^Dd^[.e\d^[.f^Dd^\^FTd^[.^Pld^[.^B|d^[.n^\d^[.d4d^[.H|d^[.^Rtd^[.^N^Td^Td^[.^Td^[.{,d^[.+dd^[.`Td^[..^Dd^[.s\d^[.^ETd^[.^Z^Ld^[.
> > $d^[.YTd^\^L^Dd^[.1^Dd^[.^O^\d^[.^PDd^[.^L^\d^\
> > dd^[
> > Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
> > 4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
> > 4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[Q<d^[U^\d^[~td^[l$d^[8<d^[6^Ld^[^F4d^[^Odd^\^O$d^[^K<d^[^N<d^[^K^Dd^[^W4d^[_,d^[p^Dd^[+^\d^[N,d^[$Td^[~^\d^[eLd^[NTd^[
> > ,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
> > ,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
> > <d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[.w^Ld^[.bLd^[.L\d^[.
> > Dd^[.dld^[..<d^[.Lld^\ td^\4d^\ld^^Td^\d^\ <d^\
> > ^Dd^Ld^$d^[.,d^[.dd^[.^\d^[.Td^\
> > 10468/12000 HSRP (Standby)
> >
> > Interrupt level stacks:
> > Level Called Unused/Size Name
> > 1 2648551315 6280/9000 Network interfaces
> > 2 0 9000/9000 DMA/Timer Interrupt
> > 3 185107 7472/9000 PA Management Int Handler
> > 4 1715750501 8444/9000 Console Uart
> > 5 0 9000/9000 OIR/Error Interrupt
> > 7 3207930022 8532/9000 NMI Interrupt Handler
> >
> > Spurious interrupts: 233
> > rtr1#
> >
> > and on a different router:
> >
> > rtr1.chi#sh stacks
> > Minimum process stacks:
> > Free/Size Name
> > [....]
> > 3500/6000
> > 7160/9000 5,<$/jDSw_h 5,< 5,< 5,< 5,< 5,< d(X d(X 5,< 5,< 5,< 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< d'X 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,< 5,<V7;@ 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^BT 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,<Q6mT 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,<#^Qz|#^Qy|#^Qy| 5,<#^Qx|#^Qx| 5,<%Dtx%Dtx%Dtx%Dtx%Dsx%Dsx%Dsx%Dsx 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,< 5,<%Dsx 5,< 5,< 5,<%Drx 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,<#^Qw|#^Qw|#^Qv| 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,<S3^R@ 5,< 5,< 5,<Vo^BT d'X d&X d&X d%X 5,< 5,< 5,<&S<<!A`t 5,< 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<#W:x#W9x
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^AT-4,4 5,<PN^P^T 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,< 5,<Vo^ATVo
> > 5316/6000 BGP Accepter
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

rodunn at cisco

Aug 20, 2008, 6:34 AM

Post #5 of 5 (529 views)
Permalink

Re: debugging stack corruption [In reply to]

If you pull that cable on the aux/console do you still see the
saem thing?

On Tue, Aug 19, 2008 at 11:49:43AM -0700, bill fumerola wrote:
> On Tue, Aug 19, 2008 at 10:41:05AM -0400, Rodney Dunn wrote:
> > How are you getting this output?
>
> ssh rtr1
> en
> sh stacks
>
> > If you ssh/telnet to it and run the command do you get th esame output?
>
> it is not signal noise (serial spew, ip corruption, etc).
>
> > That's not stack corruption to me.
>
> i'll try and profile the exec process, but i'm not so good w/ profiling
> and tracing w/o at least symbols.
>
> there is also the matter of the 30% solid EXEC process. however, the
> switch that device is attached to (both in network and by serial via
> rtr1:aux<>sw1:cons) is exhibiting the same behavior. it could be a feedback
> loop on the serial connection, but i've tried turning all of that down
> and still no relief. the jump occurred to both at the same time.
>
> it could just be corruption in the display, but the CPU spike is what
> made me investigate in the first place.
>
> -- bill
>
> > > rtr1#show stacks
> > > Minimum process stacks:
> > > Free/Size Name
> [...]
> > > 3360/6000
> > > d^\ytd^[.^P^Ld^\zTd^[.`Dd^[.I$d^\^[.Td^[.T^Dd^\y^Dd^\^P<d^[.@Td^[.p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[.e\d^[.NTd^[.)$d^[.pLd^[.|^\d^\
> > > ,d^[.mdd^\^Nld^\
> > > dd^[. 4d^[.Q<d^[.^V^\d^\1dd^[.1<d^[.O4d^[.|Dd^\^P<d^[.^Ydd^[.e\d^[.)$d^[.NTd^[.
> > > 4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
> > > ,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
> > > ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
> > > ,d^[.^R^Dd^[.^Fld^[.\d^[.b^Td^[.^LDd^\^P^Dd^[.^B4d^[.^NLd^[.^Y,d^[.^Kdd^\
> > > ^\d^\^CDd^[.s^Td^[.^A^\d^[.U,d^[.j,d^[.~^Dd^\^QDd^[.Jtd^[.~Ld^[.|^Td^[.,Dd^^\d^[.rld^[.R|d^[.{Dd^[.
> > > \d^[.^Add^[.^Q\d^[.^QLd^[.
> > > ld^[.ttd^[.zdd^\,$d^[.^Vdd^[.)4d^\34d^[.wLd^[.m,d^[.^Z|d^[.\,d^[.g|d^[.y|d^[.^D<d^[.x$d^[.^^Dd^[.
> > > ld^[.^Bld^[.RDd^[.ptd^[.^Q$d^[.v4d^\^Ptd^[.^VTd^[.7$d^\1td^[.P$d^[.uTd^[.^VTd^[.zdd^[.7$d^[.z,d^[.z^\d^[.ytd^[.@Td^[.<^Dd^\,$d^\+Dd^\,4d^[.^D<d^\`^Dd^[.^VTd^[.k4d^[.P^Td^[.a$d^[.$d^[.^V^\d^[.^Utd^[.mdd^[.^Ytd^[.|^Ld^[.^L^Ld^\^ALd^[.#^Dd^[.e\d^[.f^Dd^\^FTd^[.^Pld^[.^B|d^[.n^\d^[.d4d^[.H|d^[.^Rtd^[.^N^Td^Td^[.^Td^[.{,d^[.+dd^[.`Td^[..^Dd^[.s\d^[.^ETd^[.^Z^Ld^[.
> > > $d^[.YTd^\^L^Dd^[.1^Dd^[.^O^\d^[.^PDd^[.^L^\d^\
> > > dd^[.
> > > Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
> > > 4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
> > > 4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[.Q<d^[.U^\d^[.~td^[.l$d^[.8<d^[.6^Ld^[.^F4d^[.^Odd^\^O$d^[.^K<d^[.^N<d^[.^K^Dd^[.^W4d^[._,d^[.p^Dd^[.+^\d^[.N,d^[.$Td^[.~^\d^[.eLd^[.NTd^[.
> > > ,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
> > > ,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
> > > <d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[.w^Ld^[.bLd^[.L\d^[.
> > > Dd^[.dld^[..<d^[.Lld^\ td^\4d^\ld^^Td^\d^\ <d^\
> > > ^Dd^Ld^$d^[.,d^[.dd^[.^\d^[.Td^\
> > > 6856/9000
> > > d^\^[.Td^[.T^Dd^\y^Dd^\^P<d^[.@Td^[.p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[.e\d^[.NTd^[.)$d^[.pLd^[.|^\d^\
> > > ,d^[.mdd^\^Nld^\
> > > dd^[. 4d^[.Q<d^[.^V^\d^\1dd^[.1<d^[.O4d^[.|Dd^\^P<d^[.^Ydd^[.e\d^[.)$d^[.NTd^[.
> > > 4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
> > > ,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
> > > ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
> > > Minimum process stacks:
> > > Free/Size Name
> > > ,d^[.^R^Dd^[.^Fld^[.\d^[.b^Td^[.^LDd^\^P^Dd^[.^B4d^[.^NLd^[.^Y,d^[.^Kdd^\
> > > ^\d^\^CDd^[.s^Td^[.^A^\d^[.U,d^[.j,d^[.~^Dd^\^QDd^[.Jtd^[.~Ld^[.|^Td^[.,Dd^^\d^[.rld^[.R|d^[.{Dd^[.
> > > \d^[.^Add^[.^Q\d^[.^QLd^[.
> > > ld^[.ttd^[.zdd^\,$d^[.^Vdd^[.)4d^\34d^[.wLd^[.m,d^[.^Z|d^[.\,d^[.g|d^[.y|d^[.^D<d^[.x$d^[.^^Dd^[.
> > > ld^[.^Bld^[.RDd^[.ptd^[.^Q$d^[.v4d^\^Ptd^[.^VTd^[.7$d^\1td^[.P$d^[.uTd^[.^VTd^[.zdd^[.7$d^[.z,d^[.z^\d^[.ytd^[.@Td^[.<^Dd^\,$d^\+Dd^\,4d^[.^D<d^\`^Dd^[.^VTd^[.k4d^[.P^Td^[.a$d^[.$d^[.^V^\d^[.^Utd^[.mdd^[.^Ytd^[.|^Ld^[.^L^Ld^\^ALd^[.#^Dd^[.e\d^[.f^Dd^\^FTd^[.^Pld^[.^B|d^[.n^\d^[.d4d^[.H|d^[.^Rtd^[.^N^Td^Td^[.^Td^[.{,d^[.+dd^[.`Td^[..^Dd^[.s\d^[.^ETd^[.^Z^Ld^[.
> > > $d^[.YTd^\^L^Dd^[.1^Dd^[.^O^\d^[.^PDd^[.^L^\d^\
> > > dd^[.
> > > Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
> > > 4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
> > > 4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[.Q<d^[.U^\d^[.~td^[.l$d^[.8<d^[.6^Ld^[.^F4d^[.^Odd^\^O$d^[.^K<d^[.^N<d^[.^K^Dd^[.^W4d^[._,d^[.p^Dd^[.+^\d^[.N,d^[.$Td^[.~^\d^[.eLd^[.NTd^[.
> > > ,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
> > > ,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
> > > <d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[.w^Ld^[.bLd^[.L\d^[.
> > > Dd^[.dld^[..<d^[.Lld^\ td^\4d^\ld^^Td^\d^\ <d^\
> > > ^Dd^Ld^$d^[.,d^[.dd^[.^\d^[.Td^\
> > > 10468/12000 HSRP (Standby)
> > >
> > > Interrupt level stacks:
> > > Level Called Unused/Size Name
> > > 1 2648551315 6280/9000 Network interfaces
> > > 2 0 9000/9000 DMA/Timer Interrupt
> > > 3 185107 7472/9000 PA Management Int Handler
> > > 4 1715750501 8444/9000 Console Uart
> > > 5 0 9000/9000 OIR/Error Interrupt
> > > 7 3207930022 8532/9000 NMI Interrupt Handler
> > >
> > > Spurious interrupts: 233
> > > rtr1#
> > >
> > > and on a different router:
> > >
> > > rtr1.chi#sh stacks
> > > Minimum process stacks:
> > > Free/Size Name
> > > [....]
> > > 3500/6000
> > > 7160/9000 5,<$/jDSw_h 5,< 5,< 5,< 5,< 5,< d(X d(X 5,< 5,< 5,< 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< d'X 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,< 5,<V7;@ 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^BT 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,<Q6mT 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,<#^Qz|#^Qy|#^Qy| 5,<#^Qx|#^Qx| 5,<%Dtx%Dtx%Dtx%Dtx%Dsx%Dsx%Dsx%Dsx 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,< 5,<%Dsx 5,< 5,< 5,<%Drx 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,<#^Qw|#^Qw|#^Qv| 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,<S3^R@ 5,< 5,< 5,<Vo^BT d'X d&X d&X d%X 5,< 5,< 5,<&S<<!A`t 5,< 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<#W:x#W9x
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^AT-4,4 5,<PN^P^T 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,< 5,<Vo^ATVo
> > > 5316/6000 BGP Accepter
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads