Mailing List Archive: nsp: juniper

bfd = busted failure detection :)

Index | Next | Previous | View Threaded

ras at e-gerbil

Nov 21, 2009, 12:22 PM

Post #1 of 24 (2396 views)
Permalink

bfd = busted failure detection :)

Is there a way to see stats on bfd sessions, such as the number of
probes lost? I'm trying to figure out why I can't reliably keep bfd
running between two Juniper's without getting a ton of false positives,
even with very high detection thresholds. Nothing useful in "show bfd
session extensive" (below), and I'm not seeing anything obvious in the
pfe shell for bfd stats (I'm running ppm delegate, wondering if there
are possible issues there).

Detect Transmit
Address State Interface Time Interval Multiplier
x.x.x.x Up xe-7/0/0.0 5.000 0.500 10
Client ISIS L2, TX interval 0.500, RX interval 0.500
Session up time 00:12:13, previous down time 00:00:17
Local diagnostic NbrSignal, remote diagnostic CtlExpire
Remote state Up, version 1
Replicated
Min async interval 0.500, min slow interval 1.000
Adaptive async TX interval 0.500, RX interval 0.500
Local min TX interval 0.500, minimum RX interval 0.500, multiplier 10
Remote min TX interval 0.500, min RX interval 0.500, multiplier 10
Local discriminator 174, remote discriminator 84
Echo mode disabled/inactive
Remote is control-plane independent

--
Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

nkhambal at juniper

Nov 21, 2009, 12:53 PM

Post #2 of 24 (2366 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

Hi Richard,

Just talking from this router perspective, it looks like the remote end
router has problem receiving BFD packets from this router. It signaled the
BFD session down because of that.

You can start by looking at egress stats at the on the local router. See if
there are any ttp queue drops (software queue drops) in "show pfe statistics
traffic" any queuing drops on the egress interface.

On 11/21/09 12:22 PM, "Richard A Steenbergen" <ras [at] e-gerbil> wrote:

> Is there a way to see stats on bfd sessions, such as the number of
> probes lost? I'm trying to figure out why I can't reliably keep bfd
> running between two Juniper's without getting a ton of false positives,
> even with very high detection thresholds. Nothing useful in "show bfd
> session extensive" (below), and I'm not seeing anything obvious in the
> pfe shell for bfd stats (I'm running ppm delegate, wondering if there
> are possible issues there).
>
> Detect Transmit
> Address State Interface Time Interval
> Multiplier
> x.x.x.x Up xe-7/0/0.0 5.000 0.500 10
> Client ISIS L2, TX interval 0.500, RX interval 0.500
> Session up time 00:12:13, previous down time 00:00:17
> Local diagnostic NbrSignal, remote diagnostic CtlExpire
> Remote state Up, version 1
> Replicated
> Min async interval 0.500, min slow interval 1.000
> Adaptive async TX interval 0.500, RX interval 0.500
> Local min TX interval 0.500, minimum RX interval 0.500, multiplier 10
> Remote min TX interval 0.500, min RX interval 0.500, multiplier 10
> Local discriminator 174, remote discriminator 84
> Echo mode disabled/inactive
> Remote is control-plane independent

_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

nkhambal at juniper

Nov 21, 2009, 1:05 PM

Post #3 of 24 (2367 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

[Hit send accidently before completing the email]

You can narrow down the pfe stats per FPC using the "fpc" knob in the "show
pfe statistics traffic" output.

At the remote end, you can look for any input errors (framing, CRC etcs) at
the interface level. Then look for any drops at the route lookup level and
PFE CPU level. Check if PFE CPU is being overrun due to some excess host
bound traffic. You can check "show pfe statistics error" on both side
routers along with "show pfe statistics traffic fpc <slot>" to check if any
ASIC blocks are having issues and they are dropping packets for this
interface/PFE. Also, check the CPU and memory utilization of FPCs on either
sides using "show chassis fpc" command.

Thanks
Nilesh.

On 11/21/09 12:53 PM, "Nilesh Khambal" <nkhambal [at] juniper> wrote:

> Hi Richard,
>
> Just talking from this router perspective, it looks like the remote end
> router has problem receiving BFD packets from this router. It signaled the
> BFD session down because of that.
>
> You can start by looking at egress stats at the on the local router. See if
> there are any ttp queue drops (software queue drops) in "show pfe statistics
> traffic" any queuing drops on the egress interface.
>
>
> On 11/21/09 12:22 PM, "Richard A Steenbergen" <ras [at] e-gerbil> wrote:
>
>> Is there a way to see stats on bfd sessions, such as the number of
>> probes lost? I'm trying to figure out why I can't reliably keep bfd
>> running between two Juniper's without getting a ton of false positives,
>> even with very high detection thresholds. Nothing useful in "show bfd
>> session extensive" (below), and I'm not seeing anything obvious in the
>> pfe shell for bfd stats (I'm running ppm delegate, wondering if there
>> are possible issues there).
>>
>> Detect Transmit
>> Address State Interface Time Interval
>> Multiplier
>> x.x.x.x Up xe-7/0/0.0 5.000 0.500 10
>> Client ISIS L2, TX interval 0.500, RX interval 0.500
>> Session up time 00:12:13, previous down time 00:00:17
>> Local diagnostic NbrSignal, remote diagnostic CtlExpire
>> Remote state Up, version 1
>> Replicated
>> Min async interval 0.500, min slow interval 1.000
>> Adaptive async TX interval 0.500, RX interval 0.500
>> Local min TX interval 0.500, minimum RX interval 0.500, multiplier 10
>> Remote min TX interval 0.500, min RX interval 0.500, multiplier 10
>> Local discriminator 174, remote discriminator 84
>> Echo mode disabled/inactive
>> Remote is control-plane independent
>
> _______________________________________________
> juniper-nsp mailing list juniper-nsp [at] puck
> https://puck.nether.net/mailman/listinfo/juniper-nsp

_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

ras at e-gerbil

Nov 21, 2009, 3:16 PM

Post #4 of 24 (2368 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Sat, Nov 21, 2009 at 12:53:58PM -0800, Nilesh Khambal wrote:
> Hi Richard,
>
> Just talking from this router perspective, it looks like the remote
> end router has problem receiving BFD packets from this router. It
> signaled the BFD session down because of that.

There are actually two particular interfaces between this pair of
routers (both MX960s running 9.4R3, both circuits are long-haul ~70ms
latency) that are flapping because of BFD. The interesting part is that
they both land on different DPCs (on both ends), there are other
circuits between these same devices which are not having BFD issues, and
I ran regular RE based pings between the devices (with src/dst set
correctly to force traffic over the links in question) and didn't record
any loss when BFD thought that it was detecting a failure.

The other side sees:

Session up time 02:39:23, previous down time 00:00:16
Local diagnostic CtlExpire, remote diagnostic NbrSignal

Session up time 02:39:36, previous down time 00:00:28
Local diagnostic CtlExpire, remote diagnostic NbrSignal

Of course this is far from the only problem I have with BFD (just the
one I have right in front of me :P), there are other devices with
similar but more intermittent issues (a few times a day, seemingly at
random) so I'm interested in some better options for debugging it.

> You can start by looking at egress stats at the on the local router.
> See if there are any ttp queue drops (software queue drops) in "show
> pfe statistics traffic" any queuing drops on the egress interface.

No software drops on either device, only a small and non-incrementing
number of hardware input drops and info cell drops.

> At the remote end, you can look for any input errors (framing, CRC
> etcs) at the interface level. Then look for any drops at the route
> lookup level and PFE CPU level. Check if PFE CPU is being overrun due
> to some excess host bound traffic. You can check "show pfe statistics
> error" on both side routers along with "show pfe statistics traffic
> fpc <slot>" to check if any ASIC blocks are having issues and they are
> dropping packets for this interface/PFE. Also, check the CPU and
> memory utilization of FPCs on either sides using "show chassis fpc"
> command.

Already checked the obvious stuff, no interface drops, no incrementing
errors (these devices have been up for a while, so there are a few old
ones from previous circuit flaps), the interfaces are nowhere close to
full, and ping over the affected links comes back clean.

PFE CPU/memory seems perfectly normal on all DPCs on both devices, they
all look pretty much like this:

Temp CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt DRAM (MB) Heap Buffer
0 Online 41 11 0 1024 27 30
1 Online 40 11 0 1024 26 30
2 Online 37 12 0 1024 26 30
3 Online 36 10 0 1024 26 31
4 Online 36 15 0 1024 26 30
5 Online 38 12 0 1024 26 30
6 Empty
7 Online 39 15 0 1024 26 30
8 Online 38 14 0 1024 26 30

One of the circuits has a few Ipktwr Drops (the 1836 I-chip, the other
port is something completely unrelated) reported on one side, the other
is clean:

Ipktwr Drops: 1836 3 0 5010

I was hoping for some kind of ppm delegate and/or bfd counter stats that
show exactly how many hellos are being missed, similar to what Cisco
has:

Holddown (hits): 4979(0), Hello (hits): 999(745796)
Rx Count: 745694, Rx Interval (ms) min/max/avg: 564/1228/883 last: 16 ms ago
Tx Count: 745698, Tx Interval (ms) min/max/avg: 756/1000/880 last: 652 ms ago
Elapsed time watermarks: -1 0 (last: 0)
Registered protocols: ISIS TE/FRR
Uptime: 1w0d
Last packet: Version: 1 - Diagnostic: 0
State bit: Up - Demand bit: 0
Poll bit: 0 - Final bit: 0
Multiplier: 5 - Length: 24
My Discr.: 1 - Your Discr.: 1
Min tx interval: 999000 - Min rx interval: 999000
Min Echo interval: 999000

--
Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

ras at e-gerbil

Dec 4, 2009, 12:40 PM

Post #5 of 24 (2291 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Sat, Nov 21, 2009 at 05:16:57PM -0600, Richard A Steenbergen wrote:
> On Sat, Nov 21, 2009 at 12:53:58PM -0800, Nilesh Khambal wrote:
> > Hi Richard,
> >
> > Just talking from this router perspective, it looks like the remote
> > end router has problem receiving BFD packets from this router. It
> > signaled the BFD session down because of that.
>
> There are actually two particular interfaces between this pair of
> routers (both MX960s running 9.4R3, both circuits are long-haul ~70ms
> latency) that are flapping because of BFD. The interesting part is that
> they both land on different DPCs (on both ends), there are other
> circuits between these same devices which are not having BFD issues, and
> I ran regular RE based pings between the devices (with src/dst set
> correctly to force traffic over the links in question) and didn't record
> any loss when BFD thought that it was detecting a failure.

FYI I found the root problem and hereby take back any comments impugning
BFD's reputation. It turns out there actually WAS some kind of pfe bug
which was causing intermittent blackholing of traffic for a few seconds
at a time at seemingly random intervals several times a day. Ping from
the affected devices didn't catch the issue becuase of the re->pfe
forwarding path, only traffic routed entirely via pfe was being
affected. BFD was actually doing its job and detected the failures that
were too short to be noticed by normal routing protocols. I discovered
the issue on several MX960s (mostly running 9.2R4, but one pair was
running 9.4R3), and upgrading them to 9.5R3 seems to solve it (or
perhaps it was just the pfe rstart that did it, remains to be seen).

--
Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

ross at kallisti

Dec 8, 2009, 4:54 PM

Post #6 of 24 (2257 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Fri, Dec 04, 2009 at 02:40:14PM -0600, Richard A Steenbergen wrote:
> FYI I found the root problem and hereby take back any comments impugning
> BFD's reputation. It turns out there actually WAS some kind of pfe bug
> which was causing intermittent blackholing of traffic for a few seconds
> at a time at seemingly random intervals several times a day. Ping from
> the affected devices didn't catch the issue becuase of the re->pfe
> forwarding path, only traffic routed entirely via pfe was being
> affected. BFD was actually doing its job and detected the failures that
> were too short to be noticed by normal routing protocols. I discovered
> the issue on several MX960s (mostly running 9.2R4, but one pair was
> running 9.4R3), and upgrading them to 9.5R3 seems to solve it (or
> perhaps it was just the pfe rstart that did it, remains to be seen).

Is there a PR number for this issue? Sounds like it could be a drag.

Ross

--
Ross Vandegrift
ross [at] kallisti

"If the fight gets hot, the songs get hotter. If the going gets tough,
the songs get tougher."
--Woody Guthrie

Attachments:

signature.asc (0.19 KB)

ras at e-gerbil

Dec 8, 2009, 10:27 PM

Post #7 of 24 (2256 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Tue, Dec 08, 2009 at 07:54:49PM -0500, Ross Vandegrift wrote:
> On Fri, Dec 04, 2009 at 02:40:14PM -0600, Richard A Steenbergen wrote:
> > FYI I found the root problem and hereby take back any comments impugning
> > BFD's reputation. It turns out there actually WAS some kind of pfe bug
> > which was causing intermittent blackholing of traffic for a few seconds
> > at a time at seemingly random intervals several times a day. Ping from
> > the affected devices didn't catch the issue becuase of the re->pfe
> > forwarding path, only traffic routed entirely via pfe was being
> > affected. BFD was actually doing its job and detected the failures that
> > were too short to be noticed by normal routing protocols. I discovered
> > the issue on several MX960s (mostly running 9.2R4, but one pair was
> > running 9.4R3), and upgrading them to 9.5R3 seems to solve it (or
> > perhaps it was just the pfe rstart that did it, remains to be seen).
>
> Is there a PR number for this issue? Sounds like it could be a drag.

Maybe, but I'm not sufficiently motivated to try and explain the issue
to JTAC to find out. :) FWIW we've now upgraded 3 of the MX960s that had
the issue from 9.2R4->9.5R3 and it resolved things completely. 9.2R4 is
a giant buggy mess for a lot of others reasons anyways, there is really
no good reason to still be running it. The only reason we still had it
deployed anywhere was the grief caused by the logical-routers ->
logical-systems rename in 9.3 (which broke backwards compatibility for
commit scripts), this was just extra motivation to get some long overdue
upgrades done.

--
Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

ras at e-gerbil

Dec 9, 2009, 12:10 AM

Post #8 of 24 (2253 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Wed, Dec 09, 2009 at 12:27:55AM -0600, Richard A Steenbergen wrote:
> Maybe, but I'm not sufficiently motivated to try and explain the issue
> to JTAC to find out. :) FWIW we've now upgraded 3 of the MX960s that had
> the issue from 9.2R4->9.5R3 and it resolved things completely. 9.2R4 is
> a giant buggy mess for a lot of others reasons anyways, there is really
> no good reason to still be running it. The only reason we still had it
> deployed anywhere was the grief caused by the logical-routers ->
> logical-systems rename in 9.3 (which broke backwards compatibility for
> commit scripts), this was just extra motivation to get some long overdue
> upgrades done.

Oh and btw I take back all the nice things I said about progress being
made to resolve the slow fib install / krt queue blocking bug. It is
still alive and well in 9.5R3, and blackholing traffic more than ever
(just recorded a good 10 minutes of it).

--
Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

pekkas at netcore

Dec 9, 2009, 1:04 AM

Post #9 of 24 (2253 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Wed, 9 Dec 2009, Richard A Steenbergen wrote:
> Oh and btw I take back all the nice things I said about progress being
> made to resolve the slow fib install / krt queue blocking bug. It is
> still alive and well in 9.5R3, and blackholing traffic more than ever
> (just recorded a good 10 minutes of it).

Do you have a bug open on this? We did investigate it (on 9.3
though), but the result was "it's working as designed", Juniper
couldn't replicate the issue and we had to close the case :-(.

--
Pekka Savola "You each name yourselves king, yet the
Netcore Oy kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

ras at e-gerbil

Dec 9, 2009, 1:40 AM

Post #10 of 24 (2251 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Wed, Dec 09, 2009 at 11:04:59AM +0200, Pekka Savola wrote:
> On Wed, 9 Dec 2009, Richard A Steenbergen wrote:
> >Oh and btw I take back all the nice things I said about progress being
> >made to resolve the slow fib install / krt queue blocking bug. It is
> >still alive and well in 9.5R3, and blackholing traffic more than ever
> >(just recorded a good 10 minutes of it).
>
> Do you have a bug open on this? We did investigate it (on 9.3
> though), but the result was "it's working as designed", Juniper
> couldn't replicate the issue and we had to close the case :-(.

Nothing current, it ended up spread out over a bunch of cases which all
got closed with no real resolution to the root problem. Everyone I've
talked to at Juniper knows that the problem exists, they just don't know
why, and I don't know of any currently ongoing effort to fix it. Maybe
it's time we all get organized and start raising the attention level for
this very serious issue.

--
Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

mtinka at globaltransit

Dec 9, 2009, 1:50 AM

Post #11 of 24 (2252 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Wednesday 09 December 2009 05:40:00 pm Richard A
Steenbergen wrote:

> Nothing current, it ended up spread out over a bunch of
> cases which all got closed with no real resolution to
> the root problem. Everyone I've talked to at Juniper
> knows that the problem exists, they just don't know why,
> and I don't know of any currently ongoing effort to fix
> it. Maybe it's time we all get organized and start
> raising the attention level for this very serious issue.

I'd be willing to help if we can offline this to a
reproduction in my lab.

I have a case that will have been open for 1 year, if
February 2010 comes and we still haven't fixed it. So I know
what it's like :-).

Cheers,

Mark.

Attachments:

signature.asc (0.82 KB)

davidtball at gmail

Dec 9, 2009, 8:13 AM

Post #12 of 24 (2247 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

Do your KRT queues eventually flush though? Is it just a slow
control->fwding thing when large route updates occur? I've done 2
upgrades in as many years to resolve a KRT related bug, but that
resulted in the queue NEVER emptying. It's apparently related to a
residual variable being set after an RPD restart (caused by another
bug) resulting in a kernel/rpd inconsistency. I'm told mine is
resolved in 9.5R3 (PR291407), but I got nervous when I read Richard's
earlier post.

David

2009/12/9 Mark Tinka <mtinka [at] globaltransit>:
> On Wednesday 09 December 2009 05:40:00 pm Richard A
> Steenbergen wrote:
>
>> Nothing current, it ended up spread out over a bunch of
>> cases which all got closed with no real resolution to
>> the root problem. Everyone I've talked to at Juniper
>> knows that the problem exists, they just don't know why,
>> and I don't know of any currently ongoing effort to fix
>> it. Maybe it's time we all get organized and start
>> raising the attention level for this very serious issue.
>
> I'd be willing to help if we can offline this to a
> reproduction in my lab.
>
> I have a case that will have been open for 1 year, if
> February 2010 comes and we still haven't fixed it. So I know
> what it's like :-).
>
> Cheers,
>
> Mark.
>
> _______________________________________________
> juniper-nsp mailing list juniper-nsp [at] puck
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

ras at e-gerbil

Dec 9, 2009, 3:21 PM

Post #13 of 24 (2245 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Wed, Dec 09, 2009 at 09:13:28AM -0700, David Ball wrote:
> Do your KRT queues eventually flush though? Is it just a slow
> control->fwding thing when large route updates occur? I've done 2
> upgrades in as many years to resolve a KRT related bug, but that
> resulted in the queue NEVER emptying. It's apparently related to a
> residual variable being set after an RPD restart (caused by another
> bug) resulting in a kernel/rpd inconsistency. I'm told mine is
> resolved in 9.5R3 (PR291407), but I got nervous when I read Richard's
> earlier post.

The behavior we've always seen (from mid 7.x's until today) is that
something seems to "block" the KRT queue while the pending changes keep
piling up, then eventually whatever is causing the blockage clears and
all the routes quickly install immediately thereafter. I saw the exact
same behavior last night during an upgrade to 9.5R3, 263k routes stuck
in Pending state for a hair over 10 minutes, then they all synced in
just a few seconds. But I think it's actually getting worse, because in
older versions the routes that were stuck in pending state didn't seem
to be advertised to peers. This time it seemed to advertise the routes
even though it didn't have them installed in hardware, resulting in
blackholing of traffic.

> 2009/12/9 Mark Tinka <mtinka [at] globaltransit>:
> >
> > I'd be willing to help if we can offline this to a
> > reproduction in my lab.
> >
> > I have a case that will have been open for 1 year, if
> > February 2010 comes and we still haven't fixed it. So I know
> > what it's like :-).

I've personally never had any luck reproducing it in the lab, so I
understand Juniper's frustration. It seems to require a complexity of
routes, ports, and/or protocols which we simply don't have the time or
money to reproduce in the lab, but I can reproduce it in the field (with
undesired customer impact of course) nearly every time I reboot a
router. Maybe we just need to help provide them with an example
configuration that they can try to reproduce themselves.

As for the cases which have been open for over 1 year... I had quite a
few a year ago, JTAC was really dropping the ball in absolutely abysmal
ways. IMHO it has gotten significantly better over the last year, both
in JTAC and code quality, but they're still pretty hit or miss in the
initial stages. The sad part was none of my issues were complex problems
that actually took a year to resolve, they were all relatively simple
problems that took JTAC a year and several escalations just to find
someone competent who could actually read, understand, and follow my
instructions to reproduce the problem.

--
Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

ross at kallisti

Dec 11, 2009, 11:50 AM

Post #14 of 24 (2249 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Wed, Dec 09, 2009 at 05:21:21PM -0600, Richard A Steenbergen wrote:
> I've personally never had any luck reproducing it in the lab, so I
> understand Juniper's frustration. It seems to require a complexity of
> routes, ports, and/or protocols which we simply don't have the time or
> money to reproduce in the lab, but I can reproduce it in the field (with
> undesired customer impact of course) nearly every time I reboot a
> router. Maybe we just need to help provide them with an example
> configuration that they can try to reproduce themselves.

Hmmm, I may have just reproduced something like this in the lab. I
had two static routes for 10.57.55.0/24 and realized I hadn't applied
a per-packet load balancing policy to the forwarding table export. So
I wrote a policy and applied it to the forwarding-table export. This
has been stuck in the KRT queue for over four hours now.

This is different than the indirect next-hop change, but I wonder if
its related. Note that interesting error "EPERM -- Jtree walk in
progress".

Ross

{master:0}
rvandegrift [at] lab-420> show route 10.57.55.0 extensive

inet.0: 8 destinations, 8 routes (7 active, 0 holddown, 1 hidden)
10.57.55.0/24 (1 entry, 1 announced)
TSI:
KRT queued (deferred) change
10.57.55.0/24 -> {172.16.23.2}=>{172.16.23.2, 172.16.23.3}
in-kernel 10.57.55.0/24 -> {172.16.23.2}
*Static Preference: 5
Next hop type: Router
Next-hop reference count: 2
Next hop: 172.16.23.2 via me0.0, selected
Next hop: 172.16.23.3 via me0.0
State: <Active Int Ext>
Age: 8:49
Task: RT
Announcement bits (1): 0-KRT
AS path: I

{master:0}
rvandegrift [at] lab-420> show krt queue
Routing table add queue: 0 queued
Interface add/delete/change queue: 0 queued
Indirect next hop add/change: 0 queued
MPLS add queue: 0 queued
Indirect next hop delete: 0 queued
High-priority deletion queue: 0 queued
High-priority change queue: 0 queued
High-priority add queue: 0 queued
Normal-priority indirect next hop queue: 0 queued
Normal-priority deletion queue: 0 queued
Normal-priority composite next hop deletion queue: 0 queued
Normal-priority change queue: 1 queued
CHANGE FROM gf 1 inst id 0 10.57.55.0/24 nexthop
172.16.23.2, me0.0
(15)
error 'EPERM -- Jtree walk in progress'
TO gf 1 inst id 0 10.57.55.0/24 nexthops
172.16.23.2, me0.0
172.16.23.3, me0.0
(15)
error 'EPERM -- Jtree walk in progress'
Normal-priority add queue: 0 queued
Routing table delete queue: 0 queued

--
Ross Vandegrift
ross [at] kallisti

"If the fight gets hot, the songs get hotter. If the going gets tough,
the songs get tougher."
--Woody Guthrie

Attachments:

signature.asc (0.19 KB)

ras at e-gerbil

Dec 13, 2009, 1:11 AM

Post #15 of 24 (2223 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Fri, Dec 11, 2009 at 02:50:51PM -0500, Ross Vandegrift wrote:
> On Wed, Dec 09, 2009 at 05:21:21PM -0600, Richard A Steenbergen wrote:
> > I've personally never had any luck reproducing it in the lab, so I
> > understand Juniper's frustration. It seems to require a complexity of
> > routes, ports, and/or protocols which we simply don't have the time or
> > money to reproduce in the lab, but I can reproduce it in the field (with
> > undesired customer impact of course) nearly every time I reboot a
> > router. Maybe we just need to help provide them with an example
> > configuration that they can try to reproduce themselves.
>
> Hmmm, I may have just reproduced something like this in the lab. I
> had two static routes for 10.57.55.0/24 and realized I hadn't applied
> a per-packet load balancing policy to the forwarding table export. So
> I wrote a policy and applied it to the forwarding-table export. This
> has been stuck in the KRT queue for over four hours now.
>
> This is different than the indirect next-hop change, but I wonder if
> its related. Note that interesting error "EPERM -- Jtree walk in
> progress".

That one is pretty different from the usual slowness issue that seems to
be affecting most people. I just cleared bgp sessions on a router to
demonstrate the issue, which you can portions of any time you make a
major routing change. Unfortunately (for my demonstration) this router
was pretty small and didn't exhibit any stalls in processing fib
updates. The performance was pretty acceptable, fully syncing in under a
minute. I'm sure the simultanious loss of IGP routes and having more
complex routing protocol configurations has something to do with it too.

At any rate, what you'll see while the paths are trying to change is
something like this:

2.0.0.0/16 +[BGP/170] 00:00:31, MED 0, localpref 100, from x.x.x.x
AS path: ### ### I
> to x.x.x.x via xe-2/1/0.54
to x.x.x.x via xe-2/2/0.54
-[BGP/170] 00:03:10, MED 0, localpref 100, from x.x.x.x
AS path: ### ### I
> to x.x.x.x via xe-2/1/0.54

And in a show bgp summary, you'll see the routes stuck in pending state
(again this one processed the routes quite fast, this is just for demo
purposes):

Groups: 3 Peers: 7 Down peers: 7
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0 0 0 0 0 0 13831

I've sent Juniper the krt queue as the stall is happening many dozens of
times in the past, but it basically just looks like a router syncing a
bunch of new routes, like this:

Indirect next hop add/change: 0 queued
MPLS add queue: 0 queued
Indirect next hop delete: 0 queued
High-priority deletion queue: 0 queued
High-priority change queue: 0 queued
High-priority add queue: 0 queued
Normal-priority indirect next hop queue: 0 queued
Normal-priority deletion queue: 0 queued
Normal-priority composite next hop deletion queue: 0 queued
Normal-priority change queue: 0 queued
Normal-priority add queue: 53504 queued
ADD gf 1 inst id 0 62.56.0.0/17 type 3
ADD gf 1 inst id 0 62.49.0.0/16 type 3
ADD gf 1 inst id 0 84.19.96.0/19 type 3
ADD gf 1 inst id 0 85.91.40.0/22 type 3
ADD gf 1 inst id 0 85.91.48.0/20 type 3
ADD gf 1 inst id 0 83.104.0.0/14 type 3
ADD gf 1 inst id 0 80.176.0.0/15 type 3
ADD gf 1 inst id 0 80.255.192.0/19 type 3
ADD gf 1 inst id 0 146.179.0.0/18 type 3
ADD gf 1 inst id 0 34.252.140.0/22 type 3
ADD gf 1 inst id 0 147.108.200.0/21 type 3
ADD gf 1 inst id 0 146.23.177.0/24 type 3
ADD gf 1 inst id 0 146.23.178.0/24 type 3
ADD gf 1 inst id 0 134.144.108.0/24 type 3
ADD gf 1 inst id 0 138.32.240.0/22 type 3
ADD gf 1 inst id 0 32.97.185.0/24 type 3
ADD gf 1 inst id 0 52.124.0.0/21 type 3
etc etc

The problem is that something seems to cause this process to stall for
several minutes at a time, leaving the router in limbo without routes
installed in the fib, and in some cases blackholing packets as a result.

--
Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

ras at e-gerbil

Dec 14, 2009, 1:23 AM

Post #16 of 24 (2207 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Sun, Dec 13, 2009 at 03:11:29AM -0600, Richard A Steenbergen wrote:
> That one is pretty different from the usual slowness issue that seems to
> be affecting most people. I just cleared bgp sessions on a router to
> demonstrate the issue, which you can portions of any time you make a
> major routing change. Unfortunately (for my demonstration) this router
> was pretty small and didn't exhibit any stalls in processing fib
> updates. The performance was pretty acceptable, fully syncing in under a
> minute. I'm sure the simultanious loss of IGP routes and having more
> complex routing protocol configurations has something to do with it too.

Oh what good timing, just had to reboot a router tonight to recover from
a differnet Juniper bug (enabling graceful-switchover on a 9.5R3 box
caused blackholing of traffic, disabling it didn't fix it, had to reboot
the box to clear the issue which of course blew away all the state, so
there will be no finding the root cause). But it did provide a perfect
example of the FIB blocking issue, with the vast majority of the routing
table blocking for over 13 minutes before finally installing within a
few seconds.

Here we are at just past the 13 minute mark, BGP fully synchronized, but
the vast majority of the routing table not actually installed to FIB:

Groups: 65 Peers: 92 Down peers: 15
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0 2793497 333891 0 0 0 292429
inetflow.0 27 27 0 0 0 4
inet6.0 9438 2075 0 0 0 811

Here is the show krt queue from the same time, showing almost nothing in
the queue. A followup command a second later showed completely different
items in the queue, leading one to believe that the krt queue was not
stuck.

Routing table add queue: 0 queued
Interface add/delete/change queue: 0 queued
Indirect next hop add/change: 0 queued
MPLS add queue: 0 queued
Indirect next hop delete: 2 queued
DELETE index 1048789
DELETE index 1048790
High-priority deletion queue: 0 queued
High-priority change queue: 0 queued
High-priority add queue: 0 queued
Normal-priority indirect next hop queue: 0 queued
Normal-priority deletion queue: 0 queued
Normal-priority composite next hop deletion queue: 0 queued
Normal-priority change queue: 0 queued
Normal-priority add queue: 7 queued
ADD gf 1 inst id 0 173.164.0.0/19 type 3
(20)
ADD gf 1 inst id 0 173.162.16.0/20 type 3
(20)
ADD gf 1 inst id 0 173.160.64.0/19 type 3
(20)
ADD gf 1 inst id 0 217.168.224.0/20 type 3
(20)
ADD gf 1 inst id 0 209.211.136.0/24 nexthop
x.x.x.x, xe-7/1/0.0
(19)
ADD gf 1 inst id 0 208.45.191.0/24 nexthop
x.x.x.x, xe-7/1/0.0
(19)
ADD gf 1 inst id 0 208.45.190.0/24 nexthop
x.x.x.x, xe-7/1/0.0
(19)
Routing table delete queue: 0 queued

Here is an example of a route which has been stuck trying to install for
over 8 minutes (first entry in a show route, the rest all look roughly
the same though):

2.0.0.0/16 +[BGP/170] 00:08:40, MED 0, localpref 200, from xx.xx.xxx.xxx
AS path: 5413 12654 I
> to xx.xx.xxx.xx via xe-3/2/0.0, label-switched-path XXXXX
to xx.xx.xxx.xx via xe-3/2/0.0, label-switched-path XXXXX
to xx.xx.xxx.xx via xe-3/2/0.0, label-switched-path XXXXX
to xx.xx.xxx.xx via xe-3/2/0.0, label-switched-path XXXXX
to xx.xx.xxx.xx via ae0.50, label-switched-path Bypass->xx.xx.xxx.xx->xx.xx.xxx.xx
to xx.xx.xxx.xx via ae0.50, label-switched-path Bypass->xx.xx.xxx.xx->xx.xx.xxx.xx
to xx.xx.xxx.xx via ae0.50, label-switched-path Bypass->xx.xx.xxx.xx->xx.xx.xxx.xx
to xx.xx.xxx.xx via ae0.50, label-switched-path Bypass->xx.xx.xxx.xx->xx.xx.xxx.xx

The above is pretty representative of the issue, which has been going on
in one form or another since around the mid 7.x's (confirmed by dozens
of people I've talked to who saw the same behavior beginning at around
the same time).

--
Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

hoogen82 at gmail

Dec 14, 2009, 10:23 AM

Post #17 of 24 (2204 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

Thanks for all the great info Richard...

-Hoogen

On Mon, Dec 14, 2009 at 1:23 AM, Richard A Steenbergen <ras [at] e-gerbil>wrote:

> On Sun, Dec 13, 2009 at 03:11:29AM -0600, Richard A Steenbergen wrote:
> > That one is pretty different from the usual slowness issue that seems to
> > be affecting most people. I just cleared bgp sessions on a router to
> > demonstrate the issue, which you can portions of any time you make a
> > major routing change. Unfortunately (for my demonstration) this router
> > was pretty small and didn't exhibit any stalls in processing fib
> > updates. The performance was pretty acceptable, fully syncing in under a
> > minute. I'm sure the simultanious loss of IGP routes and having more
> > complex routing protocol configurations has something to do with it too.
>
> Oh what good timing, just had to reboot a router tonight to recover from
> a differnet Juniper bug (enabling graceful-switchover on a 9.5R3 box
> caused blackholing of traffic, disabling it didn't fix it, had to reboot
> the box to clear the issue which of course blew away all the state, so
> there will be no finding the root cause). But it did provide a perfect
> example of the FIB blocking issue, with the vast majority of the routing
> table blocking for over 13 minutes before finally installing within a
> few seconds.
>
> Here we are at just past the 13 minute mark, BGP fully synchronized, but
> the vast majority of the routing table not actually installed to FIB:
>
> Groups: 65 Peers: 92 Down peers: 15
> Table Tot Paths Act Paths Suppressed History Damp State
> Pending
> inet.0 2793497 333891 0 0 0
> 292429
> inetflow.0 27 27 0 0 0
> 4
> inet6.0 9438 2075 0 0 0
> 811
>
> Here is the show krt queue from the same time, showing almost nothing in
> the queue. A followup command a second later showed completely different
> items in the queue, leading one to believe that the krt queue was not
> stuck.
>
> Routing table add queue: 0 queued
> Interface add/delete/change queue: 0 queued
> Indirect next hop add/change: 0 queued
> MPLS add queue: 0 queued
> Indirect next hop delete: 2 queued
> DELETE index 1048789
> DELETE index 1048790
> High-priority deletion queue: 0 queued
> High-priority change queue: 0 queued
> High-priority add queue: 0 queued
> Normal-priority indirect next hop queue: 0 queued
> Normal-priority deletion queue: 0 queued
> Normal-priority composite next hop deletion queue: 0 queued
> Normal-priority change queue: 0 queued
> Normal-priority add queue: 7 queued
> ADD gf 1 inst id 0 173.164.0.0/19 type 3
> (20)
> ADD gf 1 inst id 0 173.162.16.0/20 type 3
> (20)
> ADD gf 1 inst id 0 173.160.64.0/19 type 3
> (20)
> ADD gf 1 inst id 0 217.168.224.0/20 type 3
> (20)
> ADD gf 1 inst id 0 209.211.136.0/24 nexthop
> x.x.x.x, xe-7/1/0.0
> (19)
> ADD gf 1 inst id 0 208.45.191.0/24 nexthop
> x.x.x.x, xe-7/1/0.0
> (19)
> ADD gf 1 inst id 0 208.45.190.0/24 nexthop
> x.x.x.x, xe-7/1/0.0
> (19)
> Routing table delete queue: 0 queued
>
> Here is an example of a route which has been stuck trying to install for
> over 8 minutes (first entry in a show route, the rest all look roughly
> the same though):
>
> 2.0.0.0/16 +[BGP/170] 00:08:40, MED 0, localpref 200, from
> xx.xx.xxx.xxx
> AS path: 5413 12654 I
> > to xx.xx.xxx.xx via xe-3/2/0.0, label-switched-path
> XXXXX
> to xx.xx.xxx.xx via xe-3/2/0.0, label-switched-path
> XXXXX
> to xx.xx.xxx.xx via xe-3/2/0.0, label-switched-path
> XXXXX
> to xx.xx.xxx.xx via xe-3/2/0.0, label-switched-path
> XXXXX
> to xx.xx.xxx.xx via ae0.50, label-switched-path
> Bypass->xx.xx.xxx.xx->xx.xx.xxx.xx
> to xx.xx.xxx.xx via ae0.50, label-switched-path
> Bypass->xx.xx.xxx.xx->xx.xx.xxx.xx
> to xx.xx.xxx.xx via ae0.50, label-switched-path
> Bypass->xx.xx.xxx.xx->xx.xx.xxx.xx
> to xx.xx.xxx.xx via ae0.50, label-switched-path
> Bypass->xx.xx.xxx.xx->xx.xx.xxx.xx
>
> The above is pretty representative of the issue, which has been going on
> in one form or another since around the mid 7.x's (confirmed by dozens
> of people I've talked to who saw the same behavior beginning at around
> the same time).
>
> --
> Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
> GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
> _______________________________________________
> juniper-nsp mailing list juniper-nsp [at] puck
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

mtinka at globaltransit

Dec 14, 2009, 10:59 PM

Post #18 of 24 (2187 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Monday 14 December 2009 05:23:45 pm Richard A Steenbergen
wrote:

> Oh what good timing, just had to reboot a router tonight
> to recover from a differnet Juniper bug (enabling
> graceful-switchover on a 9.5R3 box caused blackholing of
> traffic, disabling it didn't fix it, had to reboot the
> box to clear the issue which of course blew away all the
> state, so there will be no finding the root cause).

Sorry to side-track the thread a bit, but curious whether
you had NSR or Graceful Restart enabled when the bug hit.

We chose to wait for 9.5R4, but would like to be sure this
won't be an issue when we upgrade. However, we're a Graceful
Switchover + Graceful Restart house. No NSR.

Cheers,

Mark.

Attachments:

signature.asc (0.82 KB)

ras at e-gerbil

Dec 15, 2009, 12:16 AM

Post #19 of 24 (2192 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Tue, Dec 15, 2009 at 02:59:04PM +0800, Mark Tinka wrote:
> On Monday 14 December 2009 05:23:45 pm Richard A Steenbergen
> wrote:
>
> > Oh what good timing, just had to reboot a router tonight
> > to recover from a differnet Juniper bug (enabling
> > graceful-switchover on a 9.5R3 box caused blackholing of
> > traffic, disabling it didn't fix it, had to reboot the
> > box to clear the issue which of course blew away all the
> > state, so there will be no finding the root cause).
>
> Sorry to side-track the thread a bit, but curious whether
> you had NSR or Graceful Restart enabled when the bug hit.
>
> We chose to wait for 9.5R4, but would like to be sure this
> won't be an issue when we upgrade. However, we're a Graceful
> Switchover + Graceful Restart house. No NSR.

Neither, they were both off when graceful switchover was turned on
(which caused the breakage). The best way to do a non-ISSU upgrade is to
pre-stage the new code/config on the backup RE, then do a non-graceful
switchover. It still reboots the PFE, but it avoids the jinstall upgrade
time, or the RE bootup time if you were doing it from a cold boot.

I've actually seen similar bugs in the past, where graceful-switchover
caused blackholing of live traffic. It would stop blackholing when we
rebooted the backup RE, then started blackholing again when the backup
RE came back online. JTAC was unable to reproduce it, and we were
unwilling to turn gracful-switchover back on (and start blackholing
traffic again :P), so eventually they just closed the case with no
resolution of the actual problem. In this particular incident, we turned
graceful-switchover on for (4) recently upgraded 9.5R3 boxes, and only
one of the started blackholing traffic. But unlike the previous issues,
it didn't recover when g-s was turned off this time. If it makes you
feel any better I highly doubt 9.5R4 will fix the problem (though at
least it seems to be pretty rare). :)

As for GR vs NSR, we're actually in the process of turning GR off in
favor of NSR. So far, in very limited tests mind you, ISSU has actually
worked for us without anything exploding or catching on fire (surprising
I know :P). Also, NSR you can turn on and off without causing any impact
to the router, but GR causes protocol bounces when you turn it on/off.
Thus we decided it is better to ditch GR now, with the hope of working
NSR and widespread ISSU success in the future. :)

--
Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

mtinka at globaltransit

Dec 15, 2009, 12:53 AM

Post #20 of 24 (2182 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Tuesday 15 December 2009 04:16:23 pm Richard A
Steenbergen wrote:

> As for GR vs NSR, we're actually in the process of
> turning GR off in favor of NSR. So far, in very limited
> tests mind you, ISSU has actually worked for us without
> anything exploding or catching on fire (surprising I
> know :P). Also, NSR you can turn on and off without
> causing any impact to the router, but GR causes protocol
> bounces when you turn it on/off. Thus we decided it is
> better to ditch GR now, with the hope of working NSR and
> widespread ISSU success in the future. :)

We've always been in favour of the NSR concept since
inception, but the reason we didn't choose it at the time
was because of limited protocol support (early days of JUNOS
9.x). Also, only a handful of boxes on the Cisco side
support(ed) NSR, yet we like to standardize feature sets
even between vendors.

As at today, NSR supports nearly all major protocols in
JUNOS, so it's come back to the table for consideration.
Your feedback is certainly encouraging.

Until then, it's Graceful Restart for now (which, for better
or worse, we've been happy with these past 2 years).

Cheers,

Mark.

Attachments:

signature.asc (0.82 KB)

ras at e-gerbil

Dec 15, 2009, 4:04 PM

Post #21 of 24 (2172 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Tue, Dec 15, 2009 at 04:53:44PM +0800, Mark Tinka wrote:
> We've always been in favour of the NSR concept since
> inception, but the reason we didn't choose it at the time
> was because of limited protocol support (early days of JUNOS
> 9.x). Also, only a handful of boxes on the Cisco side
> support(ed) NSR, yet we like to standardize feature sets
> even between vendors.
>
> As at today, NSR supports nearly all major protocols in
> JUNOS, so it's come back to the table for consideration.
> Your feedback is certainly encouraging.
>
> Until then, it's Graceful Restart for now (which, for better
> or worse, we've been happy with these past 2 years).

It's been slowly trickling in over the last couple years, with each new
code rev. The last protocol support we were waiting on for NSR was RSVP,
which finally came in 9.6. Of course we aren't running 9.6 anywhere yet,
but NSR+ISSU has been working reasonably well in tests of prior versions
with only modest RSVP traffic disruptions, which is still better than
nothing. The only thing to note is that the process takes FOREVER to
complete, reminds me of trying to do GR back in the day of RE-2.0's,
where it took 30 minutes to sync up after a restart. When 9.6R3 comes
out we're planning to start limited deployment experiments in a couple
select test sites, and with any luck our next round of widespread
upgrades will be something where NSR+ISSU is fully working.

--
Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

judah.scott.iam at gmail

Dec 15, 2009, 4:40 PM

Post #22 of 24 (2169 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

In lab tests, RSVP success during ISSU was hit-or-miss. There were several
cases that is did work but It seemed to be not entirely based on
configuration (some degree of random failure). AFAIK it still hasn't been
officially listed as supported in release notes or upgrade guides. Am I
missing something?

-J Scott

On Tue, Dec 15, 2009 at 4:04 PM, Richard A Steenbergen <ras [at] e-gerbil>wrote:

> On Tue, Dec 15, 2009 at 04:53:44PM +0800, Mark Tinka wrote:
> > We've always been in favour of the NSR concept since
> > inception, but the reason we didn't choose it at the time
> > was because of limited protocol support (early days of JUNOS
> > 9.x). Also, only a handful of boxes on the Cisco side
> > support(ed) NSR, yet we like to standardize feature sets
> > even between vendors.
> >
> > As at today, NSR supports nearly all major protocols in
> > JUNOS, so it's come back to the table for consideration.
> > Your feedback is certainly encouraging.
> >
> > Until then, it's Graceful Restart for now (which, for better
> > or worse, we've been happy with these past 2 years).
>
> It's been slowly trickling in over the last couple years, with each new
> code rev. The last protocol support we were waiting on for NSR was RSVP,
> which finally came in 9.6. Of course we aren't running 9.6 anywhere yet,
> but NSR+ISSU has been working reasonably well in tests of prior versions
> with only modest RSVP traffic disruptions, which is still better than
> nothing. The only thing to note is that the process takes FOREVER to
> complete, reminds me of trying to do GR back in the day of RE-2.0's,
> where it took 30 minutes to sync up after a restart. When 9.6R3 comes
> out we're planning to start limited deployment experiments in a couple
> select test sites, and with any luck our next round of widespread
> upgrades will be something where NSR+ISSU is fully working.
>
> --
> Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
> GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
> _______________________________________________
> juniper-nsp mailing list juniper-nsp [at] puck
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

toasty at dragondata

Dec 15, 2009, 9:03 PM

Post #23 of 24 (2168 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Dec 9, 2009, at 5:21 PM, Richard A Steenbergen wrote:
>
> The behavior we've always seen (from mid 7.x's until today) is that
> something seems to "block" the KRT queue while the pending changes
> keep
> piling up, then eventually whatever is causing the blockage clears and
> all the routes quickly install immediately thereafter. I saw the exact
> same behavior last night during an upgrade to 9.5R3, 263k routes stuck
> in Pending state for a hair over 10 minutes, then they all synced in
> just a few seconds. But I think it's actually getting worse, because
> in
> older versions the routes that were stuck in pending state didn't seem
> to be advertised to peers. This time it seemed to advertise the routes
> even though it didn't have them installed in hardware, resulting in
> blackholing of traffic.

I went back and forth on this forever (pestering you while doing it),
because it was affecting us badly on old M20s. My "lab" boxes would
never ever show the problem, but it would happen in on the production
routers. I finally gave up and decided to figure out what the
difference was between my production configuration and the lab
simulation by slowly changing my production config to match the nearly
identical lab config.

The problem went away when I removed a BGP session with a peer that
was extremely slow to accept routes, and we were exchanging full
tables with each other. I think it was some kind of deadlock where the
peer wasn't accepting routes because it was blocked trying to send me
stuff, and I was in the same boat. Snooping at the TCP layer, I didn't
see anything unusual except both peers ended up in a state where they
were advertising 0 window size to each other. The moment the KRT queue
cleared up, they finished exchanging routes and all was happy.

I can't say for certain that was the problem, but shutting down that
peer was a pretty reliable way to clear the KRT queue problem whenever
it happened.

-- Kevin

_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

ras at e-gerbil

Dec 16, 2009, 12:28 PM

Post #24 of 24 (2135 views)
Permalink

Re: bfd = busted failure detection :) [In reply to]

On Tue, Dec 15, 2009 at 11:03:08PM -0600, Kevin Day wrote:
>
> I went back and forth on this forever (pestering you while doing it),
> because it was affecting us badly on old M20s. My "lab" boxes would
> never ever show the problem, but it would happen in on the production
> routers. I finally gave up and decided to figure out what the
> difference was between my production configuration and the lab
> simulation by slowly changing my production config to match the nearly
> identical lab config.
>
> The problem went away when I removed a BGP session with a peer that
> was extremely slow to accept routes, and we were exchanging full
> tables with each other. I think it was some kind of deadlock where the
> peer wasn't accepting routes because it was blocked trying to send me
> stuff, and I was in the same boat. Snooping at the TCP layer, I didn't
> see anything unusual except both peers ended up in a state where they
> were advertising 0 window size to each other. The moment the KRT queue
> cleared up, they finished exchanging routes and all was happy.
>
> I can't say for certain that was the problem, but shutting down that
> peer was a pretty reliable way to clear the KRT queue problem whenever
> it happened.

What code was this? In theory shouldn't the routes be in a bgp queue
regardless of whats happening with the tcp layer? Should see if we can
reproduce this with modern hardware and code.

--
Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
_______________________________________________
juniper-nsp mailing list juniper-nsp [at] puck
https://puck.nether.net/mailman/listinfo/juniper-nsp

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads