Mailing List Archive: NANOG: users

Resilience - How many BGP providers

Index | Next | Previous | View Threaded

adel at baklawasecrets

Nov 11, 2009, 9:14 AM

Post #1 of 8 (651 views)
Permalink

Resilience - How many BGP providers

Hi,

After recent discussions on the list, I've been thinking about the affects
of multiple BGP feeds to the overall resilience of Internet connectivity
for my organisation.Ā So originally when I looked at the design
proposals, there was a provision in there for four connections with the
same Internet provider.Ā Thinking about it and with the valuable input of
members on this list, it was obvious that multiple connections from the
same provider defeated the aim of providing resilience.

So having come to the decision to use two providers and BGP peer with
both, I'm wondering how much more resilience I would get by peering
withĀ more than twoĀ providers.Ā So will it significantly increase my
resilience by peering with three providers for example, as both of the
upstreams I choose will be multihomed to other providers.Ā Especially as
I am only looking at peering out of the UK.

Hope the above makes sense.

Adel

dylan.ebner at crlmed

Nov 11, 2009, 9:27 AM

Post #2 of 8 (617 views)
Permalink

RE: Resilience - How many BGP providers [In reply to]

You question has many caveats. Just having two providers does not necessarily get you more resiliency. If you have two providers and they are terminating on the same router, then you still have a SPOF problem. You also need to look at pysical paths as well. If you have two (or three) providers and they are using a common carrier, then you have a problem as well. For example, GLBX has a small prescence in the Minneapolis metro. If I were to use them as a provider, they would use Qwest as a last mile. If my other provider is Qwest (which it is), I may not have path divergence.
Facilities are important too. We have three upstreams; Qwest, MCI and ATT. The facility only has two entrances, so that means two of these are in the same conduit. IF you only have one entrance, all you connections are going to run through that conduit, and that makes you susceptable to a rouge backhoe.

You are on the right track to question your resilancy. Some upstreams can offer good resilancy with multiple feeds. Others cannot. I would start with your provider and see what you are getting. Maybe you already have path divergence, sperate last miles, and multiple paths in the isp core. If you go with multiple providers, you want to make sure you don't risk losing something you already have.

-----Original Message-----
From: adel [at] baklawasecrets [mailto:adel [at] baklawasecrets]
Sent: Wednesday, November 11, 2009 11:14 AM
To: nanog [at] nanog
Subject: Resilience - How many BGP providers

Hi,

After recent discussions on the list, I've been thinking about the affects
of multiple BGP feeds to the overall resilience of Internet connectivity
for my organisation. So originally when I looked at the design
proposals, there was a provision in there for four connections with the
same Internet provider. Thinking about it and with the valuable input of
members on this list, it was obvious that multiple connections from the
same provider defeated the aim of providing resilience.

So having come to the decision to use two providers and BGP peer with
both, I'm wondering how much more resilience I would get by peering
with more than two providers. So will it significantly increase my
resilience by peering with three providers for example, as both of the
upstreams I choose will be multihomed to other providers. Especially as
I am only looking at peering out of the UK.

Hope the above makes sense.

Adel

jay at west

Nov 11, 2009, 9:49 AM

Post #3 of 8 (610 views)
Permalink

Re: Resilience - How many BGP providers [In reply to]

Dylan Ebner wrote:
> IF you only have one entrance, all you connections are going to run through that conduit, and that makes you susceptable to a rouge backhoe.

Not just the rouge ones. The big yellow ones are far more common and
can do just as much damage.

--
Jay Hennigan - CCIE #7880 - Network Engineering - jay [at] impulse
Impulse Internet Service - http://www.impulse.net/
Your local telephone and internet company - 805 884-6323 - WB6RDV

scg at gibbard

Nov 11, 2009, 11:18 AM

Post #4 of 8 (608 views)
Permalink

Re: Resilience - How many BGP providers [In reply to]

The thing to remember about redundancy is that it's a statistical game
rather than a magic formula.

You can be reasonably sure that any single component will go down at some
point. Nothing works perfectly. Few things last forever.

If you have two fairly reliable components, and if they're suffciently
isolated from eachother that they won't be broken by the same event, it's
much less likely that they'll both break at the same time. That means
that if one breaks, and you're not unlucky, you'll have time to fix it
before the other breaks.

If you have three components, the chances of all three being broken at
once are even less than the chances of two of them being broken at once.
With four, you're even safer, and so on and so forth. But once you get
beyond two, you hit a point of diminishing returns pretty quickly.

That doesn't mean you should always do two of any given component. Some
things may be so important that you're not willing to take that level of
risk and are willing to spend significantly more money to get a small
amount more protection. Some things may be sufficiently unimportant that
you're willing to deal with occasional outages, and you can get by without
a spare (few people -- with obvious exceptions who we don't need to hear
about right now -- have fully redundant home connectivity, for instance).
It's just a matter of understanding the risks, and doing the cost-benefit
analysis to determine how much protection you need and how much you're
willing to pay for it.

-Steve

On Wed, 11 Nov 2009, adel [at] baklawasecrets wrote:

>
>
> Hi,
>
> After recent discussions on the list, I've been thinking about the affects
> of multiple BGP feeds to the overall resilience of Internet connectivity
> for my organisation.Ā So originally when I looked at the design
> proposals, there was a provision in there for four connections with the
> same Internet provider.Ā Thinking about it and with the valuable input of
> members on this list, it was obvious that multiple connections from the
> same provider defeated the aim of providing resilience.
>
> So having come to the decision to use two providers and BGP peer with
> both, I'm wondering how much more resilience I would get by peering
> withĀ more than twoĀ providers.Ā So will it significantly increase my
> resilience by peering with three providers for example, as both of the
> upstreams I choose will be multihomed to other providers.Ā Especially as
> I am only looking at peering out of the UK.
>
> Hope the above makes sense.
>
> Adel
>
>

adel at baklawasecrets

Nov 11, 2009, 3:41 PM

Post #5 of 8 (607 views)
Permalink

RE: Resilience - How many BGP providers [In reply to]

I suppose I could take the whole resilience thing further and further and further. One of the replies used a phrase which I thing captured the problem quite nicely: "diminishing returns".
Basically I could spend lots and lots of money to try and eliminate all single points of failure. Clearly I don't have the money to do this and what I'm really trying to establish is at what
point do the returns start to diminish with regards to obtaining multiple transit providers. The answer appears to be "it depends". So if getting a third BGP peering with divergent paths,
separate last mile, separate facility and separate router will increase costs by 5x but only increase resilience by 0.001% is it really worth it? I'm trying to quantify the resilience of my
Internet connectivity and quantify the effects of adding more providers. Now to run through my case:

- I have one facility to locate BGP routers at. Thats not changing for the moment.
- I can afford two BGP routers.
- The facility I'm located at tell me they have divergent fibre paths and multiple entries into the facility. (Still need to verify this by getting them to walk the routes with me)
- I am going to take transit from two upstreams.
- I could ask the question as to whether I can peer with separate routers on each of the upstreams. i.e. to protect against router failures on their side.
- I will make sure that neither upstream peers with the other directly. (Does this give me some AS path redundancy?)

So from the above:

- I have no resilience with regards to datacentre location. i.e. if a plane fell out of the sky etc., I'm done.
- I can afford some BGP router resilience on my side. So I should be able to continue working if a router failure which only affects one of my routers occurs.
- I have some resilience in terms of actual fibre paths to the facilites where I will be picking up the BGP feeds from. (to be verified)
- I have some "AS resilience" if this is the right term. So if the AS of one of my upstreams drops off the face of the Internet, I can still get to the Internet through the AS of my other
provider
- Peering with separate routers may give me some resilience for router failure on the side of my upstreams? (not totally sure on this)

In this situation, if I add another peering with another upstream, am I really getting much return in terms of resilience? Or should I spend this money examining the many other SPOFs in
my architecture? I'm perfectly sure there is absolutely no point me peering with 6 providers, but maybe some gains in peering with 3? I'm trying to figure out at what point is adding
another peering in my case a waste of money.

I haven't gone into switch and power redundancy, because I "think" I understand it. I wanted to concentrate on the multiple upstreams question. Heads starting to whirl right about now.

Adel

On Wed 5:27 PM , "Dylan Ebner" dylan.ebner [at] crlmed sent:
>
> You question has many caveats. Just having two providers does not
> necessarily get you more resiliency. If you have two providers and they are
> terminating on the same router, then you still have a SPOF problem. You
> also need to look at pysical paths as well. If you have two (or three)
> providers and they are using a common carrier, then you have a problem as
> well. For example, GLBX has a small prescence in the Minneapolis metro. If
> I were to use them as a provider, they would use Qwest as a last mile. If
> my other provider is Qwest (which it is), I may not have path
> divergence.Facilities are important too. We have three upstreams; Qwest, MCI and ATT.
> The facility only has two entrances, so that means two of these are in the
> same conduit. IF you only have one entrance, all you connections are going
> to run through that conduit, and that makes you susceptable to a rouge
> backhoe.
> You are on the right track to question your resilancy. Some upstreams can
> offer good resilancy with multiple feeds. Others cannot. I would start with
> your provider and see what you are getting. Maybe you already have path
> divergence, sperate last miles, and multiple paths in the isp core. If you
> go with multiple providers, you want to make sure you don't risk losing
> something you already have.
>
>
>
> -----Original Message-----
> From: adel [at] baklawasecrets [adel@
> baklawasecrets.com] Sent: Wednesday, November 11, 2009 11:14 AM
> To: nanog [at] nanog
> rgSubject: Resilience - How many BGP providers
>
>
>
> Hi,
>
> After recent discussions on the list, I've been thinking about the
> affectsof multiple BGP feeds to the overall resilience of Internet
> connectivityfor my organisation.Ā So originally when I looked at the design
> proposals, there was a provision in there for four connections with the
> same Internet provider.Ā Thinking about it and with the valuable input
> ofmembers on this list, it was obvious that multiple connections from the
> same provider defeated the aim of providing resilience.
>
> So having come to the decision to use two providers and BGP peer with
> both, I'm wondering how much more resilience I would get by peering
> withĀ more than twoĀ providers.Ā So will it significantly
> increase myresilience by peering with three providers for example, as both of the
> upstreams I choose will be multihomed to other providers.Ā Especially
> asI am only looking at peering out of the UK.
>
> Hope the above makes sense.
>
> Adel
>
>
>
>
>

hiersd at gmail

Nov 11, 2009, 4:05 PM

Post #6 of 8 (605 views)
Permalink

Re: Resilience - How many BGP providers [In reply to]

It is wise to stack the deck in your favor, but you'll never really
know how much real redundancy you've purchased:

http://www.atis.org/ndai/ATIS_NDAI_Final_Report_2006.pdf

David

On Wed, Nov 11, 2009 at 3:41 PM, <adel [at] baklawasecrets> wrote:
> I suppose I could take the whole resilience thing further and further and further. One of the replies used a phrase which I thing captured the problem quite nicely: "diminishing returns".
> Basically I could spend lots and lots of money to try and eliminate all single points of failure. Clearly I don't have the money to do this and what I'm really trying to establish is at what
> point do the returns start to diminish with regards to obtaining multiple transit providers. The answer appears to be "it depends". So if getting a third BGP peering with divergent paths,
> separate last mile, separate facility and separate router will increase costs by 5x but only increase resilience by 0.001% is it really worth it? I'm trying to quantify the resilience of my
> Internet connectivity and quantify the effects of adding more providers. Now to run through my case:
>
> - I have one facility to locate BGP routers at. Thats not changing for the moment.
> - I can afford two BGP routers.
> - The facility I'm located at tell me they have divergent fibre paths and multiple entries into the facility. (Still need to verify this by getting them to walk the routes with me)
> - I am going to take transit from two upstreams.
> - I could ask the question as to whether I can peer with separate routers on each of the upstreams. i.e. to protect against router failures on their side.
> - I will make sure that neither upstream peers with the other directly. (Does this give me some AS path redundancy?)
>
> So from the above:
>
> - I have no resilience with regards to datacentre location. i.e. if a plane fell out of the sky etc., I'm done.
> - I can afford some BGP router resilience on my side. So I should be able to continue working if a router failure which only affects one of my routers occurs.
> - I have some resilience in terms of actual fibre paths to the facilites where I will be picking up the BGP feeds from. (to be verified)
> - I have some "AS resilience" if this is the right term. So if the AS of one of my upstreams drops off the face of the Internet, I can still get to the Internet through the AS of my other
> provider
> - Peering with separate routers may give me some resilience for router failure on the side of my upstreams? (not totally sure on this)
>
> In this situation, if I add another peering with another upstream, am I really getting much return in terms of resilience? Or should I spend this money examining the many other SPOFs in
> my architecture? I'm perfectly sure there is absolutely no point me peering with 6 providers, but maybe some gains in peering with 3? I'm trying to figure out at what point is adding
> another peering in my case a waste of money.
>
> I haven't gone into switch and power redundancy, because I "think" I understand it. I wanted to concentrate on the multiple upstreams question. Heads starting to whirl right about now.
>
> Adel
>
>
> On Wed 5:27 PM , "Dylan Ebner" dylan.ebner [at] crlmed sent:
>>
>> You question has many caveats. Just having two providers does not
>> necessarily get you more resiliency. If you have two providers and they are
>> terminating on the same router, then you still have a SPOF problem. You
>> also need to look at pysical paths as well. If you have two (or three)
>> providers and they are using a common carrier, then you have a problem as
>> well. For example, GLBX has a small prescence in the Minneapolis metro. If
>> I were to use them as a provider, they would use Qwest as a last mile. If
>> my other provider is Qwest (which it is), I may not have path
>> divergence.Facilities are important too. We have three upstreams; Qwest, MCI and ATT.
>> The facility only has two entrances, so that means two of these are in the
>> same conduit. IF you only have one entrance, all you connections are going
>> to run through that conduit, and that makes you susceptable to a rouge
>> backhoe.
>> You are on the right track to question your resilancy. Some upstreams can
>> offer good resilancy with multiple feeds. Others cannot. I would start with
>> your provider and see what you are getting. Maybe you already have path
>> divergence, sperate last miles, and multiple paths in the isp core. If you
>> go with multiple providers, you want to make sure you don't risk losing
>> something you already have.
>>
>>
>>
>> -----Original Message-----
>> From: adel [at] baklawasecrets [adel@
>> baklawasecrets.com] Sent: Wednesday, November 11, 2009 11:14 AM
>> To: nanog [at] nanog
>> rgSubject: Resilience - How many BGP providers
>>
>>
>>
>> Hi,
>>
>> After recent discussions on the list, I've been thinking about the
>> affectsof multiple BGP feeds to the overall resilience of Internet
>> connectivityfor my organisation. So originally when I looked at the design
>> proposals, there was a provision in there for four connections with the
>> same Internet provider. Thinking about it and with the valuable input
>> ofmembers on this list, it was obvious that multiple connections from the
>> same provider defeated the aim of providing resilience.
>>
>> So having come to the decision to use two providers and BGP peer with
>> both, I'm wondering how much more resilience I would get by peering
>> with more than two providers. So will it significantly
>> increase myresilience by peering with three providers for example, as both of the
>> upstreams I choose will be multihomed to other providers. Especially
>> asI am only looking at peering out of the UK.
>>
>> Hope the above makes sense.
>>
>> Adel
>>
>>
>>
>>
>>
>
>
>

tore at linpro

Nov 12, 2009, 12:11 AM

Post #7 of 8 (598 views)
Permalink

Re: Resilience - How many BGP providers [In reply to]

* adel [at] baklawasecrets

> - I could ask the question as to whether I can peer with separate
> routers on each of the upstreams. i.e. to protect against router
> failures on their side.

If you're getting transit from two different upstreams, you're pretty
much guaranteed to be connected to two different routers. Unless you're
thinking about establishing redundant connections to each provider, that is.

What you should ensure, though, is that the PoPs of the two upstreams
are not found in the same physical building (or neighbourhood for that
matter), and that the fibres that connects you to those PoPs never
cross - it doesn't really help that much with two trenches on each side
of your building if the paths converge 1km away from it. You might also
want to consider getting the fibres from two different providers to
guard against contract-related disputes, unexpected bankruptcies, or
similar that would cause the fibre provider to terminating/suspending
your service.

> - I will make sure that neither upstream peers with the other
> directly.

This does not make any sense, if you're talking about peering. Peering
is a good thing for reliability and performance. I see from the rest of
your e-mail that you're mixing up the terms peering and transit, though,
so if you're taking about your provider A purchasing transit from
provider B, it makes perfect sense - at least if provider A is _only_
getting transit from B. If on the other hand provider A is getting
transit from C, D, and E in addition to B, it's not really a problem.

It might also be the case that A and B both get transit from C only,
which would make C a single point of failure for you.

Best regards,
--
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

ras at e-gerbil

Nov 12, 2009, 11:44 AM

Post #8 of 8 (591 views)
Permalink

Re: Resilience - How many BGP providers [In reply to]

On Wed, Nov 11, 2009 at 11:18:20AM -0800, Steve Gibbard wrote:
> If you have three components, the chances of all three being broken at
> once are even less than the chances of two of them being broken at
> once. With four, you're even safer, and so on and so forth. But once
> you get beyond two, you hit a point of diminishing returns pretty
> quickly.

Not only that, but you have to ask yourself what are the chances that
all these extra components will become extra points of failure and
actually increase the likelihood of something going wrong. I know a lot
of folks who have gotten themselves into a lot of trouble buying transit
from everyone they can possibly buy from, thinking it will make their
network more reliable. In most cases all it does is make their network
more unstable. The more transit paths you have out there, the more
likely you are to have something flap and wipe you out w/flap dampening,
and the more likely you are to see any single event cause a massive
amount of churn. I've seen people with 8 transit providers appear to
others on the internet as though they flapped 100+ times over a single
session flap, because of all the churn as the network reconverges. More
transit providers also means more 95th calculations, and thus a higher
bill, but that is another story for another day. :)

--
Richard A Steenbergen <ras [at] e-gerbil> http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads