Mailing List Archive: NANOG: users

Failover how much complexity will it add?

1 2

View All

Index | Next | Previous | View Threaded

adel at baklawasecrets

Nov 8, 2009, 3:51 AM

Post #1 of 42 (1749 views)
Permalink

Failover how much complexity will it add?

HI,

I was recently brought onto a project where some failover is desired, but I think that the number of connections provisioned is excessive. Also hoping to get some guidance with regards to how well I can get the failover to actually work. So currently 4 X 100Mb/s Internet connections have been provisioned. One is to be used for general Internet, out of the organisation, it also terminates VPNs from remote sites belonging to the organisation and some publicly accessible servers -routed DMZ and translated IPs. Second Internet connection to be used for a separate system which has a site-to-site VPN to a third party support vendor. Internet connections 3 and 4 are currently thought of as providing backups for one and two. Both connections firewalled by a Juniper SSG of some description.

Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are:

- Should I make sure that the backup Internet connection is from a separate provider?

- How can I acheive a failover which doesn't require me to change all the remote VPN endpoints in case of a failover? Its possible to configure failover VPNs on the Junipers, which should take care of this, but how do I take care of the DMZ hosts and external translation?

- In fact I think I'm asking what are my options with regard to failover between one Internet connection and the other?

I'm hoping to figure out whether adding an extra Internet connection actually gives us that much, in fact whether it justifies the complexity and spend.

Many Thanks for your comments.

Adel

bpfankuch at cpgreeley

Nov 8, 2009, 7:23 AM

Post #2 of 42 (1706 views)
Permalink

RE: Failover how much complexity will it add? [In reply to]

>> -----Original Message-----
>> From: adel [at] baklawasecrets [mailto:adel [at] baklawasecrets]
>> Sent: Sunday, November 08, 2009 4:52 AM
>> To: nanog [at] nanog
>> Subject: Failover how much complexity will it add?
>>
>> HI,
>>
>> I was recently brought onto a project where some failover is desired, but I think that the number of connections provisioned
>> is excessive. Also hoping to get some guidance with regards to how well I can get the failover to actually work. So currently
>> 4 X 100Mb/s Internet connections have been provisioned. One is to be used for general Internet, out of the organisation, it
>> also terminates VPNs from remote sites belonging to the organisation and some publicly accessible servers -routed DMZ and
>> translated IPs. Second Internet connection to be used for a separate system which has a site-to-site VPN to a third party
>> support vendor. Internet connections 3 and 4 are currently thought of as providing backups for one and two. Both connections
>> firewalled by a Juniper SSG of some description.
>>
>> Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make
>> sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection
>> and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to
>> do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and
>> reliability issues. So my questions regarding this are:
>>
>> - Should I make sure that the backup Internet connection is from a separate provider?
>>

Yes yes yes yes a thousand times yes. Depending on the criticality of internet connectivity you should also aim to have your redundant
connections coming from a complete separate direction. Example, fiber from Level 3 come from the north in a dedicated conduit and
fiber from Verizon coming in a dedicated conduit from the south of the building. Why? Put simply we had construction ignore the
painted lines and dig up our conduit a few years back. At that point we have 4 bonded T1's from a single carrier. That was a long
couple of days... Carrier diversity is not a bad thing, spend some time shopping an additional provider. Make sure they operate their
own network for last mile, and also make sure they don’t piggyback off the same network your main carrier does anywhere locally.
Comcast Ethernet, Verizon and Cogent make great secondary connections when you need high availability. You don’t need your
secondary to have 99.999% uptime. 97% is usually good enough if it's on a separate network. I wouldn't sway from the big names
for your primary connections either.

>>
>> - How can I acheive a failover which doesn't require me to change all the remote VPN endpoints in case of a failover? Its
>> possible to configure failover VPNs on the Junipers, which should take care of this, but how do I take care of the DMZ hosts and
>> external translation?
>>

With recent experience with the Juniper SSG VPN functions put nicely they suck. VPN failover is in there, but we had issues with the
tunnel staying active for extended periods of time. Also depending on if you do a route based or a policy based VPN, it becomes so
much of a headache. We used 2 SSG550 devices as a proof of concept and the one thing which annoyed me to no end was the complete and
total crap options within then VPN configuration. When I typically set up a VPN, I use a SonicWall NSA or E-class device (yes I know
hiss boo) or an ASA. Saying that the Juniper was lacking was a complete understatement. I personally would completely avoid even
attempting VPN failover within a Juniper device. I will say they are rock solid though for generic firewall functionality, just try
to keep the config simple or they turn into giant slow dogs.

>>
>> - In fact I think I'm asking what are my options with regard to failover between one Internet connection and the other?
>>

Considering you have 4x 100mbit lines, have you looked at BGP? Even if you drop line 2 and its associated backup, you have 2x 100mbit
lines. Or even if you have 3 unique carriers with a 100mbit from each of them it makes BGP very appealing. I think this would be an
ideal situation for a BGP setup using a couple of small routers. You could probably get away with something as small as a Cisco 3825
for each connection (purely redundancy). If the Cisco name scares you Juniper routers are great as well. Don’t forget Vyatta!

If you do BGP, you have 1 VPN to configure, you have 1 tunnel to configure, there is no VPN failover configuration and hopefully you
are not pushing more than 1 subnet across the VPN otherwise you end up doing a route based VPN instead of a policy based VPN and you
will be significantly happier. That’s a Juniper headache for another day however.

>>
>> I'm hoping to figure out whether adding an extra Internet connection actually gives us that much, in fact whether it justifies the
>> complexity and spend.
>>

What you really need to know to ask this is how much is Mr. Customer going to yell and scream if someone cuts only internet connection
and it's going to be down for about 3 days while they patch the conduit and fiber.

>>
>> Many Thanks for your comments.
>>
>> Adel

jmaimon at ttec

Nov 8, 2009, 7:47 AM

Post #3 of 42 (1715 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

adel [at] baklawasecrets wrote:
> HI,
>
>
> Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are:
>

I wouldnt jump to any conclusions that everything will work properly if
you are terminating multiple connections directly on the SSG, what with
egress likely being different than the ingress, even if you are using
the same IP range (BGP) on all the links.

You could really be asking for trouble if you are planning on using a
different ISP provided IP range on each connection for each purpose.

Front it all with routers that can policy route, whether or not you also
use BGP.

Joe

sethm at rollernet

Nov 8, 2009, 9:23 AM

Post #4 of 42 (1702 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

adel [at] baklawasecrets wrote:
> HI,
>
> I was recently brought onto a project where some failover is desired, but I think that the number of connections provisioned is excessive. Also hoping to get some guidance with regards to how well I can get the failover to actually work. So currently 4 X 100Mb/s Internet connections have been provisioned. One is to be used for general Internet, out of the organisation, it also terminates VPNs from remote sites belonging to the organisation and some publicly accessible servers -routed DMZ and translated IPs. Second Internet connection to be used for a separate system which has a site-to-site VPN to a third party support vendor. Internet connections 3 and 4 are currently thought of as providing backups for one and two. Both connections firewalled by a Juniper SSG of some description.
>
> Now I couldn't get any good answers as to why Internet connections 1 and 2 need to be separate. I think the idea was to make sure that there was enough bandwidth for the third party support VPN. I feel that I can consolidate this into one connection and just use rate limiting to reserve some portion of the bandwidth on the connection and this should be fine. Now if I was to do this then I can make a case for just having one backup Internet connection. However I'm still concerned about failover and reliability issues. So my questions regarding this are:
>
> - Should I make sure that the backup Internet connection is from a separate provider?
>
> - How can I acheive a failover which doesn't require me to change all the remote VPN endpoints in case of a failover? Its possible to configure failover VPNs on the Junipers, which should take care of this, but how do I take care of the DMZ hosts and external translation?
>
> - In fact I think I'm asking what are my options with regard to failover between one Internet connection and the other?

Forget all of that and just multihome to two separate providers with
BGP. Also make sure that of the providers you choose that one is not a
customer of the other. Instant, painless redundancy. Having multiple
circuits to one provider *will not* back anything up if that provider
has an outage as they are %99.999 likely to be part of the same larger
circuit and certainly share the same infrastructure at the provider.

>
> I'm hoping to figure out whether adding an extra Internet connection actually gives us that much, in fact whether it justifies the complexity and spend.
>

Only if you calculate the cost (money, time, angry customers, etc.) of
an outage to be greater than the cost of additional connectivity.

> Many Thanks for your comments.
>

~Seth

asr+nanog at latency

Nov 8, 2009, 9:29 AM

Post #5 of 42 (1701 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

On 2009-11-08-10:23:41, Blake Pfankuch <bpfankuch [at] cpgreeley> wrote:
> Make sure they operate their own network for last mile
[...]
> I wouldn't sway from the big names for your primary connections
> either.

Because ownership of the provider/subsidiary delivering the last mile
means one hand is talking to the other, and you're going to get good
service and reliability as a result? And "big names" never have any
peering-related spats and always deliver the best possible end-user
experience, right? :-)

(Some good points further on, though important we don't lead the OP
down the wrong path or with a false sense of security there...)

-a

adel at baklawasecrets

Nov 8, 2009, 9:34 AM

Post #6 of 42 (1700 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

Thanks for all your comments guys. With regards to bgp I did
think about placing two bgp routers in front of the ssg's. However
my limited understanding makes me think that if I had two bgp
connections from different providers I would still have issues. So
I guess that if my primary Internet goes down I lose connectivity
to all the publicly addressed devices on that connection. Like
dmz hosts and so on. I would be interested to hear how this
can be avoided if at all or do I have to use the same provider.

I should add that we currently have provisioned two ssg in ha
mode. Also is terminating bgp on the ssg also an option? I really
like the flexibility of route based VPN with addresable tun interfaces.

Thanks

adel
On Sun 3:47 PM , "Joe Maimon" jmaimon [at] ttec sent:
>
>
> adel@
> baklawasecrets.com wrote:> HI,
> >
> >
> > Now I couldn't get any good answers as to why
> Internet connections 1 and 2 need to be separate. I think the idea was to
> make sure that there was enough bandwidth for the third party support VPN.
> I feel that I can consolidate this into one connection and just use rate
> limiting to reserve some portion of the bandwidth on the connection and
> this should be fine. Now if I was to do this then I can make a case for
> just having one backup Internet connection. However I'm still concerned
> about failover and reliability issues. So my questions regarding this
> are:>
>
> I wouldnt jump to any conclusions that everything will work properly if
> you are terminating multiple connections directly on the SSG, what with
> egress likely being different than the ingress, even if you are using
> the same IP range (BGP) on all the links.
>
> You could really be asking for trouble if you are planning on using a
> different ISP provided IP range on each connection for each purpose.
>
> Front it all with routers that can policy route, whether or not you also
> use BGP.
>
>
> Joe
>
>
>
>
>

John.Herbert at ins

Nov 8, 2009, 10:09 AM

Post #7 of 42 (1702 views)
Permalink

RE: Failover how much complexity will it add? [In reply to]

Seth Mattinen [sethm [at] rollernet] said:

>Forget all of that and just multihome to two separate providers with BGP
--Assuming that you're advertising PI space or can work around that appropriately with your providers, I agree, that's the ideal situation.

>Having multiple circuits to one provider *will not* back anything up if that provider
>has an outage as they are %99.999 likely to be part of the same larger circuit
--True - if you don't specify otherwise when you're ordering, then why would they make the effort? Comments made in some of the other responses in this thread are also valid even with a single service provider - diverse entry points into your facility, diverse upstream circuit routing, and homing to different POPs - which may mean backhauling your secondary circuit away from your local POP and taking a hit for the higher latency on that second link. The moral of this is that whether you're using one provider or more than one, state your diversity requirements clearly up front, and then stay involved and make sure that what's presented to you is _actually_ diverse (oldsflash: even the best intentioned people sometimes make mistakes, especially when there's a handoff to a different last mile provider who may not have been clear on the requirement ). Of course, all of this is potentially wasted effort if the data center you're providing connectivity for does not also maintain the same kind of diversity itself in terms of power, connectivity, architecture, etc.

>and certainly share the same infrastructure at the provider.
--If you enter a single provider's network at diverse points, then that local infrastructure isn't the same at least. But by the same measure, if that provider has a major BGP issue for example, then yeah - they're both screwed... in which case we loop back to the dual provider scenario you mentioned in the first place :)

Ultimately choosing the appropriate solution will boil down to the what level of service unavailability one can tolerate in the first place, and put a business value on that impact. From that one can derive technical options, then go cap in hand with a business case to the poor soul paying the bill ;-)

j.

sethm at rollernet

Nov 8, 2009, 10:19 AM

Post #8 of 42 (1703 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

adel [at] baklawasecrets wrote:
> Thanks for all your comments guys. With regards to bgp I did
> think about placing two bgp routers in front of the ssg's. However
> my limited understanding makes me think that if I had two bgp
> connections from different providers I would still have issues. So
> I guess that if my primary Internet goes down I lose connectivity
> to all the publicly addressed devices on that connection. Like
> dmz hosts and so on. I would be interested to hear how this
> can be avoided if at all or do I have to use the same provider.
>

No, you will announce the same IP addresses (minimum of a /24 which you
can easily obtain from one upstream just by saying "I want to multihome"
if you don't already have a /24) over both. That's the whole point of
multihoming. If cost is an issue you can just use one BGP speaking
router. If you multihome there is no "primary" like you're thinking.

~Seth

Valdis.Kletnieks at vt

Nov 8, 2009, 11:46 AM

Post #9 of 42 (1705 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

On Sun, 08 Nov 2009 08:23:41 MST, Blake Pfankuch said:
> I wouldn't sway from the big names for your primary connections either.

This is, of course, dependent on the OP's location and budget. I know when we
were getting our NLR connection set up, there was a fair amount of "You want
40G worth of DWDM *where*?" involved, and the resulting topology was...
complicated. At least at one time, there were places where our provider was
running our link across lambdas of a subsidiary of ours, which are going across
physical fiber owned by the provider... turtles all the way down. ;)

adel at baklawasecrets

Nov 8, 2009, 12:17 PM

Post #10 of 42 (1700 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

Thanks Seth and James,

Things are getting a lot clearer. The BGP multihoming solution sounds like exactly what I want. I have more questions :-)

Now I suppose I would get my allocation from RIPE as I am UK based?

Do I also need to apply for an AS number?

As the IP block is "mine", it is ISP independent. i.e. I can take it with me when I decide to use two completely different ISPs?

Is the obtaining of this IP block, what is referred to as PI space?

Of course internally I split the /24 up however I want - /28 for untrust range and maybe a routed DMZ block etc.?

Assuming I apply for IP block and AS number, whats involved and how long does it take to get these babies?

I know the SSG550's have BGP capabilites. As I have two of these in HA mode, does it make sense to do the BGP on these, or should I get dedicated BGP routers?

Fixing the internal routing policy so traffic is directed at the active BGP connection. Whats involved here, preferring one BGP link over the other?

Thanks again, I obviously need to do some reading of my own, but all the suggestions so far have been very valuable and definitely seem to be pointing in some
fruitful directions.

Adel

On Sun 6:31 PM , "James Hess" mysidia [at] gmail sent:
> On Sun, Nov 8, 2009 at 11:34 AM, <adel@
> baklawasecrets.com> wrote:[..]
> > connections from different providers I would
> still have issues. So> I guess that if my primary Internet goes down I
> lose connectivity> to all the publicly addressed devices on that
> connection. Like> dmz hosts and so on. I would be interested
> to hear how this> can be avoided if at all or do I have to use the
> same provider.
> You assign multi-homed IP address space to your publicly addressed
> devices,which are not specific to either ISP. You announce to both ISPs, and
> you accept some routes from both ISPs.
>
> You get multi-homed IPs, either by having an existing ARIN allocation,
> or getting a /22 from ARIN (special allocation available for
> multi-homing), or ask for a /24 from ISP A or ISP B for
> multihoming.
>
>
> If Link A fails, the BGP session eventually times out and dies: ISP
> A's BGP routers withdraw the routes, the IP addresses are then
> associated only with provider B.
>
> And you design your internal routing policy to direct traffic
> within your network to the router with an active BGP session.
>
> Link A's failure is _not_ a total non-event, but a 3-5 minute partial
> disruption, while the BGP session times out and updates occur in other
> people's routers, is minimal compared to a 3 day outage, if serious
> repairs to upstream fiber are required.
>
>
> --
> -J
>
>
>

ken.gilmour at gmail

Nov 8, 2009, 12:24 PM

Post #11 of 42 (1698 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

Hi Adel

There are companies like packet exchange (www.packetexchange.net)
(whom i have personally used) who will do all of the legwork for you,
such as applying for the ASN, address space, transit agreements, and
get the tail connections directly to your building. You just need to
pay them and buy the equipment (which they can also provide). Probably
easier in the long run.

NOTE: I am not an employee, or paid affiliate of packet exchange... I
have used them for services and am promoting them due to my own good
experiences with their services.

Regards,

Ken

2009/11/8 <adel [at] baklawasecrets>:
> Thanks Seth and James,
>
> Things are getting a lot clearer. �The BGP multihoming solution sounds like exactly what I want. �I have more questions :-)
>
> Now I suppose I would get my allocation from RIPE as I am UK based?
>
> Do I also need to apply for an AS number?
>
> As the IP block is "mine", it is ISP independent. �i.e. I can take it with me when I decide to use two completely different ISPs?
>
> Is the obtaining of this IP block, what is referred to as PI space?
>
> Of course internally I split the /24 up however I want - /28 for untrust range and maybe a routed DMZ block etc.?
>
> Assuming I apply for IP block and AS number, whats involved and how long does it take to get these babies?
>
> I know the SSG550's have BGP capabilites. �As I have two of these in HA mode, does it make sense to do the BGP on these, or should I get dedicated BGP routers?
>
> Fixing the internal routing policy so traffic is directed at the active BGP connection. �Whats involved here, preferring one BGP link over the other?
>
> Thanks again, I obviously need to do some reading of my own, but all the suggestions so far have been very valuable and definitely seem to be pointing in some
> fruitful directions.
>
> Adel
>
>
>
> On Sun � 6:31 PM , "James Hess" mysidia [at] gmail sent:
>> On Sun, Nov 8, 2009 at 11:34 AM, �<adel@
>> baklawasecrets.com> wrote:[..]
>> > connections from different providers I would
>> still have issues. �So> I guess that if my primary Internet goes down I
>> lose connectivity> to all the publicly addressed devices on that
>> connection. Like> dmz hosts and so on. �I would be interested
>> to hear how this> can be avoided if at all or do I have to use the
>> same provider.
>> You assign multi-homed IP address space to your publicly addressed
>> devices,which are not specific to either ISP. You announce to both ISPs, �and
>> you accept some routes from both ISPs.
>>
>> You get multi-homed IPs, either by having an existing ARIN allocation,
>> or getting a /22 from ARIN �(special allocation available for
>> multi-homing), or �ask for a /24 from �ISP A or ISP B �for
>> multihoming.
>>
>>
>> If �Link A fails, the BGP session eventually times out and dies: ISP
>> A's �BGP routers withdraw the routes, �the IP addresses are then
>> associated only with provider B.
>>
>> And you design your internal routing policy �to �direct �traffic
>> within your network to the router with an active BGP session.
>>
>> Link A's failure is _not_ a total non-event, �but a 3-5 minute partial
>> disruption, while the BGP session times out and updates occur in other
>> people's routers, is minimal compared to �a �3 day outage, if serious
>> repairs to upstream fiber are required.
>>
>>
>> --
>> -J
>>
>>
>>
>
>
>

adel at baklawasecrets

Nov 8, 2009, 12:54 PM

Post #12 of 42 (1697 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

Hi,

Thanks for the info on UKNOF. I've started a thread there with regards to RIPE and obtaining ASN numbers and so on., as
this is I guess quite UK specific.

Adel

On Sun 8:40 PM , Arnold Nipper <arnold [at] nipper> wrote:

> Hi Adel,
>
> On 08.11.2009 21:24 Ken Gilmour wrote
>
> > There are companies like packet exchange (www.packetexchange.net [1])
>
> I could also comment on PacketExchange, but I do not. If you get more UK
> specific now you may perhaps want to post to UKNOF
> (http://lists.uknof.org.uk/cgi-bin/mailman/listinfo/uknof/) [2] as well.
>
> For _independant_ consultancy you may want to have a look at Netsumo
> (http://www.netsumo.com/) [3] Ask for Andy Davidson.
>
> Best regards,
> Arnold
> --
> Arnold Nipper / nIPper consulting, Sandhausen, Germany
> email: arnold [at] nipper phone: +49 6224 9259 299
> mobile: +49 172 2650958 fax: +49 6224 9259 333
>
>
>
> Links:
> ------
> [1]
> http://webmail.123-reg.co.uk/parse.php?redirect=http://www.packetexchange.n
> et[2]
> http://webmail.123-reg.co.uk/parse.php?redirect=http://lists.uknof.org.uk/c
> gi-bin/mailman/listinfo/uknof/%29[3]
> http://webmail.123-reg.co.uk/parse.php?redirect=http://www.netsumo.com/%29
>
>

adel at baklawasecrets

Nov 8, 2009, 1:00 PM

Post #13 of 42 (1698 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

Don't think I sent the below to the list, so resending:

Thanks Seth and James,

Things are getting a lot clearer. The BGP multihoming solution sounds like exactly what I want. I have more questions :-)

Now I suppose I would get my allocation from RIPE as I am UK based?

Do I also need to apply for an AS number?

As the IP block is "mine", it is ISP independent. i.e. I can take it with me when I decide to use two
completely different ISPs?

Is the obtaining of this IP block, what is referred to as PI space?

Of course internally I split the /24 up however I want - /28 for untrust range and maybe a routed DMZ block
etc.?

Assuming I apply for IP block and AS number, whats involved and how long does it take to get these babies?>

I know the SSG550's have BGP capabilites. As I have two of these in HA mode, does it make sense to do the BGP
on these, or should I get dedicated BGP routers?

Fixing the internal routing policy so traffic is directed at the active BGP connection. Whats involved here,
preferring one BGP link over the other?

Thanks again, I obviously need to do some reading of my own, but all the suggestions so far have been very valuable
and definitely seem to be pointing in some fruitful directions.

Adel

On Sun 6:31 PM , James Hess <mysidia [at] gmail> wrote:

> On Sun, Nov 8, 2009 at 11:34 AM, wrote:
> [..]
> > connections from different providers I would still have issues. So
> > I guess that if my primary Internet goes down I lose connectivity
> > to all the publicly addressed devices on that connection. Like
> > dmz hosts and so on. I would be interested to hear how this
> > can be avoided if at all or do I have to use the same provider.
>
> You assign multi-homed IP address space to your publicly addressed
> devices,
> which are not specific to either ISP. You announce to both ISPs, and
> you accept some routes from both ISPs.
>
> You get multi-homed IPs, either by having an existing ARIN allocation,
> or getting a /22 from ARIN (special allocation available for
> multi-homing), or ask for a /24 from ISP A or ISP B for
> multihoming.
>
> If Link A fails, the BGP session eventually times out and dies: ISP
> A's BGP routers withdraw the routes, the IP addresses are then
> associated only with provider B.
>
> And you design your internal routing policy to direct traffic
> within your network to the router with an active BGP session.
>
> Link A's failure is _not_ a total non-event, but a 3-5 minute partial
> disruption, while the BGP session times out and updates occur in other
> people's routers, is minimal compared to a 3 day outage, if serious
> repairs to upstream fiber are required.
>
> --
> -J
>
>
>

adel at baklawasecrets

Nov 8, 2009, 1:39 PM

Post #14 of 42 (1698 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

Hi,

Ok thanks for clearing that up. I'm getting some good feedback on applying for PI and ASN through Ripe LIRs over on the UKNOF so I think I have a handle on this.
With regards to BGP and using separate BGP routers. I am announcing my PI space to my upstreams, but I don't need to carry a full Internet routing table, correct?
So I can get away with some "lightweight" BGP routers not being an ISP if that makes sense?

Adel

On Sun 9:26 PM , Ken Gilmour <ken.gilmour [at] gmail> wrote:

> Hey,
>
> Yes you apply to RIPE for your allocation. You should ask them for a
> /20 since it's the same price for that as a /24 if you can justify it
> (at least with LACNIC where i now get my allocations)...
>
> You will also need to apply for an ASN
>
> Correct- the block belongs to you and as long as you contact the
> transit provider from the address listed in WHOIS then you should be
> able to set up a new agreement easily.
>
> Yes the block is PI space (provider independent)
>
> It can take up to 1 month to get your assignments.
>
> I would recommend getting some different routers for this. I use
> OpenBSD in some of my locations which is extremely easy to work with.
> I also have some old NS-208 devices running ScreenOS for internal BGP
> in one other location. I would not recommend using any router with
> less than 1GB of RAM for BGP. in HA Mode you can connect the two
> tails, one to each SSG (if they are in active active mode) and
> announce it that way (check out anycast), we also do this :).
>
> The way BGP works is that both connections are active at the same
> time, there is no primary and backup, if one goes down you just have
> one less to receive traffic over and more traffic on the other, but
> unless you stop announcing from one connection traffic will go over
> both.
>
> Regards,
>
> Ken
>
> 2009/11/8 :
> > Don't think I sent the below to the list, so resending:
> >
> > Thanks Seth and James,
> >
> > Things are getting a lot clearer. The BGP multihoming solution
> sounds like exactly what I want. I have more questions :-)
> >
> > Now I suppose I would get my allocation from RIPE as I am UK based?
> >
> > Do I also need to apply for an AS number?
> >
> > As the IP block is "mine", it is ISP independent. i.e. I can take
> it with me when I decide to use two
> > completely different ISPs?
> >
> > Is the obtaining of this IP block, what is referred to as PI space?
> >
> > Of course internally I split the /24 up however I want - /28 for
> untrust range and maybe a routed DMZ block
> > etc.?
> >
> > Assuming I apply for IP block and AS number, whats involved and how
> long does it take to get these babies?>
> >
> > I know the SSG550's have BGP capabilites. As I have two of these in
> HA mode, does it make sense to do the BGP
> > on these, or should I get dedicated BGP routers?
> >
> > Fixing the internal routing policy so traffic is directed at the
> active BGP connection. Whats involved here,
> > preferring one BGP link over the other?
> >
> > Thanks again, I obviously need to do some reading of my own, but
> all the suggestions so far have been very valuable
> > and definitely seem to be pointing in some fruitful directions.
> >
> > Adel
> >
> >
> >
> >
> > On Sun 6:31 PM , James Hess wrote:
> >
> >> On Sun, Nov 8, 2009 at 11:34 AM, wrote:
> >> [..]
> >> > connections from different providers I would still have issues. So
> >> > I guess that if my primary Internet goes down I lose connectivity
> >> > to all the publicly addressed devices on that connection. Like
> >> > dmz hosts and so on. I would be interested to hear how this
> >> > can be avoided if at all or do I have to use the same provider.
> >>
> >> You assign multi-homed IP address space to your publicly addressed
> >> devices,
> >> which are not specific to either ISP. You announce to both ISPs, and
> >> you accept some routes from both ISPs.
> >>
> >> You get multi-homed IPs, either by having an existing ARIN allocation,
> >> or getting a /22 from ARIN (special allocation available for
> >> multi-homing), or ask for a /24 from ISP A or ISP B for
> >> multihoming.
> >>
> >> If Link A fails, the BGP session eventually times out and dies: ISP
> >> A's BGP routers withdraw the routes, the IP addresses are then
> >> associated only with provider B.
> >>
> >> And you design your internal routing policy to direct traffic
> >> within your network to the router with an active BGP session.
> >>
> >> Link A's failure is _not_ a total non-event, but a 3-5 minute partial
> >> disruption, while the BGP session times out and updates occur in other
> >> people's routers, is minimal compared to a 3 day outage, if serious
> >> repairs to upstream fiber are required.
> >>
> >> --
> >> -J
> >>
> >>
> >>
> >
> >
>
>
>

sethm at rollernet

Nov 8, 2009, 1:53 PM

Post #15 of 42 (1699 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

adel [at] baklawasecrets wrote:
> Hi,
>
> Thanks for the info on UKNOF. I've started a thread there with regards to RIPE and obtaining ASN numbers and so on., as
> this is I guess quite UK specific.
>

You will need an AS number regardless of what path you get your
addresses from to multihome. In ARIN land the minimum for a multihomed
end-site is /22, so if I were to do this here, I would ask one of the
upstreams for a /24. I don't know the first thing about RIPE policy.

~Seth

sethm at rollernet

Nov 8, 2009, 2:01 PM

Post #16 of 42 (1697 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

adel [at] baklawasecrets wrote:
> Hi,
>
> Ok thanks for clearing that up. I'm getting some good feedback on applying for PI and ASN through Ripe LIRs over on the UKNOF so I think I have a handle on this.
> With regards to BGP and using separate BGP routers. I am announcing my PI space to my upstreams, but I don't need to carry a full Internet routing table, correct?
> So I can get away with some "lightweight" BGP routers not being an ISP if that makes sense?
>

Most will give you three choices: full routes, partial routes (internal,
their customers) with default, and default only. If you can't swing full
routes then I would go for partial routes as it will at least send
traffic for each ISP and their customers directly to them rather than
randomly over the other link. It all depends on what you're going to use
as your BGP speaking platform.

~Seth

adel at baklawasecrets

Nov 8, 2009, 2:13 PM

Post #17 of 42 (1699 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

I think partial routes makes perfect sense, makes sense that traffic for customers who are connected to each of my upstreams should go out of
the correct BGP link as long as they are up! Now I need to start thinking of BGP router choices, sure I have a plethora of choices :-(

On Sun 10:01 PM , Seth Mattinen <sethm [at] rollernet> wrote:

> adel [at] baklawasecrets wrote:
> > Hi,
> >
> > Ok thanks for clearing that up. I'm getting some good feedback on
> applying for PI and ASN through Ripe LIRs over on the UKNOF so I think I
> have a handle on this.
> > With regards to BGP and using separate BGP routers. I am announcing my
> PI space to my upstreams, but I don't need to carry a full Internet
> routing table, correct?
> > So I can get away with some "lightweight" BGP routers not being an ISP
> if that makes sense?
> >
>
> Most will give you three choices: full routes, partial routes (internal,
> their customers) with default, and default only. If you can't swing full
> routes then I would go for partial routes as it will at least send
> traffic for each ISP and their customers directly to them rather than
> randomly over the other link. It all depends on what you're going to use
> as your BGP speaking platform.
>
> ~Seth
>
>
>

sethm at rollernet

Nov 8, 2009, 2:18 PM

Post #18 of 42 (1700 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

adel [at] baklawasecrets wrote:
> I think partial routes makes perfect sense, makes sense that traffic for customers who are connected to each of my upstreams should go out of
> the correct BGP link as long as they are up! Now I need to start thinking of BGP router choices, sure I have a plethora of choices :-(
>

Personally I'll always go for full routes if the router has enough
memory (software based) or TCAM space (hardware based). Cheaper to do on
software platforms though. An entry level Cisco 2811 can take full
tables from multiple upstreams with 786MB RAM or even 512. It won't push
100 meg of mixed traffic though.

~Seth

adel at baklawasecrets

Nov 8, 2009, 2:39 PM

Post #19 of 42 (1692 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

So if my requirements are as follows:

- BGP router capable of holding full Internet routing table. (whether I go for partial or full, I think I want something with full capability).

- Capable of pushing 100meg plus of mixed traffic.

What are my options? I want to exclude openbsd, or linux with quagga. Probably looking at Cisco or Juniper products, but interested
in any other alternatives people suggest. I realise this is quite a broad question, but hoping this will provide a starting point. Oh and
if I have missed any specs I should have included above, please let me know.

Thanks

Adel

On Sun 10:18 PM , Seth Mattinen <sethm [at] rollernet> wrote:

> adel [at] baklawasecrets wrote:
> > I think partial routes makes perfect sense, makes sense that traffic
> for customers who are connected to each of my upstreams should go out of
> > the correct BGP link as long as they are up! Now I need to start
> thinking of BGP router choices, sure I have a plethora of choices :-(
> >
>
> Personally I'll always go for full routes if the router has enough
> memory (software based) or TCAM space (hardware based). Cheaper to do on
> software platforms though. An entry level Cisco 2811 can take full
> tables from multiple upstreams with 786MB RAM or even 512. It won't push
> 100 meg of mixed traffic though.
>
> ~Seth
>
>
>

adel at baklawasecrets

Nov 8, 2009, 2:39 PM

Post #20 of 42 (1689 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

John.Herbert at ins

Nov 8, 2009, 2:46 PM

Post #21 of 42 (1692 views)
Permalink

RE: Failover how much complexity will it add? [In reply to]

>________________________________________
>From: adel [at] baklawasecrets [adel [at] baklawasecrets]

>- BGP router capable of holding full Internet routing table. (whether I go for partial or full,
>I think I want something with full capability).

--Capable of holding _2_ full internet routing tables if you are looking for diversity. (just being picky ;-)

j.

frederick at dahype

Nov 8, 2009, 3:30 PM

Post #22 of 42 (1689 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

There are any problems with quagga+BSD/Linux that you know or something
like that?

Or in your scenario a "cisco/juniper box" is a requirement?

I'm asking this because I'm always running BGP with upstreams providers
using quagga on BSD and everything is fine until now.

--------------------------------------------------
From: <adel [at] baklawasecrets>
Sent: Sunday, November 08, 2009 8:39 PM
To: <nanog [at] nanog>
Subject: Re: Failover how much complexity will it add?

>
> So if my requirements are as follows:
>
> - BGP router capable of holding full Internet routing table. (whether I
> go for partial or full, I think I want something with full capability).
>
> - Capable of pushing 100meg plus of mixed traffic.
>
> What are my options? I want to exclude openbsd, or linux with quagga.
> Probably looking at Cisco or Juniper products, but interested
> in any other alternatives people suggest. I realise this is quite a broad
> question, but hoping this will provide a starting point. Oh and
> if I have missed any specs I should have included above, please let me
> know.
>
> Thanks
>
> Adel

adel at baklawasecrets

Nov 8, 2009, 3:36 PM

Post #23 of 42 (1698 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

Basically the organisation that I'm working for will not have the skills in house to support a linux or bsd box. They will have trouble
with supporting the BGP configuration, however I don't think they will be happy with me if I leave them with a linux box when they
don't have linux/unix resource internally. At least with a Cisco or Juniper they are familiar with IOS and it won't be too foreign to them.

On Sun 11:30 PM , "Renato Frederick" <frederick [at] dahype> wrote:

> There are any problems with quagga+BSD/Linux that you know or something
> like that?
>
> Or in your scenario a "cisco/juniper box" is a requirement?
>
> I'm asking this because I'm always running BGP with upstreams providers
> using quagga on BSD and everything is fine until now.
>
> --------------------------------------------------
> From:
> Sent: Sunday, November 08, 2009 8:39 PM
> To:
> Subject: Re: Failover how much complexity will it add?
>
> >
> > So if my requirements are as follows:
> >
> > - BGP router capable of holding full Internet routing table. (whether I
>
> > go for partial or full, I think I want something with full capability).
> >
> > - Capable of pushing 100meg plus of mixed traffic.
> >
> > What are my options? I want to exclude openbsd, or linux with quagga.
> > Probably looking at Cisco or Juniper products, but interested
> > in any other alternatives people suggest. I realise this is quite a
> broad
> > question, but hoping this will provide a starting point. Oh and
> > if I have missed any specs I should have included above, please let me
> > know.
> >
> > Thanks
> >
> > Adel
>
>
>

adel at baklawasecrets

Nov 9, 2009, 2:53 AM

Post #24 of 42 (1675 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

You will laugh, but the budget at the moment looks like £13k. Impossible? Do only linux and openbsd solutions remain in the mix for this pittance?

On Sun 11:47 PM , Dale Rumph <daler [at] ibbs> wrote:

> What does your budget look like? A pair of Cisco 7246vxr's with G1's
> sitting on the edge of the network would be very effective and still allow
> expansion. Or you could go up to the 7609. However this gear may be
> slightly overkill. You might be ok with a 3660 enterprise and a ton of
> ram. I have done single sessions on them but not with the level of HA your
> looking for.
>
> Just my 2c
>
> ----- Original Message -----
> From: adel [at] baklawasecrets
> To: nanog [at] nanog
> Sent: Sun Nov 08 18:36:31 2009
> Subject: Re: Failover how much complexity will it add?
>
> Basically the organisation that I'm working for will not have the skills
> in house to support a linux or bsd box. They will have trouble
> with supporting the BGP configuration, however I don't think they will be
> happy with me if I leave them with a linux box when they
> don't have linux/unix resource internally. At least with a Cisco or
> Juniper they are familiar with IOS and it won't be too foreign to them.
>
> On Sun 11:30 PM , "Renato Frederick" wrote:
>
> > There are any problems with quagga+BSD/Linux that you know or something
>
> > like that?
> >
> > Or in your scenario a "cisco/juniper box" is a requirement?
> >
> > I'm asking this because I'm always running BGP with upstreams providers
>
> > using quagga on BSD and everything is fine until now.
> >
> > --------------------------------------------------
> > From:
> > Sent: Sunday, November 08, 2009 8:39 PM
> > To:
> > Subject: Re: Failover how much complexity will it add?
> >
> > >
> > > So if my requirements are as follows:
> > >
> > > - BGP router capable of holding full Internet routing table. (whether
> I
> >
> > > go for partial or full, I think I want something with full
> capability).
> > >
> > > - Capable of pushing 100meg plus of mixed traffic.
> > >
> > > What are my options? I want to exclude openbsd, or linux with quagga.
>
> > > Probably looking at Cisco or Juniper products, but interested
> > > in any other alternatives people suggest. I realise this is quite a
> > broad
> > > question, but hoping this will provide a starting point. Oh and
> > > if I have missed any specs I should have included above, please let
> me
> > > know.
> > >
> > > Thanks
> > >
> > > Adel
> >
> >
> >
>
>
>

adel at baklawasecrets

Nov 9, 2009, 3:32 AM

Post #25 of 42 (1674 views)
Permalink

Re: Failover how much complexity will it add? [In reply to]

Looking at two 100Mbit/s BGP connections, so I think I want something that will do more than 100 but nowhere close to a gig. So full routing table capability
with throughput of mixed traffic around 200Mbit/s. If that makes sense. Do the 2850s fall into that sort of price point?

Adel

On Mon 11:13 AM , Joe Abley <jabley [at] hopcount> wrote:

> On 2009-11-09, at 19:53, adel [at] baklawasecrets wrote:
>
> > You will laugh, but the budget at the moment looks like £13k.
> > Impossible? Do only linux and openbsd solutions remain in the mix
> > for this pittance?
>
> I don't see an indication of the traffic you need to push (maybe I
> deleted a message too enthusiastically) but check the 2800 series from
> cisco. The 2850 will take full tables and has gigabit interfaces, but
> don't expect them to do wire speed. Other 2800s suffer from reduced
> RAM, but perhaps you don't need full tables.
>
> Also look at Juniper J-series boxes, and maybe Force10 S-series boxes.
>
> There's a healthy market in used cisco gear in most places I have ever
> visited, if you don't need new.
>
> Joe
>
>
>

1 2

View All

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads