Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: Cisco: NSP

Sharing router uplinks?

 

 

Cisco nsp RSS feed   Index | Next | Previous | View Threaded


enelsonm5 at yahoo

Aug 1, 2012, 8:23 AM

Post #1 of 12 (936 views)
Permalink
Sharing router uplinks?

I have always thought it is a best practice to not put servers or PCs on links/subnets that connect routers together. I also have always thought that router to router links should be 1:1. For example, the link from a top-of-rack or end-of-row router to the data center core should be a dedicated link. 

I have run into a situation where there is insistence that both of these practices not be observed. I am being asked to put many router uplinks on a single subnet connected to a single port on the core router. I am also being asked to put a web server on this same subnet. 

What do others think of this?  I have been unable to find anything on the web that says anything for or against. If anyone knows of authoritative guidelines on the web about this I would be very interested.


_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


saku at ytti

Aug 1, 2012, 8:46 AM

Post #2 of 12 (884 views)
Permalink
Re: Sharing router uplinks? [In reply to]

On (2012-08-01 08:23 -0700), Erik Nelson wrote:

> What do others think of this?  I have been unable to find anything on the web that says anything for or against. If anyone knows of authoritative guidelines on the web about this I would be very interested.

It is doable but not optimal. It is questionable if any benefits can be
extracted in well designed network this way.

- Usually core ports with no HQoS cost significantly less than edge ports
with HQoS. If you share core and edge, you need VLAN separation, which
means if you plan to run any QoS in core, you need HQoS cards making this
/more/ expensive than not sharing.

- You can possibly save port counts, but you lose hardware liveliness
detection, as you have switch between core devices. Which means high
convergence time (lower quality) and possibly increased complexity (BFD).

--
++ytti
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


peter.hicks at poggs

Aug 1, 2012, 9:21 AM

Post #3 of 12 (885 views)
Permalink
Re: Sharing router uplinks? [In reply to]

On 1 Aug 2012, at 16:23, Erik Nelson wrote:

> I have run into a situation where there is insistence that both of these practices not be observed. I am being asked to put many router uplinks on a single subnet connected to a single port on the core router. I am also being asked to put a web server on this same subnet.
>
> What do others think of this? I have been unable to find anything on the web that says anything for or against. If anyone knows of authoritative guidelines on the web about this I would be very interested.


Which of the two devices do you set as the default gateway on the web server? (rhetorical question)


Peter


_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


enelsonm5 at yahoo

Aug 1, 2012, 9:48 AM

Post #4 of 12 (887 views)
Permalink
Re: Sharing router uplinks? [In reply to]

The gateway is one of my issues with it.  I can't really tell what they expect. They might be expecting that there is no traffic to the server from the downstream side (a bad assumption), or that the core router will just "handle it" (icmp redirect maybe, except I like to turn that off).



----- Original Message -----
From: Peter Hicks <peter.hicks [at] poggs>
To: Erik Nelson <enelsonm5 [at] yahoo>
Cc: "cisco-nsp [at] puck" <cisco-nsp [at] puck>
Sent: Wednesday, August 1, 2012 12:21 PM
Subject: Re: [c-nsp] Sharing router uplinks?


On 1 Aug 2012, at 16:23, Erik Nelson wrote:

> I have run into a situation where there is insistence that both of these practices not be observed. I am being asked to put many router uplinks on a single subnet connected to a single port on the core router. I am also being asked to put a web server on this same subnet.
>
> What do others think of this?  I have been unable to find anything on the web that says anything for or against. If anyone knows of authoritative guidelines on the web about this I would be very interested.


Which of the two devices do you set as the default gateway on the web server?  (rhetorical question)


Peter

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


enelsonm5 at yahoo

Aug 1, 2012, 9:48 AM

Post #5 of 12 (891 views)
Permalink
Re: Sharing router uplinks? [In reply to]

The gateway is one of my issues with it.  I can't really tell what they expect. They might be expecting that there is no traffic to the server from the downstream side (a bad assumption), or that the core router will just "handle it" (icmp redirect maybe, except I like to turn that off).



----- Original Message -----
From: Peter Hicks <peter.hicks [at] poggs>
To: Erik Nelson <enelsonm5 [at] yahoo>
Cc: "cisco-nsp [at] puck" <cisco-nsp [at] puck>
Sent: Wednesday, August 1, 2012 12:21 PM
Subject: Re: [c-nsp] Sharing router uplinks?


On 1 Aug 2012, at 16:23, Erik Nelson wrote:

> I have run into a situation where there is insistence that both of these practices not be observed. I am being asked to put many router uplinks on a single subnet connected to a single port on the core router. I am also being asked to put a web server on this same subnet.
>
> What do others think of this?  I have been unable to find anything on the web that says anything for or against. If anyone knows of authoritative guidelines on the web about this I would be very interested.


Which of the two devices do you set as the default gateway on the web server?  (rhetorical question)


Peter

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


SeniorJ at bennettjones

Aug 1, 2012, 1:53 PM

Post #6 of 12 (880 views)
Permalink
Re: Sharing router uplinks? [In reply to]

Putting a web server (or any other) host device on the same subnet causes reachability issues to other subnets -- hacks/workarounds include ICMP redirects, static routing tables, and proxy arp on the subnet. A server won't know which 'router' to take to get to which subnet. This is an administrative disaster as you have to either permit ICMP redirects explicitly (Operating systems shouldn't/don't support this by default anymore), turn on evil proxy arp, have a full mesh IGP, or enable static routes on the hosts.

Hosts should only have a single exit point out of a subnet, through a router(or two, using FHRP).

As far as shared 'router' vlans or subnets, this is completely normal and common for distribution/core networks.

-JP Senior

-----Original Message-----
From: cisco-nsp-bounces [at] puck [mailto:cisco-nsp-bounces [at] puck] On Behalf Of Erik Nelson
Sent: 01 August 2012 9:23 AM
To: cisco-nsp [at] puck
Subject: [c-nsp] Sharing router uplinks?

I have always thought it is a best practice to not put servers or PCs on links/subnets that connect routers together. I also have always thought that router to router links should be 1:1. For example, the link from a top-of-rack or end-of-row router to the data center core should be a dedicated link. 

I have run into a situation where there is insistence that both of these practices not be observed. I am being asked to put many router uplinks on a single subnet connected to a single port on the core router. I am also being asked to put a web server on this same subnet. 

What do others think of this?  I have been unable to find anything on the web that says anything for or against. If anyone knows of authoritative guidelines on the web about this I would be very interested.


_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
The contents of this message may contain confidential and/or privileged
subject matter. If this message has been received in error, please contact
the sender and delete all copies. Like other forms of communication,
e-mail communications may be vulnerable to interception by unauthorized
parties. If you do not wish us to communicate with you by e-mail, please
notify us at your earliest convenience. In the absence of such
notification, your consent is assumed. Should you choose to allow us to
communicate by e-mail, we will not take any additional security measures
(such as encryption) unless specifically requested.


_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


wayne at tuckerlabs

Aug 2, 2012, 8:18 AM

Post #7 of 12 (861 views)
Permalink
Re: Sharing router uplinks? [In reply to]

On Wed, Aug 1, 2012 at 1:53 PM, JP Senior <SeniorJ [at] bennettjones> wrote:
> Putting a web server (or any other) host device on the same subnet causes reachability issues to
> other subnets -- hacks/workarounds include ICMP redirects, static routing tables, and proxy arp on
> the subnet. A server won't know which 'router' to take to get to which subnet. This is an
> administrative disaster as you have to either permit ICMP redirects explicitly (Operating systems
> shouldn't/don't support this by default anymore), turn on evil proxy arp, have a full mesh IGP, or
> enable static routes on the hosts.

Things also get ugly if that web server is hijacked - with a little
ARP spoofing it can get access to transit traffic.


> As far as shared 'router' vlans or subnets, this is completely normal and common for
> distribution/core networks.

I've found that a lot of NMSs don't handle the shared segments well.
Point to point links are easy to plot and monitor - both because
they're 1:1 and because if your IGP does adjacencies you can monitor
for neighbors != 1.

:w
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


scott at granados-llc

Aug 2, 2012, 8:23 AM

Post #8 of 12 (861 views)
Permalink
Re: Sharing router uplinks? [In reply to]

I second this, I think that point to point links make for the best connection type in this use for all the reasons mentioned ands also for the simplicity. There's something to be said for keeping the core (and network) as simple as possible as long as the functionality is there. I don't see what the original posters customer gains with the design and I think in general it's all downside.

On Aug 2, 2012, at 11:18 AM, Wayne Tucker <wayne [at] tuckerlabs> wrote:

> On Wed, Aug 1, 2012 at 1:53 PM, JP Senior <SeniorJ [at] bennettjones> wrote:
>> Putting a web server (or any other) host device on the same subnet causes reachability issues to
>> other subnets -- hacks/workarounds include ICMP redirects, static routing tables, and proxy arp on
>> the subnet. A server won't know which 'router' to take to get to which subnet. This is an
>> administrative disaster as you have to either permit ICMP redirects explicitly (Operating systems
>> shouldn't/don't support this by default anymore), turn on evil proxy arp, have a full mesh IGP, or
>> enable static routes on the hosts.
>
> Things also get ugly if that web server is hijacked - with a little
> ARP spoofing it can get access to transit traffic.
>
>
>> As far as shared 'router' vlans or subnets, this is completely normal and common for
>> distribution/core networks.
>
> I've found that a lot of NMSs don't handle the shared segments well.
> Point to point links are easy to plot and monitor - both because
> they're 1:1 and because if your IGP does adjacencies you can monitor
> for neighbors != 1.
>
> :w
> _______________________________________________
> cisco-nsp mailing list cisco-nsp [at] puck
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/


_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


mark.tinka at seacom

Oct 1, 2012, 6:29 AM

Post #9 of 12 (621 views)
Permalink
Re: Sharing router uplinks? [In reply to]

On Thursday, August 02, 2012 05:23:51 PM Scott Granados
wrote:

> I second this, I think that point to point links make for
> the best connection type in this use for all the reasons
> mentioned ands also for the simplicity. There's
> something to be said for keeping the core (and network)
> as simple as possible as long as the functionality is
> there. I don't see what the original posters customer
> gains with the design and I think in general it's all
> downside.

Agree that point-to-point links between your core and edge
devices are simpler, but they add considerable cost to your
core router, particularly if the individual links are not
fully utilized.

It "may" also stem from the fact that between 100Mbps
Ethernet switches and Gig-E Ethernet switches, ATM formed
many a switched core backbone on the Internet back in the
day, in order to leapfrog the 100Mbps barrier. Of course,
those had to be point-to-point links, but I digress...

I've run core environments based on few 10Gbps ports between
the core routers and switches, and multiple Gig-E or 10Gbps
ports between the edge and core switches, on shared subnets.
No issues to complain about, be they monitoring, be they
convergence. Depending on PoP size, switches have ranged
from big chassis-based units to 1U desktop-sized platforms.
But I will concede that this type of topology is not
immediately intuitive in core routing environments, and I
have come across such situations also.

An academic argument would also be that you use up a lot
more IP addresses than you might have in point-to-point
topologies, depending on the scale. But like I said,
"academic" :-).

Cheers,

Mark.
Attachments: signature.asc (0.82 KB)


mark.tinka at seacom

Oct 1, 2012, 6:29 AM

Post #10 of 12 (621 views)
Permalink
Re: Sharing router uplinks? [In reply to]

On Wednesday, August 01, 2012 05:46:30 PM Saku Ytti wrote:

> It is doable but not optimal. It is questionable if any
> benefits can be extracted in well designed network this
> way.

I've had discussions with other operators in the past, who
have been used to have a direct interconnect between routers
in the PoP, and optionally, links to an Ethernet switch to
aggregate downstream devices.

I've tended to re-use the switched capacity and ignore the
point-to-point link between the routers, at all bandwidth
levels. Been doing that for the last several years, and no
major drama - especially if you're not fully utilizing the
ports on a point-to-point link.

Certain topologies will require traffic mapping between
routers more consistently, but those would be few and far
between at the scale points affecting this decision,
depending on the operation, e.g., MPLS-TE, IGP-based load
sharing from upstream/downstream of the core network, e.t.c.

> - You can possibly save port counts,...

Which is not an unreasonable goal, particularly if ports
cost additional money, space or power that one might not
have.

> but you lose
> hardware liveliness detection, as you have switch
> between core devices. Which means high convergence time
> (lower quality) and possibly increased complexity (BFD).

I've been running such topologies for several years, and IS-
IS + BFD have not inconvenienced me once, whether in drills
or real failure scenarios.

Cheers,

Mark.
Attachments: signature.asc (0.82 KB)


mark.tinka at seacom

Oct 1, 2012, 6:30 AM

Post #11 of 12 (621 views)
Permalink
Re: Sharing router uplinks? [In reply to]

On Thursday, August 02, 2012 05:18:54 PM Wayne Tucker wrote:

> I've found that a lot of NMSs don't handle the shared
> segments well. Point to point links are easy to plot and
> monitor - both because they're 1:1 and because if your
> IGP does adjacencies you can monitor for neighbors != 1.

Agree, but it's a compromise between what your NMS complains
about and whether you can afford such a topology.

Imagine situations where some of the routers in the core are
busier than others, and with a point-to-point topology, you
have to provision multiple links (probably as LAG's) between
the core router and some edge routers, while the rest will
run as single or fewer links. And all the while, you have
certain ports on the core router that are heavily
underutilized.

The problem becomes worse if your PoP grows with a lot more
such devices. Perhaps you host a CDN, it's a video head end,
it's your only PoP so all customers terminate there, e.t.c.

Cheers,

Mark.
Attachments: signature.asc (0.82 KB)


mark.tinka at seacom

Oct 1, 2012, 6:30 AM

Post #12 of 12 (630 views)
Permalink
Re: Sharing router uplinks? [In reply to]

On Wednesday, August 01, 2012 05:23:05 PM Erik Nelson wrote:

> I have always thought it is a best practice to not put
> servers or PCs on links/subnets that connect routers
> together. I also have always thought that router to
> router links should be 1:1. For example, the link from a
> top-of-rack or end-of-row router to the data center core
> should be a dedicated link.
>
> I have run into a situation where there is insistence
> that both of these practices not be observed. I am being
> asked to put many router uplinks on a single subnet
> connected to a single port on the core router. I am also
> being asked to put a web server on this same subnet.
>
> What do others think of this? I have been unable to find
> anything on the web that says anything for or against.
> If anyone knows of authoritative guidelines on the web
> about this I would be very interested.

Within a PoP, I've normally had 2x core switches. All edge
and peering routers will share the same VLAN on each core
switch, but different VLAN's between the switches (the
switches are interlinked, but this is superfluous, as
traffic never follows that path anyway due to all routers
multihoming to both switches).

The core routers also connect to the core switches, into the
appropriate VLAN on each core switch in order to be on the
same subnet as the edge and peering routers on that
particular switch.

The reason we use VLAN's on the core switches is because in
some PoP's, there are border routers too. Those will live in
a separate VLAN from the edge and peering network, along
with the core router interfaces to match (in the past, a PoP
would have 4x core switches to support this, but routers
have now gotten more powerful and major upstream providers
all support 10Gbps hand-offs now).

So yes, I agree that sharing a subnet between your
infrastructure makes sense, i.e., between your core routers,
border routers, edge routers, peering routers, e.t.c.,
particularly if you find that you're massively
underutilizing port bandwidth in a point-to-point topology.

However, I also wouldn't recommend placing your services on
the same subnet as your core infrastructure, particularly if
you can afford not to.

What I have done at previous employers is to have a so-
called "Production" router, which hooks into the core switch
also. This router is responsible for aggregating downstream
traffic from services-oriented infrastructure, e.g., DNS,
RADIUS, FTP, HTTP/HTTPS, SMTP, POP3, DHCP, e.t.c. This way,
you keep your services separate from your transport network,
in case it makes sense for you to do so.

In smaller networks with little cash to spend, everything is
collapsed into a single router, including transit and
peering traffic, customer edge termination, services
aggregation, e.t.c. It will work, but presents challenges
that only you will discover down the line.

Hope this helps.

Cheers,

Mark.
Attachments: signature.asc (0.82 KB)

Cisco nsp RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.