Mailing List Archive: Cisco: NSP

Please help clarify bus/fabric terminology on the 6500/7600

Index | Next | Previous | View Threaded

cnsp at shreddedmail

Nov 5, 2009, 11:11 PM

Post #1 of 8 (159 views)
Permalink

Please help clarify bus/fabric terminology on the 6500/7600

I'm trying to wrap my brain around Cisco's document on the 6500/7600
technology:

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd80673385.html

Terminology on the bus architecture and switch fabric are becoming less
confusing to me the more I read it, but I'm still not comfortable with my
level of understanding.
What I think the document says is:

- The 32Gbs shared bus is the path between the supervisor and individual
line cards. Line cards do not move data between each other; traffic must
pass through the Sup.
- The raw capacity of the 32Gbs bus is just that; 32Gbs across the entire
bus, combined across all cards
- The switch fabric is single or dual channel 20Gbs, dual channel just
allowing higher port/speed-density on the line cards
- The 20Gbs fabric is used to transfer traffic directly between
DFC-enabled line cards, bypassing the Sup.
- The 20Gbs fabric is not shared, each DFC line card can talk to any other
DFC line card at 20Gbs up to a potential aggregate of 720Gbs
- CEF and dCEF simply refer to whether the line card has a DFC
- CEF256 using 8Gbs of the fabric, CEF720 uses 20Gbs
- "Classic" line cards use only the 32Gbs bus.
- Usage of 8Gbs or 20Gbs on the fabric is dependent on the line card and
the Sup.
- Sup720 allows 20Gbs, others are only 8Gbs
- Mixed 8/20Gbs line cards can be used. 20Gbs is not lost if 8Gbs is
present.
- The full "720Gbs" capacity is all dual-fabric line cards with DFCs

I'm most confused on the 8Gbs limit and how it relates to the Supervisor and
line cards. Other discussions I've had indicate that some combination of
line cards can bring the whole system down to the lowest common
denominator. Am I on track? Where does oversubscription on line cards come
in? Is there something else I haven't covered?

Sorry for the laundry list. I'd rather make sure I'm clear in my head
before the design, then find a gotcha after it is too late.

Thanks!
_______________________________________________
cisco-nsp mailing list cisco-nsp[at]puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

gert at greenie

Nov 6, 2009, 1:04 AM

Post #2 of 8 (145 views)
Permalink

Re: Please help clarify bus/fabric terminology on the 6500/7600 [In reply to]

Hi,

On Thu, Nov 05, 2009 at 11:11:56PM -0800, Rick Ernst wrote:
> What I think the document says is:
[..]

As far as I understand the architecture, all of this is correct :-)

> - Mixed 8/20Gbs line cards can be used. 20Gbs is not lost if 8Gbs is
> present.
>
> I'm most confused on the 8Gbs limit and how it relates to the Supervisor and
> line cards.

65xx cards (like the WS-X6516) have an 8Gbps fabric connection, 67xx cards
(WS-X6724-SFP) have 20Gbps fabric connection.

Sup2+SFM has 8Gbps fabric.

Sup720 has 20Gbps fabric that can also run at 8Gbps - and *as far as I
understand* - this is independent among line cards, so you can have one
WS-X6516 running at 8Gbps and one WS-X6724-SFP running at 20Gbps.

> Other discussions I've had indicate that some combination of
> line cards can bring the whole system down to the lowest common
> denominator.

There's two sides to "lowest common denominator" - bus/fabric (so if you
have a Sup2 without fabric module, only shared bus for you...) - and
PFC revision. There's Sup720/3A, /3B and /3C, and all of these can come
with "-XL".

So - if you have a Sup720/3C-XL with 1 million TCAM entries and 96k MAC
table entries, and add a line card that has an DFC-3A on it, the whole
system will fall down to "3A, no XL" level -> no MPLS, 256k TCAM entries,
32k MAC table entries.

This is only relevant if you have DFCs in the system.

> Am I on track? Where does oversubscription on line cards come
> in? Is there something else I haven't covered?

Oversubscription is the next independent gotcha - for example, the
6724 card has 24 gbit ports, but only 20 gbps fabric connection
(which is not that bad, given that in practice, nobody runs all 24 ports
at 100% line rate all the time).

The 6708 10G card has 8x10 gbit externally, but only 40 gbps fabric
connection - but it has a DFC and can do local switching without
going to the fabric, so depending on your traffic pattern, it's more
or less oversubscribed...

The 6704 10G card has 40 gbps fabric connection, but a somewhat slow
internal ASIC, so it won't do more than 35 Gbit/s (or so) in total...

So you also need to take into account the specifics of the line card
you're planning to use.

> Sorry for the laundry list. I'd rather make sure I'm clear in my head
> before the design, then find a gotcha after it is too late.

The architecture *is* a bit confusing :-)

gert
--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert[at]greenie.muc.de
fax: +49-89-35655025 gert[at]net.informatik.tu-muenchen.de

cnsp at shreddedmail

Nov 6, 2009, 4:15 AM

Post #3 of 8 (140 views)
Permalink

Re: Please help clarify bus/fabric terminology on the 6500/7600 [In reply to]

Thanks (and Gert, too),

So,

- The 32Gbs bus is shared and the PFC on the sup does the forwarding
- The switch fabric is on the Sup; DFC cards use the fabric, others use the
PFC
- The fabric is limited to 8 or 20Gbs depending on Sup; CEF 256cards use
8Gbs, CEF720 uses 20gbs if the Sup supports it
- Lowest-common-denominator applies to DFC cards; you get the DFC features,
capabilities and TCAM of the least capable card/Sup, but you can
mix-and-match
- If I happen to install a 256K DFC in a 1M TCAM system, can the DFC be
forced off; 1M TCAM via the 32Gbs bus?
- The 32Gbs bus and 20Gbs fabric are total capacity; could you push
1Gbs/31Gbs on the 32Gbs bus?
- Design considerations need to include Sup level, PFC, DFC, 32Gbs shared,
8gbs/20Gbs fabric

>From a practical viewpoint, I'm currently pushing a little less than ~800Mbs
in+out at about 120Kpps. It's getting to be too much for my current software
forwarding, especially during D/DoS. A Sup720-3BXL gives me 1M routes in
TCAM, and 15Gbs/30Mpps forwarding in the PFC. Control-plane and data-plane
separation, with data-plane in hardware. I could use any combination of line
cards and still be significantly ahead of my current utilization. As the
bits get bigger and faster, I can offload forwarding onto DFC-enabled cards,
but I'd need to start with DFCs that also have the large TCAM, otherwise
I'm still using the 32Gbs bus and the PFC.

For D/DoS purposes, policing is handled in hardware at the port ASIC. If a
1Gbs-connected network were to go nuts and was throttled to 1Mbs, neither
the bus nor fabric would see the .99Gbs?

Rick

On Thu, Nov 5, 2009 at 11:42 PM, Asbjorn Hojmark <Asbjorn[at]hojmark.org>wrote:

> On Thu, 5 Nov 2009 23:11:56 -0800, you wrote:
>
> > - The raw capacity of the 32Gbs bus is just that; 32Gbs across the
> > entire bus, combined across all cards
>
> Well, actually it's 16 Gbps shared bus. (The "32G" is marketing, and
> even more so here, because it's not full duplex; it's a bus).
>
> > - The switch fabric is single or dual channel 20Gbs, dual channel
> > just allowing higher port/speed-density on the line cards
>
> Each slot gets two fabric connections, but some cards use only one of
> those. The channels can be 8 og 20G.
>
> > - The 20Gbs fabric is used to transfer traffic directly between
> > DFC-enabled line cards, bypassing the Sup.
>
> Not really. With a DFC, the lookup is done on the line card (instead
> of on the PFC on the Sup), but the forwarding is still done via the
> fabric (which is also on the Sup).
>
> > - The full "720Gbs" capacity is all dual-fabric line cards with DFCs
>
> 9 slots * 2 channels/slot * 20G/channel * 2 for marketing = 720.
>
> The bandwidth is not (directly) dependant on the precense of DFCs,
> only the forwarding capacity. (And other resources, such as NetFlow
> table space).
>
> -A
>
_______________________________________________
cisco-nsp mailing list cisco-nsp[at]puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

nick at inex

Nov 6, 2009, 4:30 AM

Post #4 of 8 (139 views)
Permalink

Re: Please help clarify bus/fabric terminology on the 6500/7600 [In reply to]

On 06/11/2009 08:11, Rick Ernst wrote:
> - The 20Gbs fabric is used to transfer traffic directly between
> DFC-enabled line cards, bypassing the Sup.

Not quite. All fabric enabled cards can transfer traffic directly to each
other, as it's a crossbar fabric. The difference between DFC and non-DFC
enabled cards is that on a DFC enabled card, the destination path lookup is
done locally on the card, whereas on a non DFC card, the internal path
lookup is done by the sup720, and the line cards use the 32Gb bus as an
out-of-band data channel for doing internal lookups. The destination path
lookup just tells the card which physical destination fabric path to use
when sending the packet from one 20G fabric channel to another.

As each packet triggers a destination lookup, on a busy box pushing many
mpps, the 32Gb bus can get saturated by lookup requests, and if this
happens you need to use DFCs to move the lookup functionality away from the
sup720 and into the line card. So using a DFC will not affect switching
speed unless you are pushing a very large number of pps.

> Other discussions I've had indicate that some combination of
> line cards can bring the whole system down to the lowest common
> denominator.

That used to be the case in certain configurations a long time ago, but not
any more.

> Am I on track? Where does oversubscription on line cards come
> in? Is there something else I haven't covered?

Oversubscription on line cards just means that there is more edge switching
capacity than the line card can actually handle. So on a 6148-ge-tx card,
the card has 48 gig ports, but in fact it can only handle 8G of traffic
(and even then, with a strong tailwind). On a fabric card, you have either
1x or 2x 20G channels from the line card into the fabric. This means that
if you have more edge bandwidth being used than fabric capacity available,
you can run into over subscription problems. In practice, this tends not
to be a problem on the 6724 / 6748 cards (whether TX or SFP), because on an
imix system, you'll statistically only rarely run into full port saturation
problems with every card on the box pushing line rate or near line rate.
Oversubscription on fabric enabled cards tends to be more of a problem with
10GE line cards for a variety of reasons - there's lots of talk about this
in the list archives, to which I refer you.

Nick
_______________________________________________
cisco-nsp mailing list cisco-nsp[at]puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

geoff at pendery

Nov 6, 2009, 5:55 AM

Post #5 of 8 (139 views)
Permalink

Re: Please help clarify bus/fabric terminology on the 6500/7600 [In reply to]

Well you're off to an excellent start. �Others have added some good
clarifications and details, but so far I don't see this one answered:

"Other discussions I've had indicate that some combination of line
cards can bring the whole system down to the lowest common
denominator."

My guess is that this is referring to the Fabric/Bus mode, for the chassis.
It's described on the link you sent, if you search to "Cisco Catalyst
6500 Architecture: Bus Switching Modes"

As Nick Hilliard explained, the bus is used, even with all fabric
cards, for communication between the Sup and the line cards. The Sup
first determines which of the three modes to use for communication.

If you have a Sup with no fabric (like Sup 1A, or Sup 2 w/o SFM, or
Sup 32) the switch will run in "Flow-Through" mode, meaning that each
time a packet is received, the entire packet is sent on the shared
bus, so it's seen by the Sup and all line cards. This will only get
you up to 15 Mpps, and a theoretical max of 32 Gbps (likely lower in
practice).

If you have a fabric Sup and fabric line cards, but at least one
Classic line card, the switch will drop into "Truncated" mode. This
is likely what someone was referring to when they told you "lowest
common denominator". The classic cards will still send the whole
packet, like in flow-through mode, but the fabric cards will send only
the headers, and send the data portion to the Sup via fabric. This is
still limited to 15 Mpps, but because the data flows via fabric, you
can squeeze some extra bandwidth out.

Lastly, if you have no Classic cards present in the chassis, it can go
into Compact mode, where only compressed headers are sent via the bus,
all data flows via fabric. This gets you up to 30 Mpps and your
theoretical 720 Gbps of total forwarding capacity.

Here's some sample output from a chassis with all fabric (and in this
case, dCEF) cards:

hostname#show fabric
show fabric active:
Active fabric card in slot 5
No backup fabric card in the system
show fabric mode:
Global switching mode is Compact
dCEF mode is not enforced for system to operate
Fabric module is not �required for system to operate
Modules are allowed to operate in bus mode
Truncated mode is allowed, due to presence of DFC module
Module Slot � � Switching Mode
�� 1 � � � � � � � � � � dCEF
�� 2 � � � � � � � � � � dCEF
�� 5 � � � � � � � � � � dCEF
�� 9 � � � � � � � � � � dCEF

-Geoff

On Fri, Nov 6, 2009 at 1:11 AM, Rick Ernst <cnsp[at]shreddedmail.com> wrote:
>
> I'm trying to wrap my brain around Cisco's document on the 6500/7600
> technology:
>
> http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd80673385.html
>
> Terminology on the bus architecture and switch fabric are becoming less
> confusing to me the more I read it, but I'm still not comfortable with my
> level of understanding.
> What I think the document says is:
>
> �- The 32Gbs shared bus is the path between the supervisor and individual
> line cards. Line cards do not move data between each other; traffic must
> pass through the Sup.
> �- The raw capacity of the 32Gbs bus is just that; 32Gbs across the entire
> bus, combined across all cards
> �- The switch fabric is single or dual channel 20Gbs, dual channel just
> allowing higher port/speed-density on the line cards
> �- The 20Gbs fabric is used to transfer traffic directly between
> DFC-enabled line cards, bypassing the Sup.
> �- The 20Gbs fabric is not shared, each DFC line card can talk to any other
> DFC line card at 20Gbs up to a potential aggregate of 720Gbs
> �- CEF and dCEF simply refer to whether the line card has a DFC
> �- CEF256 using 8Gbs of the fabric, CEF720 uses 20Gbs
> �- "Classic" line cards use only the 32Gbs bus.
> �- Usage of 8Gbs or 20Gbs on the fabric is dependent on the line card and
> the Sup.
> �- Sup720 allows 20Gbs, others are only 8Gbs
> �- Mixed 8/20Gbs line cards can be used. 20Gbs is not lost if 8Gbs is
> present.
> �- The full "720Gbs" capacity is all dual-fabric line cards with DFCs
>
> I'm most confused on the 8Gbs limit and how it relates to the Supervisor and
> line cards. �Other discussions I've had indicate that some combination of
> line cards can bring the whole system down to the lowest common
> denominator. �Am I on track? Where does oversubscription on line cards come
> in? Is there something else I haven't covered?
>
> Sorry for the laundry list. �I'd rather make sure I'm clear in my head
> before the design, then find a gotcha after it is too late.
>
> Thanks!
> _______________________________________________
> cisco-nsp mailing list �cisco-nsp[at]puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp[at]puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

gert at greenie

Nov 6, 2009, 7:13 AM

Post #6 of 8 (138 views)
Permalink

Re: Please help clarify bus/fabric terminology on the 6500/7600 [In reply to]

Hi,

On Fri, Nov 06, 2009 at 04:15:39AM -0800, Rick Ernst wrote:
> - The 32Gbs bus is shared and the PFC on the sup does the forwarding

Yes.

> - The switch fabric is on the Sup; DFC cards use the fabric, others use the
> PFC

Not exactly. Fabric-enabled cards will always use the fabric to transport
the packets directly to the destination line card. If the card has no DFC,
it will use the bus(!) to do the destination lookup via the Sup PFC.

(I'm a bit unclear on how fabric-only cards transport packets to bus-only
cards, tho).

> - If I happen to install a 256K DFC in a 1M TCAM system, can the DFC be
> forced off; 1M TCAM via the 32Gbs bus?

As far as I know, no. If you have no DFC, you need a CFC on the card.

> - The 32Gbs bus and 20Gbs fabric are total capacity; could you push
> 1Gbs/31Gbs on the 32Gbs bus?

Hmmm?

> - Design considerations need to include Sup level, PFC, DFC, 32Gbs shared,
> 8gbs/20Gbs fabric

Yes. If you have enough traffic that it matters...

> >From a practical viewpoint, I'm currently pushing a little less than ~800Mbs
> in+out at about 120Kpps. It's getting to be too much for my current software
> forwarding, especially during D/DoS. A Sup720-3BXL gives me 1M routes in
> TCAM, and 15Gbs/30Mpps forwarding in the PFC. Control-plane and data-plane
> separation, with data-plane in hardware. I could use any combination of line
> cards and still be significantly ahead of my current utilization. As the
> bits get bigger and faster, I can offload forwarding onto DFC-enabled cards,
> but I'd need to start with DFCs that also have the large TCAM, otherwise
> I'm still using the 32Gbs bus and the PFC.

Yes.

> For D/DoS purposes, policing is handled in hardware at the port ASIC. If a
> 1Gbs-connected network were to go nuts and was throttled to 1Mbs, neither
> the bus nor fabric would see the .99Gbs?

I think this depends on card type. A bus-only card has no other way to
decide what to do with the packet than "put it on the bus".

On a fabric/CFC card, you'll see the headers on the bus, but not the packets.

A DFC card will drop the packet right away.

(I might be mistaken here)

gert

--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert[at]greenie.muc.de
fax: +49-89-35655025 gert[at]net.informatik.tu-muenchen.de

cnsp at shreddedmail

Nov 6, 2009, 12:06 PM

Post #7 of 8 (128 views)
Permalink

Re: Please help clarify bus/fabric terminology on the 6500/7600 [In reply to]

Thanks everybody for all the feedback and information. Between that and the
white paper I'm starting to feel comfortable with my decision-making
process. At worst, I can start asking more intelligent questions (and be
able to vet the answers) of Cisco.

One piece that I'm still unclear on is on CEF256/fabric cards and
connectivity to the rest of the system. The white paper says:
"- CEF256: The line card in this mode supports a connection into the
32-Gbps shared bus and the switch fabric-these line cards will use the
switch fabric for data switching when the Supervisor Engine 720 is
present-if a Supervisor Engine 32 is present it will revert back to using
the 32-Gbps shared bus."

The way that is written, a CEF256 card in a Sup720 equipped chassis will use
the 8Gbs fabric to move data around. In a sparsely populated (eg 2 CEF256
cards) system there is more capacity on the shared 32Gbs bus than on the
fabric. Am I misreading/misunderstanding?

Does forcing the card into flow-through mode address this?

Rick

On Fri, Nov 6, 2009 at 5:55 AM, Geoffrey Pendery <geoff[at]pendery.net> wrote:

>
> If you have a Sup with no fabric (like Sup 1A, or Sup 2 w/o SFM, or
> Sup 32) the switch will run in "Flow-Through" mode, meaning that each
> time a packet is received, the entire packet is sent on the shared
> bus, so it's seen by the Sup and all line cards. This will only get
> you up to 15 Mpps, and a theoretical max of 32 Gbps (likely lower in
> practice).
>
> If you have a fabric Sup and fabric line cards, but at least one
> Classic line card, the switch will drop into "Truncated" mode. This
> is likely what someone was referring to when they told you "lowest
> common denominator". The classic cards will still send the whole
> packet, like in flow-through mode, but the fabric cards will send only
> the headers, and send the data portion to the Sup via fabric. This is
> still limited to 15 Mpps, but because the data flows via fabric, you
> can squeeze some extra bandwidth out.
>
> Lastly, if you have no Classic cards present in the chassis, it can go
> into Compact mode, where only compressed headers are sent via the bus,
> all data flows via fabric. This gets you up to 30 Mpps and your
> theoretical 720 Gbps of total forwarding capacity.
>
>
>
_______________________________________________
cisco-nsp mailing list cisco-nsp[at]puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

gert at greenie

Nov 6, 2009, 1:37 PM

Post #8 of 8 (128 views)
Permalink

Re: Please help clarify bus/fabric terminology on the 6500/7600 [In reply to]

Hi,

On Fri, Nov 06, 2009 at 12:06:36PM -0800, Rick Ernst wrote:
> The way that is written, a CEF256 card in a Sup720 equipped chassis will use
> the 8Gbs fabric to move data around. In a sparsely populated (eg 2 CEF256
> cards) system there is more capacity on the shared 32Gbs bus than on the
> fabric.

Correct.

> Does forcing the card into flow-through mode address this?

No idea (we only have a few CEF256 cards, and they are in Sup2-no-SFM
or in Sup32 switches).

gert
--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany gert[at]greenie.muc.de
fax: +49-89-35655025 gert[at]net.informatik.tu-muenchen.de

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com