Mailing List Archive: Cisco: NSP

Increasing hold-queue to alleviate microbursts with small hardware queues

Index | Next | Previous | View Threaded

jneiberger at gmail

Aug 16, 2012, 11:02 PM

Post #1 of 8 (727 views)
Permalink

Increasing hold-queue to alleviate microbursts with small hardware queues

This has come up a few times recently. We continue to run into new
situations where we see lots of output queue drops on 6748 blades,
especially in cases where a 10g link is feeding a 1g link. We see OQDs
long before the interface approaches anything close to line rate on
average. Cisco has never recommended this when I've talked to them
about it, but I have seen a recommendation elsewhere to use something
like "hold-queue 1024 out" on the 1g interfaces to smooth out those
microbursts. I think that will add a tiny amount of latency to the
path, but would there be any other significant performance impacts?
Would this even do what we want? I'm not sure how it would interact
with the hardware queues. It seems as if it creates software-based
buffer space in front of the hardware output queues to store any
bursty traffic. Is it possible that the solution to this particular
problem is this simple?

I can only assume that since Cisco has never recommended trying this,
there must be a reason for it. Either it won't work at all or it has
unfortunate side effects. But I'm considering giving it a try.

Any thoughts?

Many thanks, as usual!

John
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

ltd at aristanetworks

Aug 17, 2012, 12:06 AM

Post #2 of 8 (710 views)
Permalink

Re: Increasing hold-queue to alleviate microbursts with small hardware queues [In reply to]

On Fri, Aug 17, 2012 at 4:02 PM, John Neiberger <jneiberger [at] gmail>wrote:

> Would this even do what we want?

It won't do anything.
'hold-queue' is only for software based forwarding platforms.

it would help if you had a burst of packets back-to-back going for software
forwarding, but if you have that happening, you have bigger problems.

cheers,

lincoln.
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

p.mayers at imperial

Aug 17, 2012, 1:37 AM

Post #3 of 8 (704 views)
Permalink

Re: Increasing hold-queue to alleviate microbursts with small hardware queues [In reply to]

On 08/17/2012 07:02 AM, John Neiberger wrote:
> This has come up a few times recently. We continue to run into new
> situations where we see lots of output queue drops on 6748 blades,
> especially in cases where a 10g link is feeding a 1g link. We see OQDs
> long before the interface approaches anything close to line rate on
> average.

As I'm sure you're aware, but just for the archives, average rate is
pretty irrelevant in this scenario. All that matters is the burst rate
at the ingress (10 gig) port.

6748 cards have ~1.3Mb of per-port output buffer, which at 9Gbit/sec (10
in minus 1 out) will fill in ~1.1 milliseconds. At some fraction of that
load (e.g. 5 in, 1 out) it'll scale appropriately.

If you are doing QoS with the default queue parameters, you'll get
(possibly much) less than that.

> bursty traffic. Is it possible that the solution to this particular
> problem is this simple?

As others have said, no. It won't do anything.

You're sure you've either disabled QoS or reconfigured the queues to
allocate space appropriately i.e. in proportion to the offered load?

TBH microburst problems on this platform don't come up on the list very
often, as far as I can see. It's usually the lower-end catalysts.
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

jneiberger at gmail

Aug 17, 2012, 6:48 AM

Post #4 of 8 (701 views)
Permalink

Re: Increasing hold-queue to alleviate microbursts with small hardware queues [In reply to]

Thanks, I suspected that was the case or it would have been mentioned
before. We have played around with different queuing parameters and queue
depths, but I'm trying to find some other potential solutions.

Would a shaping service policy work for this? I'm wondering what the impact
would be on microbursts if we enabled shaping. Would that give us any
additional short-term buffer space?

Thanks again,
John
On Aug 17, 2012 2:44 AM, "Phil Mayers" <p.mayers [at] imperial> wrote:

> On 08/17/2012 07:02 AM, John Neiberger wrote:
>
>> This has come up a few times recently. We continue to run into new
>> situations where we see lots of output queue drops on 6748 blades,
>> especially in cases where a 10g link is feeding a 1g link. We see OQDs
>> long before the interface approaches anything close to line rate on
>> average.
>>
>
> As I'm sure you're aware, but just for the archives, average rate is
> pretty irrelevant in this scenario. All that matters is the burst rate at
> the ingress (10 gig) port.
>
> 6748 cards have ~1.3Mb of per-port output buffer, which at 9Gbit/sec (10
> in minus 1 out) will fill in ~1.1 milliseconds. At some fraction of that
> load (e.g. 5 in, 1 out) it'll scale appropriately.
>
> If you are doing QoS with the default queue parameters, you'll get
> (possibly much) less than that.
>
> bursty traffic. Is it possible that the solution to this particular
>> problem is this simple?
>>
>
> As others have said, no. It won't do anything.
>
> You're sure you've either disabled QoS or reconfigured the queues to
> allocate space appropriately i.e. in proportion to the offered load?
>
> TBH microburst problems on this platform don't come up on the list very
> often, as far as I can see. It's usually the lower-end catalysts.
> ______________________________**_________________
> cisco-nsp mailing list cisco-nsp [at] puck
> https://puck.nether.net/**mailman/listinfo/cisco-nsp<https://puck.nether.net/mailman/listinfo/cisco-nsp>
> archive at http://puck.nether.net/**pipermail/cisco-nsp/<http://puck.nether.net/pipermail/cisco-nsp/>
>
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

p.mayers at imperial

Aug 17, 2012, 6:57 AM

Post #5 of 8 (702 views)
Permalink

Re: Increasing hold-queue to alleviate microbursts with small hardware queues [In reply to]

On 17/08/12 14:48, John Neiberger wrote:
> Thanks, I suspected that was the case or it would have been mentioned
> before. We have played around with different queuing parameters and
> queue depths, but I'm trying to find some other potential solutions.
>
> Would a shaping service policy work for this? I'm wondering what the
> impact would be on microbursts if we enabled shaping. Would that give us
> any additional short-term buffer space?

Can you be more specific here? Where would you shape?
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

jneiberger at gmail

Aug 17, 2012, 7:29 AM

Post #6 of 8 (699 views)
Permalink

Re: Increasing hold-queue to alleviate microbursts with small hardware queues [In reply to]

On Aug 17, 2012 7:57 AM, "Phil Mayers" <p.mayers [at] imperial> wrote:
>
> On 17/08/12 14:48, John Neiberger wrote:
>>
>> Thanks, I suspected that was the case or it would have been mentioned
>> before. We have played around with different queuing parameters and
>> queue depths, but I'm trying to find some other potential solutions.
>>
>> Would a shaping service policy work for this? I'm wondering what the
>> impact would be on microbursts if we enabled shaping. Would that give us
>> any additional short-term buffer space?
>
>
> Can you be more specific here? >Where would you shape?

I was wondering if an outbound shaping policy on the 1g links would smooth
out the peaks of those bursts prior to them hitting the small hardware
queue. I'm just kind of thinking out loud.

Our approach at the moment is to map everything to the same queue and give
it all the queue space, except for the 15 percent that the priority queue
gets. We can't disable mls qos because other features depend on it.
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

p.mayers at imperial

Aug 17, 2012, 7:36 AM

Post #7 of 8 (699 views)
Permalink

Re: Increasing hold-queue to alleviate microbursts with small hardware queues [In reply to]

On 17/08/12 15:29, John Neiberger wrote:

> > Can you be more specific here? >Where would you shape?
>
> I was wondering if an outbound shaping policy on the 1g links would
> smooth out the peaks of those bursts prior to them hitting the small
> hardware queue. I'm just kind of thinking out loud.

Not really. Shaping happens at the output of the queue, so it can't help
you with the queue overflowing.

On the 6500, packets are

1. received
2. put into an ingress queue
3. forwarding lookup takes place
4. packets are moved across the fabric into an egress queue
5. egress queueing takes place

Steps 1-3 don't really "block" because the fabric is non-blocking. So
the ingress queueing is largely useless on this platform. So, your only
room for dealing with output speed/load mismatches is in the egress queue.

You'd need to ingress shape (which is not possible) or egress shape on
the upstream box. And shaping is not widely available.

TBH, you either need a box with much bigger buffers, or move to a 10g port.
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

jneiberger at gmail

Aug 17, 2012, 7:41 AM

Post #8 of 8 (700 views)
Permalink

Re: Increasing hold-queue to alleviate microbursts with small hardware queues [In reply to]

On Fri, Aug 17, 2012 at 8:36 AM, Phil Mayers <p.mayers [at] imperial> wrote:
> On 17/08/12 15:29, John Neiberger wrote:
>
>> > Can you be more specific here? >Where would you shape?
>>
>> I was wondering if an outbound shaping policy on the 1g links would
>> smooth out the peaks of those bursts prior to them hitting the small
>> hardware queue. I'm just kind of thinking out loud.
>
>
> Not really. Shaping happens at the output of the queue, so it can't help you
> with the queue overflowing.
>
> On the 6500, packets are
>
> 1. received
> 2. put into an ingress queue
> 3. forwarding lookup takes place
> 4. packets are moved across the fabric into an egress queue
> 5. egress queueing takes place
>
> Steps 1-3 don't really "block" because the fabric is non-blocking. So the
> ingress queueing is largely useless on this platform. So, your only room for
> dealing with output speed/load mismatches is in the egress queue.
>
> You'd need to ingress shape (which is not possible) or egress shape on the
> upstream box. And shaping is not widely available.
>
> TBH, you either need a box with much bigger buffers, or move to a 10g port.

Thanks. That's really what I figured, but I was hoping to find some
creative way to alleviate the issue since in many cases we can't
upgrade the link speed or hardware, at least in the short term. That's
definitely going to be my recommendation in the long term.

Thanks again!
John
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads