Mailing List Archive: Cisco: NSP

Network performance question - TCP Window issue?

Index | Next | Previous | View Threaded

cisconsp_list at hotmail

Apr 29, 2012, 12:34 AM

Post #1 of 10 (1037 views)
Permalink

Network performance question - TCP Window issue?

Hi Guys,

(Apologies if this is too off-topic for the cisco list)

Have 3 POPs A,B+C (A connected to B via 200Mb Eth, B connected to C via 250Mb Eth), all POPs are 7200's with 2960's and 3560's

Latency:

A -> B, 2 - 3 m/sec (No packet loss, minimal jitter)
B -> C, 12 - 13 m/sec (No packet loss, minimal jitter)

Performing wget or iperf I see the following (Have linux servers at each POP)

Pulling file from B->A (A's linux server is on 10/100 port) I get 9.59M/s (So basically 100Mbit)
Pulling file from C->B I get 8.30M/s, but it does fluctuate up+down considerably
Pulling file from C->A I get a flatline at 2.16M/s (Multiple sessions and all attain this speed, but no faster and doesnt fluctuate)

Im not seeing any excessive errors/drops nor any duplex issues on any of the switches - Is TCP Window size the cause of the huge speed difference I see from C->B vs C->A, even though the latency difference between the 2 is minimal?(i.e. 3m/sec?) or does the extra hop cause the issue?

thanks in advance for any advice.

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

andrew at 2sheds

Apr 29, 2012, 1:23 AM

Post #2 of 10 (996 views)
Permalink

Re: Network performance question - TCP Window issue? [In reply to]

On 29/04/2012, at 5:34 PM, CiscoNSP_list CiscoNSP_list wrote:
> Have 3 POPs A,B+C (A connected to B via 200Mb Eth, B connected to C via 250Mb Eth), all POPs are 7200's with 2960's and 3560's
>
> Latency:
>
> A -> B, 2 - 3 m/sec (No packet loss, minimal jitter)
> B -> C, 12 - 13 m/sec (No packet loss, minimal jitter)

There is a huge latency difference between A-B and B-C. (about 5 times!)
How far are these sites apart?

This might also make some good reading:

http://knol.google.com/k/understanding-network-and-internet-latency

Cheers

Andrew

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

kka at netuse

Apr 29, 2012, 1:43 AM

Post #3 of 10 (997 views)
Permalink

Re: Network performance question - TCP Window issue? [In reply to]

On Sun, 29 Apr 2012 18:04 +1030, CiscoNSP_list CiscoNSP_list wrote:
>
> Performing wget or iperf I see the following (Have linux servers at each POP)
>
> Pulling file from B->A (A's linux server is on 10/100 port) I get 9.59M/s (So basically 100Mbit)
> Pulling file from C->B I get 8.30M/s, but it does fluctuate up+down considerably
> Pulling file from C->A I get a flatline at 2.16M/s (Multiple sessions and all attain this speed, but no faster and doesnt fluctuate)
>
>
> Im not seeing any excessive errors/drops nor any duplex issues on any of the switches

Did you run your iperf tests also with UDP? (The numbers don't look
like it.)

With TCP you won't see many drops on your switches, it will adjust - and
you will see less throughput.

With iperf available at all three sites I would run tests with UDP streams.
This won't find the maximum bandwith automatically, you have to set a
bandwidth for testing and see if you have any packet loss.

Keep in mind that your carrier might police on ethernet bandwidth,
iperf measures IP throuput.

Best regards,
Klaus Kastens

--
Klaus Kastens NetUSE AG
Dr.-Hell-Str. 6, D-24107 Kiel, Germany
Fon: +49 431 2390 400 (07:00 UTC - 17:00 UTC)
Fax: +49 431 2390 499

Vorstand: Dr. Joerg Posewang (Vorsitz), Dr. Roland Kaltefleiter, Andreas Seeger
Aufsichtsrat: Dr. Dirk Lukas (Vorsitz)
Sitz der AG: Kiel, HRB 5358 USt.ID: DE156073942

Diese E-Mail enthaelt vertrauliche oder rechtlich geschuetzte Informationen.
Das unbefugte Kopieren dieser E-Mail oder die unbefugte Weitergabe der
enthaltenen Informationen ist nicht gestattet.

The information contained in this message is confidential or protected by
law. Any unauthorised copying of this message or unauthorised distribution
of the information contained herein is prohibited.

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

cisconsp_list at hotmail

Apr 29, 2012, 1:47 AM

Post #4 of 10 (997 views)
Permalink

Re: Network performance question - TCP Window issue? [In reply to]

>
> On 29/04/2012, at 5:34 PM, CiscoNSP_list CiscoNSP_list wrote:
> > Have 3 POPs A,B+C (A connected to B via 200Mb Eth, B connected to C via 250Mb Eth), all POPs are 7200's with 2960's and 3560's
> >
> > Latency:
> >
> > A -> B, 2 - 3 m/sec (No packet loss, minimal jitter)
> > B -> C, 12 - 13 m/sec (No packet loss, minimal jitter)
>
> There is a huge latency difference between A-B and B-C. (about 5 times!)
> How far are these sites apart?

A+B about 70K's
B+C about 1000K's

I've just checked in the opposite direction (C -> A) and I see ~4.5M/s - A's on a 10/100 port, C's on a Gig port...so pulling from C->A would be Gig, 250, 200, 100 (where I see 2.15M/s), but in the other direction (pulling from A->C) would be 100, 200, 250, Gig...

>
> This might also make some good reading:
>
> http://knol.google.com/k/understanding-network-and-internet-latency
>

Thanks - Will have a read.

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

cisconsp_list at hotmail

Apr 29, 2012, 2:22 AM

Post #5 of 10 (988 views)
Permalink

Re: Network performance question - TCP Window issue? [In reply to]

>
> Did you run your iperf tests also with UDP? (The numbers don't look
> like it.)
>
> With TCP you won't see many drops on your switches, it will adjust - and
> you will see less throughput.
>
> With iperf available at all three sites I would run tests with UDP streams.
> This won't find the maximum bandwith automatically, you have to set a
> bandwidth for testing and see if you have any packet loss.
>
> Keep in mind that your carrier might police on ethernet bandwidth,
> iperf measures IP throuput.

Thanks Klaus - No, did not test with udp...here it is:
(With 100M had too many drops - 80M was the best:)[ 3] local xxx.xxx.73.54 port 45790 connected with xxx.xxx.65.2 port 5001[ ID] Interval Transfer Bandwidth[ 3] 0.0-10.0 sec 95.4 MBytes 80.0 Mbits/sec[ 3] Sent 68029 datagrams[ 3] Server Report:[ 3] 0.0-10.0 sec 95.4 MBytes 80.0 Mbits/sec 0.044 ms 1/68028 (0.0015%)[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
So to be able to see similar performance with tcp, I will need to adjust tcp window correct?

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

sledge121 at gmail

Apr 29, 2012, 4:41 AM

Post #6 of 10 (992 views)
Permalink

Re: Network performance question - TCP Window issue? [In reply to]

I have seen this before, it's called bandwidth delay product and is linked to window size, let us know the tcp results after you have adjusted.

Sent from my iPad

On 29 Apr 2012, at 10:22, CiscoNSP_list CiscoNSP_list <cisconsp_list [at] hotmail> wrote:

>
>
>
>>
>> Did you run your iperf tests also with UDP? (The numbers don't look
>> like it.)
>>
>> With TCP you won't see many drops on your switches, it will adjust - and
>> you will see less throughput.
>>
>> With iperf available at all three sites I would run tests with UDP streams.
>> This won't find the maximum bandwith automatically, you have to set a
>> bandwidth for testing and see if you have any packet loss.
>>
>> Keep in mind that your carrier might police on ethernet bandwidth,
>> iperf measures IP throuput.
>
> Thanks Klaus - No, did not test with udp...here it is:
> (With 100M had too many drops - 80M was the best:)[ 3] local xxx.xxx.73.54 port 45790 connected with xxx.xxx.65.2 port 5001[ ID] Interval Transfer Bandwidth[ 3] 0.0-10.0 sec 95.4 MBytes 80.0 Mbits/sec[ 3] Sent 68029 datagrams[ 3] Server Report:[ 3] 0.0-10.0 sec 95.4 MBytes 80.0 Mbits/sec 0.044 ms 1/68028 (0.0015%)[ 3] 0.0-10.0 sec 1 datagrams received out-of-order
> So to be able to see similar performance with tcp, I will need to adjust tcp window correct?
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp [at] puck
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

jneiberger at gmail

Apr 29, 2012, 10:57 AM

Post #7 of 10 (982 views)
Permalink

Re: Network performance question - TCP Window issue? [In reply to]

The timing of this is coincidental. I've been helping to troubleshoot
a similar problem at work for days. Let's say we have three servers,
A, B and C. We transfer files between them and here is what we see:

A to B: Fast (around 18 MB/s)
B to A: Slow (around 1 MB/s)
A to C: Slow (around 1MB/s)
C to A: Fast (around 18MB/s)

In our case, Server A is fast when sending to B but not when sending
to C. C can send at a high speed when sending back to A, though.

We've checked everything we can think of. The paths aren't the same.
One path goes through a firewall, another path goes through GRE
tunnels. There are no TCP retransmits and we've verified that MTU
isn't the problem. The firewall can't be the problem because it's only
in the path of one set of transfers. All the TCP settings we've
checked on the servers seem to be the same, although I'm not a server
guy. Someone else has been checking those. The endpoints are on 1-gig
links but it's 10-gig the whole way between them. There is about 50ms
round-trip latency in all cases.

I have no idea what could account for this behavior.

On Sun, Apr 29, 2012 at 5:41 AM, <sledge121 [at] gmail> wrote:
> I have seen this before, it's called bandwidth delay product and is linked to window size, let us know the tcp results after you have adjusted.
>
>
> Sent from my iPad
>
> On 29 Apr 2012, at 10:22, CiscoNSP_list CiscoNSP_list <cisconsp_list [at] hotmail> wrote:
>
>>
>>
>>
>>>
>>> Did you run your iperf tests also with UDP? (The numbers don't look
>>> like it.)
>>>
>>> With TCP you won't see many drops on your switches, it will adjust - and
>>> you will see less throughput.
>>>
>>> With iperf available at all three sites I would run tests with UDP streams.
>>> This won't find the maximum bandwith automatically, you have to set a
>>> bandwidth for testing and see if you have any packet loss.
>>>
>>> Keep in mind that your carrier might police on ethernet bandwidth,
>>> iperf measures IP throuput.
>>
>> Thanks Klaus - No, did not test with udp...here it is:
>> (With 100M had too many drops - 80M was the best:)[ �3] local xxx.xxx.73.54 port 45790 connected with xxx.xxx.65.2 port 5001[ ID] Interval � � � Transfer � � Bandwidth[ �3] �0.0-10.0 sec �95.4 MBytes �80.0 Mbits/sec[ �3] Sent 68029 datagrams[ �3] Server Report:[ �3] �0.0-10.0 sec �95.4 MBytes �80.0 Mbits/sec �0.044 ms � �1/68028 (0.0015%)[ �3] �0.0-10.0 sec �1 datagrams received out-of-order
>> So to be able to see similar performance with tcp, I will need to adjust tcp window correct?
>>
>>
>> _______________________________________________
>> cisco-nsp mailing list �cisco-nsp [at] puck
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
> _______________________________________________
> cisco-nsp mailing list �cisco-nsp [at] puck
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

andrew at 2sheds

Apr 29, 2012, 2:18 PM

Post #8 of 10 (974 views)
Permalink

Re: Network performance question - TCP Window issue? [In reply to]

On 30/04/2012, at 3:57 AM, John Neiberger wrote:

> The timing of this is coincidental. I've been helping to troubleshoot
> a similar problem at work for days. Let's say we have three servers,
> A, B and C. We transfer files between them and here is what we see:
>
> A to B: Fast (around 18 MB/s)
> B to A: Slow (around 1 MB/s)
> A to C: Slow (around 1MB/s)
> C to A: Fast (around 18MB/s)
>
> In our case, Server A is fast when sending to B but not when sending
> to C. C can send at a high speed when sending back to A, though.
> account for this behavior.

How are you testing this speed? What servers with what connections with which operating system versions?
What is exactly the list of all hardware/ interfaces/ connections between these 3 boxes.
You need to draw this up.

This however is probably a different issue to the original one listed.
_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

lukasz at bromirski

Apr 29, 2012, 3:39 PM

Post #9 of 10 (974 views)
Permalink

Re: Network performance question - TCP Window issue? [In reply to]

On 2012-04-29 19:57, John Neiberger wrote:
> The timing of this is coincidental. I've been helping to troubleshoot
> a similar problem at work for days. Let's say we have three servers,
> A, B and C. We transfer files between them and here is what we see:
>
> A to B: Fast (around 18 MB/s)
> B to A: Slow (around 1 MB/s)
> A to C: Slow (around 1MB/s)
> C to A: Fast (around 18MB/s)
>
> In our case, Server A is fast when sending to B but not when sending
> to C. C can send at a high speed when sending back to A, though.

Typical problems with the different speed depending on the direction
are caused either by duplex mismatch at access port (test, don't
trust what one side tells you!) or problems with negotiating the
TCP windows size (depending on the TCP/IP stack, application and tool
you may get a number of different results).

--
"There's no sense in being precise when | �ukasz Bromirski
you don't know what you're talking | jid:lbromirski [at] jabber
about." John von Neumann | http://lukasz.bromirski.net

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

jneiberger at gmail

Apr 29, 2012, 3:57 PM

Post #10 of 10 (971 views)
Permalink

Re: Network performance question - TCP Window issue? [In reply to]

On Sun, Apr 29, 2012 at 4:39 PM, Łukasz Bromirski <lukasz [at] bromirski> wrote:
> On 2012-04-29 19:57, John Neiberger wrote:
>>
>> The timing of this is coincidental. I've been helping to troubleshoot
>> a similar problem at work for days. Let's say we have three servers,
>> A, B and C. We transfer files between them and here is what we see:
>>
>> A to B: Fast (around 18 MB/s)
>> B to A: Slow (around 1 MB/s)
>> A to C: Slow (around 1MB/s)
>> C to A: Fast (around 18MB/s)
>>
>> In our case, Server A is fast when sending to B but not when sending
>> to C. C can send at a high speed when sending back to A, though.
>
>
> Typical problems with the different speed depending on the direction
> are caused either by duplex mismatch at access port (test, don't
> trust what one side tells you!) or problems with negotiating the
> TCP windows size (depending on the TCP/IP stack, application and tool
> you may get a number of different results).

We checked all the usual stuff and haven't found anything wrong
anywhere in the path despite looking at this from a number of
different angles for days. However, Drew emailed me off-list and
suggested that we look to see if apf is running on the linux servers.
This has a known "feature" that disables TCP window scaling, which
would certainly cause what we're seeing. I'm going to have a server
guy check it tomorrow.

_______________________________________________
cisco-nsp mailing list cisco-nsp [at] puck
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads