Mailing List Archive: nsp: extreme

Packet loss/drop investigation

Index | Next | Previous | View Threaded

youssef.ghorbal at gmail

Aug 8, 2012, 11:08 AM

Post #1 of 7 (1512 views)
Permalink

Packet loss/drop investigation

Hello,

I'm currently investigating random audio loss on some Avaya phones
attached to x460 stacks.
The phone reports packet loss when the audio is gone (which seems legit)
The audio loss always happens in one way (from "outside the building"
to he IP phone) and lasts 10s

I need to know what kind of switch statistics/counters can be
relevant to confirm or deny packet loss on the switch side.

show ports rxerros / txerros / congestion / anomaly are all null
=> does this mean that there is no packet loss on the stack ? and
thus the packet loss is happening somewhere upstream ?

Any advice would be really appreciated

Youssef Ghorbal
_______________________________________________
extreme-nsp mailing list
extreme-nsp [at] puck
https://puck.nether.net/mailman/listinfo/extreme-nsp

erik at bais

Aug 8, 2012, 12:32 PM

Post #2 of 7 (1474 views)
Permalink

Re: Packet loss/drop investigation [In reply to]

Hi Youssef,

Best thing to do, is to do some testing using a tool like iperf (which is a traffic generator) is you think that you are having some issues on the switches...

Best way to check if all packets are actually entering and leaving the switches is to do the following:

Create an access-list on the switch using the following command:

edit policy pkt
entry AtoB{
if {
source-address <insert IP address A>/32;
destination-address <insert IP addres B>/32;
} then {
permit;
count AtoB;
}
}
entry BtoA {
if {
source-address <insert IP address B>/32;
destination-address <insert IP address A>/32;
} then {
permit;
count BtoA;
}
}

Save the policy and check the syntax using: check policy pkt

Now apply the access-list to the ports using:

configure access-list pkt ports 1:1 ingress
configure access-list pkt ports 1:2 ingress
configure access-list pkt ports 1:3 ingress

These ports should match the ingress and egress ports that you are using for the testing. It doesn't matter if you are putting it on other ports, as it doesn't block anything as the default policy with the XoS access-lists are allow.
The only thing you are doing it counting the packets that match the policy.

You can show the counters using: show accesss-list counters

If you are now sending packets from A2B or from B2A, you should see everything going into the switch and moving out of the switch.
If there are packets dropped, this is the way to see it.

After every test you can clear the counters using clear access-list counter

Hope this helps.

Kind regards,
Erik Bais
A2B Internet

_______________________________________________
extreme-nsp mailing list
extreme-nsp [at] puck
https://puck.nether.net/mailman/listinfo/extreme-nsp

marcin at leon

Aug 8, 2012, 3:05 PM

Post #3 of 7 (1468 views)
Permalink

Re: Packet loss/drop investigation [In reply to]

Youssef Ghorbal wrote:
> Hello,
>
> I'm currently investigating random audio loss on some Avaya phones
> attached to x460 stacks.
> The phone reports packet loss when the audio is gone (which seems legit)
> The audio loss always happens in one way (from "outside the building"
> to he IP phone) and lasts 10s

Is it 10s and than it is back, or after 10s of call you loose voice in
one direction ?
This could be some SIP issue, that RTP is redirected somewhere or,
firewall drops one way traffic because of some timeout (i.e. NAT problems).

Describe your case in more details.

Regards,
Marcin

>
> I need to know what kind of switch statistics/counters can be
> relevant to confirm or deny packet loss on the switch side.
>
> show ports rxerros / txerros / congestion / anomaly are all null
> => does this mean that there is no packet loss on the stack ? and
> thus the packet loss is happening somewhere upstream ?
>
> Any advice would be really appreciated
>
> Youssef Ghorbal
> _______________________________________________
> extreme-nsp mailing list
> extreme-nsp [at] puck
> https://puck.nether.net/mailman/listinfo/extreme-nsp
>

_______________________________________________
extreme-nsp mailing list
extreme-nsp [at] puck
https://puck.nether.net/mailman/listinfo/extreme-nsp

youssef.ghorbal at gmail

Aug 8, 2012, 3:56 PM

Post #4 of 7 (1479 views)
Permalink

Re: Packet loss/drop investigation [In reply to]

> Is it 10s and than it is back, or after 10s of call you loose voice in one
> direction ?

Yeah, it's 10s and than it's back.
The 10s audio loss happens randomly during the call, sometimes after 1
or 2 minutes of call establishement but sometimes after 6 minutes.
The only constant is the 10s consecutive one way audio loss.

> This could be some SIP issue, that RTP is redirected somewhere or, firewall
> drops one way traffic because of some timeout (i.e. NAT problems).

It's H323 and not SIP.
No firewall in the way (call and session manager are on the same IP
segment) and the problem occurs on internal calls and external ones
(GSM)
Moreover, the problem does not happen with "other" access switchs (Cisco)

I suspect some RTP packet loss somehow in the path and I want to
identify the culprit.

> Describe your case in more details.

The problem happens in this configuration

Avaya (9608) <-100Mb-> x460 stack <-10Gb-> x650 <-10Gb-> Brocade MLX
<-1Gb-> Cisco <-> Avaya (Call/session manager)

The problem does not happen with this configuration

Avaya (9608) <-100Mb-> Cisco <-1Gb-> Brocade MLX <-1Gb-> Cisco <->
Avaya (Call/session manager)

Same phone, same core switch and same Avaya servers. The only change
is the Extreme stuff (x460 and x650) and maybe the 10G nics.

Thank you for your assistance.

Youssef
_______________________________________________
extreme-nsp mailing list
extreme-nsp [at] puck
https://puck.nether.net/mailman/listinfo/extreme-nsp

marcin at leon

Aug 9, 2012, 2:04 AM

Post #5 of 7 (1471 views)
Permalink

Re: Packet loss/drop investigation [In reply to]

Youssef Ghorbal wrote:
>> Is it 10s and than it is back, or after 10s of call you loose voice in one
>> direction ?
>
> Yeah, it's 10s and than it's back.
> The 10s audio loss happens randomly during the call, sometimes after 1
> or 2 minutes of call establishement but sometimes after 6 minutes.
> The only constant is the 10s consecutive one way audio loss.
>
>> This could be some SIP issue, that RTP is redirected somewhere or, firewall
>> drops one way traffic because of some timeout (i.e. NAT problems).
>
> It's H323 and not SIP.
> No firewall in the way (call and session manager are on the same IP
> segment) and the problem occurs on internal calls and external ones
> (GSM)
> Moreover, the problem does not happen with "other" access switchs (Cisco)
>
> I suspect some RTP packet loss somehow in the path and I want to
> identify the culprit.
>
>> Describe your case in more details.
>
> The problem happens in this configuration
>
> Avaya (9608) <-100Mb-> x460 stack <-10Gb-> x650 <-10Gb-> Brocade MLX
> <-1Gb-> Cisco <-> Avaya (Call/session manager)
>
> The problem does not happen with this configuration
>
> Avaya (9608) <-100Mb-> Cisco <-1Gb-> Brocade MLX <-1Gb-> Cisco <->
> Avaya (Call/session manager)
>
> Same phone, same core switch and same Avaya servers. The only change
> is the Extreme stuff (x460 and x650) and maybe the 10G nics.
>
> Thank you for your assistance.

Well, the only what I can propose is to mirror port where phone is and
dump the traffic. Than - analzye.

Marcin
_______________________________________________
extreme-nsp mailing list
extreme-nsp [at] puck
https://puck.nether.net/mailman/listinfo/extreme-nsp

youssef.ghorbal at gmail

Aug 9, 2012, 9:15 AM

Post #6 of 7 (1475 views)
Permalink

Re: Packet loss/drop investigation [In reply to]

> Well, the only what I can propose is to mirror port where phone is and
> dump the traffic. Than - analzye.

I'm pretty sure that there is packet loss. The traffic dump will
confirm that but I'm not sure I'll be able to see where it's
happening.

I'll do it anyway to confirm the packet loss and see if there any
802.1p pause packets flying around. After that I'll give Erik's
suggestion a shot.

I'll let you know whenever I found out the root cause of this.

Youssef
_______________________________________________
extreme-nsp mailing list
extreme-nsp [at] puck
https://puck.nether.net/mailman/listinfo/extreme-nsp

youssef.ghorbal at gmail

Aug 10, 2012, 10:20 AM

Post #7 of 7 (1476 views)
Permalink

Re: Packet loss/drop investigation [In reply to]

More on the subject.
I've done mirroring on the port directly connected to the phone, the
traffic dump shows clearly egress packet loss for 10s.
Now, I've done mirroring on the uplink port of the stack and the
traffic dump does not show any ingress packet loss.

somehow on the way between the ingress port (the uplink) and the
egress port (connected to the phone)

The traffic dump shows also that during the 10s packet loss a huge
rate of arp requests hit the switch (~2000/s)

This leads to some DOS protect features that get triggred on the
stack, the problem is that I have no DOS/flood protection enabled in
the configuration. Also, all DOS/Flood related counters I'm aware of,
are null (show dos-protect, show ip-security, show ports rate-limit,
show ports congestion)

I'll try to generate fake arp at high speed rates and see if the
problem triggers. If the problem is reproductible at will, it will be
more easy to push diag further.

Youssef
--------------------
On Thu, Aug 9, 2012 at 6:15 PM, Youssef Ghorbal
<youssef.ghorbal [at] gmail> wrote:
>> Well, the only what I can propose is to mirror port where phone is and
>> dump the traffic. Than - analzye.
>
> I'm pretty sure that there is packet loss. The traffic dump will
> confirm that but I'm not sure I'll be able to see where it's
> happening.
>
> I'll do it anyway to confirm the packet loss and see if there any
> 802.1p pause packets flying around. After that I'll give Erik's
> suggestion a shot.
>
> I'll let you know whenever I found out the root cause of this.
>
> Youssef
_______________________________________________
extreme-nsp mailing list
extreme-nsp [at] puck
https://puck.nether.net/mailman/listinfo/extreme-nsp

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads