Mailing List Archive: NANOG: users

Consistent asymetric latency on monitoring?

Index | Next | Previous | View Threaded

nanog at shreddedmail

Oct 21, 2009, 8:07 AM

Post #1 of 8 (60 views)
Permalink

Consistent asymetric latency on monitoring?

Although the implementation is Cisco-specific, this feels more appropriate
for NANOG.

We've started rolling out a state-wide monitoring system based on Cisco's
"IP SLA" feature set. Out of 5 sites deployed so far (different locations,
different providers), we are consistently seeing one-way latency mirror the
opposite direction. As source-destination latency goes up,
destination-source latency goes down and vice versa.

Myself and the monitoring team have ripped apart the OIDs, IP SLA
configuration, and monitoring system. We've also built an ad-hoc system to
compare the results. It's still consistent behavior. It's not a true
mirror; there is definitely variation between the data collection, but at
the 10,000 foot level, there is an obvious and consistent mirror to the
data.

The network topology is independant service providers all providing backhaul
to a local ethernet exchange.

Has anybody seen this type of behavior? We are solidly convinced that we are
using the proper OIDs and making the proper transformations of the data.
The two remaining causes appear to be either "natural behavior of the links"
and/or "artifact in the IP SLA mechanism".

Any ideas?

Thanks!

perry at coders

Oct 21, 2009, 6:31 PM

Post #2 of 8 (50 views)
Permalink

Re: Consistent asymetric latency on monitoring? [In reply to]

Rick Ernst wrote:
> Although the implementation is Cisco-specific, this feels more appropriate
> for NANOG.
>
> We've started rolling out a state-wide monitoring system based on Cisco's
> "IP SLA" feature set. Out of 5 sites deployed so far (different locations,
> different providers), we are consistently seeing one-way latency mirror the
> opposite direction. As source-destination latency goes up,
> destination-source latency goes down and vice versa.
>
> Myself and the monitoring team have ripped apart the OIDs, IP SLA
> configuration, and monitoring system. We've also built an ad-hoc system to
> compare the results. It's still consistent behavior. It's not a true
> mirror; there is definitely variation between the data collection, but at
> the 10,000 foot level, there is an obvious and consistent mirror to the
> data.
>
> The network topology is independant service providers all providing backhaul
> to a local ethernet exchange.
>
> Has anybody seen this type of behavior? We are solidly convinced that we are
> using the proper OIDs and making the proper transformations of the data.
> The two remaining causes appear to be either "natural behavior of the links"
> and/or "artifact in the IP SLA mechanism".
>
> Any ideas?
>

Having never used cisco's IP SLA (or even read about it), take this with
a sack of salt.

I assume this product works by having a packet with a timestamp sent
from the source to the destination where it is timestamped again and
either sent back, or another packet is sent in the other direction. The
difference between the two timestamps gives you the latency in that
direction.

Now, how are your clocks syncronised? are they synchronised using NTP?
or something better (GPS?) If one of your clocks is drifting with
respect to the other then you'll see this effect. Does your clock drift
because NTP is failing to keep the clock well syncronised when it's
connection to it's parent NTP server is saturated?

nanog at daork

Oct 21, 2009, 6:40 PM

Post #3 of 8 (50 views)
Permalink

Re: Consistent asymetric latency on monitoring? [In reply to]

On 22/10/2009, at 2:31 PM, Perry Lorier wrote:

> I assume this product works by having a packet with a timestamp sent
> from the source to the destination where it is timestamped again and
> either sent back, or another packet is sent in the other direction.
> The difference between the two timestamps gives you the latency in
> that direction.

I believe a packet is sent, and the target router responds with a
timestamp.

But yeah, timestamps are being compared.

I'm with Perry though - sounds like your clocks are drifting.

--
Nathan Ward

nanog at shreddedmail

Oct 21, 2009, 8:03 PM

Post #4 of 8 (50 views)
Permalink

Re: Consistent asymetric latency on monitoring? [In reply to]

Resent, since I responded from the wrong address:
---
The basic operation of IP SLA is as surmised; payload with timestamps
and other telemetry data is sent to a 'responder' which manipulates
the payload, including adding its own timestamps, and returns the
altered payload.

I had to do a mental walk-through, but I think I see how drift can
cause this. I'm going to generate some artificial data, graph it, and
see if it matches the general waveshape I'm seeing.

I purposefully have the traffic generators ntp syncing against the
responders. I thought that would keep the clocks more closely in sync.
I don't necessarily care if the time is 'right', just that it's the
same. What kind of difference should I expect if I sync both
generators and responders against the same source, or not sync the
responder? I'm thinking that having one source with constant drift may
be better than both devices trying to walk/correct the time.

Thanks for the input!

On Wed, Oct 21, 2009 at 8:01 PM, Rick Ernst <ernst[at]shreddedmail.com> wrote:

> Resent, since I responded from the wrong address:
> ---
> The basic operation of IP SLA is as surmised; payload with timestamps
> and other telemetry data is sent to a 'responder' which manipulates
> the payload, including adding its own timestamps, and returns the
> altered payload.
>
> I had to do a mental walk-through, but I think I see how drift can
> cause this. I'm going to generate some artificial data, graph it, and
> see if it matches the general waveshape I'm seeing.
>
> I purposefully have the traffic generators ntp syncing against the
> responders. I thought that would keep the clocks more closely in sync.
> I don't necessarily care if the time is 'right', just that it's the
> same. What kind of difference should I expect if I sync both
> generators and responders against the same source, or not sync the
> responder? I'm thinking that having one source with constant drift may
> be better than both devices trying to walk/correct the time.
>
> Thanks for the input!
>
>
> On Wed, Oct 21, 2009 at 7:55 PM, Rick Ernst <ernst[at]shreddedmail.com>wrote:
>
>> The basic operation of IP SLA is as surmised; payload with timestamps
>> and other telemetry data is sent to a 'responder' which manipulates
>> the payload, including adding its own timestamps, and returns the
>> altered payload.
>>
>> I had to do a mental walk-through, but I think I see how drift can
>> cause this. I'm going to generate some artificial data, graph it, and
>> see if it matches the general waveshape I'm seeing.
>>
>> I purposefully have the traffic generators ntp syncing against the
>> responders. I thought that would keep the clocks more closely in sync.
>> I don't necessarily care if the time is 'right', just that it's the
>> same. What kind of difference should I expect if I sync both
>> generators and responders against the same source, or not sync the
>> responder? I'm thinking that having one source with constant drift may
>> be better than both devices trying to walk/correct the time.
>>
>> Thanks for the input!
>>
>>
>> On Wednesday, October 21, 2009, Nathan Ward <nanog[at]daork.net> wrote:
>> > On 22/10/2009, at 2:31 PM, Perry Lorier wrote:
>> >
>> >
>> > I assume this product works by having a packet with a timestamp sent
>> from the source to the destination where it is timestamped again and either
>> sent back, or another packet is sent in the other direction. The difference
>> between the two timestamps gives you the latency in that direction.
>> >
>> >
>> > I believe a packet is sent, and the target router responds with a
>> timestamp.
>> >
>> > But yeah, timestamps are being compared.
>> >
>> > I'm with Perry though - sounds like your clocks are drifting.
>> >
>> > --
>> > Nathan Ward
>> >
>> >
>>
>
>

swmike at swm

Oct 21, 2009, 10:51 PM

Post #5 of 8 (49 views)
Permalink

Re: Consistent asymetric latency on monitoring? [In reply to]

On Wed, 21 Oct 2009, Rick Ernst wrote:

> Has anybody seen this type of behavior? We are solidly convinced that we
> are using the proper OIDs and making the proper transformations of the
> data. The two remaining causes appear to be either "natural behavior of
> the links" and/or "artifact in the IP SLA mechanism".

I've been using IP SLA for years (right now under 12.4) and I have not
seen behaviour that mirrors what you see. I often see one-way latency go
up without the other way doing so.

You should start by looking in "show ip sla (monitor) op" and see what
values you see in the router, that might give you more information
regarding where the problem might be (your polling system or if the IP SLA
agent is actually reporting what you see).

--
Mikael Abrahamsson email: swmike[at]swm.pp.se

perry at coders

Oct 21, 2009, 11:10 PM

Post #6 of 8 (49 views)
Permalink

Re: Consistent asymetric latency on monitoring? [In reply to]

Rick Ernst wrote:
> Resent, since I responded from the wrong address:
> ---
> The basic operation of IP SLA is as surmised; payload with timestamps
> and other telemetry data is sent to a 'responder' which manipulates
> the payload, including adding its own timestamps, and returns the
> altered payload.
>

Yup :) It's the obvious way to do it :)

> I had to do a mental walk-through, but I think I see how drift can
> cause this. I'm going to generate some artificial data, graph it, and
> see if it matches the general waveshape I'm seeing.
>
> I purposefully have the traffic generators ntp syncing against the
> responders. I thought that would keep the clocks more closely in sync.
> I don't necessarily care if the time is 'right', just that it's the
> same.

This causes major problems. What you're actually measuring here is how
well ntp can keep the clock sync'd under assymetric latency. ntp is
trying to do it's own measurements of one way delay, without the help of
clocks to measure clock drift as well. As you can see from your graphs
ntp is not coping[1].

You are far better to have each end sync to a local stratum 1 or stratum
2 ntp source, preferably one over a different link to the one under
test. If you don't have a local stratum 1/2 time source at each end,
you might be able find one over a local exchange or other less congested
link. If this is very important to you then you should consider looking
at running your own stratum 1 clocks at each end syncronised off
something like GPS, CDMA or a T1 clock.

> What kind of difference should I expect if I sync both
> generators and responders against the same source, or not sync the
> responder? I'm thinking that having one source with constant drift may
> be better than both devices trying to walk/correct the time.
>

Most hardware clocks in PC's/routers/switches etc have pretty atrocious
amounts of drift if left to free run[2], sometimes in the order of
seconds or occasionally minutes per week. To get useful numbers you
really do need to syncronise them to /something/. Synchronising them to
each other causes problems as ntp I think (I could be wrong) assumes
mostly symmetrical latency, and if the latency isn't symmetric assumes
it's because one clock is running fast/slow and will alter the clock's
speed to account for it. The great thing about ntp stratum 1 servers is
that by definition they have more or less the same time no matter where
they are, so synchronising each against a local ntp server will be a
much much better solution. If possible you should consider peering with
at least 3 upstreams, preferably 4(!)[3] other ntp servers.

[1]: To be fair it's a hard problem. Anything that involves time just
gets more and more complicated the more you look at it, ntp is extremely
clever and probably knows more about time than I'd ever want to know,
but you're making it's job hard.

[2]: http://vancouver-webpages.com/time/ /
http://vancouver-webpages.com/time/ltmhist.png

[3]:
http://twiki.ntp.org/bin/view/Support/SelectingOffsiteNTPServers#Section_5.3.3.

rdobbins at arbor

Oct 22, 2009, 3:59 AM

Post #7 of 8 (44 views)
Permalink

Re: Consistent asymetric latency on monitoring? [In reply to]

On Oct 21, 2009, at 11:03 PM, Rick Ernst wrote:

> I thought that would keep the clocks more closely in sync. I don't
> necessarily care if the time is 'right', just that it's the same.

ntp is a pretty basic operational requirement for any network,
irrespective of the use of IP SLA, is it not?

-----------------------------------------------------------------------
Roland Dobbins <rdobbins[at]arbor.net> // <http://www.arbornetworks.com>

Sorry, sometimes I mistake your existential crises for technical
insights.

-- xkcd #625

nanog at shreddedmail

Oct 22, 2009, 4:25 AM

Post #8 of 8 (44 views)
Permalink

Re: Consistent asymetric latency on monitoring? [In reply to]

Lots of good info, and a nice mind-dump that gives me a whole host of other
things that need to be looked at... Umm. "thanks" :)

On Wed, Oct 21, 2009 at 11:10 PM, Perry Lorier <perry[at]coders.net> wrote:

> Rick Ernst wrote:
>
>> Resent, since I responded from the wrong address:
>> ---
>> The basic operation of IP SLA is as surmised; payload with timestamps
>> and other telemetry data is sent to a 'responder' which manipulates
>> the payload, including adding its own timestamps, and returns the
>> altered payload.
>>
>>
>
> Yup :) It's the obvious way to do it :)
>
> I had to do a mental walk-through, but I think I see how drift can
>> cause this. I'm going to generate some artificial data, graph it, and
>> see if it matches the general waveshape I'm seeing.
>>
>> I purposefully have the traffic generators ntp syncing against the
>> responders. I thought that would keep the clocks more closely in sync.
>> I don't necessarily care if the time is 'right', just that it's the
>> same.
>>
>
> This causes major problems. What you're actually measuring here is how
> well ntp can keep the clock sync'd under assymetric latency. ntp is trying
> to do it's own measurements of one way delay, without the help of clocks to
> measure clock drift as well. As you can see from your graphs ntp is not
> coping[1].
>
> You are far better to have each end sync to a local stratum 1 or stratum 2
> ntp source, preferably one over a different link to the one under test. If
> you don't have a local stratum 1/2 time source at each end, you might be
> able find one over a local exchange or other less congested link. If this
> is very important to you then you should consider looking at running your
> own stratum 1 clocks at each end syncronised off something like GPS, CDMA or
> a T1 clock.
>
> What kind of difference should I expect if I sync both
>> generators and responders against the same source, or not sync the
>> responder? I'm thinking that having one source with constant drift may
>> be better than both devices trying to walk/correct the time.
>>
>>
>
> Most hardware clocks in PC's/routers/switches etc have pretty atrocious
> amounts of drift if left to free run[2], sometimes in the order of seconds
> or occasionally minutes per week. To get useful numbers you really do need
> to syncronise them to /something/. Synchronising them to each other causes
> problems as ntp I think (I could be wrong) assumes mostly symmetrical
> latency, and if the latency isn't symmetric assumes it's because one clock
> is running fast/slow and will alter the clock's speed to account for it.
> The great thing about ntp stratum 1 servers is that by definition they have
> more or less the same time no matter where they are, so synchronising each
> against a local ntp server will be a much much better solution. If possible
> you should consider peering with at least 3 upstreams, preferably 4(!)[3]
> other ntp servers.
>
> [1]: To be fair it's a hard problem. Anything that involves time just gets
> more and more complicated the more you look at it, ntp is extremely clever
> and probably knows more about time than I'd ever want to know, but you're
> making it's job hard.
>
> [2]: http://vancouver-webpages.com/time/ /
> http://vancouver-webpages.com/time/ltmhist.png
>
> [3]:
> http://twiki.ntp.org/bin/view/Support/SelectingOffsiteNTPServers#Section_5.3.3
> .
>

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com