Mailing List Archive: NANOG: users

XO Outage

Index | Next | Previous | View Threaded

sharp at sharpone

Sep 22, 2008, 2:35 PM

Post #1 of 5 (1532 views)
Permalink

XO Outage

We are seeing some issues w/ XO/Savvis peering..

Trace from XO to Savvis IP space (64.75.10.151)

Keys: Help Display mode Restart statistics Order of fields quit

Packets Pings
Host
Loss% Snt Last Avg Best Wrst StDev
1. scrubbed

0.0% 6 0.6 0.5 0.4 0.6 0.1
2.
ip65-44-114-97.z114-44-65.customer.algx.net
0.0% 6 1.3 1.3 1.2 1.4 0.1
3. ???

Trace from Savvis to XO IP space (65.44.114.97)

1. scrubbed

0.0% 38 0.4 0.4 0.3 0.5 0.1
2.
64.41.199.129
0.0% 37 1.0 24.0 0.6 330.2 80.4
3.
hr1-ge-7-47.santaclarasc5.savvis.net
0.0% 37 0.7 1.4 0.6 27.3 4.4
4.
er1-te-1-0-0.sanjose3equinix.savvis.net
0.0% 37 0.7 5.2 0.6 140.3 23.2
5.
cr1-tenge-0-7-5-0.sanfrancisco.savvis.net
2.7% 37 2.9 4.0 2.6 16.6 2.5
6.
cr2-pos-0-0-3-3.dallas.savvis.net
0.0% 37 42.6 43.1 42.3 51.4 1.4
7.
dpr1-ge-4-0-0.dallasequinix.savvis.net
0.0% 37 43.1 44.8 42.9 76.9 6.7
8.
er1-te-2-1.dallasequinix.savvis.net
0.0% 37 43.3 49.2 42.8 233.6 31.6
9.
208.175.175.90
0.0% 37 43.0 42.8 42.6 43.6 0.2
10.
65.106.1.102
75.0% 37 43.5 46.5 43.4 62.9 6.3
11.
65.106.1.101
0.0% 37 43.4 47.8 43.2 112.3 12.5
12.
65.106.0.41
0.0% 37 57.5 65.1 57.1 177.3 21.0
13.
65.106.1.73
0.0% 37 57.4 66.5 57.1 162.1 24.2
14. ???

Trying to call into XO and they aren't even taking calls, they mention
something about network issues in Spokane. Any ideas as to what is going
on/ETA to fix?

--Justin

mike at digg

Sep 22, 2008, 2:40 PM

Post #2 of 5 (1432 views)
Permalink

Re: XO Outage [In reply to]

yeah, we noticed it as well. looks like they're back now. i noticed
their routes drop off at about 14:16 PT and come back just after 14:30 PT.

+m

Justin Sharp wrote:
> We are seeing some issues w/ XO/Savvis peering..
>
> Trace from XO to Savvis IP space (64.75.10.151)
>
> Keys: Help Display mode Restart statistics Order of fields quit
>
> Packets Pings
> Host
> Loss% Snt Last Avg Best Wrst StDev
> 1. scrubbed
>
> 0.0% 6 0.6 0.5 0.4 0.6 0.1
> 2.
> ip65-44-114-97.z114-44-65.customer.algx.net
> 0.0% 6 1.3 1.3 1.2 1.4 0.1
> 3. ???
>
>
> Trace from Savvis to XO IP space (65.44.114.97)
>
> 1. scrubbed
>
> 0.0% 38 0.4 0.4 0.3 0.5 0.1
> 2.
> 64.41.199.129
> 0.0% 37 1.0 24.0 0.6 330.2 80.4
> 3.
> hr1-ge-7-47.santaclarasc5.savvis.net
> 0.0% 37 0.7 1.4 0.6 27.3 4.4
> 4.
> er1-te-1-0-0.sanjose3equinix.savvis.net
> 0.0% 37 0.7 5.2 0.6 140.3 23.2
> 5.
> cr1-tenge-0-7-5-0.sanfrancisco.savvis.net
> 2.7% 37 2.9 4.0 2.6 16.6 2.5
> 6.
> cr2-pos-0-0-3-3.dallas.savvis.net
> 0.0% 37 42.6 43.1 42.3 51.4 1.4
> 7.
> dpr1-ge-4-0-0.dallasequinix.savvis.net
> 0.0% 37 43.1 44.8 42.9 76.9 6.7
> 8.
> er1-te-2-1.dallasequinix.savvis.net
> 0.0% 37 43.3 49.2 42.8 233.6 31.6
> 9.
> 208.175.175.90
> 0.0% 37 43.0 42.8 42.6 43.6 0.2
> 10.
> 65.106.1.102
> 75.0% 37 43.5 46.5 43.4 62.9 6.3
> 11.
> 65.106.1.101
> 0.0% 37 43.4 47.8 43.2 112.3 12.5
> 12.
> 65.106.0.41
> 0.0% 37 57.5 65.1 57.1 177.3 21.0
> 13.
> 65.106.1.73
> 0.0% 37 57.4 66.5 57.1 162.1 24.2
> 14. ???
>
> Trying to call into XO and they aren't even taking calls, they mention
> something about network issues in Spokane. Any ideas as to what is
> going on/ETA to fix?
>
> --Justin
>
>

mike.lyon at gmail

Sep 22, 2008, 2:43 PM

Post #3 of 5 (1441 views)
Permalink

Re: XO Outage [In reply to]

Also seeing some issues with XO out here:

Tracing route to yahoo.com [68.180.206.184]
over a maximum of 30 hops:

1 2 ms 1 ms 1 ms 10.100.20.1
2 2 ms 1 ms 2 ms sjcisr01-int.wyse.com [10.100.1.15]
3 3 ms 2 ms 2 ms 132.237.245.1
4 3 ms 3 ms 2 ms 207.88.3.37.ptr.us.xo.net [207.88.3.37]
5 7 ms 3 ms 3 ms p3-0-0.mar2.fremont-ca.us.xo.net [207.88.80.181]

6 16 ms 5 ms 3 ms p4-0-0.rar2.sanjose-ca.us.xo.net [65.106.5.137]

7 12 ms 14 ms 11 ms p6-0-0.rar1.la-ca.us.xo.net [65.106.0.17]
8 17 ms 11 ms 11 ms p0-0-0d0.rar2.la-ca.us.xo.net [65.106.1.50]
9 44 ms 50 ms 52 ms p6-0-0.rar1.dallas-tx.us.xo.net [65.106.0.13]
10 * * * Request timed out.
11 44 ms 44 ms 44 ms 207.88.12.150.ptr.us.xo.net [207.88.12.150]
12 53 ms 85 ms 52 ms 206.111.5.42.ptr.us.xo.net [206.111.5.42]
^C

On Mon, Sep 22, 2008 at 2:35 PM, Justin Sharp <sharp [at] sharpone> wrote:
> We are seeing some issues w/ XO/Savvis peering..
>
> Trace from XO to Savvis IP space (64.75.10.151)
>
> Keys: Help Display mode Restart statistics Order of fields quit
>
> Packets Pings
> Host
> Loss% Snt Last Avg Best Wrst StDev
> 1. scrubbed
> 0.0% 6 0.6 0.5 0.4 0.6 0.1
> 2. ip65-44-114-97.z114-44-65.customer.algx.net
> 0.0% 6 1.3 1.3 1.2 1.4 0.1
> 3. ???
>
>
> Trace from Savvis to XO IP space (65.44.114.97)
>
> 1. scrubbed
> 0.0% 38 0.4 0.4 0.3 0.5 0.1
> 2. 64.41.199.129
> 0.0% 37 1.0 24.0 0.6 330.2 80.4
> 3. hr1-ge-7-47.santaclarasc5.savvis.net
> 0.0% 37 0.7 1.4 0.6 27.3 4.4
> 4. er1-te-1-0-0.sanjose3equinix.savvis.net
> 0.0% 37 0.7 5.2 0.6 140.3 23.2
> 5. cr1-tenge-0-7-5-0.sanfrancisco.savvis.net
> 2.7% 37 2.9 4.0 2.6 16.6 2.5
> 6. cr2-pos-0-0-3-3.dallas.savvis.net
> 0.0% 37 42.6 43.1 42.3 51.4 1.4
> 7. dpr1-ge-4-0-0.dallasequinix.savvis.net
> 0.0% 37 43.1 44.8 42.9 76.9 6.7
> 8. er1-te-2-1.dallasequinix.savvis.net
> 0.0% 37 43.3 49.2 42.8 233.6 31.6
> 9. 208.175.175.90
> 0.0% 37 43.0 42.8 42.6 43.6 0.2
> 10. 65.106.1.102
> 75.0% 37 43.5 46.5 43.4 62.9 6.3
> 11. 65.106.1.101
> 0.0% 37 43.4 47.8 43.2 112.3 12.5
> 12. 65.106.0.41
> 0.0% 37 57.5 65.1 57.1 177.3 21.0
> 13. 65.106.1.73
> 0.0% 37 57.4 66.5 57.1 162.1 24.2
> 14. ???
>
> Trying to call into XO and they aren't even taking calls, they mention
> something about network issues in Spokane. Any ideas as to what is going
> on/ETA to fix?
>
> --Justin
>
>
>

sharp at sharpone

Sep 22, 2008, 2:48 PM

Post #4 of 5 (1436 views)
Permalink

Re: XO Outage [In reply to]

Seems like Savvis/XO routes have been restored, albeit through an
unusual Dallas route (usually takes an XO Los Angeles route)..

1. scrubbed

0.0% 204 0.4 0.7 0.3 7.0 0.9
2.
ip65-44-114-97.z114-44-65.customer.algx.net
0.0% 204 1.8 1.8 1.1 8.4 1.2
3.
ip65-46-48-29.z48-46-65.customer.algx.net
0.0% 204 3.3 6.3 2.8 96.4 9.0
4.
p6-0-0-0.mar1.saltlake-ut.us.xo.net
0.5% 204 8.1 14.4 3.1 161.0 26.3
5.
p4-1-0-0.rar1.denver-co.us.xo.net
0.0% 204 15.5 25.6 15.3 191.4 26.7
6.
p0-0-0d0.rar2.denver-co.us.xo.net
0.0% 204 16.2 22.4 16.1 129.2 14.4
7.
p1-0-0.rar2.dallas-tx.us.xo.net
0.0% 204 45.3 38.3 29.9 201.3 21.3
8.
te-3-4-0.rar3.dallas-tx.us.xo.net
66.0% 204 30.3 34.2 29.6 116.2 11.6
9.
207.88.12.146.ptr.us.xo.net
0.0% 204 33.1 32.7 29.4 106.7 8.5
10.
208.175.175.89
0.0% 204 29.8 35.7 29.5 195.4 20.0
11.
dpr1-ge-2-0-0.dallasequinix.savvis.net
0.0% 204 34.3 33.8 29.6 115.2 10.5
12.
cr2-tengige0-7-5-0.dallas.savvis.net
1.5% 204 30.2 33.3 29.7 68.6 5.4
13.
cr2-loopback.sfo.savvis.net
0.0% 203 75.1 209.8 74.0 3270. 493.7
14.
er1-te-2-0-1.sanjose3equinix.savvis.net
0.0% 203 71.8 75.6 71.7 219.5 11.7
15.
hr1-te-2-0-0.santaclarasc5.savvis.net
0.0% 203 72.1 75.5 72.0 194.0 9.4
16.
216.34.2.226
0.0% 203 72.3 105.4 72.1 723.7 102.5
17.
64.41.199.140
28.6% 203 77.5 74.9 72.0 183.1 9.7
18. scrubbed

0.0% 203 73.8 76.0 72.4 207.9 11.3

Hop 8 looks busy..

--Justin

Mike Lyon wrote:
> Also seeing some issues with XO out here:
>
> Tracing route to yahoo.com [68.180.206.184]
> over a maximum of 30 hops:
>
> 1 2 ms 1 ms 1 ms 10.100.20.1
> 2 2 ms 1 ms 2 ms sjcisr01-int.wyse.com [10.100.1.15]
> 3 3 ms 2 ms 2 ms 132.237.245.1
> 4 3 ms 3 ms 2 ms 207.88.3.37.ptr.us.xo.net [207.88.3.37]
> 5 7 ms 3 ms 3 ms p3-0-0.mar2.fremont-ca.us.xo.net [207.88.80.181]
>
> 6 16 ms 5 ms 3 ms p4-0-0.rar2.sanjose-ca.us.xo.net [65.106.5.137]
>
> 7 12 ms 14 ms 11 ms p6-0-0.rar1.la-ca.us.xo.net [65.106.0.17]
> 8 17 ms 11 ms 11 ms p0-0-0d0.rar2.la-ca.us.xo.net [65.106.1.50]
> 9 44 ms 50 ms 52 ms p6-0-0.rar1.dallas-tx.us.xo.net [65.106.0.13]
> 10 * * * Request timed out.
> 11 44 ms 44 ms 44 ms 207.88.12.150.ptr.us.xo.net [207.88.12.150]
> 12 53 ms 85 ms 52 ms 206.111.5.42.ptr.us.xo.net [206.111.5.42]
> ^C
>
>
>
> On Mon, Sep 22, 2008 at 2:35 PM, Justin Sharp <sharp [at] sharpone> wrote:
>
>> We are seeing some issues w/ XO/Savvis peering..
>>
>> Trace from XO to Savvis IP space (64.75.10.151)
>>
>> Keys: Help Display mode Restart statistics Order of fields quit
>>
>> Packets Pings
>> Host
>> Loss% Snt Last Avg Best Wrst StDev
>> 1. scrubbed
>> 0.0% 6 0.6 0.5 0.4 0.6 0.1
>> 2. ip65-44-114-97.z114-44-65.customer.algx.net
>> 0.0% 6 1.3 1.3 1.2 1.4 0.1
>> 3. ???
>>
>>
>> Trace from Savvis to XO IP space (65.44.114.97)
>>
>> 1. scrubbed
>> 0.0% 38 0.4 0.4 0.3 0.5 0.1
>> 2. 64.41.199.129
>> 0.0% 37 1.0 24.0 0.6 330.2 80.4
>> 3. hr1-ge-7-47.santaclarasc5.savvis.net
>> 0.0% 37 0.7 1.4 0.6 27.3 4.4
>> 4. er1-te-1-0-0.sanjose3equinix.savvis.net
>> 0.0% 37 0.7 5.2 0.6 140.3 23.2
>> 5. cr1-tenge-0-7-5-0.sanfrancisco.savvis.net
>> 2.7% 37 2.9 4.0 2.6 16.6 2.5
>> 6. cr2-pos-0-0-3-3.dallas.savvis.net
>> 0.0% 37 42.6 43.1 42.3 51.4 1.4
>> 7. dpr1-ge-4-0-0.dallasequinix.savvis.net
>> 0.0% 37 43.1 44.8 42.9 76.9 6.7
>> 8. er1-te-2-1.dallasequinix.savvis.net
>> 0.0% 37 43.3 49.2 42.8 233.6 31.6
>> 9. 208.175.175.90
>> 0.0% 37 43.0 42.8 42.6 43.6 0.2
>> 10. 65.106.1.102
>> 75.0% 37 43.5 46.5 43.4 62.9 6.3
>> 11. 65.106.1.101
>> 0.0% 37 43.4 47.8 43.2 112.3 12.5
>> 12. 65.106.0.41
>> 0.0% 37 57.5 65.1 57.1 177.3 21.0
>> 13. 65.106.1.73
>> 0.0% 37 57.4 66.5 57.1 162.1 24.2
>> 14. ???
>>
>> Trying to call into XO and they aren't even taking calls, they mention
>> something about network issues in Spokane. Any ideas as to what is going
>> on/ETA to fix?
>>
>> --Justin
>>
>>
>>
>>

zaid at zaidali

Sep 22, 2008, 2:49 PM

Post #5 of 5 (1458 views)
Permalink

Re: XO Outage [In reply to]

I am seeing it on my end also:

traceroute: Warning: www.cnn.com has multiple addresses; using
157.166.224.25
traceroute to www.cnn.com (157.166.224.25), 64 hops max, 40 byte packets
1 hq-rtr1.genius.local (64.244.66.1) 0.891 ms 0.429 ms 0.449 ms
2 ip65-46-253-157.z253-46-65.customer.algx.net (65.46.253.157) 1.856
ms 2.860 ms 1.881 ms
3 p3-0-0.mar2.fremont-ca.us.xo.net (207.88.80.181) 16.922 ms 2.041
ms 2.013 ms
4 p4-3-0.rar2.sanjose-ca.us.xo.net (65.106.5.161) 2.637 ms 2.192 ms
2.823 ms
5 p6-0-0.rar1.la-ca.us.xo.net (65.106.0.17) 10.308 ms 10.258 ms
10.386 ms
6 207.88.13.22.ptr.us.xo.net (207.88.13.22) 10.931 ms 10.535 ms
10.037 ms
7 *^C

Justin Sharp wrote:
> We are seeing some issues w/ XO/Savvis peering..
>
> Trace from XO to Savvis IP space (64.75.10.151)
>
> Keys: Help Display mode Restart statistics Order of fields quit
>
> Packets Pings
> Host
> Loss% Snt Last Avg Best Wrst StDev
> 1. scrubbed
>
> 0.0% 6 0.6 0.5 0.4 0.6 0.1
> 2.
> ip65-44-114-97.z114-44-65.customer.algx.net
> 0.0% 6 1.3 1.3 1.2 1.4 0.1
> 3. ???
>
>
> Trace from Savvis to XO IP space (65.44.114.97)
>
> 1. scrubbed
>
> 0.0% 38 0.4 0.4 0.3 0.5 0.1
> 2.
> 64.41.199.129
> 0.0% 37 1.0 24.0 0.6 330.2 80.4
> 3.
> hr1-ge-7-47.santaclarasc5.savvis.net
> 0.0% 37 0.7 1.4 0.6 27.3 4.4
> 4.
> er1-te-1-0-0.sanjose3equinix.savvis.net
> 0.0% 37 0.7 5.2 0.6 140.3 23.2
> 5.
> cr1-tenge-0-7-5-0.sanfrancisco.savvis.net
> 2.7% 37 2.9 4.0 2.6 16.6 2.5
> 6.
> cr2-pos-0-0-3-3.dallas.savvis.net
> 0.0% 37 42.6 43.1 42.3 51.4 1.4
> 7.
> dpr1-ge-4-0-0.dallasequinix.savvis.net
> 0.0% 37 43.1 44.8 42.9 76.9 6.7
> 8.
> er1-te-2-1.dallasequinix.savvis.net
> 0.0% 37 43.3 49.2 42.8 233.6 31.6
> 9.
> 208.175.175.90
> 0.0% 37 43.0 42.8 42.6 43.6 0.2
> 10.
> 65.106.1.102
> 75.0% 37 43.5 46.5 43.4 62.9 6.3
> 11.
> 65.106.1.101
> 0.0% 37 43.4 47.8 43.2 112.3 12.5
> 12.
> 65.106.0.41
> 0.0% 37 57.5 65.1 57.1 177.3 21.0
> 13.
> 65.106.1.73
> 0.0% 37 57.4 66.5 57.1 162.1 24.2
> 14. ???
>
> Trying to call into XO and they aren't even taking calls, they mention
> something about network issues in Spokane. Any ideas as to what is going
> on/ETA to fix?
>
> --Justin
>
>

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact Gossamer Threads