Mailing List Archive: NANOG: users

Level 3 - "legacy" Wiltel/Looking Glass bandwidth

Index | Next | Previous | View Threaded

scott at doc

Jul 1, 2009, 11:15 PM

Post #1 of 11 (836 views)
Permalink

Level 3 - "legacy" Wiltel/Looking Glass bandwidth

We're looking at getting connectivity via Level 3 in a particular
datacenter, but we're being told that it's "legacy Wiltel/Looking Glass"
rather than "true" Level 3.

Given that both of these acquisitions occurred years ago should I be
worried, or is this "legacy" connectivity the same as L3 at any other
datacenter?

Scott.

drais at icantclick

Jul 2, 2009, 6:18 AM

Post #2 of 11 (779 views)
Permalink

Re: Level 3 - "legacy" Wiltel/Looking Glass bandwidth [In reply to]

On Wed, 1 Jul 2009, Scott Howard wrote:

> We're looking at getting connectivity via Level 3 in a particular
> datacenter, but we're being told that it's "legacy Wiltel/Looking Glass"
> rather than "true" Level 3.
>
> Given that both of these acquisitions occurred years ago should I be
> worried, or is this "legacy" connectivity the same as L3 at any other
> datacenter?

As recently as a year ago, I had circuit issues in a L3 gateway facility
(-not- an aquisition facility). It took 8 hours, and a VP level
escalation to get resolved. The excuse that -every- tech save the last
one gave? "we don't have access to some of the legacy [wiltel] equipment
in the path, we can't diagnose further"

YMMV, etc etc etc. But full integration may still be far from
complete...

[.full disclosure: L3's purchase of Wiltel, then Telcove and Progress,
destroyed my formerly reasonable opinion of L3 as they suddenly became the
monopoly player in my town and were completely unable to deliver or
maintain anything. later issues in L3's own Gateway facilities further
enforced my low opinion of them]

---
david raistrick http://www.netmeister.org/news/learn2quote.html
drais[at]icantclick.org http://www.expita.com/nomime.html

george at montco

Jul 2, 2009, 7:47 AM

Post #3 of 11 (781 views)
Permalink

Re: Level 3 - "legacy" Wiltel/Looking Glass bandwidth [In reply to]

As a Level 3 customer who had connectivity to some legacy Wiltel equipment I
can also attest that service levels were pretty bad following that merger.
We had a few memorable outages where escalations were necessary to get
things resolved and folks familiar with the equipment were hard to find. In
meeting with our account team we were able to get them to agree to roll us
off the old gear which was really inadequate anyway for the purpose which
happened to be an Option A MPLS NNI. It took them a long time to get that
done but since then I would say that service levels have returned to the
normal albeit somewhat less than desirable levels.

We have had relatively few problems with our vanilla transit connections
though. The same goes for some long haul that they inherited with the
Broadwing acquisition. There were some glitches with some of the fiber
grooming they did a while back but that seems to have passed.

I also seem to be receiving less maintenance notifications overall from
Level 3 as a general trend. Hope it continues.

George

jml at packetpimp

Jul 2, 2009, 7:57 AM

Post #4 of 11 (781 views)
Permalink

Re: Level 3 - "legacy" Wiltel/Looking Glass bandwidth [In reply to]

We're not very happy with Level3 anymore either, terrible support, no
RFO is ever given, tickets are closed with no explanation at all. They
bought so many providers close together that they have a lot of work to
do to integrate everything into a workable set of products.

George Carey wrote:
> As a Level 3 customer who had connectivity to some legacy Wiltel
> equipment I can also attest that service levels were pretty bad
> following that merger. We had a few memorable outages where
> escalations were necessary to get things resolved and folks familiar
> with the equipment were hard to find. In meeting with our account team
> we were able to get them to agree to roll us off the old gear which
> was really inadequate anyway for the purpose which happened to be an
> Option A MPLS NNI. It took them a long time to get that done but since
> then I would say that service levels have returned to the normal
> albeit somewhat less than desirable levels.
> We have had relatively few problems with our vanilla transit
> connections though. The same goes for some long haul that they
> inherited with the Broadwing acquisition. There were some glitches
> with some of the fiber grooming they did a while back but that seems
> to have passed.
> I also seem to be receiving less maintenance notifications overall
> from Level 3 as a general trend. Hope it continues.
>
> George
>

dhubbard at dino

Jul 2, 2009, 8:01 AM

Post #5 of 11 (780 views)
Permalink

RE: Level 3 - "legacy" Wiltel/Looking Glass bandwidth [In reply to]

From: nanog[at]nanog.org
>
> We're not very happy with Level3 anymore either, terrible support, no
> RFO is ever given, tickets are closed with no explanation at
> all. They
> bought so many providers close together that they have a lot
> of work to
> do to integrate everything into a workable set of products.

Haven't had any support issues, but we've had a billing
dispute that's been going back and forth for 14 months now,
yes, months, and during that time we've gone through three
different sales reps, the newest one brought on more than
three months ago and I've yet to hear from the guy.

Good times...

David

alex at blastro

Jul 2, 2009, 8:22 AM

Post #6 of 11 (777 views)
Permalink

Re: Level 3 - "legacy" Wiltel/Looking Glass bandwidth [In reply to]

We were a former Wiltel customer that was bought out by Level 3. Wiltel
service had been great, and then as soon as level 3 took over, things
went downhill. We had GigE service, and wanted another line, but Level
3 said they didn't want to sell any more ports on Wiltel gear. We also
had serious problems even when we tried to disconnect. They didn't have
access to any of the Wiltel records, so they couldn't even tell who had
ordered what. Big mess overall. I'm not going to be working with Level
3 anymore if I can avoid it, and I definitely wouldn't get in on legacy
Wiltel stuff under Level 3.

Alex Thurlow
Blastro Networks

http://www.blastro.com
http://www.roxwel.com
http://www.yallwire.com

On 7/2/2009 10:01 AM, David Hubbard wrote:
> From: nanog[at]nanog.org
>
>> We're not very happy with Level3 anymore either, terrible support, no
>> RFO is ever given, tickets are closed with no explanation at
>> all. They
>> bought so many providers close together that they have a lot
>> of work to
>> do to integrate everything into a workable set of products.
>>
>
> Haven't had any support issues, but we've had a billing
> dispute that's been going back and forth for 14 months now,
> yes, months, and during that time we've gone through three
> different sales reps, the newest one brought on more than
> three months ago and I've yet to hear from the guy.
>
> Good times...
>
> David
>
>

will at loopfree

Jul 2, 2009, 9:38 AM

Post #7 of 11 (777 views)
Permalink

Re: Level 3 - "legacy" Wiltel/Looking Glass bandwidth [In reply to]

We have an old Wiltel DS3 homed from their Anaheim POP... all traffic that's
not going to destinations on the old 7911 backbone seems to be backhauled to
Level3 in San Jose before getting anywhere else, like so:

1 (internal)
2 (internal)
3 (internal)
4 anhmca1wcx1-atm10-0-0.wcg.net (64.200.142.169) 8.017 ms 8.596 ms 8.138 ms
5 anhmca1wcx3-oc48.wcg.net (64.200.143.65) 9.325 ms 8.056 ms 9.485 ms
6 64.200.249.122 (64.200.249.122) 79.108 ms 251.438 ms 222.596 ms
7 64.200.249.142 (64.200.249.142) 16.357 ms 16.167 ms 17.004 ms
8 te-3-2-70.car4.SanJose1.Level3.net (4.68.110.25) 12.568 ms 13.195 ms 13.384 ms

So they don't seem to interconnect 7911 and 3356 in Los Angeles. We complained
about this a year ago and they basically said, "tough, you bought IP transit,
we're giving you IP transit".

Anyway, so in the meantime we bougt a new gig-e to 3356 in San Jose...

1 (internal)
2 (internal)
3 (internal)
4 (internal)
5 ge-x-x.car4.SanJose1.Level3.net (4.71.x.x) 9.063 ms 7.780 ms 8.460 ms

So it almost looks like my gig-e in San Jose is off the same router they
backhaul 7911/Anaheim via.

I wouldn't take a new connection to 7911 unless you're okay with this sort of
thing. Performance has been fine other than the 5-10ms of extra latency, but the
asthetics bug me more than anything. If they communicated more about what
they're doing on the 7911 net and what the PLAN is for transitioning things, I'd
be happier. But it's been this way for years now so we're just disconencting the
old one, which is probably what they're waiting for everyone to do rather than
really merge the networks.

-Will

On Wed, Jul 01, 2009 at 11:15:42PM -0700, Scott Howard wrote:
> Date: Wed, 1 Jul 2009 23:15:42 -0700
> Subject: Level 3 - "legacy" Wiltel/Looking Glass bandwidth
> From: Scott Howard <scott[at]doc.net.au>
> To: nanog[at]nanog.org
>
> We're looking at getting connectivity via Level 3 in a particular
> datacenter, but we're being told that it's "legacy Wiltel/Looking Glass"
> rather than "true" Level 3.
>
> Given that both of these acquisitions occurred years ago should I be
> worried, or is this "legacy" connectivity the same as L3 at any other
> datacenter?
>
> Scott.

eric.nowland at wyoming

Jul 2, 2009, 10:21 AM

Post #8 of 11 (771 views)
Permalink

Re: Level 3 - "legacy" Wiltel/Looking Glass bandwidth [In reply to]

Will Orton wrote:
> We have an old Wiltel DS3 homed from their Anaheim POP... all traffic that's
> not going to destinations on the old 7911 backbone seems to be backhauled to
> Level3 in San Jose before getting anywhere else, like so:
>
> 1 (internal)
> 2 (internal)
> 3 (internal)
> 4 anhmca1wcx1-atm10-0-0.wcg.net (64.200.142.169) 8.017 ms 8.596 ms 8.138 ms
> 5 anhmca1wcx3-oc48.wcg.net (64.200.143.65) 9.325 ms 8.056 ms 9.485 ms
> 6 64.200.249.122 (64.200.249.122) 79.108 ms 251.438 ms 222.596 ms
> 7 64.200.249.142 (64.200.249.142) 16.357 ms 16.167 ms 17.004 ms
> 8 te-3-2-70.car4.SanJose1.Level3.net (4.68.110.25) 12.568 ms 13.195 ms 13.384 ms
>
> So they don't seem to interconnect 7911 and 3356 in Los Angeles. We complained
> about this a year ago and they basically said, "tough, you bought IP transit,
> we're giving you IP transit".
>
> Anyway, so in the meantime we bougt a new gig-e to 3356 in San Jose...
>
>
> 1 (internal)
> 2 (internal)
> 3 (internal)
> 4 (internal)
> 5 ge-x-x.car4.SanJose1.Level3.net (4.71.x.x) 9.063 ms 7.780 ms 8.460 ms
>
>
We have had the same issues with the Broadwing network. Our OC-3
connections are in Wyoming and they lugged everything through Salt Lake
and then to San Jose adding about 30 - 50 ms. We placed orders with
Level 3 to move them onto the Level 3 POP's last year and so far we have
one of them up that they finished last week.

> So it almost looks like my gig-e in San Jose is off the same router they
> backhaul 7911/Anaheim via.
>
> I wouldn't take a new connection to 7911 unless you're okay with this sort of
> thing. Performance has been fine other than the 5-10ms of extra latency, but the
> asthetics bug me more than anything. If they communicated more about what
> they're doing on the 7911 net and what the PLAN is for transitioning things, I'd
> be happier. But it's been this way for years now so we're just disconencting the
> old one, which is probably what they're waiting for everyone to do rather than
> really merge the networks.
>
> -Will
>
>
> On Wed, Jul 01, 2009 at 11:15:42PM -0700, Scott Howard wrote:
>
>> Date: Wed, 1 Jul 2009 23:15:42 -0700
>> Subject: Level 3 - "legacy" Wiltel/Looking Glass bandwidth
>> From: Scott Howard <scott[at]doc.net.au>
>> To: nanog[at]nanog.org
>>
>> We're looking at getting connectivity via Level 3 in a particular
>> datacenter, but we're being told that it's "legacy Wiltel/Looking Glass"
>> rather than "true" Level 3.
>>
>> Given that both of these acquisitions occurred years ago should I be
>> worried, or is this "legacy" connectivity the same as L3 at any other
>> datacenter?
>>
>> Scott.
>>
>
>
>
We have also had the fun of the musical sales reps as we are on our 4th
one since they took of Broadwing...

Thanks
Eric Nowland
Sr Network Engineer
Cerento/Wyoming.com

up at 3

Jul 4, 2009, 8:20 AM

Post #9 of 11 (715 views)
Permalink

Re: Level 3 - "legacy" Wiltel/Looking Glass bandwidth [In reply to]

On Wed, 1 Jul 2009, Scott Howard wrote:

> We're looking at getting connectivity via Level 3 in a particular
> datacenter, but we're being told that it's "legacy Wiltel/Looking Glass"
> rather than "true" Level 3.
>
> Given that both of these acquisitions occurred years ago should I be
> worried, or is this "legacy" connectivity the same as L3 at any other
> datacenter?

While I cannot speak directly to their treatment of former Wiltel
customers, I can tell you that once they acquired Broadwing, service in
their Norristown, PA data center went from not-so-great to completely
unacceptable. IIRC, we've had about 6 multi-hour outages in the past
year.

Apparently, that data center is connected to their Philly POP via a
Foundry Big Iron switch that suffers from broadcast storms periodically,
which can only be fixed by their dispatching a tech to Philly to
power-cycle it, which for some reason takes from 1 to 4 hours. Why
they're not familiar with remote-power cycling equipment is beyond me, let
alone why they haven't resolved the issue properly, despite having
supposedly replaced hardware at one point.

My 3 year contract is up next month, after which I am so out of there.
The fact that L3 tried to double their price on me in the middle of that
contract, only backing down after getting two lawyers involved, didn't
help my opinion of them as a company, either.

James Smallacombe PlantageNet, Inc. CEO and Janitor
up[at]3.am http://3.am
=========================================================================

justin at justinshore

Jul 7, 2009, 4:17 AM

Post #10 of 11 (644 views)
Permalink

Re: Level 3 - "legacy" Wiltel/Looking Glass bandwidth [In reply to]

Scott Howard wrote:
> We're looking at getting connectivity via Level 3 in a particular
> datacenter, but we're being told that it's "legacy Wiltel/Looking Glass"
> rather than "true" Level 3.
>
> Given that both of these acquisitions occurred years ago should I be
> worried, or is this "legacy" connectivity the same as L3 at any other
> datacenter?

We were initially homed to the old Telcove network. Never had any
trouble with those guys. When Level3 bought them they canned all the
local IP folks. That forced you to work with the remaining overworked
IP folks back on the East coast (Angela and a guy who's name I forget).
Their local transport techs are good but there are very few of them
left now, as compared to seeing dozens of their trucks roam the streets
daily.

We eventually asked to be moved off of 19094 and onto 3356. The extra
Telcove hop made for some less preferred and inefficient routing. All
they did was extend 3356 to the local 7600 though. The single Wichita
7600 gets on the old ring in a very fugly way. Working != correct,
proper, reliable or SP-grade.

We just turned up a new 200Mbps circuit to them. Wichita was flagged as
not allowing any more high-speed circuits so they provisioned our
circuit on a new ring to St Louis. I'm actually glad that's the case.
I'm hoping that it's more stable than the KC-Wichita-Houston-Dallas
ring has been in the past. We had several complete and partial outages
(read: dozen plus in 2 years time) on that ring. The most recent was a
few months ago when we suddenly lost all but about 2000 routes from L3.
I spent close to 6 hours on that problem in the wee hours, trying to
get someone to diagnose the issue.

When we turned up the new 200Mbps circuit we asked for a way to do a
speedtest on it. We got nothing. Apparently L3 forced Telcove to take
down their own speedtest site which were pointed to after we turned up
the 100Mbps circuit. L3 apparently doesn't offer a speedtest site of
their own. I find that to be completely unacceptable. Every time we
tried to take this position we got the same old line of "we've got
everything in the path configured correctly; you'll get the full
200Mbps" to which I'd reply with a reminder that we got the same
assurance when we turned up the 100Mbps with them a year prior only to
later discover a cap of around 50Mbps somewhere in the middle. Our
account team's hands are tied. There isn't anything that they can do
about it. I've got it documented in email so if we suddenly flatline
again at some percentage under 100% we'll raise an unholy hell with them.

We've also had significant problems getting some planned maintenance
notifications after the fact (ie, after the window and what appears like
an outage to us).

YMMV but if I had the choice I'd try to get connected to the real L3
backbone and not that of an acquisition. The acquisition networks were
probably much more reliable before they got bought. Now that they've
been stripped of their resources the legacy edges are showing signs of
old age, alzheimer's, and senile dementia. It's a shame to see them
reduced to that.

Justin

streiner at cluebyfour

Jul 7, 2009, 7:24 AM

Post #11 of 11 (639 views)
Permalink

Re: Level 3 - "legacy" Wiltel/Looking Glass bandwidth [In reply to]

On Tue, 7 Jul 2009, Justin Shore wrote:

> Every time we tried to take this position we got the same old line of
> "we've got everything in the path configured correctly; you'll get the
> full 200Mbps" to which I'd reply with a reminder that we got the same
> assurance when we turned up the 100Mbps with them a year prior only to
> later discover a cap of around 50Mbps somewhere in the middle. Our
> account team's hands are tied. There isn't anything that they can do
> about it. I've got it documented in email so if we suddenly flatline
> again at some percentage under 100% we'll raise an unholy hell with them.

I ran into similar problems with Verizon on a long-haul EPL circuit a few
years ago. Speed problems were aggravated by the fact that this
was still a very new service for them, so almost none of their NCC
folks were trained in provisioning or troubleshooting them. Assuming that
L3 provisions circuits like this over a SONET transport, then they're
basically bonding four STS-1s into a 200 meg logical pipe. If the
bonding and mapping isn't done properly all the way through the system,
including the end terminals where the Ethernet <-> SONET conversions
happen, then you don't get the full bandwidth. You might also see other
odd side effects like unexplainable jitter, occasional retransmits, or
other throughput killers.

jms

Index | Next | Previous | View Threaded

Interested in having your list archived? Contact lists@gossamer-threads.com