Login | Register For Free | Help
Search for: (Advanced)

Mailing List Archive: NANOG: users

Fire, Power loss at Fisher Plaza in Seattle

 

 

NANOG users RSS feed   Index | Next | Previous | View Threaded


darren at bolding

Jul 3, 2009, 4:19 AM

Post #1 of 21 (1913 views)
Permalink
Fire, Power loss at Fisher Plaza in Seattle

Fisher Plaza, a self-styled carrier hotel in Seattle, and home to multiple
datacenter and colocation providers, has had a major issue in one of its
buildings late last night, early this morning.
The best information I am aware of is that there was a failure in the
main/generator transfer switch which resulted in a fire. The sprinkler
system activated. From speaking to the fire battalion chief, I am under the
impression that Seattle Fire did use water on the fire as well, but I am
unsure of this.

Given the failure location, generator power was not available, and cooling
failed. UPS power to systems continued, and I can personally vouch that
they held out for well over an hour. When we were able to access our
equipment, ambient air temps were well over 100 degrees in the room our
equipment is located in.

At least some, if not many circuits were affected. Several large
co-location providers and other datacenters are located in the facility,
these facilities have no power.

As this was the main/generator switch, and it is now highly damaged, the
circuits in the area are damaged, and the entire area is doused in water, a
rapid restoration of power does not seem likely. Fisher Plaza's phone
numbers now result in fast-busy signals, so I have no recent update from
them directly.

Interestingly, this building is also the production studios for several
Seattle TV and radio stations.

There is no ETA for resolution.

--D

--
-- Darren Bolding --
-- darren [at] bolding --


joe at disconformity

Jul 3, 2009, 4:40 AM

Post #2 of 21 (1862 views)
Permalink
Re: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

Multiple folks on Twitter who are in the area are reporting a 5-6 hour
ETA.

-Joe

--
Joe Richards <joe [at] disconformity>
--
ipv4: http://www.disconformity.net [ 72.29.169.48/28 ]
ipv6: http://ipv6.disconformity.net [ 2001:48c0:1001:1::/64 ]
blog: http://www.mainlined.org


sethm at rollernet

Jul 3, 2009, 10:00 AM

Post #3 of 21 (1854 views)
Permalink
Re: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

Darren Bolding wrote:
>
> Interestingly, this building is also the production studios for several
> Seattle TV and radio stations.
>
> There is no ETA for resolution.
>

Apparently it took authorize.net with it, too:
http://twitter.com/authorizenet

~Seth


dhubbard at dino

Jul 3, 2009, 10:05 AM

Post #4 of 21 (1855 views)
Permalink
RE: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

From: Seth Mattinen [mailto:sethm [at] rollernet]
>
> Apparently it took authorize.net with it, too:
> http://twitter.com/authorizenet
>
> ~Seth

No technical explanation of course but it also took down
their 'backup facility' according to them on twitter;
I assume some bad routing/DNS if they do actually have
a backup facility. Lots of online stores are
offline right now because of this, and the holiday is
unfortunately keeping those store owners from knowing
they are not making sales right now. Life in ecommerce...

David


tomb at byrneit

Jul 3, 2009, 10:20 AM

Post #5 of 21 (1852 views)
Permalink
RE: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

This begs the question of what basic parameters should be for a "carrier
hotel" or co-lo.

Given that we're getting designated "Critical Infrastructure", we'd
getter start coming up with some, or we'll have them defined for us.

The old NEBS standards were too much of a straightjacket, but the
current situation, where any buffoon who wants to can claim to be
something they aren't (redundant and reliable) undermines the business
of those who actually spend the money, and make the effort, to provide a
true "carrier grade" co-lo.

This is life in the current Internet: Overpromise, and Underdeliver.


>-----Original Message-----
>From: David Hubbard [mailto:dhubbard [at] dino]
>Sent: Friday, July 03, 2009 10:05 AM
>To: NANOG list
>Subject: RE: Fire, Power loss at Fisher Plaza in Seattle
>
>From: Seth Mattinen [mailto:sethm [at] rollernet]
>>
>> Apparently it took authorize.net with it, too:
>> http://twitter.com/authorizenet
>>
>> ~Seth
>
>No technical explanation of course but it also took down
>their 'backup facility' according to them on twitter;
>I assume some bad routing/DNS if they do actually have
>a backup facility. Lots of online stores are
>offline right now because of this, and the holiday is
>unfortunately keeping those store owners from knowing
>they are not making sales right now. Life in ecommerce...
>
>David


sethm at rollernet

Jul 3, 2009, 10:40 AM

Post #6 of 21 (1849 views)
Permalink
Re: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

Tomas L. Byrnes wrote:
> This begs the question of what basic parameters should be for a "carrier
> hotel" or co-lo.
>
> Given that we're getting designated "Critical Infrastructure", we'd
> getter start coming up with some, or we'll have them defined for us.
>
> The old NEBS standards were too much of a straightjacket, but the
> current situation, where any buffoon who wants to can claim to be
> something they aren't (redundant and reliable) undermines the business
> of those who actually spend the money, and make the effort, to provide a
> true "carrier grade" co-lo.

Absolutely. Then your pricing is so far out of whack with the apparent
competition that it's hard to get customers when it appears one can get
the same/better for far less. Me, personally, I just don't say things
like "100% uptime" or claim to be a carrier-grade facility. But I think
that scares people off when my competitors (and I've seen the insides of
some of the horrid trash heaps they call a NOC) claim they do.


>
> This is life in the current Internet: Overpromise, and Underdeliver.
>

"Our flywheel systems are so failure-proof and thinking outside the box
that we don't need a silly battery UPS that can cold-start!"

I know outages and related discussion end up attracting the off-topic
hammer here on NANOG, but I do find them interesting and worthwhile.

~Seth


mksmith at adhost

Jul 3, 2009, 10:57 AM

Post #7 of 21 (1849 views)
Permalink
RE: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

-----Original Message-----
From: Tomas L. Byrnes [mailto:tomb [at] byrneit]
Sent: Fri 7/3/2009 10:20 AM
To: David Hubbard; NANOG list
Subject: RE: Fire, Power loss at Fisher Plaza in Seattle

This begs the question of what basic parameters should be for a "carrier
hotel" or co-lo.

Given that we're getting designated "Critical Infrastructure", we'd
getter start coming up with some, or we'll have them defined for us.

----

I think the more important question is, "what do you consider redundancy?" We have facilities in Plaza East (no down) and Plaza West (unaffected). If you are critical infrastructure there is no amount of redundancy that you should offload onto a colo provider. Instead, you build your redundancy across different data centers, different providers, different everything. If you rely on a single provider for any of the aforementioned then you have built in at least one single point of failure, regardless of the resiliency of the underlying provider.

My .02, worth almost every penny.

Mike


eriks at nationalfastfreight

Jul 3, 2009, 11:13 AM

Post #8 of 21 (1847 views)
Permalink
RE: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

http://seattletimes.nwsource.com/html/localnews/2009415571_apwafisherpla
zafire1stldwritethru.html




-----Original Message-----
From: David Hubbard [mailto:dhubbard [at] dino]
Sent: Friday, July 03, 2009 1:05 PM
To: NANOG list
Subject: RE: Fire, Power loss at Fisher Plaza in Seattle

From: Seth Mattinen [mailto:sethm [at] rollernet]
>
> Apparently it took authorize.net with it, too:
> http://twitter.com/authorizenet
>
> ~Seth

No technical explanation of course but it also took down
their 'backup facility' according to them on twitter;
I assume some bad routing/DNS if they do actually have
a backup facility. Lots of online stores are
offline right now because of this, and the holiday is
unfortunately keeping those store owners from knowing
they are not making sales right now. Life in ecommerce...

David


herrin-nanog at dirtside

Jul 3, 2009, 11:49 AM

Post #9 of 21 (1850 views)
Permalink
Re: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

On Fri, Jul 3, 2009 at 1:20 PM, Tomas L. Byrnes<tomb [at] byrneit> wrote:
> This begs the question of what basic parameters should be for a
> "carrier hotel" or co-lo. [...] The old NEBS standards were too much
> of a straightjacket.

Tomas,

There is a useful standard: ANSI/TIA-942. It offers specifications for
four tiers of data centers ranging from tier 1 (a basic data center
with no redundancy) to tier 4 (fully fault tolerant).

http://www.tiaonline.org/standards/catalog/search.cfm?standards_criteria=TIA-942
(the 2005 one)

Judging from http://www.techlinks.net/community/articles/article/1-article-submission-forms/14833-a-quick-primer-on-data-center-tier-classifications
there's even research that projects what sort of annual downtime you
can expect for each of the tiers described by the standard.

When I walk into a data center, I make a habit of asking which tier
they achieve, at least for the HVAC and electrical systems. And then I
ask to see the components which the tier claim says they should have.

Regards,
Bill Herrin


--
William D. Herrin ................ herrin [at] dirtside bill [at] herrin
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004


sean at donelan

Jul 3, 2009, 12:22 PM

Post #10 of 21 (1844 views)
Permalink
Re: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

On Fri, 3 Jul 2009, William Herrin wrote:
> There is a useful standard: ANSI/TIA-942. It offers specifications for
> four tiers of data centers ranging from tier 1 (a basic data center
> with no redundancy) to tier 4 (fully fault tolerant).

Are you better off with a single "tier 4" data center, multiple
"tier 1" data centers, or something in between?

Distance and quantity versus complexity and scaling versus cost and risk.
Sometimes no matter what you choose, you might be wrong.

Earth is a single point of failure, where is your backup site?


stuart at tech

Jul 3, 2009, 1:10 PM

Post #11 of 21 (1847 views)
Permalink
Re: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

> Earth is a single point of failure, where is your backup site?

This reminds me of the 1996 thread about how MAE-East still had no
generator. Same topic, roughly, some of the same people (hi, Sean).

Sure, the line about the Earth SPOF is catchy, but in terms of more
likely scenarios: how many people stand *outside* the "tier 4"
datacenter and imagine a fire marshal pointing at the building and
saying, "Turn *that* off, now." I've seen that happen a couple times
since the WilTel POP thing in 1996.

Stephen


tomb at byrneit

Jul 3, 2009, 1:21 PM

Post #12 of 21 (1848 views)
Permalink
RE: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

>
>Earth is a single point of failure, where is your backup site?

[TLB:] Given that all my customers are on Earth, I don't need one if my
customers also are "down".


jeffrey.lyon at blacklotus

Jul 3, 2009, 1:29 PM

Post #13 of 21 (1847 views)
Permalink
Re: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

Wasn't Authorize.net affected by this? We received a support ticket
about why Authorize.net is down today (I don't know either, I don't
ask too many questions).

Jeff

On Fri, Jul 3, 2009 at 4:21 PM, Tomas L. Byrnes<tomb [at] byrneit> wrote:
>
>
>>
>>Earth is a single point of failure, where is your backup site?
>
> [TLB:] Given that all my customers are on Earth, I don't need one if my
> customers also are "down".
>
>
>
>



--
Jeffrey Lyon, Leadership Team
jeffrey.lyon [at] blacklotus | http://www.blacklotus.net
Black Lotus Communications of The IRC Company, Inc.

Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th
at Booth #401.


bc-list at beztech

Jul 3, 2009, 1:29 PM

Post #14 of 21 (1846 views)
Permalink
Re: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

Yes it was.

On Jul 3, 2009, at 4:29 PM, Jeffrey Lyon wrote:

> Wasn't Authorize.net affected by this? We received a support ticket
> about why Authorize.net is down today (I don't know either, I don't
> ask too many questions).
>
> Jeff
>
> On Fri, Jul 3, 2009 at 4:21 PM, Tomas L. Byrnes<tomb [at] byrneit>
> wrote:
>>
>>
>>>
>>> Earth is a single point of failure, where is your backup site?
>>
>> [TLB:] Given that all my customers are on Earth, I don't need one
>> if my
>> customers also are "down".
>>
>>
>>
>>
>
>
>
> --
> Jeffrey Lyon, Leadership Team
> jeffrey.lyon [at] blacklotus | http://www.blacklotus.net
> Black Lotus Communications of The IRC Company, Inc.
>
> Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th
> at Booth #401.
>


bicknell at ufp

Jul 3, 2009, 1:39 PM

Post #15 of 21 (1849 views)
Permalink
Re: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

In a message written on Fri, Jul 03, 2009 at 03:22:14PM -0400, Sean Donelan wrote:
> Are you better off with a single "tier 4" data center, multiple
> "tier 1" data centers, or something in between?

It depends entirely on your dependency on connectivity.

One extreme is something like a Central Office. Lots of cables
from end-sites terminate in the building. Having a duplicate of
the head end termination equipment on the opposite coast is darn
near useless. If the building goes down, the users going through
it go down. "Tier 4" is probably a good idea.

The other extreme is a pure content play (YouTube, Google Search).
Users don't care which data center they hit (within reason), and
indeed often don't know. You're better off having data centers
spread out all over, both so you're more likely to only loose one
at a time, but also so that the loss of one is relatively unimportant.
Once you're already in this architecture, Tier 1 is generally
cheaper.

There are two problems though. First, most folks don't fit neatly
in one of these buckets. They have some ties to local infrastructure,
and some items which are not tied. Latency as a performance penality
is very subjective. A backup 1000 miles away is fine for many
things, and very bad for some things.

Second, most folks don't have choices. It would be nice if most
cities had three each Tier 1, 2, 3 and 4 data centers available so
there was choice and competition but that's rare.

Very few companies consider these choices rationally; often because
choices are made by different groups. I am amazed how many times
inside of an ISP the folks deploying the DNS and mail servers are
firewalled from the folks deploying the network, to the point where
you have to get to the President to reach common management. This
leads to them making choices in opposite directions that end up
costing extra money the company, and often resulting in a much lower
uptimes than expected. Having the network group deploy a single point
of failure to the "Tier 4" data center the server guys required is,
well, silly.

However, more important than all of this is testing your infrastructure.
Would you feel comfortable walking into your data center and ripping
the power cable out of some bit of equipment at random _right now_?
If not, you have no faith your equipment will work in an outage.

--
Leo Bicknell - bicknell [at] ufp - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/


darren at bolding

Jul 3, 2009, 1:57 PM

Post #16 of 21 (1849 views)
Permalink
Re: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

Power to some of the affected sections of the building has been restored via
existing onsite generators. The central power risers cannot be connected to
current generators in a timely manner due to excessive damage to the
electrical switching equipment (and those generators may still be in
standing water). These provide power to a number of colocated systems.
Temporary generators are on order to be connected to the central risers,
and the site expects that to be complete sometime late this evening. As
best I can tell, there is still no utility power connected to any of the
systems.
The AC systems (chiller and crac) are currently not working. It is not
clear to me whether these will be brought back on line when the temporary
generators are available, but I am assuming so.

It was pleasant to see the general positive attitude, sharing of information
and offers of assistance that were made by representatives of the various
tenants, customers and carriers that were on the scene. The usual suspects
(companies and individuals) stepped up and took care of things, as they
always seem to.

--D

On Fri, Jul 3, 2009 at 1:39 PM, Leo Bicknell <bicknell [at] ufp> wrote:

> In a message written on Fri, Jul 03, 2009 at 03:22:14PM -0400, Sean Donelan
> wrote:
> > Are you better off with a single "tier 4" data center, multiple
> > "tier 1" data centers, or something in between?
>
> It depends entirely on your dependency on connectivity.
>
> One extreme is something like a Central Office. Lots of cables
> from end-sites terminate in the building. Having a duplicate of
> the head end termination equipment on the opposite coast is darn
> near useless. If the building goes down, the users going through
> it go down. "Tier 4" is probably a good idea.
>
> The other extreme is a pure content play (YouTube, Google Search).
> Users don't care which data center they hit (within reason), and
> indeed often don't know. You're better off having data centers
> spread out all over, both so you're more likely to only loose one
> at a time, but also so that the loss of one is relatively unimportant.
> Once you're already in this architecture, Tier 1 is generally
> cheaper.
>
> There are two problems though. First, most folks don't fit neatly
> in one of these buckets. They have some ties to local infrastructure,
> and some items which are not tied. Latency as a performance penality
> is very subjective. A backup 1000 miles away is fine for many
> things, and very bad for some things.
>
> Second, most folks don't have choices. It would be nice if most
> cities had three each Tier 1, 2, 3 and 4 data centers available so
> there was choice and competition but that's rare.
>
> Very few companies consider these choices rationally; often because
> choices are made by different groups. I am amazed how many times
> inside of an ISP the folks deploying the DNS and mail servers are
> firewalled from the folks deploying the network, to the point where
> you have to get to the President to reach common management. This
> leads to them making choices in opposite directions that end up
> costing extra money the company, and often resulting in a much lower
> uptimes than expected. Having the network group deploy a single point
> of failure to the "Tier 4" data center the server guys required is,
> well, silly.
>
> However, more important than all of this is testing your infrastructure.
> Would you feel comfortable walking into your data center and ripping
> the power cable out of some bit of equipment at random _right now_?
> If not, you have no faith your equipment will work in an outage.
>
> --
> Leo Bicknell - bicknell [at] ufp - CCIE 3440
> PGP keys at http://www.ufp.org/~bicknell/
>



--
-- Darren Bolding --
-- darren [at] bolding --


tme at americafree

Jul 3, 2009, 1:59 PM

Post #17 of 21 (1849 views)
Permalink
Re: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

On Jul 3, 2009, at 4:29 PM, Ben Carleton wrote:

> Yes it was.
>
> On Jul 3, 2009, at 4:29 PM, Jeffrey Lyon wrote:
>
>> Wasn't Authorize.net affected by this? We received a support ticket
>> about why Authorize.net is down today (I don't know either, I don't
>> ask too many questions).
>>

Authorize.net was for a while completely off the air, and companies
that relied upon them
were not getting credit card authorizations (and, thus, no ecommerce).
I think it is still only
partially functional.

Authorize.net has been communicating with customers mostly
(entirely ?) with twitter - they are

@AuthorizeNet with a hash tab of #authorizenet

If you go there, you will see a lot of status messages like

#authorizenet (cont.) Do not manually submit ARB transactions b/c you
run the risk of your merchants being double billed.
10 minutes ago from web

(i.e., 4:47 EDT).

You will also see a lot of posts from annoyed people if you search on
#authorizenet

It's an interesting use of Web 2.0 for emergency communications.

Regards
Marshall


>> Jeff
>>
>> On Fri, Jul 3, 2009 at 4:21 PM, Tomas L. Byrnes<tomb [at] byrneit>
>> wrote:
>>>
>>>
>>>>
>>>> Earth is a single point of failure, where is your backup site?
>>>
>>> [TLB:] Given that all my customers are on Earth, I don't need one
>>> if my
>>> customers also are "down".
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Jeffrey Lyon, Leadership Team
>> jeffrey.lyon [at] blacklotus | http://www.blacklotus.net
>> Black Lotus Communications of The IRC Company, Inc.
>>
>> Look for us at HostingCon 2009 in Washington, DC on August 10th -
>> 12th
>> at Booth #401.
>>
>
>
>


jeffrey.lyon at blacklotus

Jul 3, 2009, 2:11 PM

Post #18 of 21 (1849 views)
Permalink
Re: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

That's a great idea, use some lame Web 2.0 trend to communicate with
actual real life customers. </sarcasm>

Jeff


On Fri, Jul 3, 2009 at 4:59 PM, Marshall Eubanks<tme [at] americafree> wrote:
>
> On Jul 3, 2009, at 4:29 PM, Ben Carleton wrote:
>
>> Yes it was.
>>
>> On Jul 3, 2009, at 4:29 PM, Jeffrey Lyon wrote:
>>
>>> Wasn't Authorize.net affected by this? We received a support ticket
>>> about why Authorize.net is down today (I don't know either, I don't
>>> ask too many questions).
>>>
>
> Authorize.net was for a while completely off the air, and companies that
> relied upon them
> were not getting credit card authorizations (and, thus, no ecommerce). I
> think it is still only
> partially functional.
>
> Authorize.net has been communicating with customers mostly (entirely ?) with
> twitter - they are
>
> @AuthorizeNet with a hash tab of #authorizenet
>
> If you go there, you will see a lot of status messages like
>
> #authorizenet (cont.) Do not manually submit ARB transactions b/c you run
> the risk of your merchants being double billed.
> 10 minutes ago from web
>
> (i.e., 4:47 EDT).
>
> You will also see a lot of posts from annoyed people if you search on
>  #authorizenet
>
> It's an interesting use of Web 2.0 for emergency communications.
>
> Regards
> Marshall
>
>
>>> Jeff
>>>
>>> On Fri, Jul 3, 2009 at 4:21 PM, Tomas L. Byrnes<tomb [at] byrneit> wrote:
>>>>
>>>>
>>>>>
>>>>> Earth is a single point of failure, where is your backup site?
>>>>
>>>> [TLB:] Given that all my customers are on Earth, I don't need one if my
>>>> customers also are "down".
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Jeffrey Lyon, Leadership Team
>>> jeffrey.lyon [at] blacklotus | http://www.blacklotus.net
>>> Black Lotus Communications of The IRC Company, Inc.
>>>
>>> Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th
>>> at Booth #401.
>>>
>>
>>
>>
>
>
>
>



--
Jeffrey Lyon, Leadership Team
jeffrey.lyon [at] blacklotus | http://www.blacklotus.net
Black Lotus Communications of The IRC Company, Inc.

Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th
at Booth #401.


mike at m5computersecurity

Jul 3, 2009, 2:22 PM

Post #19 of 21 (1838 views)
Permalink
RE: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

On Fri, 2009-07-03 at 13:21 -0700, Tomas L. Byrnes wrote:
>
> >
> >Earth is a single point of failure, where is your backup site?
>
> [TLB:] Given that all my customers are on Earth, I don't need one if my
> customers also are "down".

Bad Day !


tme at americafree

Jul 3, 2009, 4:23 PM

Post #20 of 21 (1835 views)
Permalink
Re: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

On Jul 3, 2009, at 5:11 PM, Jeffrey Lyon wrote:

> That's a great idea, use some lame Web 2.0 trend to communicate with
> actual real life customers. </sarcasm>
>

I would assume they figured it was better than just remaining silent.

Regards
Marshall

> Jeff
>
>
> On Fri, Jul 3, 2009 at 4:59 PM, Marshall Eubanks<tme [at] americafree>
> wrote:
>>
>> On Jul 3, 2009, at 4:29 PM, Ben Carleton wrote:
>>
>>> Yes it was.
>>>
>>> On Jul 3, 2009, at 4:29 PM, Jeffrey Lyon wrote:
>>>
>>>> Wasn't Authorize.net affected by this? We received a support ticket
>>>> about why Authorize.net is down today (I don't know either, I don't
>>>> ask too many questions).
>>>>
>>
>> Authorize.net was for a while completely off the air, and companies
>> that
>> relied upon them
>> were not getting credit card authorizations (and, thus, no
>> ecommerce). I
>> think it is still only
>> partially functional.
>>
>> Authorize.net has been communicating with customers mostly
>> (entirely ?) with
>> twitter - they are
>>
>> @AuthorizeNet with a hash tab of #authorizenet
>>
>> If you go there, you will see a lot of status messages like
>>
>> #authorizenet (cont.) Do not manually submit ARB transactions b/c
>> you run
>> the risk of your merchants being double billed.
>> 10 minutes ago from web
>>
>> (i.e., 4:47 EDT).
>>
>> You will also see a lot of posts from annoyed people if you search on
>> #authorizenet
>>
>> It's an interesting use of Web 2.0 for emergency communications.
>>
>> Regards
>> Marshall
>>
>>
>>>> Jeff
>>>>
>>>> On Fri, Jul 3, 2009 at 4:21 PM, Tomas L. Byrnes<tomb [at] byrneit>
>>>> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Earth is a single point of failure, where is your backup site?
>>>>>
>>>>> [TLB:] Given that all my customers are on Earth, I don't need
>>>>> one if my
>>>>> customers also are "down".
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Jeffrey Lyon, Leadership Team
>>>> jeffrey.lyon [at] blacklotus | http://www.blacklotus.net
>>>> Black Lotus Communications of The IRC Company, Inc.
>>>>
>>>> Look for us at HostingCon 2009 in Washington, DC on August 10th -
>>>> 12th
>>>> at Booth #401.
>>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
>
>
> --
> Jeffrey Lyon, Leadership Team
> jeffrey.lyon [at] blacklotus | http://www.blacklotus.net
> Black Lotus Communications of The IRC Company, Inc.
>
> Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th
> at Booth #401.
>
>

Regards
Marshall Eubanks
CEO / AmericaFree.TV


frnkblk at iname

Jul 4, 2009, 1:43 PM

Post #21 of 21 (1752 views)
Permalink
RE: Fire, Power loss at Fisher Plaza in Seattle [In reply to]

Leo:

That presumes the people at the bottom care about the big picture. I'd
hazard to guess that the people who participate in this listserv *do* care,
but if Dilbert has even any correlation to real life, then just as often
there are people who build DNS/DHCP/etc redundantly because that's what they
were told to do, but don't have the initiative or weren't directed to look
at the bigger picture. At the end of the day it depends on these grunts'
managers to be sure that things are being designed and implemented properly.
And even then, we know how personalities, resources, and internal inertia
more of than not develop and grow networks that aren't BCP-friendly.

Frank

-----Original Message-----
From: Leo Bicknell [mailto:bicknell [at] ufp]
Sent: Friday, July 03, 2009 3:40 PM
To: NANOG list
Subject: Re: Fire, Power loss at Fisher Plaza in Seattle

<snip>

Very few companies consider these choices rationally; often because
choices are made by different groups. I am amazed how many times
inside of an ISP the folks deploying the DNS and mail servers are
firewalled from the folks deploying the network, to the point where
you have to get to the President to reach common management. This
leads to them making choices in opposite directions that end up
costing extra money the company, and often resulting in a much lower
uptimes than expected. Having the network group deploy a single point
of failure to the "Tier 4" data center the server guys required is,
well, silly.

<snip>

NANOG users RSS feed   Index | Next | Previous | View Threaded
 
 


Interested in having your list archived? Contact Gossamer Threads
 
  Web Applications & Managed Hosting Powered by Gossamer Threads Inc.