Disaster Recovery Planning

Better get cracking on some disaster recovery planning then!

If nothing else, I was a little surprised they didn't (or at least didn't mention) having a fail over plan in place already. Seems like "Prepare for the worst, hope for the best" would have been the logical game plan.

Its called business continuity plan and tbh most companies drop the ball in that regard.

Edge cases? That's why they are called disasters. That's why there's disaster planning. Ever read https://www.ready.gov/ ?

How is this mode not a standard part of their disaster recovery plan? Especially in sf and the bay area they need to assume an earthquake is going to take out a lot of infrastructure. Did they not take into account this would happen?

Can’t emphasize this enough! Outages will eventually happen, you need to plan for them.

Maybe not if they didn't have a tested recovery plan for whatever failure scenario they are experiencing.

All sorts of accidents are expected. We take them into account. For example, if you operate a large data center, you simply assume that a certain number of drives and motherboards will fail, and build it into the plan.No one planned for Fukushima to be flooded. No on planned for one of Chernobyl's reactor cores to explode. (The phrase "impossible" was tossed around a lot; it took a good deal of thinking to finally recognize the design flaw.)

Depends on how much spare capacity they had. Being one failure away from going down is an emergency situation at many places.

nah. documentation only goes so far.you handle this by having and exercising a disaster recovery plan and having (and listening to) competent people that would spot this issue in a jiffy.