Building things this way strikes me as expensive. At Netflix's scale, it pays off, but for systems that don't serve as many requests I'm forced to wonder whether just avoiding the cloud might be more cost-effective.
Well, you really have to figure out how much it would cost your site/service to be down for 2 days straight. Or maybe a week (PSN). Would this design pay for itself in preventing that loss?
...and then multiply that cost by the probability that this will happen to find the expected payoff. If the work costs substantially more than the payoff, don't bother. If substantially less, you're negligent if you don't do it. (The Learned Hand Rule http://aler.oxfordjournals.org/content/7/2/523.full)
A lot of it simply building in redundancy, it has little to do directly with the cloud. And yes, redundancy is always costly, especially if done right. You just need to decide whether your business plan will benefit from it, and what parts are necessary.
The rest of it is making sure your app or site fails gracefully; that is that failure of one part doesn't bring down the whole. That can be expensive to retrofit, but actually should have been designed in from the beginning, as it is a generally accepted part of good design for anything running over a network.
This has nothing to do specifically with cloud computing. What is mentioned in netflix's case is but the idea (randomly killing a service) can be used in a more traditional architecture aswel.
"Building things this way strikes me as expensive."
That is a qualitative statement. It implies a value proposition in your head between engineering effort involved in doing CM style disaster prep vs product benefit.
Operationally, not doing some level of CM is like paying for operations with "Lottery Checks". A Lottery check has a payee and a nominal amount but where the amount is actually printed, there is a scratch off box. Sometimes when you scratch it off its for a lot more money than you intended to spend :-).
But it is very hard to talk rationally about "we're spending two engineers here to do nothing but try to randomly break the system and get bugs fixed that would cover for that problem." Because the problem is self inflicted it seems like a waste of money, and there is no guarantee that they will have found and fixed the problem which is going to kill you in the future. However, if you run an experiment enough times, you eventually achieve the solution. Think of it as the Monte Carlo method of systems test. Its a good thing and it helps people sleep at night.
And when the world does go pear shaped like it did with AWS here you may find yourself yawning rather than panicking, and that feels very good indeed.
