Most importantly, I wouldn't let the staffing levels get this low.
At this point in its life, reddit should have 6-12 programmers/system administrators + a few support staff, compared with the 3 they have at the moment.
That way they won't be agonizing over the choice between devoting their resources to keeping reddit running in the short term, or to moving reddit away from EC2 for long term stability.
Ah yes, the telltale sign of a large bureaucracy: people spending $150 of one kind of money to avoid spending $100 of a different kind of money (in this case, payroll budget versus tech-infrastructure budget).
Well, I'm a SysAdmin/hardware guy myself, so obviously, I'd buy my own hardware and run it. Once a month I'd spin up my off-site backups (that would be on ec2) to make sure I had somewhere to go if the shit hit the fan at my co-lo.
Of course, I've spent my life dealing with infrastructure, so running a rack or two (or ten) servers is going to be a lot cheaper/easier for me than it would be for someone without that experience. The economics for people unwilling to gain that experience will depend entirely on scale; e.g. do they have enough servers that the lower running cost of owning hardware would pay for someone to manage that sort of thing. (and yes, hiring other people has overhead in and of itself.)
Generally, as much as possible, I avoid building complexity. I find that having a single point of failure with a backup that can be manually brought in to place (such as an asynchronously replicated database) is quite often more reliable than fancy home-made SAN solutions. In general, you need to be /very careful/ of complex redundant systems. In fact, I approach it somewhat like crypto. As much as possible, I don't build it myself. use well-used open-source tools with well-known failure modes.
The other thing to think of is failure domains. Sure, think about single points of failure, but more importantly, think about what goes down if that single point fails. For instance, in my current setup, each xen host is a single point of failure... but one going down won't take down anything else. I've seen other people design similar systems with improvised SAN setups, thinking "oh, if one node dies, I'll boot the guests on another!" the problem is that if that san goes down, everyone is toast, while if one of my hosts goes down, we're talking maybe 1/40th of my customers who are out of action.
I've seen drbd setups where a guest mirrored itself locally and to a second server... It sounded like a great idea, but the system turned out to be less reliable than my dumb local storage setup, as weirdness in drbd and how drbd dealt with disk and network issues would cause lockups that were much more frequent than the hardware failures that would take down whole nodes in my local disk setup.
In fact, I have a very strong suspicion, born of hard experience and smoking pagers, of SANs that cost less than mid-sized bay-area condos. And I'm pretty cheap, so that means local storage for me. There has been a lot of activity in that field lately, so I'm very carefully exploring it again, but I certainly wouldn't count on some homemade san as being more reliable than local disk.
(to be fair, my experience lies with "expensive SANs, used expensive SANs, and homemade SANs." and my only good experience was with the first of those. I don't have a lot of experience with low-cost commercial SANs.)
Morever, I'm pretty suspicious of disk over the network schemes, even when the expensive "real" SANs are involved. NFS is the only scheme I really trust; it's older than I am. It has problems, but we know about all of them. Overall, I think NFS handles network blips much better than any of the block device over the network schemes I've used. And you do see network blips. The network simply isn't as reliable as your sata cable, and the block subsystem isn't designed to deal with devices that are temporarily unavailable.
I've seen a lot of 'clever' redundant setups built by people who are much smarter than I am... quite often, their setup ends up becoming less reliable than my "dumb" systems.
> I find that having a single point of failure with a backup
> that can be manually brought in to place (such as an
> asynchronously replicated database) is quite often more
> reliable than fancy home-made SAN solutions.
As someone who has invested heavily in implementing HA systems and then seen them be the root of increased downtime, I have come to the same conclusion. A simple fail-over system that will result in short downtime & require limited manual intervention is in many cases the best route.
If you had to run a site like Reddit, what would you do?