If you're considering Heroku, don't automatically dismiss it because of all this. For one, it's unlikely your site/app will ever be as big as RapGenius. That's not a shot, just reality. They ran into problems as an edge case. Is Heroku and their architecture at fault? Hell yeah it is, but I have faith that they will fix it.
Why?
Because I really think that when push comes to shove, Heroku was actually trying to do the right thing with the changes they made and perhaps didn't consider or understand some of the ramifications of the changes they made to the Rails community. They may have fallen in love a little too much with the new node.js hotness and the like. Their CORE audience is startups/new business where Rails is very popular and they understand what this has done to their reputation. If they don't address this in a serious way it will damage their business severely.
I don't personally use Heroku but I have used it in the past and would not hesitate to use it on an appropriate project.
As for me, this story excludes Heroku from any future consideration. Why?
There are two reasons for paying a significant premium to a a platform provider: a) you don't want to do this job yourself, b) someone knows better how to do it.
This story showed me that (b) isn't true. Now, I'm all for learning and growing and improving, but not on my business and not on my applications, and especially not while I'm paying a premium for it, being promised "scalability".
Fwiw our site isn't as big as rap genius and we see highly variable performance and timeouts even on requests that on average only take 100ms. I'm not ready to blame heroku entirely since there's a lot of moving parts that still could be at fault, but after optimizing as much as we could we still see the behavior. It feels like the routing layer could be the most likely problem. Summary: you might see highly variable request time regardless of traffic levels.
Why?
Because I really think that when push comes to shove, Heroku was actually trying to do the right thing with the changes they made and perhaps didn't consider or understand some of the ramifications of the changes they made to the Rails community. They may have fallen in love a little too much with the new node.js hotness and the like. Their CORE audience is startups/new business where Rails is very popular and they understand what this has done to their reputation. If they don't address this in a serious way it will damage their business severely.
I don't personally use Heroku but I have used it in the past and would not hesitate to use it on an appropriate project.