"The point is that training deep nets carries a big cost, in both computational and debugging time. Such expense doesn’t make sense for lots of day-to-day prediction problems and the ROI of tweaking a deep net to them, even when tweaking small networks, might be too low. "

As a Masters student now training deep models for a little while now, I think this point is underemphasized. Doing something novel (so, not just image classification) requires a TON of engineering, not to mention the research considerations. And there are so many tiny decisions and hyperparameters, that even when I thought I had considerable domain knowledge I found it very lacking. I guess it should not be surprising given that 'Deep Learning' refers to a very broad set of models only related by having a learned hierarchical representation. There are a few problems where you can use existing deep learning almost off the shelf (most notably image classification, segmentation), but for most applications I think we're not there yet. As long as this remains true (which I suspect will be for a long time), SVMs and decision trees and linear models are still definitely worth knowing and understanding.