It sounds to me like you are in a software company with a lot of inexperience.
I have seen people spin up massive map-reduce clusters to perform something AWK could do in half the time. And I have seen spend enormous amounts of time training neural nets for something a simple SVM could do. In both cases (and most similar cases I have seen) a lot of needless time and energy was expended.
But why? This is my hypothesis anyways...
There seems to be two primary pools of people who feed into the software engineering. Those from engineering disciplines and those from research disciplines. Engineering schools can be fairly cut throat; limiting the number of candidates who can move on at various stages, or designing courses to reduce the number of enrolled students. Those from research disciplines are required to create novel ideas which are worth publishing just to graduate their programs. And those that actually like academia believe (and rightfully so) that creating new ideas is the way towards notoriety and success.
That means a lot of the craftsmen who would prefer making tiny refinements on well understood paradigms are simply weeded out from the discipline before they can even enter it.
I have known very smart people who quit engineering in college because they could simply not stand being in a room of people where everybody wanted to be the "smartest in the room".
Of course... the incessant CV polishing in Silicon Valley probably doesn't help either.
I have seen people spin up massive map-reduce clusters to perform something AWK could do in half the time. And I have seen spend enormous amounts of time training neural nets for something a simple SVM could do. In both cases (and most similar cases I have seen) a lot of needless time and energy was expended.
But why? This is my hypothesis anyways...
There seems to be two primary pools of people who feed into the software engineering. Those from engineering disciplines and those from research disciplines. Engineering schools can be fairly cut throat; limiting the number of candidates who can move on at various stages, or designing courses to reduce the number of enrolled students. Those from research disciplines are required to create novel ideas which are worth publishing just to graduate their programs. And those that actually like academia believe (and rightfully so) that creating new ideas is the way towards notoriety and success.
That means a lot of the craftsmen who would prefer making tiny refinements on well understood paradigms are simply weeded out from the discipline before they can even enter it.
I have known very smart people who quit engineering in college because they could simply not stand being in a room of people where everybody wanted to be the "smartest in the room".
Of course... the incessant CV polishing in Silicon Valley probably doesn't help either.