The fact that agile would be overrun by bureaucrats and vendors and thus turned into the same old bullshit was not only predictable but predicted from the outset. The new bullshit may not be quite as bad as the old, but it will take decades if not centuries for the real principles of software development to be worked out at a macro level.
A big part of the problem is the myth of "organizational change" that is endlessly propagated by consultants and enthusiasts. I've seen fathomless amounts of time and energy spent on trying to do this. It is largely poured down a drain. Inefficient, anticreative organizations don't change, except by dying. The idea that they can somehow go through an enlightenment experience and become productive places that are good at making software is extremely seductive. People are willing to put a lot of money into it and there's a steady supply of gurus happy to take the other side of that bargain. Note that the gurus don't produce software themselves, they just talk (at high hourly rates) about how you should do it. From what I observed (and I saw a lot of it) this is mostly mutual self-deception.
The way we will get better at software overall is not by "organizational change" in the places that suck at it, but by starting new organizations (e.g. startups) to provide new outlets for creative energy, some of which will succeed and render the old ways obsolete.
Happily, the situation at the individual level is not as gloomy as at the macro level. We are free to do things differently if we want to.
I produce software on a regular basis. And I'm also one of those high-priced gurus you mention.
It's not black or white. Organizations are full of good, intelligent people who want what's best for both the people and the organization. It's just that everybody has a different idea of what that is.
I wish it were as simple as "adapt or die" but many times it's "adapt enough and survive" which means big companies do a little bit of everything.
And it's the agile true believers that are many times behind this make-agile-just-another-heavy-process movement. Take a look at the various lists for defining if a team is "really" doing scrum or not. Read up on what some of the practitioners want to do. The agile community is doing a lot of this to itself, large organization or not.
So big companies pay us gurus to come in and try to change everything. And then nobody wants to change. And nobody gets fired if people miss a sprint. And everybody wants to perfectly define what agile is. And upper management wants to start tracking what metrics make "good" teams and what make "bad" agile teams. If you're lucky, you improve productivity by 5-8 percent using whatever generic agile principles the teams can accept.
That's a winning preposition for everybody. Doesn't feel great, and there is no break in the clouds with sunlight streaming in and music playing, but from a cost-benefit perspective it's a worthwhile cause.
Yes, agile is full of wonks preaching TDD that haven't coded in 20 years. Or analysts with 2 years of experience apply agile with a checklist. But that's where the movement is at right now -- it's spread from emphasizing delivery to being a little bit of something for everybody.
I didn't say it was simple; I said it was mutual self-deception. That being said, I did find a simple solution: I got out. The effect is comparable to what I imagine someone would feel if they were cured of athsma and able to breathe.
Once again, I don't think it's mutual self-deception. People see that agile principles work. Some don't understand how it applies to their job. Some, quite frankly, view agile as a bunch of new age, feel-good crap. In a big organization, life is about compromises. That can look like self-deception, but it's not.
I'm not a big corporate guy, even though I work with them, so I understand the freedom of getting out. When I'm in startup mode I'm about a hundred times more happy and productive than working in a large group.
But corporations still exist, and people still need to work in them. Some people want the imagined security that things will not change -- that they can punch a clock every day and work at the same place for 30 years.
Now you and I know that stability is an illusion, but it's one that people cling to.
A big part of the problem is the myth of "organizational change" that is endlessly propagated by consultants and enthusiasts. I've seen fathomless amounts of time and energy spent on trying to do this. It is largely poured down a drain. Inefficient, anticreative organizations don't change, except by dying. The idea that they can somehow go through an enlightenment experience and become productive places that are good at making software is extremely seductive. People are willing to put a lot of money into it and there's a steady supply of gurus happy to take the other side of that bargain. Note that the gurus don't produce software themselves, they just talk (at high hourly rates) about how you should do it. From what I observed (and I saw a lot of it) this is mostly mutual self-deception.
The way we will get better at software overall is not by "organizational change" in the places that suck at it, but by starting new organizations (e.g. startups) to provide new outlets for creative energy, some of which will succeed and render the old ways obsolete.
Happily, the situation at the individual level is not as gloomy as at the macro level. We are free to do things differently if we want to.