Carson Holmes made an interesting post <a href=”http://tech.groups.yahoo.com/group/agilemodeling/message/8986″ target=”_blank”>Big Consultants Use “Lean” Buzz to Take Advantage of Bureaucrats</a>

My response:

And you can bet that when Big Company and Big Consulting Inc fills out an industry survey about the success of the projects there, it will be all rosy.

Too often I have seen execs roll out large deployments of enterprise-wide systems &mdash; largely I think to get it on their resumes &mdash; with little true ability to actually measure the pre- and post-outcomes. This is just seemingly normal for large, bureaucratic organizations &mdash; even if they are ostensibly capitalist/free-market driven.

I think the problem is that very few organizations (I know of none, but there must be some) truly knows if the IT budget is delivering business value <i>as good as it could be</i> in such large organizations. For example, do you think CIOs from two companies can discuss the output of their IT staff on an absolute basis? Someone from the outside might be able to discern that one organization appears to do twice as much work as the other, with fewer people. But even that is more of a gut feel than a measurement of a consistent unit of measure.

If the output of development were easy to see and measure (e.g., painting walls or laying carpet), it would be easy to examine return on investment. Eventually, competitors that do a better job of efficiently leveraging technology for business advantage will win out in the free marketplace. But it could take years or decades to reveal bad decisions (think Y2K) — by this time the responsible individuals are long gone, or have been promoted up the chain.

(Even in the small, many times an individual developer may not be around long enough to learn the consequences of some decisions/code they made two years ago &mdash; hence not being able to learn and grow from that experience. Instead, more often than not, they see and learn from other people’s mistakes. Yet they are unable to understand the original thought processes that went into that decision, because the author is long gone.)

Until our industry is able to resolve the conundrum of how to compare expenditure versus return on IT, it will be easy for Big Consultants Inc to do Selling By Buzzword, and easy for IT execs to do Management By Magazine.

It gets worse in large bureaucracies, like the government, where there are no market forces to expunge wasteful practices.

We are indeed a nascent industry.

Larry Constantine wrote a “Real Data” post wherein he posited a question (summarizing):

What would it mean to the agile community IF CMM level 5 and RUP practices actually worked significantly better than agile ones? Would that mean we should switch horses? Or would it mean we should revise agile to incorporate the best parts of other practice traditions? (And maybe shed some baggage at the same time?) Or doe the agile community have the TRUE answers, regardless of the facts?

Larry,

without reading the other 20 posts, here’s my gut reaction.

I would surmise that the success or failure of a project being attributed to the methodology is but one aspect to consider.

In a multivariate analysis of variance, just how much impact does the methodology have? Heck, we can’t hardly even say two projects are alike in too many dimensions.

In my opinion, it isn’t the process that matters nearly as much as the people. Ineptitude will never be trumped by CMM 5. Ineptitude may merely take longer to surface in a heavyweight process. Frankly, I have always felt that agile methods fare the best in the hands of really keen teams, and can fail miserably in the hands of rank amateurs who don’t “get it.” Folks that don’t “get it” are protected inside large process.

I would question the results (even without any a priori knowledge of them — since you revealed nothing in your post):

• The data is anecdotal and unscientific (no double-blind studies)
• Was the data unwittingly tainted by those who provided it?
  • Did all the successful small agile projects pass on reporting because they don’t keep such minutia?
  • Did all firms that keep such data naturally want to look good?
  • Did all firms that spend LARGE sums of money on heavy processes sugarcoat their results so they can justify this expenditure?
• Just who keeps the total cost of ownership? really.
• Who keeps the total cost of maintaining a CMM 5 level shop?
• What about projects that failed and go under-reported — with either methodology?
• Exactly how much data do you have on truly longitudinal studies?
  • How many legacy systems that have been in place for 20 years were in the study?
  • How many 10 year-long agile projects have their been?
• Is there pressure on the data analysts to have the results appeal to the paid recipients/members?

The cynic in me also says — IT DOESN’T MATTER. You can’t legislate responsible behavior, and you can’t dictate a foolproof software development process.

The agile concepts are about as irrefutable as Newton’s Laws, so if you told me CMM 5 was the Holy Grail, I wouldn’t change one sentence on the agile manifesto. Agile is all about doing the best thing that works in your current situation. CMM5 certainly does not apply to 100% of the software projects… whereas the agile mindset does.

The best we can do is share experiences and loudly proclaim “caveat emptor” and “YMMV.”

I wonder… is it a bit like free market forces/capitalism — where you can have “sky’s the limit” wealth and abject poverty (agile); versus socialism where the range from best to worst is much narrower (heavyweight, prescriptive)?

I think we could use the Super/Freakonomics guys to help tease out interesting causal relationships between the findings and reality.