Larry Constantine wrote a “Real Data” post wherein he posited a question (summarizing):
What would it mean to the agile community IF CMM level 5 and RUP practices actually worked significantly better than agile ones? Would that mean we should switch horses? Or would it mean we should revise agile to incorporate the best parts of other practice traditions? (And maybe shed some baggage at the same time?) Or doe the agile community have the TRUE answers, regardless of the facts?
without reading the other 20 posts, here’s my gut reaction.
I would surmise that the success or failure of a project being attributed to the methodology is but one aspect to consider.
In a multivariate analysis of variance, just how much impact does the methodology have? Heck, we can’t hardly even say two projects are alike in too many dimensions.
In my opinion, it isn’t the process that matters nearly as much as the people. Ineptitude will never be trumped by CMM 5. Ineptitude may merely take longer to surface in a heavyweight process. Frankly, I have always felt that agile methods fare the best in the hands of really keen teams, and can fail miserably in the hands of rank amateurs who don’t “get it.” Folks that don’t “get it” are protected inside large process.
I would question the results (even without any a priori knowledge of them — since you revealed nothing in your post):
- The data is anecdotal and unscientific (no double-blind studies)
- Was the data unwittingly tainted by those who provided it?
- Did all the successful small agile projects pass on reporting because they don’t keep such minutia?
- Did all firms that keep such data naturally want to look good?
- Did all firms that spend LARGE sums of money on heavy processes sugarcoat their results so they can justify this expenditure?
- Just who keeps the total cost of ownership? really.
- Who keeps the total cost of maintaining a CMM 5 level shop?
- What about projects that failed and go under-reported — with either methodology?
- Exactly how much data do you have on truly longitudinal studies?
- How many legacy systems that have been in place for 20 years were in the study?
- How many 10 year-long agile projects have their been?
- Is there pressure on the data analysts to have the results appeal to the paid recipients/members?
The cynic in me also says — IT DOESN’T MATTER. You can’t legislate responsible behavior, and you can’t dictate a foolproof software development process.
The agile concepts are about as irrefutable as Newton’s Laws, so if you told me CMM 5 was the Holy Grail, I wouldn’t change one sentence on the agile manifesto. Agile is all about doing the best thing that works in your current situation. CMM5 certainly does not apply to 100% of the software projects… whereas the agile mindset does.
The best we can do is share experiences and loudly proclaim “caveat emptor” and “YMMV.”
I wonder… is it a bit like free market forces/capitalism — where you can have “sky’s the limit” wealth and abject poverty (agile); versus socialism where the range from best to worst is much narrower (heavyweight, prescriptive)?
I think we could use the Super/Freakonomics guys to help tease out interesting causal relationships between the findings and reality.