Considering Sprint Length

A friend of mine had an interesting situation:

  • Novel product, many unknowns
  • Multiple teams grouped into 3 product areas
  • Experience doing 3-week sprints

Only a fool would do anything other than the 30-day official sprint cycle that I saw on some website and in a few books.

(Just kidding. Unfortunately, like most of agile development, context has a tremendous impact on what you choose to do, process-wise.)

A lot could go into what the Optimal Sprint Length should be… You could ponder the dependent variables and try and guess an optimal length to optimize the independent variable(s) — which would be, what, maybe cost and rate of feature delivery and quality? You could do the “democratic process” and allow the team to vote, or even do “rock-paper-scissors” to figure out 2 or 3 weeks.

However, what if we built a continuum of sprint lengths for the sake of discussion. On the one end, we start at the idealization of doing one useful feature at a time and deploying it immediately — think simple web app. Anything longer than this is a compromise based on some (hopefully valid) reason. On the other extreme, we could wait until the entire system is done before deploying or integrating, maybe after 6 months or a year.

The cost of “batching up” the “work in process” at the upper end of long sprint lengths, is pretty obvious to everyone. I submit, that if you agree with (or experience first-hand) the premise that batching work has a non-linear impact on overall cost (including the hidden and subtle cost of everything that we know is bad with waterfall), then it stands to reason one might favor shorter cycles and less batching.

Not to digress, but the parallels exist in industry. To allow WIP to be large, and to allow certain parts of the process to run at high levels of batching, is a risk. A risk that the items in the batch, once released into the wild, are discovered to not be as valuable as first thought. Well, it’s water over the dam, time and effort you will never get back. (Think: extra features built because someone thought they would be useful, and it turned out that the marketplace thought otherwise.) Nonetheless, sometimes weighing the risks will lead you to some level of batch that makes the most sense.

There is often much more to the decision on sprint length than purely the development team. For example, what is the cost of QA? If the cost of QA is no different for 1 feature at a time versus a week’s worth of features, than QA cycle time/cost is not an issue. However, if it requires a week of QA time to regression test the system in the case of even a single small feature or bug fix, then you have a serious input into what the optimal sprint length should be.

Naturally, one could do development sprints at one frequency, and QA sprints at another… and even customer ship sprints at a completely other cycle time.

Regarding multiple teams… this is a solution that can be recursively applied, much like you would at a software architectural level. If the teams are horribly coupled, your costs will balloon and no amount of pondering sprint lengths will have a significant impact. If the work dependencies are carefully controlled between the teams, sprint length could vary between teams due to their own local reasons.

Much like the QA process can be a “tax” on each Sprint, what other taxes does your process incur? Running down to a one week sprint will likely reveal expensive parts of the process that could be ripe for improving.

So having said all of that… Here’s a thought. Why not simply agree to try out a few different lengths for enough sprints to get a feel for the differences. Try one week sprints for the next 6 weeks. Try 3 week Sprints three times. See if you can monitor metrics that will tell you what worked better. Consider that different teams might also work at different frequencies to test the “costs” of thinking the teams should be synchronized.

Much like with our USA republic, surely don’t let democratic, mob rule win the day.

Agile Won’t Be Killed

Agile is about the freedom to do the right thing.

UncleBob wrote this good piece on whether Scrum could Kill Agile.

To me, Agile is a mindset, a state of mind, a way to be. It is neither a (set of) coding practice(s), a concrete process, or a management technique.

Agile is not TDD, XP, FDD, Crystal, Scrum, Lean, Kanban, Six Sigma.

Agile is about being responsive to the current and future needs of the business within the current context, so that you can do the very best possible given your team and budget. Read these again:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

The “we value the left-hand items over the right-hand items” is precisely loose because it all depends on the context and the business needs. Believe me, we argued mightily over small wording changes in that Snowbird meeting! It’s just that in general, our collective experiences had born out the truths that we were more effective and responsive to delivering valuable software when we emphasized the left-hand items. I was coming from the government background, and I was determined to demand we push for truth and honesty in software. No more stupid 70% done and no working software to show for it!

I submit that the Agile Manifesto tenets are irrefutable ideas and will stand the test of time.

The Non-agile Manifesto

Let’s try a little negative test (if you agree to these, I’ll poke you with a hot stick):

  • We will build software with processes and tools — that way we don’t need to worry about people issues and effective communication.
    • That way we can hire any individual regardless of skill, and forgo all verbal/personal interactions in favor of solely written words. Even better if those written words are automatically transformed into code. Maybe we can get non-coder tools! After all, people are merely fungible assets/resources, and software is factory-work — with processes and tools, and a horde of low-paid serfs, we can crank it out!
  • We would rather spend exorbitant sums of money producing comprehensive documentation than have tangible, working results early on.
    • We start with complete requirements specifications (often 400 pages), that follow our company standard template.
    • Even our Use Cases follow a mandatory 3-level deep path, with proper exception and alternate paths worked out.
    • We link the requirements items into detailed design documents — which include system design diagrams, interface specifications, and detailed object models.
    • If we don’t write it all down now, we’re likely to forget what we wanted. And if we don’t do it to the n-th degree, the developers might screw it up.
    • Writing it all down up front allows us to go on vacation while the process and tools “write” the code from the detailed specs/diagrams. Sweet.
    • In addition, we love to be rewarded by reaching meaningless intermediate deadlines that we place on our 1500-node Gantt chart.
    • When we combine all of the upfront work with a important deadlines, many of the senior managers can get promoted due to their great commitment to generating reams of cool-looking documents. By the time the sh!t hits the fan when folks realize the “ship it” deadline is missed, the senior managers are no longer involved.
    • Besides, if we actually built software instead of writing all sorts of documents thinking about building software, our little ruse would be exposed! (BTW: if the human race had done nothing more than documented the ideas about how best to go about procreating, we wouldn’t be here to waste software budgets. w00t!)
  • We prefer to write a tight Statement of Work & force a strict fixed-price contract rather than actually work towards a common goal with the client.
    • The agreement is based on the 400-page, fully-specified requirements document, and we pad the cost estimate with a 400% profit margin.
    • We then hire dozens of people to argue during the Change Control Review Board monthly meetings about re-writing code to deliver what you wanted versus what you asked for when you thought you knew what you wanted (and wrote it down in that 400-page doc).
    • Contract negotiation pissing matches are such great uses of our collective resources and always result in perfect software! We love our fine print 🙂
    • With a 400% padding, the projects are too big to fail.
    • Once we are in it for 1 or 2 million and 50% done and 2x schedule overrun, who would ever say “No” to a contract extension? Who better to get you to the goal line than the same folks who squandered away your treasure, pissed away the calendar, and delivered no working software yet?
    • We like to appear like we’re just about done… Asymptote? Never heard of one.
  • There’s nothing more appealing than following a 1500-node Gantt chart. To the letter and subletter. Change sucks.
    • Especially a Gantt chart that has been built with tender loving care to include resource allocations, inter-project dependencies, and partial resource allocation assignments for matrix-style organizations.
    • We love hiring a small army to ensure that we drive the entire team to meet every micro-task deadline even when they no longer make any sense.
    • The real fun is collecting the “actuals” data from the developers assigned to each task.
    • And nothing sweeter than seeing 90% of our tasks being started, and 75% of those being 67% completed, and 25% of them being 99% completed — the roll-up summary to management is such a compelling story.
    • Changing such a beautiful plan that took 4 man-years to develop, that incorporates all of the comprehensive non-code documents, and is an appendix in the contract, is no small feat!
    • Better to produce the software according to plan even if nobody wants it that way. That’s our motto, and we’re not going to change!

Waterfall Sucks

Waterfall — as practiced — will die under it’s own weight. Waterfall is a bad idea when applied broadly to all sorts of classes of software projects. Eventually people will wake up and realize that — for many situations — there are better ways to doing software than waterfall. (Waterfall done on one feature at a time is a fine practice 🙂

I remember seeing folks who worked for Big Consulting Firm go into an organization to do all of the Use Cases for 9 months straight. Their documents were then handed off to some other firm to do the analysis and design. And finally, yet another firm would do the implementation based on the sum total of documents. Since the written word left nothing to the imagination, these projects always delivered perfectly. Oh, wait. Standish Group says that really isn’t how things worked out. This might be an okay way to go about hardware or construction projects — since you can be more deterministic up front, and there are more common parameters that are “agreeable” to most folks in terms of real interface documents (like engineering or architectural drawings are pretty specific and not ambiguous). But to anyone that has any experience in software development, waterfall seems ludicrous from Day One! Who would ever think that you could hand off all features as they went through each of the major phases with no loss of understanding? Who would ever think it is good to wait FOREVER to see any tangible results? After all, we’re NOT bending metal or breaking ground. It’s freaking software! Sigh.

Waterfall seemed ludicrous to me my very first time. I was exposed to Mil-STD-2167 while doing DOD contracting in the 80s. I think I followed along like a good little well-behaved junior engineer my first time. Probably even my second. Then my brain kicked in and said — this is stupid for our situation. We’re doing software development for R&D projects, we’re not doing assembly line work for a brick factory. We can’t know everything up front, and it is risky to think you can! So I started to look for better ways, and began to not do wasteful practices, even though they were prescribed. I began to do the right thing for the customer — and they agreed to the <ahem> “waivers.” (I was applying the “Better to beg forgiveness, than ask permission” principle I learned from my good friend and mentor, USMC Colonel Rich “Magic” Dinkel, fighter pilot extraordinaire.)

I sound like I am bashing people that did Waterfall… I don’t want to be that harsh. People did what they thought was best at the time. I think many times we fall into a trap of wishful thinking. And even in the face of overwhelming observations to the contrary, we soldier on in hopes things will work out like we want. (Besides, it’s in the contract!) Rarely is hope ever a good strategy. Maybe never? So folks doing waterfall certainly had good intentions, and it was compartmentalized enough to reward intermediate victories and give an illusion of progress, but there’s no denying it’s just a bad idea.

Agile didn’t kill Waterfall per se. Sure, maybe it helped accelerate the community’s realization that there were better ways to do software than a giant, rigid, 4-stage assembly line.

Waterfall killed waterfall.

Scrum Won’t Kill Agile

I say let the Scrum Fun continue! Make tons of money! Waste as much budget as you can. Brag that you are doing Scrum and have hired 20 CSMs to whip your developers into shape!

The more people and HR departments that think there is some magic bullet, the more companies that might fail and lose business. This makes way for smarter, leaner, more entrepreneurial start-ups to eat their lunch. After all, I would wager that very few small organizations are falling for the Scrum Will Save The Day stupidity that is seemingly gripping some (large?) portion of our community.

Convenient Cover?

Here are a few phrases that come to mind regarding the Scrum dustup, and I’ll add a new one…

  • A friend once quipped that “Off-shoring is just a cheaper way to fail” — implying that if you can’t build software locally, you probably can’t build it 12 time zones away either. But at least the costs are lower.
  • No one was ever fired for hiring IBM
  • We are doing Scrum — all the rage in Management By Magazine circles!

In other words, it is often far easier to go with the flow, follow the herd, live by the plan, and say you did what you were told. It is much harder to buck the system and call out the ineffective and wasteful activities, and continuously try to improve. It’s hard to buck inertia!

Conclusion

Nothing will ever kill agile.

Bold, I know. However, I view it like the USA’s founding principles — immutable laws of freedom. Agile is about freedom to do the right thing.

How can you ever deny the correct way to do software is to do what’s best for the business?

How can you deny the Agile Manifesto tenets?!

Successful software will continue to be developed in an agile manner, regardless of the terms that might become popularized.

Certification is a Good Thing

I read with pleasure “The Value of Certification” from Pawel Brodzinski, that emanated from a discussion about evaluating employees.

Pawel points out some obvious pros to certification, and one big con:

  • It’s objective measurement
  • It’s butt-simple (Pawel didn’t actually say this, that’s, ummm, me talkin’)
  • It’s, and I quote, “pretty much useless”

Then it dawned on me. College degrees are certificates, as are high school diplomas. So are participation trophies for the poor schlubs who didn’t win anything at tweeny league and are (defenselessly) surrounded by adults who mistakenly (IMHO) think such a trophy is meaningful to the kids (kind of like dogfood made to appeal to the purchaser). But that’s a digression…

In our software world, it is the Scrum Certification that bears the brunt of criticism — even though Pawel does not mention this particular style of paper valor. For those who pass an exam after two-days of grueling lectures, practical hands-on training (a.k.a., apprenticing) at the feet of master craftsmen, and in-depth co-op work experience at multiple cutting-edge development shops doing different projects… Oh, wait, that’s not quite fair as that really isn’t what’s going down in just two days. Crap. Scratch that tact.

Anyway, let’s just say many folks seem to be intent on hiring other folks based on Scrum certification (caveat emptor: I do not have said certification, so I am not speaking from direct experience — I’m too cheap to pay for it). I base this on my extensive analysis of randomingly seeing two hiring ads for Scrum Masters that were virtually identical in content — and buffoonery, I might add. For me, it was a big, bad smell for the organization. But, in large bureaucratic organizations (government or MyLargeCo), abdicating responsibility is honed to a fine art. Ergo, the HR departments did what they think was valid trawling for Scrumsters by evidently plagiarizing some sort of HR-i-pedia definition of the perfect Scrumster parameters. Good luck with that. These actions are a boon to your not-as-lazy-or-naive competition (oh, unless you are the Government, which has no competition, only an ever increasing appetite — I’ll not digress here, either).

So, maybe the Scrum Certificate is a bit like a Participation Trophy.

Yea, you were there. You participated. You didn’t actually compete/win or do anything meaningful in the real world to earn it, but there you have it. You should feel better about it. And it does mean that you at least did something (unlike me). Go forth and procreate little Scrumlins. (BTW: I’ve got nothing against Scrum, I think it is a great and simple approach to PM! It’s just hard to pass up the opportunity to poke fun at our own agile software world and make up little monikers as I go.)

So I can look at “certification” in at least two ways:

  1. Basing hiring or performance appraisal on a CSM certificate is pretty awesome for the individual on the receiving end. I bet they didn’t have to go into debt to get one (compared to a college degree)! (Although all of me wants to be judged on my real merits, not on some piece of paper — alas the chicken and the egg comes to mind for newbies.)
  2. If companies find a valuable correlation between a certificate (CSM or otherwise) and knowledge work, I would be surprised. In my experience, the last thing that I would qualify as a trait of software projects, is that they are “cookie-cutter” — that is, where one-size anything fits all (be it ideal Scrum process, database, language, etc.). However, sans external stupidity (like a bailout), companies will “learn” how to treat any software certificate — much as they have with college diplomas and belly buttons (i.e., everyone has one). The market will sort things out, eventually — in that I have faith.

So there you have it. Pawel says “Certification is Pretty Much Useless” — and I disagree. That very aspect of certification will serve a purpose to differentiate those that place undue trust in the paper from those that stick to good hiring practices and good employee relations. I predict the latter prevail.

ps: Pawel, I agree with you in spirit.

Scrum Dust Up

This post is in response to this:

Kripanidhi said the following on 10/25/10 9:34 AM:

Scott,

It may have been a very interesting and useful idea to have included one more question in your survey:

“How many Certified Scrum …..Masters, Teachers, etc know how to code or have ever written a piece of software that was paid for ? ”

I could easily speculate that figure at less than 50%. Can guys who have never hacked in their lifetime, call themselves Agile ? and worse still teach others how to hack and Scrum ?

Food for Thought

I view Scrum more like a form of Project Management. Nothing more. Nothing less. But I might be mistaken. I am not certified. At least not in that way.

Anyway, given that as my starting point… I am not sure if teaching how to do a particular style of Project Management (e.g., CSM) requires knowing how to do the work that will result in the end goal. AFAIK, Scrum can be applied to many things beyond software. Even for software, does a good PM have to know how to do great graphic design, or be a brilliant UX person, or be the most awesome coder/hacker? Maybe folks drawn to CSM actually realize they can do more for the project than simply do mediocre coding.

Now, would knowledge of the “how” help a manager be a better manager? Probably. But isn’t a truly good Project Manager (Scrum or otherwise) cognizant of the need to trust (but verify) their folks to do the right things? Sure, sometimes it is helpful to question an estimate, or critique a design approach, or help shape the product roadmap, or ask probing questions… But does that require intimate knowledge solely on the part of the CSM? Or does it require a good team effort? To turn the argument on its head: is taking the best coder out of the loop to be the CSM a good idea?

Despite my personal preference for (and being) generalizing specialists, I do not think it is a fair pass/fail “litmus” test to say a CSM has to be a hacker/coder/doer of everything.

Personally, I don’t think we have to challenge the CSM individual’s hacking skills. After all, do hackers inherently make better managers? Rather, we should challenge the terminology that bestows the ludicrous moniker of “Master” after a few days of classroom training. Of course, as a counterpoint, the concepts behind Scrum are simple enough to be mastered in a couple of days — that’s the beauty of a simple technique. But the skill of being a Project Manager can take a lifetime to master, Scrum or otherwise, IMHO.

I think what is worse is the blind-faith that someone who has been certified as a CSM is qualified to run an agile project. Or the illusion that everything else we do can suck, but as long as we have Scrum, we’ll succeed.

The beauty of a free market: this CSM brouhaha will sort itself out. Companies will learn one way or the other that there is no magic bullet to hiring (despite the world’s stupidest job postings for CSM folks), nor is there a silver bullet to getting projects completed successfully.

Using Basecamp in an Agile Manner

Even the lowly “To-Do List” (as can be found in Basecamp) can be used to manage an agile, Kanban-style, lightweight Software Development process.

Software Development Life Cycle

In software development, we want to make our process as visible as possible to all participants and stakeholders. To that end, behold a simple process that uses a handful of Basecamp To-Do Lists as markers for how a feature or bug is working its way through this process.

At any point in time, you can look at the “To-Do” list page for a status on your favorite new feature, task, or bug. And, if you have a chunk of features piled up in an “Iteration 1” list, you can see the number of items being worked through to completion. (There are even third-party apps that allow you to do burndown charts if they appeal to you.)

Using Basecamp's To-Do List in a simple manner

Requirements/Bugs

The business has a desired priority for features as expressed by the order of items in the “staging” lists, and especially the On-Deck list (to borrow a baseball metaphor for “What’s next”). For major new features, the business and the developers will work out a rough Release Plan. But for simple enhancements and bugs, the business can simply add desired items to the lists.

Development

The Developer will pick a new task off of the top of the On-Deck list. There will usually be a conversation around the issue to ensure proper understanding (unless it is self-explanatory). Ideally, we get the business/testers helping to write User Acceptance Tests in the BDD form of Given… When… Then… The Developer moves this feature into the “In Progress” list. If desired/possible, the user could add a date estimate for the task.

The Developer will create tests, be they Cucumber, RSpec, or UnitTest, or all 3 types. If possible, the Testers can help write the up-front acceptance tests, so that they are intimate with the feature/bug. The Developer writes code until the tests pass.

Once the issue is complete, and the tests are passing (and you didn’t break any other tests), the Developer: commits the code, updates the TEST server, and does a smoke test to ensure things still work as expected. The Developer then moves the task to the “To Be QA’d” list, where the Testers will now verify the issue is indeed complete and correct on TEST. Close collaboration between Tester and Developer is warranted here, as the nature of the fix and any developer tests can be explained so that testing can be as effective as possible. Plus we want to maximize use of automated tests, and minimize any unnecessary manual tests (as they are time consuming and expensive) given the nature of the code changes.

Testing

If the QA passes, the Tester moves the task to the “To Be Released” list. Depending on the nature of the feature, the Business can also help to “test” and ensure things work as expected. If the test fails, then the task is moved back to the On-Deck list and the details added to the issue as comments (if simple) and image uploads (add image link to the comments), or to the issue tracker if it is more complex (even though it is technically not an issue against the released product).

Release

The Developer will schedule the move for “To Be Released” tasks to the PRODuction box. This may require coordination with the client’s IT team if they have internal controls. Following release, the application is smoke tested to be sure the basics are still functioning. If any problems arise, the server can be rolled back, or fixed ASAP, based on the nature of the error.

Once the items are released to PROD, you can indicate that they are “complete.” If the issues came from a list that you want to keep for posterity, simply drag the completed task to the original list (“Iteration 1” for example). Now you can see the items in that list that have been released (i.e., completed).

Summary

I’ve used Jira and much more elaborate SDLC schemes in the past. But a recent Rails project with just a few folks on the team, found us trying out Basecamp. By creating various lists, we can mimic the Kanban-style wall board that Greenhooper provides within Jira. Because the lists allow simple ordering to show priority, and you can drag-and-drop between lists, and you can track your time, I think we have a very lightweight, winning combination.

This process is equally at home doing one feature all the way through, or an entire iteration’s worth of features in a bigger chunk. (Basically dependent on the cost of testing and the appetite of the end-user for changes.)

SDLC process[/caption]