MongoMapper $or and Set keys

Leave a reply

Found an interesting issue (?) with $or and mongomapper — and a work-around.

Given (lots omitted):

class Patient
  include MongoMapper::Document
  ...
  key :count_public_encounters, Integer, :default => 0
  # Provide maps to the related "doctor_num(s)" unique IDs.
  key :doctor_num_list, Set, :index => true
  # ...and related "group_nums(s)"
  key :group_num_list, Set, :index => true
  ...
end

In testing, the following basic syntax worked for an AND condition to find public patient count for doctors in a given list:

n = by_docs = Patient.where( :count_public_encounters.gt =>0, :doctor_num_list.in => doc_list).count

Since patients could be associated with a doctor, a group, or both, I wanted to find public patient counts where the group number matched, OR the doctor numbers were in the list.

n = Patient.where(:count_public_encounters.gt => 0,
        :$or =>[{:group_num_list.in => [grp.group_num]},
                {:doctor_num_list.in =>[doc_list]}]).count

After all, this syntax for array query works as a solo part of an AND query. But no luck 🙁

Turns out, the following syntax worked:

n = Patient.where(:count_public_encounters.gt => 0,
        :$or => [{'group_num_list' => { '$in' =>[grp.group_num]} },
                {'doctor_num_list'  => { '$in' =>doc_list}}]).count

Not gonna ask why…

Don’t Fight Ruby!

2 Replies

In this blog post, “Adding Type Checking to Ruby,” I spit up a little bit in my mouth. I don’t want to be harsh… I can appreciate Daniel’s different approach to Ruby.

Mind you, I’m no Ruby whiz (yet), but look at this example:

def create_name(fname, lname)
  raise "fname must be a String" unless fname.kind_of? String
  raise "lname must be a String" unless lname.kind_of? String

  fname + " " + lname
end

I really couldn’t get past this first example. We’re seriously in dire need of checking that a first and last name are passed in as String types? I know it is just an example, but it is a lousy real-world example. It’s a bit like saying we need TDD and proceeding to show test-driving an accessor. I need more red meat!

Okay, so then there is some “Type” library that bundles up some helper methods to make the ugliness above go away:

typesig String, String
def create_name(fname, lname)
  fname + " " + lname
end

Except, the ugliness didn’t go away because it is the concept that I find hard to embrace.

I think the author is attempting to coerce Ruby to act as if it was statically typed. Static typing can be a helpful crutch, but more often than not, it just gets in the way. And if you like it that much, then use a language where it is built-in already. But certainly don’t pollute your Ruby code with pseudo-static type checking on every method!

I would advise against focusing on that static typing retro-fit effort.

In doing Ruby development, I have found it better (and easier) to put your labor into:

  1. writing clear, object-oriented code
  2. driving your code test-first (BDD/TDD)

Let your tests tell you things are working right. For example, if you wanted to ensure the class that had the first and last name in it behaved properly, you could write RSpec tests:

it "should show patient name as 'first last'" do
  @patient.name.should == 'John Johnson'
end

And if you really wanted to test that the values were Strings you could protect the creation of that object. For example, in a “validate” method that ensures that if a “Setting” instance is to represent a Boolean, than its value better be set to “true” or “false:”

# Settings to be used in the program
# Access in the program as follows: s("Company")
class Setting
 include MongoMapper::Document
...
 # Attributes ::::::::::::::::::::::::::::::::::::::::::::::::::::::
 # What the user sees as a label
 key :label, String, :unique => true, :required => true
 # How we reference it in code
 key :identifier, String, :unique => true, :required => true
 # Data type
 key :field_type, String,  :default => "String"
 # Store any kind of object in the value field.
 # This is what we will use to substitute for the setting
 key :value, Object,  :serialize => true

 # Validations :::::::::::::::::::::::::::::::::::::::::::::::::::::
 validates_inclusion_of :field_type,  :within => FIELD_TYPES
 validate :validates_as_boolean

  def validates_as_boolean
    if field_type == 'Boolean'
      errors.add(:value, "must be set to true or false for Boolean type") unless value == true || value == false
    end
  end
...

Adding a contract into every method as a matter of course, is just a waste of time and a polluting of the code “forest.”

IMHO, let Ruby breathe the freedom it was meant to have.

The Agile Parallel

Leave a reply

For nations that have known virtually nothing about individual freedom and only ever experienced tyranny, you cannot throw a switch and suddenly declare the citizenry ready for self-rule. Same goes for non-agile organizations.

The USA is probably the rarest of rare, in that we actually embarked on just such an experiment in 1776:

Can man rule themselves (without a strong tyrant*)?

* such as the Kings of Western Europe, the Czars of Russia, the Sultans of the Ottoman Empire, the Emperors of assorted Chinese Dynasties, etc?

Maybe the same applies to companies as to governments as to any large organization?

After reading “The 5000 Year Leap” and many other books about the founding of the USA, I have developed an opinion that our Founders distilled the essence of Man (and our flaws) in terms of governing. Around those inherent flaws, the Founders attempted to build an organizational structure that prevents these flaws from succeeding to take hold. After all, our Founders came from a experiential base of being under Kings… but they wisely studied many forms of government, all the way back to Cicero. The last thing the Founders wanted was a strong Federal Government — or they would be right back to where they started: a Ruling Class and Kings (just under a different name).

I (*very humbly*) think the Agile Manifesto distilled the essence of Man (and our flaws) in terms of software development. It will no sooner be foolproof at preventing failures than our own representative government. Without understanding the motive forces behind the tenets of the Agile Manifesto (or the US Constitution), there is little chance that merely acting out the parts will result in lasting success.

The Founders knew that the USA will only survive as long as we have an educated citizenry that participates in trying to maintain our collective goals as a society (primarily freedom and security), while being ever vigilant about not giving away the freedom of the individual.

At the close of the Constitutional Convention, a curious Philadelphian wondering “Well, Doctor, what have we got—a Republic or a Monarchy?” to which Ben Franklin quipped: “A Republic, if you can keep it.” He wisely also asserted the challenges behind keeping it:

“In these sentiments, sir, I agree to this Constitution with all its faults; if they are such; because I think a general government necessary for us, and there is no form of government but what may be a blessing to the people if well administered; and I believe, further, that this is likely to be well administered for a course of years, and can only end in despotism, as other forms have done before it, when the people shall become so corrupted as to need despotic government, being incapable of any other.”

I have always said, “Agile is Hard.” So is freedom, as pointed out by Thomas Paine:

“Those who expect to reap the blessings of freedom, must, like men, undergo the fatigues of supporting it.”

Similarly, an organization’s software development “freedom” will be limited by the understanding that the “citizenry” has of just what makes up quality software and quality processes and self-rule.

Much as it is the rare country that can pull off self-rule like the USA, it is probably the rare large organization that can achieve the same measure of success. In the business world, there is at least a chance that poor leadership in a large organization will be self-correcting via business failure (barring unnatural outside forces). What’s worse, in my opinion, is that there is virtually no corrective force at work in our Government (or military), where bureaucracy and inefficiency can grow un-checked (except by the power of the purse). In large organizations, it is is often easy for folks to “hide” in the comfortable confines of busying oneself with doing a prescribed function (or being in incessant meetings <g>)and passing it off to the next cog in the process, with nary a care if it is an effective use of your time, or not.

The Founders believed that our rights came from God (for if they came from a man — King or otherwise — they could be taken away at any time). The intent was to give the individual as much power as possible, delegating very limited powers to government. As the structure of “government” moved further from the individual, from the town to the township to the county to the state to the federal level, the powers over the individual were necessarily diminished. After all, who knows best about what the town square should have but the town folk? It’s kind of like high cohesion and low coupling…

Similarly, the Founders built in plenty of checks and balances and distributed “power.” Much like a well-oiled Agile team that embraces clients, testing, and the quality folks as checks and balances.

When a software organization is similarly structured to permit strong cohesion at the local level, and responsibility to stakeholders, and very little coupling through the levels of the organization, good things happen. Even a large organization that skillfully permits freedom at very low levels (recursive-like delegation) in the organization gains the benefits of Agile. When the power structure is turned upside down (like in large, command-and-control bureaucracies with prescriptive process, or in the US Gov’t dictating minutiae in our daily lives), less good things will happen. I might even suggest that this sort of system results in natural selection of followers/non entrepreneur types, further exacerbating the situation. But I digress…

If the percentages during the USA Revolution are any indication, one-third were for the old way, one-third were ambivalent, and only one-third were for the new way. What that bodes for Agile sweeping aside the old way, I am not sure.

Fortunately, we “Agile Founders” did not have to “pledge to each other our Lives, our Fortunes, and our sacred Honor.”

Carry the freedom to deliver quality software at your core. The revolution starts with you!

The Agile Uprising

Leave a reply

We were discussing (on a mailing list) the Egyptian uprising and the interest in watching to see who/what stepped in to fill the vacuum of an apparent “leaderless” bit of unrest. Will there be freedom and democracy? Or yet another autocracy? I think there are some parallels to our software world.

Sometimes I wonder if some folks just — subconsciously maybe? — like to be told what to do. After all, it is easier to follow a prescriptive process that you are told will result in the desired goal, than being asked to achieve a goal on your own.

In 2001, the “Agile Manifesto” asked folks to achieve a goal, largely on their own. While there are plenty of agile practices that one can adopt as individuals and teams, there was no glaring “Rational Unified Process CD” with all the process, roles and diagrams that you could ever want to hide behind. There was no venerated “Waterfall Process” that allowed entire silos of functional teams to work on their own with little linkage to being effective at ultimately meeting the goal (until it is too late). There was no marketing-driven organization crying out for the need to be more agile. Hence, it has taken nearly a decade for Agile to become mainstream.

In 2001, Agile was just kind of thrown out there, fanning the flames of freedom for millions of software developers. For a long time, there was no real concerted effort to coalesce a new form of software development and spread it among the unwashed masses.

There didn’t need to be, or so I thought. Agile urges each individual to (using a US football metaphor) “throw the BS flag” whenever there is a need. Agile asks each individual and team to be craftsmen and do their best to achieve the client’s goal.

Agile throws off the shackles of command-and-control and bureaucratic absurdity. Agile works best from the bottom up, being “pulled” in the proper direction as we attempt to satisfy the needs of the clients — not force dictates down their throat. Agile expects and demands the best from individuals.

However, with great freedom comes great responsibility. Being agile is simple, but not easy.

I suppose some dominant command-and-control, bureaucracies were threatened by Agile uprisings, and kept the practices suppressed, driving some teams “underground.”

In the past few years, however, our community has seen a major embracing of Scrum by small and big companies alike. Maybe Scrum looks enough like routine Project Management that bureaucracies can be easily led to buy into this “agile” stuff.

So, maybe the Scrum Alliance did step in to fill that leadership vacuum after all?

Agile Bash

2 Replies

If you came to a conference with all of the original Agile Manifesto co-authors in attendance, what sort of event(s) would you like to see/participate in with said authors?

  • Nothing, we don’t really care about the past
  • Open discussions?
  • Informal gatherings at known locations throughout the day?
  • Q&A session?
  • Themed presentations to re-emphasize the true meaning of the 4 Agile Tenets?
  • Etc.

Would you want to be able to easily recognize these folks as they sit at the bar? For example, to buy them a beer 🙂

Feel free to comment below 🙂

Simple != Easy

2 Replies

In a nice, vintage post from the oh-so-wise Ron Jeffries,  as I saw this heading:

“It couldn’t be that easy!”

It led me to think, possibly a different way to say it…

“It couldn’t be that simple!”

Because I think what Ron is getting at — at least this is what I believe to be true — is that, many times, folks over-complicate things.

In addition, folks often confuse simple and easy: just because something is simple doesn’t mean it is easy.

Conversely, just because something is hard doesn’t mean it is complex. I think sometimes folks run into “hard” things to do, and decide that it has to be complex to justify the degree of “hard.”

Not so…

Keep it simple.

(Now, go look at the date of Ron’s post!)

Software Driver’s License

1 Reply

I thought of something whilst driving the other day…

A USA Drivers License is a lot like a Scrum Certificate. Basically anyone can get one with only a smattering of studying and a r-e-a-l-l-y trivial practical exam.

When people need to hire someone that can drive, they check that the cert exists (ok, I know there are commercial driver’s licenses which are ostensibly harder than what we get as normal drivers).

It doesn’t mean that they are great drivers. It doesn’t even mean that they can be trusted with much of anything. It is what it is. A simple certificate acknowledging that they passed a simple test.

So, is the issue that some hiring folks put too much emphasis on ensuring something as trivial and basic as a drivers license, when the job is indeed far more complex and nuanced than merely knowing the basics about driving?

Sounds to me like a clear signal to stay away from said company that puts undue emphasis on something trivial. That is, unless you want to work in that sort of environment.


					

		
		
	

							
	
							
	

				

			
						

				Agile Won’t Be Killed
			

										

					13 Replies				

					


				

			
UncleBob wrote this good piece on whether Scrum could Kill Agile.


To me, Agile is a mindset, a state of mind, a way to be. It is neither a (set of) coding practice(s), a concrete process, or a management technique.


Agile is not TDD, XP, FDD, Crystal, Scrum, Lean, Kanban, Six Sigma.


Agile is about being responsive to the current and future needs of the business within the current context, so that you can do the very best possible given your team and budget. Read these again:


    

  • Individuals and interactions over processes and tools
    • 

  • Working software over comprehensive documentation
    • 

  • Customer collaboration over contract negotiation
    • 

  • Responding to change over following a plan
    • 



The “we value the left-hand items over the right-hand items” is precisely loose because it all depends on the context and the business needs. Believe me, we argued mightily over small wording changes in that Snowbird meeting! It’s just that in general, our collective experiences had born out the truths that we were more effective and responsive to delivering valuable software when we emphasized the left-hand items. I was coming from the government background, and I was determined to demand we push for truth and honesty in software. No more stupid 70% done and no working software to show for it!


I submit that the Agile Manifesto tenets are irrefutable ideas and will stand the test of time.


The Non-agile Manifesto


Let’s try a little negative test (if you agree to these, I’ll poke you with a hot stick):


    

  • We will build software with processes and tools — that way we don’t need to worry about people issues and effective communication.

      

    • That way we can hire any individual regardless of skill, and forgo all verbal/personal interactions in favor of solely written words. Even better if those written words are automatically transformed into code. Maybe we can get non-coder tools! After all, people are merely fungible assets/resources, and software is factory-work — with processes and tools, and a horde of low-paid serfs, we can crank it out!
      • 

    

    • 

  • We would rather spend exorbitant sums of money producing comprehensive documentation than have tangible, working results early on.

      

    • We start with complete requirements specifications (often 400 pages), that follow our company standard template.
      • 

    • Even our Use Cases follow a mandatory 3-level deep path, with proper exception and alternate paths worked out.
      • 

    • We link the requirements items into detailed design documents — which include system design diagrams, interface specifications, and detailed object models.
      • 

    • If we don’t write it all down now, we’re likely to forget what we wanted. And if we don’t do it to the n-th degree, the developers might screw it up.
      • 

    • Writing it all down up front allows us to go on vacation while the process and tools “write” the code from the detailed specs/diagrams. Sweet.
      • 

    • In addition, we love to be rewarded by reaching meaningless intermediate deadlines that we place on our 1500-node Gantt chart.
      • 

    • When we combine all of the upfront work with a important deadlines, many of the senior managers can get promoted due to their great commitment to generating reams of cool-looking documents. By the time the sh!t hits the fan when folks realize the “ship it” deadline is missed, the senior managers are no longer involved.
      • 

    • Besides, if we actually built software instead of writing all sorts of documents thinking about building software, our little ruse would be exposed! (BTW: if the human race had done nothing more than documented the ideas about how best to go about procreating, we wouldn’t be here to waste software budgets. w00t!)
      • 

    

    • 

  • We prefer to write a tight Statement of Work & force a strict fixed-price contract rather than actually work towards a common goal with the client.

      

    • The agreement is based on the 400-page, fully-specified requirements document, and we pad the cost estimate with a 400% profit margin.
      • 

    • We then hire dozens of people to argue during the Change Control Review Board monthly meetings about re-writing code to deliver what you wanted versus what you asked for when you thought you knew what you wanted (and wrote it down in that 400-page doc).
      • 

    • Contract negotiation pissing matches are such great uses of our collective resources and always result in perfect software! We love our fine print 🙂
      • 

    • With a 400% padding, the projects are too big to fail.
      • 

    • Once we are in it for 1 or 2 million and 50% done and 2x schedule overrun, who would ever say “No” to a contract extension? Who better to get you to the goal line than the same folks who squandered away your treasure, pissed away the calendar, and delivered no working software yet?
      • 

    • We like to appear like we’re just about done… Asymptote? Never heard of one.
      • 

    

    • 

  • There’s nothing more appealing than following a 1500-node Gantt chart. To the letter and subletter. Change sucks.

      

    • Especially a Gantt chart that has been built with tender loving care to include resource allocations, inter-project dependencies, and partial resource allocation assignments for matrix-style organizations.
      • 

    • We love hiring a small army to ensure that we drive the entire team to meet every micro-task deadline even when they no longer make any sense.
      • 

    • The real fun is collecting the “actuals” data from the developers assigned to each task.
      • 

    • And nothing sweeter than seeing 90% of our tasks being started, and 75% of those being 67% completed, and 25% of them being 99% completed — the roll-up summary to management is such a compelling story.
      • 

    • Changing such a beautiful plan that took 4 man-years to develop, that incorporates all of the comprehensive non-code documents, and is an appendix in the contract, is no small feat!
      • 

    • Better to produce the software according to plan even if nobody wants it that way. That’s our motto, and we’re not going to change!
      • 

    

    • 



Waterfall Sucks


Waterfall — as practiced — will die under it’s own weight. Waterfall is a bad idea when applied broadly to all sorts of classes of software projects. Eventually people will wake up and realize that — for many situations — there are better ways to doing software than waterfall. (Waterfall done on one feature at a time is a fine practice 🙂


I remember seeing folks who worked for Big Consulting Firm go into an organization to do all of the Use Cases for 9 months straight. Their documents were then handed off to some other firm to do the analysis and design. And finally, yet another firm would do the implementation based on the sum total of documents. Since the written word left nothing to the imagination, these projects always delivered perfectly. Oh, wait. Standish Group says that really isn’t how things worked out. This might be an okay way to go about hardware or construction projects — since you can be more deterministic up front, and there are more common parameters that are “agreeable” to most folks in terms of real interface documents (like engineering or architectural drawings are pretty specific and not ambiguous). But to anyone that has any experience in software development, waterfall seems ludicrous from Day One! Who would ever think that you could hand off all features as they went through each of the major phases with no loss of understanding? Who would ever think it is good to wait FOREVER to see any tangible results? After all, we’re NOT bending metal or breaking ground. It’s freaking software! Sigh.


Waterfall seemed ludicrous to me my very first time. I was exposed to Mil-STD-2167 while doing DOD  contracting in the 80s. I think I followed along like a good little well-behaved  junior engineer my first time. Probably even my second. Then my brain kicked in and said — this  is stupid for our situation. We’re doing software development for R&D projects, we’re  not doing assembly line work for a brick factory. We can’t know everything up front, and it is risky to think you can! So I started to look  for better ways, and began to not do wasteful practices, even though  they were prescribed. I began to do the right thing for the customer — and they agreed to the <ahem> “waivers.” (I was applying the “Better to beg forgiveness, than ask permission” principle I learned from my good friend and mentor, USMC Colonel Rich “Magic” Dinkel, fighter pilot extraordinaire.)


I sound like I am bashing people that did Waterfall… I don’t want to be that harsh. People did what they thought was best at the time. I think many times we fall into a trap of wishful thinking. And even in the face of overwhelming observations to the contrary, we soldier on in hopes things will work out like we want. (Besides, it’s in the contract!) Rarely is hope ever a good strategy. Maybe never? So folks doing waterfall certainly had good intentions, and it was compartmentalized enough to reward intermediate victories and give an illusion of progress, but there’s no denying it’s just a bad idea.


Agile didn’t kill Waterfall per se. Sure, maybe it helped accelerate the community’s realization that there were better ways to do software than a giant, rigid, 4-stage assembly line.


Waterfall killed waterfall.


Scrum Won’t Kill Agile


I say let the Scrum Fun continue! Make tons of money! Waste as much budget as you can. Brag that you are doing Scrum and have hired 20 CSMs to whip your developers into shape!


The more people and HR departments that think there is some magic bullet, the more companies that might fail and lose business. This makes way for smarter, leaner, more entrepreneurial start-ups to eat their lunch. After all, I would wager that very few small organizations are falling for the Scrum Will Save The Day stupidity that is seemingly gripping some (large?) portion of our community.


Convenient Cover?


Here are a few phrases that come to mind regarding the Scrum dustup, and I’ll add a new one…


    

  • A friend once quipped that “Off-shoring is just a cheaper way to fail” — implying that if you can’t build software locally, you probably can’t build it 12 time zones away either. But at least the costs are lower.
    • 

  • No one was ever fired for hiring IBM
    • 

  • We are doing Scrum — all the rage in Management By Magazine circles!
    • 



In other words, it is often far easier to go with the flow, follow the herd, live by the plan, and say you did what you were told. It is much harder to buck the system and call out the ineffective and wasteful activities, and continuously try to improve. It’s hard to buck inertia!


Conclusion


Nothing will ever kill agile.


Bold, I know. However, I view it like the USA’s founding principles — immutable laws of freedom. Agile is about freedom to do the right thing.


How can you ever deny the correct way to do software is to do what’s best for the business?


How can you deny the Agile Manifesto tenets?!


Successful software will continue to be developed in an agile manner, regardless of the terms that might become popularized.

					

		
		
	

							
	

				

			
						

				Plan & Estimate, Only if You Must
			

										

					3 Replies				

					


				

			
It must have been the morning to challenge planning efforts, as GeePawHill did here. I was answering a post that had gotten around to discussing budgeting and planning for interruptions to fix legacy apps while teams are building new stuff.


When all of the sudden, I blurted this out:


Kanban is simple, in theory. Simply do one feature or bug fix all the  way to being released. AKA focus. But there is a lot of things that have  to be working well, for example: the cost of testing has to be low so  that it doesn’t cost a fortune in treasure or time to get in that new  feature or bug fix. (Expensive testing often leads to bundling a bunch of stuff into a given release and widening the time window, and introducing parallel development techniques.)


And, regarding budgeting in to prepare for the unknown maintenance  efforts… That’s but one way to handle things. Another way is simply  let the people who care about getting things done participate in setting  your priorities. If you were given 10 things to do in a month, and  halfway through you get an 11th thing to do. You probably don’t need to budget  for that unknown. You simply ask for help deciding which of the original  10 should get axed to make room for the new “emergency” item.


Like it or not, you are  all in it together, it is a zero sum game. If you fix the resources, fix  the schedule, the amount of work that gets done has to give when new  work is injected. Deal with it in an honest manner.


Hard to describe in words… and had to know your exact context… but what you are facing is probably more of an organizational issue than anything else.


All we software folks do for a living is produce features and fix bugs to keep paying customers happy. Just keep  trying to improve your team’s efficiencies and squeeze out wasteful practices.


When you are producing software as an internal organization, you can  sometimes be brave and forgo lots of planning stuff. If you have the  priorities set, and if you have a delivery schedule, and if you  understand your costs of releasing, you can make a pretty simple and  smooth process. I’ve done it many times in companies formerly steeped in  waterfall fantasy land. Why waste all sorts of time making estimates if you don’t have to?


The answers to these questions can help you decide how much ceremony your  processes need:


    

  • Does it matter if the team does 5 things in an iteration or does 10?
    • 

  • Does it matter if they estimated 10, but only wound up being able to do 5?
    • 

  • Or, if they bid 5 and found they could do more and gobbled up 10, is that a problem?
    • 

  • Is it more important to hit a date with at least some features?
    • 

  •  Or, is it more important to produce specific features as fast as you can, even if you miss the date?
    • 

  • Does the team need to be accurate on their estimates? How accurate and why?
    • 



Many times, we fall into traps of wanting to manage software process as  if it were a brick-making factory. Unless you live or die by doing  software as a fixed-bid project, you can often really get simple and  make software really fun in your organization by doing less ceremonial  planning crap!