Category Archives: Development

Multiple Many-to-Many Associations in MongoMapper

There was a question in the Google Group for MongoMapper, so I decided to post an answer in the form of a simple demo. You can find the source code on Github.com here.

The basic shape of the problem was this:

Users Sponsor and Attend Events

Users Sponsor and Attend Events

And the solution issues were around the multiple many-to-many associations, more or less.

For a simple one-to-many, MongoMapper has the normal:

  • User has many :events
  • Event belongs_to :user (its Owner)

But how to do the other associations? A given User can be involved with many events in different capacities:

  • Attending
  • Interested in attending
  • Likes

There are different ways to tackle these many-to-many associations.

  1. You can use a Set (to obtain the uniqueness factor) of Users that are attending or are interested.
  2. You can use an Array of instance IDs (I think this is probably the more standard technique)
class Event
  include MongoMapper::Document

  key :title, :required => true

  key :user_id
  belongs_to :user

  # One way to do it...
  key :attendees, Set
  key :interested, Set

  # Another way to do it...
  key :like_ids, Array
  many :likes, :class_name => 'User', :in => :like_ids

  def attending(a_user)
    # self.push_uniq(:attendees => a_user.id)
    attendees << a_user.id
    save
  end

  def interested_in(a_user)
    interested << a_user.id
    save
  end
...

In which direction you allow making the association, that is up to your application’s needs. For example, above you can see that an Event instance could be messaged with the user to indicate attending or interested_in. The “likes” is immediately accessible from an Event, or I could have added a wrapper method (def likes(a_user)).

And the User class has some simple retrieval methods to see what a User likes, what they are attending, and what they are interested_in:

class User
  include MongoMapper::Document

  key :name, :required => true

  many :events

  def likes
    Event.where(:like_ids => id).all
  end

  def attending
    Event.where(:attendees => id).all
  end

  def interested_in
    Event.where(:interested => id).all
  end

  def likes_event(event)
    event.likes << self
    event.save!
  end

To see how the different styles are used, you can check out the specs. For example:

Adding users who like an Event:

  it "should track interested" do
    expect {
      @event.interested_in(@jared)
      @event.interested_in(@sally)
    }.to change {@event.interested.size}.by(2)
  end

it "should allow 'likes'" do
    expect {
      @event_2.likes << @martha
    }.to change {@event_2.likes.size}.by(1)
  end

Or from the User perspective:

  it "should allow me to add an event I like" do
    @fred.likes_event(@event_2)
    @event_2.likes.size.should > 0
    @fred.likes.count.should > 0
  end

Or attendees, from the Event:

  it "should list the events I am attending" do
    [@fred, @harry].each {|u| @event.attending(u)}
    @fred.attending.count.should > 0
  end

RVM Install Troubleshooting

All I wanted to do was sit down with Mike Hartl’s “Ruby on Rails 3 Tutorial” yesterday. Hours later, with nothing to show other than a zillion open terminals, I punted and watched the Flyers beat the Sabres despite playing like girls (I know, unfair to girls!). Then I went to bed.

So I must have had a dorked up installation of RVM or something, because when I tried to install Ruby 1.9.2 and Rails 3 and upgrade RVM I ran into <ahem> “issues.” I was getting weird errors like:

error installing rails i18n requires rubygems version 1.3.5
gem_runner.rb:85:in `<top (required)>': undefined method `load_plugins'

After exploring all sorts of similar woes that other people had and not really getting that magic googullet (Google Bullet), I decided to go all nuclear (apologies to Japanese readers). I was able to run:

rvm implode

Which made “rvm -v” fail to find rvm — a good thing. It meant I had uninstalled RVM. (Or so I thought!)

I also deleted the RVM stuff from the bash_profile file(s). Just to attempt to eliminate voodoo dolls.

But I still ran into stupid errors (stupid voodoo dolls!):

error installing RVM fatal: Not a git repository (or any of the parent directories): .git

After much pain and suffering (well, only in a wimpy software relative sense), my guess is that I installed rvm as sudo at one point or installed the gem, or somehow had things “cross-eyed.” I don’t know…

I guess I should be more diligent, because in general you get spoiled by Ruby and gems and Rails and the general euphoria of how easy things are. Until they aren’t.

Today, I ended up fixing my system by looking for “rvm” on my system, and cleaning up leftovers. The crumbs from my sloppy eating at the trough of Ruby, I suppose.

I saw some directories holding rvm docs. I whacked them.

I saw some directories with rvm source, archives… Gone.

I uninstalled the rvm gem. (Oops. Who put that there?)

Finally, I retried the install:

jonsmac$ bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)

And added this to ~/.bash_profile And I restarted terminal because I always forget how to do that source ~/.bash_profile thing…

jonsmac2-2:~ jon$ rvm -v
rvm 1.6.2 by Wayne E. Seguin (wayneeseguin@gmail.com) [https://rvm.beginrescueend.com/]

Shazam! No errors. Thanks Wayne!

Now I can get past page 15 in Mike’s tutorial book!

Getting Started w/ Best Practices & Design Principles

Praveen had a seemingly innocent question here and there were lots of good answers…

If we are following Agile Methodology (Scrum) & We are in Java/J2ee Domain,

What are the Best Practises & Design Principles need to be followed
while designing the applications.

Big question is an understatement… here’s what I would advise:

  • Understand the business domain
    • because, if you get that wrong, it doesn’t much matter how well-crafted and well-tested your code is
    • Design using Separation of Concerns as the granddaddy of all patterns to sear into your brain.
    • Be sure you use a layered approach to your architecture — if I catch you putting SQL in your javascript, I’ll come over there and stab you in the toe
    • Design in the Large to avoid excessive dependencies
    • Design in the small to adhere to all those good practices like SOLID
    • Consider looking into the Domain Neutral Component — within which a shockingly huge number of domains tend to fit.
  • Architect and build the application in a highly consistent manner — much like Rails enforces.
    • If you have a dozen domain items that need the same sort of treatment from persistence to UI, please don’t get creative and do persistence 5 different ways.
    • Pick one and stick with it for the sake of consistency!
    • I’d rather a consistent implementation that is mediocre, than one with 5 different approaches (one stellar, one horrid, and the rest mediocre). Why you might ask? Because it is easier to refactor when things are consistent 🙂
  • And, of course, apply an agile process…
    • Involve a customer/proxy to ensure you understand the requirements from the business perspective, do not rely on documents — but if you must, then you need to exaggerate the agile process and make it more granular. That is, to compensate for the likelihood that the written word is ambiguous, you need to show small steps of progress on a feature and get rapid feedback to ensure you are not mis-interpreting the requirements doc.
    • Build a vertical slice of your design to prove it out and to serve as a model for how you will work on your app
    • If you have stringent performance needs, prove them out early
    • Show working code early and often
    • Get yourselves set up on an automated build
    • Try to build out the features using something like BDD or TDD
  • General
    • Be patient with your learning
    • Be impatient to see results, don’t let any team member or task go dark for more than a couple of days
    • Create the lightest-weight process possible
    • Don’t do anything to excess without frequent feedback
    • Always foster a team attitude to question everything with boldness — including my advice

Hope that helps

Class- and Field-Level Custom Validations

This is a simple example I put together for someone asking how this validation stuff works…
In this example, you can see field-level validation:

  • team must be chosen
  • name is required

And you can see a class/base-level validation:

  • at least one method of contact, a phone or an email:

Simple class example that employs some conditional validations and some regular validations:

class Registrant
  include MongoMapper::Document

  # Attributes ::::::::::::::::::::::::::::::::::::::::::::::::::::::
  key :name, String, :required => true
  key :email, String
  key :phone, String
  # Parent Info
  key :parent_name, String
  key :parent_email, String
  key :parent_phone, String
  key :street, String
  key :street2, String
  key :city, String
  key :state, String
  key :postal_code, String

  # Associations :::::::::::::::::::::::::::::::::::::::::::::::::::::
  key :team_id, ObjectId
  belongs_to :team
...
  # Validations :::::::::::::::::::::::::::::::::::::::::::::::::::::
  validate :validate_team_selection
  validate :validate_parent_contact_method
  validates_presence_of :parent_name, :street, :city, :state, :postal_code,
                                    :if => :parent_info_required?

...

  private

  def parent_info_required?
    season.parent_info_required?
  end

  def validate_parent_contact_method
    # one or the other must be provided
    if parent_phone.empty? and parent_email.empty?
      errors.add_to_base("At least one form of contact must be entered for the parent: phone or email" )
    end
  end

  def validate_contact_method
    # one or the other must be provided
    if phone.empty? and email.empty?
      errors.add_to_base("At least one form of contact must be entered: phone or email" )
    end
  end

  def validate_team_selection
    if registration_setup.require_team_at_signup
      if team_id.nil?
        errors.add(:team, "must be selected" )
      end
    end
  end
end

The above will result in always checking the name, the contacts, and the team assignment. It is actually pulled from a slightly more complex context in that a few conditionals exist to further complicate the validation logic. If you need checking only at certain points in the object life cycle, you can look to things like validate_on_create.

In the case of “Parent Info” being required, additional validations take place.

The “name” validation is a standard, out of the box field-level validation due to “:required => true” being added to the MongoMapper key.

Enjoy!

Using Basecamp in an Agile Manner

Software Development Life Cycle

In software development, we want to make our process as visible as possible to all participants and stakeholders. To that end, behold a simple process that uses a handful of Basecamp To-Do Lists as markers for how a feature or bug is working its way through this process.

At any point in time, you can look at the “To-Do” list page for a status on your favorite new feature, task, or bug. And, if you have a chunk of features piled up in an “Iteration 1” list, you can see the number of items being worked through to completion. (There are even third-party apps that allow you to do burndown charts if they appeal to you.)

Using Basecamp's To-Do List in a simple manner

Requirements/Bugs

The business has a desired priority for features as expressed by the order of items in the “staging” lists, and especially the On-Deck list (to borrow a baseball metaphor for “What’s next”). For major new features, the business and the developers will work out a rough Release Plan. But for simple enhancements and bugs, the business can simply add desired items to the lists.

Development

The Developer will pick a new task off of the top of the On-Deck list. There will usually be a conversation around the issue to ensure proper understanding (unless it is self-explanatory). Ideally, we get the business/testers helping to write User Acceptance Tests in the BDD form of Given… When… Then… The Developer moves this feature into the “In Progress” list. If desired/possible, the user could add a date estimate for the task.

The Developer will create tests, be they Cucumber, RSpec, or UnitTest, or all 3 types. If possible, the Testers can help write the up-front acceptance tests, so that they are intimate with the feature/bug. The Developer writes code until the tests pass.

Once the issue is complete, and the tests are passing (and you didn’t break any other tests), the Developer: commits the code, updates the TEST server, and does a smoke test to ensure things still work as expected. The Developer then moves the task to the “To Be QA’d” list, where the Testers will now verify the issue is indeed complete and correct on TEST. Close collaboration between Tester and Developer is warranted here, as the nature of the fix and any developer tests can be explained so that testing can be as effective as possible. Plus we want to maximize use of automated tests, and minimize any unnecessary manual tests (as they are time consuming and expensive) given the nature of the code changes.

Testing

If the QA passes, the Tester moves the task to the “To Be Released” list. Depending on the nature of the feature, the Business can also help to “test” and ensure things work as expected. If the test fails, then the task is moved back to the On-Deck list and the details added to the issue as comments (if simple) and image uploads (add image link to the comments), or to the issue tracker if it is more complex (even though it is technically not an issue against the released product).

Release

The Developer will schedule the move for “To Be Released” tasks to the PRODuction box. This may require coordination with the client’s IT team if they have internal controls. Following release, the application is smoke tested to be sure the basics are still functioning. If any problems arise, the server can be rolled back, or fixed ASAP, based on the nature of the error.

Once the items are released to PROD, you can indicate that they are “complete.” If the issues came from a list that you want to keep for posterity, simply drag the completed task to the original list (“Iteration 1” for example). Now you can see the items in that list that have been released (i.e., completed).

Summary

I’ve used Jira and much more elaborate SDLC schemes in the past. But a recent Rails project with just a few folks on the team, found us trying out Basecamp. By creating various lists, we can mimic the Kanban-style wall board that Greenhooper provides within Jira. Because the lists allow simple ordering to show priority, and you can drag-and-drop between lists, and you can track your time, I think we have a very lightweight, winning combination.

This process is equally at home doing one feature all the way through, or an entire iteration’s worth of features in a bigger chunk. (Basically dependent on the cost of testing and the appetite of the end-user for changes.)

SDLC process[/caption]

Large-scale Separation of Concerns

Found this draft post from: “Last edited on April 20, 2009 at 4:10 pm” … Wonder where I was headed? Hah, I’ll add a 2011 conclusion.

While we have enjoyed employing Separation of Concerns at many levels:

  • Design Patterns
  • Layered architecture

I wonder if it can go farther to assist us in developing robust applications.

I have been preaching a “Dual Architecture” for eons…

  • Application architecture (features)
  • Technical architecture (code, framework)

However, I think this could be taken a bit further.

We use source code, classes, tables to be an abstraction of the underlying true system being run. The source gets converted… the classes get converted, database tables get converted… to something that the machine knows how to deal with.

User gets the functionality expected, the system does the grunt work.

The user is not too worried about the “how” it works, but rather more so that it simply works.

With software, it would be nice to be able to work at a similar higher level of intention, or functional need, with less regard to the nitty gritty detail.

After all, with most business applications, the fastest moving component is technology continually changing.

A major, large-scale SoC is to separate the Functional needs from the Technology.

Users care little if the app is built on Java, .NET, Ruby, COBOL…

2011 Addition:

Of all the languages and frameworks that I have used over the past 30 years, I would say Ruby and Rails (and it’s community) with MongoDB/MongoMapper has come the closest to allowing a greater portion of development time and thought to be spent on solving the customers needs and less time on the infrastructure nuts and bolts.And it is more than the language (Ruby) and the framework (Rails). It is the entire culture and constellation of tools and gems and community that makes a difference.

In addition, it feels like I can code more to the intention of what I need the software to do, and less about the details of the language getting in my way.

 

Lactic Acid?

Jason Gorman has an interesting take on a team’s technical debt being similar to a build-up of lactic acid. That is, if you are in a team that is “unfit” and normally not used to delivering quickly, then running fast, as in sprinting through the initial iterations, will undoubtedly be painful. Much like in athletics, where you can build up “lactic acid” and experience pain and be unable to continue. Jason equates this human sort of feeling to “technical debt” building up in the development project.

While I can appreciate the analogy, avoiding technical debt in a software project is a lot more difficult than becoming more fit and increasing your lactose threshold.

Also, a team can work at a sustainable pace just by coming in at 8:30am, going home at 5pm, not checking emails on the weekend or at night, and plodding through the features and fixing bugs as they come up. Such a “sustainable-paced” team could simultaneously be building up US deficit-sized technical debt (well, okay, nobody could achieve that measure of stupidity ).

But, I *know* what Jason means. I just think that sustainability is not *the* foundation of Agile. On the contrary, I would surmise that sustainability is a side effect of having a lot of the “right things” going on in your organization, and something to strive for as part of the recipe for success.

To Prefix, or Not to Prefix

Today I had a little “tiff” with a developer about something very trivial by most accounts. Maybe I was having an off-hormone day, or my biorhythms are off.

A simple table was proposed to hold the text name behind a status property. Something as simple as:

ID STATE
1 DRAFT
2 REFERRED
n STATEn

The developer suggested “status_description” as a field name. That was all it took to chew up 25 minutes.

The IRC conversation goes something like this:

11:16 <jon> what do you propose for state name?

11:16 <sam> status_description

11:17 <jon> that’s too long of a field name

11:17 <sam> status_name then

11:17 <jon> i want to say s.name

11:17 <jon> no status_

11:18 <jon> i hate that 😉

11:18 <sam> nope, no go homey

11:18 <jon> the table tells me what it is

11:18 <sam> name is too generic

11:18 <sam> you’ll have 50 tables with “name” field

11:18 <jon> but how can it ever appear out of context?

11:18 <jon> i love 50 tables with name, it forces table aliases and context (like classes and properties)

11:21 <jon> if you repeat the table name on every field… it makes for more fun typing things like that over and over when you are writing sql

11:23 <mac> we do have xxx_id everywhere… for the primary key field

11:23 <mac> i hate it, but it *is* consistent. however, status_name is not necessary, i agree, jon

11:24 <jon> i abhor useless prefixes, it leads to lousy coding, makes the human the compiler

11:24 <jon> but that’s just me

11:25 <jon> i’m going to hibernate for a while, head back to doing C coding and structured crap so i stop thinking in OO

11:27 <sam> not that it matters but prefix has been an Oracle paradigm

11:27 <jon> it doesn’t matter. microsoft used “m_” prefixes on class members (IIRC) and i hated that too. just because a BIG company does something doesn’t mean it is a good idea. IMHO, it is a horrid paradigm to add extra stuff on everything

11:27 <sam> not really, it’s just preference

11:28 <jon> yeah, and my preference is for clean code 😉

11:28 <sam> btw: the object model doesn’t have to match the sql model

11:28 <mac> but prefixes make the SQL read poorly and is extra typing with no benefit

11:28 <sam> for instance the underscore thing I hate with oracle

11:28 <jon> why on the table Person would i want to see every field prefixed? For example, person_first, person_last, person_birthday…

11:28 <jon> of course the FIELDS IN PERSON ARE FOR PERSON

11:29 <jon> who would think otherwise if they are staring at columns like First, Last, Birthday in a table (or class) called Person?

11:29 <mac> where q.section_id = s.id is fine. i preferred “q.fk_section = s.id”

11:29 <sam> just the common id and name… that could be on 100 tables. name is very generic

11:29 <jon> yea, the beauty of tables… THEY PROVIDE CONTEXT to things like id and name

11:29 <sam> you don’t have to prefix though

11:29 <mac> p2_question.name if you avoid aliases for some reason.

11:30 <sam> it’s a DBA thing guys

11:30 <jon> Really? it is better practice, imho, to have context via table alias not prefixes. Maybe I don’t always buy all DBA rules

11:30 <jon> if there was only id,name, across 5 tables, you can’t make a mistake because you HAVE to put it in context (i.e., add a table alias)

11:31 <sam> guys this helps

11:31 <sam> what happens when I create a datawarehouse with a bunch of “name” fields

11:32 <sam> it kind of becomes a redesign effort where prefixes are then needed regardless

11:32 <sam> make sense?

11:32 <mac> use good aliases in your queries?

11:33 <jon> no, it makes *no* sense to me… when u create a datawarehouse you name things properly instead of demanding prefixes.

11:33 <jon> but don’t screw up every table with prefixes for all other use for the someday nicety to not have to make the datawarehouse column names clear

11:34 <jon> why pay the penalty in all coding all the time? please don’t spread the pain 🙂

11:34 <jon> i like to keep the crap in a small part of the app

11:34 <jon> if the dw has to have field names like person_name, agency_name… so be it

11:35 <jon> but i’d rather everybody did this consistently with no prefixes

11:36 <sam> Jon, there are countless whitepapers on proper db design, one google yields this

11:36 <sam> http://www.simple-talk.com/sql/database-administration/ten-common-database-design-mistakes/

11:36 <sam> speaks to my concern

11:37 <jon> gee where are the prefixes?

11:37 <jon> NOWHERE

11:37 <sam> CustomerName

11:37 <sam> don’t see that?

11:37 <jon> That’s one tiny example in a sea of non-examples. Not to mention this:

11:39 <jon> “Along these same lines, resist the temptation to include “metadata” in an object’s name. A name such as tblCustomer or colVarcharAddress might seem useful from a development perspective, but to the end user it is just confusing. As a developer, you should rely on being able to determine that a table name is a table name by context in the code or tool, and present to the users clear, simple, descriptive names, such as Customer and Address”

11:40 <jon> FWIW you will never sway me to the dark side of name prefixes for everything 😐

I had to go outside and toss the Jolly Ball in the snow, wind, and 5 deg F (wind chill), with my Yorkie…

Requirements Traceability Matrix

In the agile modeling forum, the subject of a “Traceability Matrix” (TM) was broached.

Fabio asked:

I am trying to elaborate a traceability matrix, of the functionalities of the system that I work with. I would like to know if someone has an example on how I can elaborate this traceability matrix. I have a draft of this matrix, that I did, but I would like some ideas, so I can improve the matrix that I elaborated.

The traceability matrix will be used more by the QA team, but I think it can be useful for the whole it team. And with the traceability matrix, any requirement changes that may be developed we know exactly where the changes are affecting taking a look at the matrix. With the matrix we will also build up our test cases.

IMHO, the TM is a holy grail of an idea. In my years of working with building tools like Together, I got requests now and then for building a Traceability feature. The first one I recall was from a big company in Texas, maybe defense related, I forget (circa 1998 or 99). I asked for details behind why they wanted it and what is was supposed to solve. I didn’t get a very good answer.

Since that time, others asked for it here and there. Though I had technology at my fingertips to build traceability into Together tools, I never found the feature worth justifying the cost. (And yes, it has to do with relating a Test to the feature, and doing code tracing…) A few years back, when I was with OptimalJ (an MDA tool), and we added SteelTrace as a front-end requirements tool. There was a technical way to relate a requirement down to the lines of code.

But so what does it buy you?

While the concept seems appealing at first glance, reality often bites. It is hard to maintain, of exponentially diminishing value as the required level detail increases. In all my years, I never once was shown an example of a TM in practice that was used for anything remotely justifying its expense.

However, that doesn’t mean that a TM is not useful to somebody, I just never ran into that group or person(s).

My suggestion, Fabio, is to revisit the true needs of the organization. Then see what is the best way to get those needs satisfied. Should some form of traceability matrix be useful, I suggest starting small and inexpensive. See what you can get easily, and see if it is useful. Conversely, think of “prototyping” the end result of a TM. Pretend to use it for some scenario for which your organization thinks it would be useful. Explore alternative ways to achieve the same result (classic manual code searching, step-wise debugger, reverse-engineering sequence diagrams, or even code-based tools that allow insane levels of search — some recent web-based offerings that i don’t recall off the top of my head). Typically, the hardest things to find that might be impacted by a requirements change (nuances of data storage, little XML tweaks here or there, meta data, resource files, things generally non-discoverable in just the source code), are not tracked in a TM anyhow.

There are many ways other than a TM to keep the team informed of how requirements are implemented through the application. Acceptance Tests that prove the existence of a specific feature are one way. Design documents and models are another — assuming you do a good job at naming things to make it clear that a method in a class is named a lot like the english language of the expected requirement. A wiki can help keep these documents front and center.

IMHO, the best way to keep the QA team informed about the detailed impact of requirements changes is through developer communication (both in the issue comments and verbally as required). And, IMO there are two schools of thought regarding keeping QA informed of details. One is that an informed QA group that knows (via the developer) that a code change they just did for Reqt X, may have caused some havoc over in Reqt B, allows the QA team to please hammer those tests for B just in case. The other QA group is to be “blind” to details of the code changes, and simply do tests and ensure things still work.

WISKY

more folks should follow the WISKY principle… NOT!

(Why Isn’t Somebody Koding Yet?)

in 1995, my team and i were architecting a C++ solution for IBM’s next-generation manufacturing software. i had a crack team of 6 senior engineers/developers.

while i was gathering requirements and building object models, sketching UI ideas, and doing release planning; others on the team were building out (coding) the technical architecture we envisaged for this radical, thin client system. i also had my team setting up some real world simulation tests to thrash the architecture about, poke and prod it, and see if it was robust enough for our use.

the client was getting restless/antsy… they wanted to get moving on development of the whole app. they wondered “why aren’t more people coding? won’t we finish sooner if you have a couple dozen people working from the start?” but in reality, it seemed they wanted to bring on the dozen or so developers that were allocated and start billing them out and to report some “progress”.

i remember telling management “flat out” that if they insisted on bringing forth the horde of developers, the developers would just sit there and do nothing until my team and i were ready for them. i explained that this up-front work was essential to getting the team propelled in the right direction, with the right architecture, and consistent coding standards and coding templates. i also mentioned that this normally required about 10% of the project effort.

i got my way. no WISKY was allowed! (Allowing a bunch of developers to begin to develop prior to understanding the architecture, coding patterns, and the basic priorities is a big mistake.)

A few weeks later, we were ready to begin and brought in the other developers. By that time, we had our thin-client, layered architecture determined, and a pattern for folks to follow.

(footnote: the 3 months upfront was pretty close to that 10% mark, for the project ended up being ~3 years, ~250 domain classes, and ~1 Million LOC. This was my first large “agile” project.)