Technical Debt

Return To Forever

went to see Chic Corea and Return to Forever last night at the Mann Music Center in Philly.

the show was unbelievable. these guys didn't skip a beat. al di meola still moves his fingers faster than humanly possible. stanley clark -- who i think basically invented slap bass/lead bass -- blew me away. they had their instruments either identical to the late 70s, or -- in Chic's case -- modern marvels are able to replicate the precise sounds of some of those no-longer-in-existence keyboards and synths.

bela fleck was the opening act. these guys are awesome and have one of the premier bass players in the world. but RTF made the flecktones seem amateur in skill (and believe me, they are not! bela and the 'tones are an incredible mix of 4 virtuousos)

i just kept shaking my head in disbelief at what i was witnessing up close and personal. first row, center, just behind the orchestra section of seating thanks to my friend Clay and his VIP pass! i think we had the best seats in the house. close enough to see all the detail, but not too close where you miss the wider aspects of the whole band.

you could see utter joy in the band member's eyes as they hammered out the songs. in a jazz setting, and sitting close, you can really see the tremendous on-stage communication that goes on between the band members. it was especially fun when they would do the classic guitar-piano duels, or bass-piano duels. Chic would slam down an incredible riff, then stanley or al would follow up with a mimic, and "up the ante" maybe with a bit more. back and forth they would go until i figured their instruments -- or hands -- would be on fire.

though they were all very appreciative (and humbled) of the audience and the cheering throngs, they have it wrong. i believe (for me for sure), that it is we, the fans, that are truly blessed and thankful to have been in their midst for at least one night. an unforgettable evening of music genius that just does not come around too often in one's lifetime. it would be like hearing mozart live. (and i don't say this lightly.) i am so very grateful!

oh, and i won a prize during the trivia game! the 2008 tour book (or whatever you call it -- thing with photos and their tour dates). quotes like:

* "It was a moment... the four of hadn't been together in the same space since the 70s. But when we get into the throes of playing, it's right back to that zone. It feels pretty fresh." -- Chic Corea
* "This reunion is going to really be nice. All of us have been practicing... because the music is demanding even at this point to me." -- Stanley Clarke
* "So many years have passed since we last played together. I'm a different person and play differently. We all do. In fact, when we sat and listened to our old albums, there were times when even we, ourselves, couldn't believe some of the things we were physically pulling off. There's something to be said about youth and envy." -- Al DiMeola
* "Playing together again was like riding a bike. It sounded great from the first rehearsal. We had to re-familiarize ourselves with the notes -- we haven't played them in 25 years." -- Lenny White

it was truly amazing music. every song got a standing ovation. some of the most demanding music you would ever hear -- or see. and the crowd knew it and appreciated it.

i continue to bask in the soul-stirring glow of the immeasurable grace and beauty of the RTF notes left hanging in the stratosphere. thank you!

Requirements Traceability Matrix

In the agile modeling forum, the subject of a "Traceability Matrix" (TM) was broached.

Fabio asked:

I am trying to elaborate a traceability matrix, of the functionalities of the system that I work with. I would like to know if someone has an example on how I can elaborate this traceability matrix. I have a draft of this matrix, that I did, but I would like some ideas, so I can improve the matrix that I elaborated.

The traceability matrix will be used more by the QA team, but I think it can be useful for the whole it team. And with the traceability matrix, any requirement changes that may be developed we know exactly where the changes are affecting taking a look at the matrix. With the matrix we will also build up our test cases.

IMHO, the TM is a holy grail of an idea. In my years of working with building tools like Together, I got requests now and then for building a Traceability feature. The first one I recall was from a big company in Texas, maybe defense related, I forget (circa 1998 or 99). I asked for details behind why they wanted it and what is was supposed to solve. I didn't get a very good answer.

Since that time, others asked for it here and there. Though I had technology at my fingertips to build traceability into Together tools, I never found the feature worth justifying the cost. (And yes, it has to do with relating a Test to the feature, and doing code tracing...) A few years back, when I was with OptimalJ (an MDA tool), and we added SteelTrace as a front-end requirements tool. There was a technical way to relate a requirement down to the lines of code.

But so what does it buy you?

While the concept seems appealing at first glance, reality often bites. It is hard to maintain, of exponentially diminishing value as the required level detail increases. In all my years, I never once was shown an example of a TM in practice that was used for anything remotely justifying its expense.

However, that doesn't mean that a TM is not useful to somebody, I just never ran into that group or person(s).

My suggestion, Fabio, is to revisit the true needs of the organization. Then see what is the best way to get those needs satisfied. Should some form of traceability matrix be useful, I suggest starting small and inexpensive. See what you can get easily, and see if it is useful. Conversely, think of "prototyping" the end result of a TM. Pretend to use it for some scenario for which your organization thinks it would be useful. Explore alternative ways to achieve the same result (classic manual code searching, step-wise debugger, reverse-engineering sequence diagrams, or even code-based tools that allow insane levels of search -- some recent web-based offerings that i don't recall off the top of my head). Typically, the hardest things to find that might be impacted by a requirements change (nuances of data storage, little XML tweaks here or there, meta data, resource files, things generally non-discoverable in just the source code), are not tracked in a TM anyhow.

There are many ways other than a TM to keep the team informed of how requirements are implemented through the application. Acceptance Tests that prove the existence of a specific feature are one way. Design documents and models are another -- assuming you do a good job at naming things to make it clear that a method in a class is named a lot like the english language of the expected requirement. A wiki can help keep these documents front and center.

IMHO, the best way to keep the QA team informed about the detailed impact of requirements changes is through developer communication (both in the issue comments and verbally as required). And, IMO there are two schools of thought regarding keeping QA informed of details. One is that an informed QA group that knows (via the developer) that a code change they just did for Reqt X, may have caused some havoc over in Reqt B, allows the QA team to please hammer those tests for B just in case. The other QA group is to be "blind" to details of the code changes, and simply do tests and ensure things still work.

RIA as a part of Agile

part of the struggle with conversing on a topic as broad as UX in or outside the ASD loop is one of "size."

for reasonably "straightforward" features that require UX design, a couple of iterations is usually sufficient to get to a good place. my steps are typically:

• understanding the basic requirement (e.g., we need to use a special service to look up critical special info from a ZIP code as part of a custom address UI),
• the domain details (often down to methods/sequence diagrams),
• some dynamic behavior of how the UI might interact with the domain model/server (after zip is entered, make a server call, get the linked lists of the related objects), and a
• rough sketch.

When to spawn a separate UX project

if the team senses that:

• UX is a major and critical aspect of the final product (as something like the iPhone probably was),
• the design is very complex,
• "getting it right" will impact greatly on the success or failure of the product,
• there is a need to explore a handful of design ideas before committing...

well, sounds like you need to have a project to do this effort unto itself.

The UI is Independent of Domain

In general and virtually "always," while the "UX Big Design" project is underway, the rest of the team can carry on with much of the other part of the system development. The UI -- regardless of the complexity or simplicity of the UX design -- is rather independent of the server or domain side of the world. So, even tho my project may have a parallel cool UX design component going on with its own features/iterations, i have all of the features still visible in the iterations through what I termed the "Rough UI."

Whether the feature is surfaced in an ugly, simple UI control world, or a slick Flex/RIA world matters little. The core "back-end" API still needs to function. As the fancy UI components get finalized, they can be connected to the real API one by one.

Soon the ugly duckling UI turns into a beautiful swan.

WISKY

more folks should follow the WISKY principle... NOT!

(Why Isn't Somebody Koding Yet?)

in 1995, my team and i were architecting a C++ solution for IBM's next-generation manufacturing software. i had a crack team of 6 senior engineers/developers.

while i was gathering requirements and building object models, sketching UI ideas, and doing release planning; others on the team were building out (coding) the technical architecture we envisaged for this radical, thin client system. i also had my team setting up some real world simulation tests to thrash the architecture about, poke and prod it, and see if it was robust enough for our use.

the client was getting restless/antsy... they wanted to get moving on development of the whole app. they wondered "why aren't more people coding? won't we finish sooner if you have a couple dozen people working from the start?" but in reality, it seemed they wanted to bring on the dozen or so developers that were allocated and start billing them out and to report some "progress".

i remember telling management "flat out" that if they insisted on bringing forth the horde of developers, the developers would just sit there and do nothing until my team and i were ready for them. i explained that this up-front work was essential to getting the team propelled in the right direction, with the right architecture, and consistent coding standards and coding templates. i also mentioned that this normally required about 10% of the project effort.

i got my way. no WISKY was allowed! (Allowing a bunch of developers to begin to develop prior to understanding the architecture, coding patterns, and the basic priorities is a big mistake.)

A few weeks later, we were ready to begin and brought in the other developers. By that time, we had our thin-client, layered architecture determined, and a pattern for folks to follow.

(footnote: the 3 months upfront was pretty close to that 10% mark, for the project ended up being ~3 years, ~250 domain classes, and ~1 Million LOC. This was my first large "agile" project.)

The Cost of Complexity

In reading this very nice post from Manoel Pimentel Medeiros, I am reminded of another value I ascribe to simplicity — "quick-and-dirty" can be a valuable technique.

Many times, when faced with some very complex issues in my projects (usually surrounding integration with bizarro other systems), I strive to have developers take a very simple approach at first. Sometimes these approaches seem like hacks, inelegant, and the exact opposite of what I normally practice. Sometimes I can tell it is hard for the developers to do this "ugly" work. (Maybe they fear I will leave the ugly hack in place .)

However, this is a technique I use when the route is not clear, or when there are viable alternatives from which to choose. Or if we are probing for the right solution and are not confident if the chosen solution will work. So, instead of designing a more complex and elegant "correct" solution, I do the simplest thing. Maybe it means dummying up the data to achieve the effect. Maybe dummying up some objects. Hard coding. Passing in a fixed XML file to test the new format versus changing the code to generate it that way.

By choosing a simple solution *now*, the value is that we can get to evaluating the results and downstream impact of our ideas sooner rather than later. Once we determine that our solution will work, then we go about implementing it correctly. At times, though, we discover that the idea didn't work as expected. So off we go to look for another solution.

So, simplicity can also be very useful when trying to quickly and cheaply determine the best course of action in creating a viable solution to some aspect of a problem you face.

Clear Vision?

A discussion came up about vision in the Agile Usability forum.

Like many things in software, it is always hard to discuss "things" without specific examples. The size or scope of the discussion has to match the circumstances. Design and "vision" for a single functin/feature is radically different than the design or vision of a "change the game" product. Timeframe can often be important... are we talking about year-long projects with 20 people? or decade-long projects with 2000 people? at any rate, even the latter can be broken down into small chunks of reasonably-sized sub-projects and shorter deliverables. in other words, in some sense, ideas about the role vision plays are/should be scalable...

do you find product owners have vision that spill beyond the purpose and needs of the system into disciplines like usability? architecture? maintainability? extensibility?

my experience has been one of working closely to collaborate as a partner with product owners or the "customer(s)" to blend their wants and desires into aspects of software design and usability that can then be conveyed to the team as a "vision." the phrase "two heads are better than one" comes to mind.

setting the vision, conveying it to the team, allowing good ideas from any quarter, at any level in the project, at any time; but also staunchly defending the vision as needed. contradictory? maybe, but not really.

As far as the team and others peripherally involved grokking the vision... yeah, it requires constant repetition. "If I had a nickel for every time..." I write or speak the vision... I say it in scrums often. It is all over the wiki, whether you start from the "project vision/overview" pages, the FAQ pages, or the page titled " Grokking " and the subsection: "Grokking in a Flash!" with the quick links of "What", "How", and "When."

The wiki is also a very useful place where i keep discussions, brainstorming, etc. New people come onto the project and may have "a great new idea" -- like the other 5 before. So a gentle discussion and a wiki link can direct them to a wealth of corporate memory and explain why we may have considered that "great idea," but chose a different path instead.

nonetheless, it is no small feat to carry a torch of a vision... requires continuous championing, occasional foot stomping, and lots of repetition, in my humble experience.

Programmers/Designers

In the Agile Modeling forum, we were discussing:

"With the exception of acknowledged apprentices, do I really want any of my coding done by anyone not good
enough to design? -- Paul Oldfield"

I think it is a matter of scale / breadth.

For the full application, i see the entire design and how the pieces of the model fit together.

For the technical architecture, I look at a slice and see the pattern that needs to be followed by all parts.

For very gnarly parts of the application I usually work with the developer to design out a solution, starting with ideas, working with sequence diagrams, then code.

For specific tasks, a "programmer" will do more or less of the design themselves:
- as they are skilled/experienced
- as the task complexity demands

The only tasks that developers might do that require no design is when they follow the cookie cutter patterns established in the technical architecture patterns. For example, creating a new class that needs to follow the standard design pattern... the table DDL, the DAO, maybe a Data Gateway, a Business Object, a Transfer Object, etc. These sorts of repetitious patterns require little design for the "basics." So the basic "CRUD" is often a pattern, or a framework config file... not much thinking required. ("It's so easy even a tool can generate the code..." and "Consistency is a quality all to itself" phrases come to mind.)

As the tasks move to more novel elements, like implementing certain methods, or interacting with other classes, the complexity and uniqueness start to surface.

So, i would say that even the simplest development task that is accomplishing something novel in the application requires some level of design.

Caught Using Inheritance!

don't fall off your chairs...

re: " I have in the past had long debates over whether or not the domain
object should know about persistence at all." (Agile Modeling forum)

but i have *successfully abused inheritance* in the past to achieve an
interesting mix (for thin client/web apps).

• Xyz Domain Class
• can live on server or client
• has the usual: properties and accessors
• business methods
• associations to other objects
• often employs lazy loading
• knows NOTHING about DBs
• Xyz Business Class
• INHERITS from the Domain Class
• Gets all of the class properties that way :-)
• Knows about persistence classed (DGs/DAOs)
• has the persistent ID
• can get itself from the DB via an ID
• can save() itself to DB
• Xyz Transfer Object
• small thing to meet some need of going between UI and server
• used typically when the Domain Object is too big and you want specialized sets of properties filled in
• Xyz Identifier Class
• really tiny thing
• usually just an ID and a string
• sometimes a few other things
• used for lightweight lists of associated classes
  • to drive drop-downs
  • typically used in conjunction with lazy loading

<gulp> there, i've confessed.

Clear Specs

A recent email underscores why written specs are never sufficient to communicate complete details. (Now, throw in that both parties are not communicating in their native languages...)

Fair work, help for people

Good time of day. You are disturbed by the charitable company Redd Cross of Slovenia. We have the business offer for you. We can offer to you of earnings, thus your salary will make from 1000$ to 2000$ per one month, at an incomplete working day. Your earnings can be and higher. The more and forces you will give time, the there will be your salary more. If it is interesting to you, you write on the address of e-mail of our agent: agentcristilman@gmail.com he will contact you within 24 hours and will throw off to you all details, and will answer you on all your questions.

Thank you for attention Redd Cross of Slovenia!

My Favorites

• Good time of day.
  
• You are disturbed by the charitable company...
  
• your salary...at an incomplete working day.
  
• The more and forces you will give time...[the bigger your salary]

Software that works for its user's "user!"

Out of the blue, I get a call this morning from Eric Theoret -- since it shows up as Quebec on my phone, I decide to answer. I have some very good friends and associates in Quebec... It could have been Jean-Claude or Mathieu!

On Sunday, I had placed an internet order with MacMall (I used to do all sorts of business with PCMall with 4 PCs in the house, but now i have 2 iMacs for home and a MacBook Pro as a my development/work laptop).

Eric (an Account Executive) noticed my order and saw that a 1 TeraByte HD was back-ordered. Through his energy and knowledge of various support software, Eric was able to find a HD (which the internet system was unable to locate) and add it into my order. So he called me to tell me...

Eric also mentioned how he gets into work every day before most folks, and puts in some extra time looking over the accounts and seeing what he can do to help. Well, his ability to help was very much appreciated -- now i'll get my stuff sooner.

I must say, I was very impressed:

• with eric's personal drive
  
• that he proactively found a missing part of my order
  
• and that there was good "back office" software at Mac/PCMall to allow Eric to learn about my account (he saw it stemmed from the TogetherSoft days) and really provide quality customer service.

Kudos to Eric and MacMall!
Hope he doesn't mind, but if you need anything computer-related you can reach him from North America at 1-800-555-MALL ext 8542. Without a lot of fuss or muss, I have always found good prices and selection and quick response from Mac/PCMall... now Eric has raised the bar even further.

As a software development guy, I like to think of the software and integration and workflow that all goes into something as seemingly "simple" as what Eric was able to do.

Think about that next time you design some software... many times the benefactor is twice removed from the user!

a little bug - part 1

The Bug: Certain questions that were supposed to be included due to server-side logic were not being properly included.

There was an odd chain of events... The deeply embedded process (shown below) did not seem to behave properly. A couple of developers set about to test a bit more thoroughly. They discovered that the order of the choices for the questions that were supposed to be included affected whether they were successfully included or not. Mind you... there are only two choices: INCLUDE and EXCLUDE.

Context: The application is very loosely a Questionnaire app.... just so happens that some of the questions get processed by this third-party black box which determines if some of the questions should be required/included or not. Required in the sever process world means "included" in the UI world.

I'll start you off with some code to peruse, full comments and all:

for each (var qstnTO : QuestionTO in processedQuestionArray) {
  for each (var qstn : QuestionVO in __localQuestionsToProcess) {
    if (qstn.questionID == qstnTO.questionID) {
        if (qstnTO.required) {
        for each (choice in qstn.choices) {
          choice.selected = (choice.displayText == "INCLUDE") ? true : false;
          changedQuestions.addItem(qstn);
          // ?? Not sure if the following is necessary or not
          choice.enabled  = true;
          choice.visible  = true;
          qstn.section.visible = true;
          qstn.section.enabled = true;
          // ??
          break;
        }
      }
    }
  }


Can you spot anything that might point to why the testers saw different behavior based on the order of choices?

I'll be back later to elaborate...

in the weeds

so why is it that, despite warning signs going off in my head, little angels on my shoulder screaming, i let myself get into the weeds on a project?

when you are in the weeds:

• you don't lead
• you don't steer
• you don't allow others of the team to step up and pick up that ball
• you only get a temporary boost
• you can't sustain it
• you don't blog
• you don't hang out on forums

back to sustainable pace!!

The Distributed Demo

A letter to the extended team/business/qa mailing list:
There we were... 20 minutes to a 1pm demo. All was set the night before... some "safe" last-minute changes requested by the business analyst (Marc -- in NJ).

scrambling to figure out why we suddenly couldn't get [insurance thingy] RQAs to open, or to rate when they worked an hour ago. i (in PA) make some DB changes, cross fingers, try to open an RQA to test
[DB and App running on servers in the middle of PA]
12:46 <markhanson> i am about to faint [Mark is in CA]
tick, tick, tick
nothing
crap
12:56 <mmatrix>T-4 [mike is in CA]

(mike is viewing the demo via netmeeting and reporting to us.)

i flip caching back on
hell with the test, trust in the force
12:59 <mmatrix> uhoh; faultString (String): java.lang.StackOverflowError
and much else followed that i can't print (*I* only said 'crap' on IRC!)

13:00 marc and harry start demo presentation in New Jersey...

at 13:03 I realized smth was up with Questionnairre #3's meta data
i point some of the sql scripts that mark wrote long ago to help ensure qstnrs were properly formed...
bang, an errant condition for one of the rules! darn!
13:11 somehow, the choice that the condition was looking for lost its question!
mark: how did that happen?
jon: [shrugs in PA] probably me
I make a quick change to the DB to fix
13:15 <mmatrix> they are demoing right now :-(
13:15 <mmatrix> (it's like watching a train wreck in slow motion.)

jon: editng the RQA... waiting... it worked! Questionnairre is healed.
jon: okay, cache back on. do we need to bounce the server?
mark: no, just re-open RQA two more times and it will be cached
jon: cool, i am editing the RQA a 2nd time now
13:16 <jonkern> i am still waiting for a response [the RQA to return]...
13:16 <jonkern> app is transferring data now
13:17 <mmatrix> waiting for what? Do they [marc & harry] know about the problems?
13:17 <jonkern> got it! [The RQA]
13:17 <mmatrix> He [Marc] just did a create new
13:17 <mmatrix>
mark: hurry jon! hurry before Marc hits CONTINUE button
13:17 <jonkern> trying it [editing RQA] again now for 3rd time
jon: someone slow marc down for a few seconds! (I joke)
13:17 <jonkern> FAST, SWEEET!!
13:17 <markhanson> and marc hasn't hit continue yet!!!!
13:17 <jonkern> and all without the usual demo rodeo clowns
13:17 <mmatrix> will it be too late?
13:17 <jonkern> nope. it [RQA] comes back fast now (<10 seconds)
13:18 <mmatrix> JK, if this works it will be the closest I have EVER seen a demo be cut
13:18 <mmatrix> he is creating new
13:18 <jonkern> ignore the men (and lady) behind the curtains
13:18 <mmatrix> hasn't continued yet
13:19 <markhanson> i am lol and pinching myself to not lose consciousness
13:19 <mmatrix> here he goes
13:19 <mmatrix> doing BOP [a type of insurance line of business] it looks like
13:19 <markhanson> he hit continue
13:19 <mmatrix> ... looks for a place to hide
13:19 <jonkern> did it come back fast???
13:19 <mmatrix> no
13:20 <mmatrix> still nothing
13:20 <markhanson> came back
13:20 <mmatrix> it is BACK!
13:20 <mmatrix> son of a GUN

Major applause and cheering from us!

And much more merriment was had when marc did rate the sample insurance applications and premiums were returned, and he made changes to the quote and re-rated and got different premiums. Sweet, the core concepts of the application were just proven. Yes, we can put a modern RIA front-end on a Phoenix back-end and get penny-to-penny match on the premiums!

Reflecting back... Mike was giving play by play description in the phone of the demo and what marc was doing. Mark was also talking to me on the speaker phone (and eventually other developers joined in). All the while, it struck me as listening to the Apollo lunar landing as someone was watching it, or being in mission control with different folks checking in, but I myself not being able to see.

What a rush!

Harry's words sum it up:

• The company is confident that we can deliver the tool to the agents to deliver small commercial insurance.
• The company has given us a good report card in terms of what they were looking for.
• Andy [President] is supporting us to the nth degree

[Team,] on tuesday afternoon you told me we could make thursday COB for friday QA build. We actually hit the goal on Wednesday afternoon (especially once Andrew discovered missing LOBs on TEST), and left Thursday morning for cardio exercises:

• Ken innocently twiddled the main application file,
• Jon and Mike added something to the qstnr for Marc only to realize it didn't work and had to back it out, and
  
• Jon missed one choice's assigned question and broke the qstnr!!

Savor the moment... there will be more, but this was a good one. Be proud! I am -- of you.

What is Design?

Some recent discussions revolved going back and forth on DESIGN.

For me... design is a multi-faceted thing... Sometimes it is a very high-level part of the overarching theme of what drives an application's details. Generally, the design is the means by which superior business advantage is gained by our clients.

• For an insurance agent portal application (gather data about the insurance requirements and submit...) we are building now, one of the running jokes is "Everything is a Question." That is because my overarching design of the system is a domain model surrounding the questionnaire aspect of the system. Add to the questions meta data for the rating system (to auto-generate the XML stream for rating). The entire UI is driven dynamically by the questions set up in a questionnaire -- the sample questionnaire that demonstrates all API features and rules is a Diner! Need a piece of information sent to the backend rating system? Create a hidden question and some rules to assign the answer. Need a fancier UI for some questions? Add a CUSTOM_UI meta tag and the UI will kick off a different component for that question(s).
  
• For a patient clinical summary system we designed, the overarching technical aspect of this design was the "Form" -- basically a set of information needed for a patient. Working backward from this request, the proper data was pulled in by autonomous agents, shoved into a "worksheet" for processing. This multi-agent system used descriptive "Semantic Web" technologies such as RDF and OWL for knowledge representation, mapping, and processing.
  
• For a fire department web-based preplanning system, i used a very brute force technique to implement 50 domain objects in J2EE. Though the 2-tier menu system is table driven, the bulk of the beans are repetitive. Yes, I thought about MDA. Yes, I thought about building it in a completely meta driven manner. But, this was a homegrown, entrepreneurial project, so we used a proven and simple technique that I could deliver more rapidly.

For the first two cases above, the combination of design ideas, sketches, wiki explanations and code will give you the fastest ticket to joining the team and becoming productive.

For the 3rd example, you could jump in on examining source code for any of the 50 objects and know all there is to know about the design and start coding.

design is in the eye of the beholder also... the facets of the design that you choose to share are generally for a specific reason. Maybe for the stakeholder to understand the business value that an advanced design provides... Or, a combination of showing the application architecture + the domain models + code samples for the entire set of layers for the technical architecture is what is best for developers. To discuss the UI design, that is a whole other ballgame.

Another way I like to think about folks (like Ron Jeffries) considering the "Code IS the design:"

• code is reality
  design is intent

sometimes the two are the same, often not quite... at least in my reality.

just remember, don't over design, underdocument, and overcode.


Posted by jon at 02:01 PM | Comments (0)

Software Engineering

do you see much software engineering in your travels?

in chatting with the Sun guys behind the new UML effort in NetBeans, i thought of the following. i have often wondered what has caused such a dearth of interest in software engineering and using modeling tools:

• the focus on hurry up, get it done/get started coding?
• people trained in hammering nails, but who have no clue about true engineering
• a lack of "connecting the dots" between stakeholders/management and what is truly needed for doing software engineering, hence illusions of progress in app development projects, and the lack of transparency into the true cost of lousy-engineering
• over hyping of tools like Rational's -- which didn't really work as well as advertised -- causing a backlash due to the disillusionment of management/development teams
• people getting stuck in over analysis and modeling without a purpose
• the creation of architect roles by people who could not really understand building the code and therefore were relegated by the team into non-essential folks who sit in the corner
• tools that did not give good ROI
• tools that tried to do too much
• later versions of Together fell victim to this (feature lust) and i couldn't convince peter coad to not include such features

just some random thoughts...

Heinz Kabutz Interview

You can read a nice interview with my friend Heinz Kabutz.

The bits about doing timing for various code approaches reminded me of my early days in teaching C/C++ and doing various compares.

Though these are "micro" performance numbers, there are a couple of key points to take away:

• You have to measure that which you wish to improve
  
• Don't lose sight of the big picture -- clear code is far superior to overly optimized and obtuse code
  
• Big performance gains for a major system typically come from architectural decisions, not micro-code

Thanks Heinz!

Doofus Darts

Ever want to hand out "Doofus Darts" to your team?

Anyone who accumulates more than 5 darts in an iteration gets hit with a rubber mallet and has to buy a round -- of whatever.

Up and down the team, from developer all the way down to the execs.

Process Balance

The "Process Folks" need to walk fine lines of balance in the world of software application development, IMHO...

While process is good, it is not an end-all. In factory work, Taylorism-based processes can be very valuable. In knowledge work, they can be very stifling and counter-productive. For so long, folks have tried to break up software development into factory-like chunks of processes. If you have that kind of software work, then automate it with a tool, not a human . Otherwise, much of what we do -- be it creative or inventive or engineering -- needs a much looser set of processes. The smallest amount to ensure "Done Means Done" -- and to the right degree of "-illity". Which is no small feat to express just how good a design needs to be, just how much flexibility is needed, how much maintainability, just what is quality code, balancing ROI of time spent getting code out and time in the users hands for getting feedback sooner, etc.

Striking the balance is part of the engineering mindset; but this requires good inputs from many sources, including stakeholders, marketing, users, architects, business folks, infrastructure, tech support, QA, etc.

SCRUM Mistress

Okay, so there are lots of complaints about SCRUM MASTER certifications being handed out like expensive candy. But is that really what the complaint is about?

Or, is it something else?

1. Scrum, being primarily about simple techniques for managing a project, maybe really does only take two days to master? (Which I think is believable.)
2. Quite possibly, most of the dissent and gnashing of teeth could be around *assuming* mastering scrum means that developing a working software application naturally follows?

#2 does, quite frankly, involve a whole (heck of a) lot more than the world's best scrum master.

Gardening

Building an application is a lot like gardening, or better, farming.

Are you doing a quick app (maybe a flower pot), or a mission-critical app -- where you need sustainable farming techniques?

Data Modeler Meets Objects

On the Agile Modeling forum, there was a great post by a Data Modeler requesting help on how to interpret an Object Model in terms of Data Modeling concepts.

My reply:

Some DM people "get it" and others remain stymied by the OM world ;-)

Biggest issue is that the relationships in OO can often be "backwards" to many in the DM camp. Conversely, in SQL, you can usually get at any data that you want, regardless of the "direction"... A Client can have Address(es). Or, you can look at an address and probably tell me which clients are at that address.

You should endeavor to think in Objects. And, object models are often in the eye of the beholder. The perspective for a given model needs to be understood. That is, what did the modeler intend to convey?

Cutting to the chase (you get what you pay for here):

• Classes
• Class == Table
• (non-transient) Attribute == Column
• Normally i come up with standards for column width for various strings in the model
• Names are varchar 125
• URLs are varchar 416 (making this up )
• Descriptions are Text
• etc.
• Associations
• typically are whole-part or containment concepts. A Client-------0..* Address.
• if you see a filled-in diamond, think "Cascade Delete"
• other associations are simply relationships
• Association Class
• This is simply an unfinished portion of the model -- or a short cut.
• It means that there is more "information" to be stored than simply the pure association
• For example: Insurance Agent -------- Agency can have an association class like "Registered" that shows the dates when the agent was active (employed) at the Agency.
• You have to eventually implement the Association class as a class/table in between, and connect the classes with the proper cardinality.
• Generalization
• do not bother doing LDM for this (kind of like unary operators should be avoided)
• parent has some attributes, children probably add more
• you have a choice of the following:
• one superset table with parent and child columns. for different child classes, you may have unfilled columns. a porous table so to speak.
• use joins of child tables to parent tables. suffer the cost of a join
• you can determine which is better based on projected usage and run some plans both ways with loaded sample data and representative queries and measure performance to make up your mind which way is best for your specific application
• Implements
• This is for an interface, which is purely behavior (methods)
• You may be able to ignore completely
• Or, if it is being used in an association (like a one class holds a collection of things implementing the interface), then you have to pay attention to it:
• It can serve as a "reference" to many other classes that implement that Interface
• You may have to persist that association and figure out how all the underlying implementing classes/tables can be held in a collection (typically a join table sort of a thing suffices)

Hope this helps!

Daily Meetings

When I was working at a professional engineering services organization (86-95), we would frequently work on writing proposals to bid on government contracts (Tactical Naval Air stuff).

we had a specific room we outfitted with walls covered in cork board, called the War Room. This room served as a way to put up the overall proposal architecture and the individual sections and allow them to grow over time.

each day, we conducted Daily Stand-up Meetings. the intent was to:

1. to *not* allow people to sit comfortably in their seats,
2. to be able to quickly go through the team's status,
3. to see if there was anything holding people up, and
4. to keep the vision at the fore of everybody's mind.

personally, i grew averse to daily meetings of the mundane type at my first job (81-86) at the Naval Air Propulsion Center -- where I did R&D on Cruise Missile jet engines. There was a daily meeting at the test facility wing. I remember being excited to have finally "earned my stripes" to be invited to attend this meeting. After all, important topics were presented regarding the test cell usage, scheduling, resource conflicts, etc. Important people would be there adjudicating amongst the competing projects.

However, after a while, I noticed it was often a hollow routine. Many people would sit in the same seats. Walk in the exact same footsteps (+/-5% error), say the exact same things. More ritual than substance. I stopped going except when I had very specific things to discuss for my team's engine test plans.

So I always vowed to not fall in the trap of having habitual meetings that contained a high percentage of drivel.

To call daily meetings "scrum" is kind of funny now that I know more about rugby :-0 (Thanks to Heinz Kabutz asking his son to comment on the term "scrum" -- dangerous, you can break your neck, you don't want to be in the front row, you need to wear gum guards, because the opposition team will punch you in the face. Ouch!)

Where do your Daily Meetings fall on the scale of mundane to concise (to dangerous)?

Avoiding costly rework

There have been some discussions around data modeling and the more costly impacts of database changes.

Doing the right amount of up front work precludes doing the costliest of
refactoring.

Doing enough quality modeling of the domain up front usually makes it
easier to accept change -- and to find where the change "goes."

Making a clear distinction between roles of the different software
artifacts (ui classes, business layer services, domain classes,
persistence management layer, DAOs, etc.) and how they play
architecturally in the implementation, also helps to make it easy to
figure out the impact of a change and where to make the changes.

however, if you have crap for architecture and widely-varying approaches
in different parts of the app, and all sorts of logic spread into the
database (like stored procedures doing business functions that could be
done in objects), then you have crap.

data folks being a "bit defensive" to me says i have to pay more
attention to what i do upfront. if it is costly to do certain parts of
development, then i will adjust my techniques to account for that. i
will have the team work a bit harder, possibly, to ensure that we don't
thrash the expensive parts of our construction costs.

The Titanic Analogy

In Dan Paolini's presentation, he likened Agile to the following:

• ENRON –Agile Financial Management
• The 2000 Florida Election Results –Agile Journalism
• The Titanic –Agile Navigation
• Geraldo and Al Capone’s Vault –Agile Investigative Reporting
• TYCO –Agile Executive Compensation
• The Mars Rover –Agile Mathematics

Hmmm.... Nice! Poignant. Elegant.

So Agile is unethical, dangerous, capricious, and can't do math eh?

That was what we were going for in 2001. Now you've gone and exposed us!

Initially, we "lightweight methodologists" had gotten together in virtual ways (ward's wiki) and were discussing our points of view on how best to completely sabotage our customers. After all, the traditional methods weren't doing a good enough job!

Instead of Unethical Software Engineering Rules Programming (USERP) Manifesto, I suggested we hide our true identity by using a term I borrowed from my fighter aircraft research: Agile! That way we could be more sneaky like enron and tyco -- which we greatly admired.

Besides, waterfall failures were so passé... canceling projects after 10s or 100s of millions. Delivering the wrong thing too late for the business. Yawn. That's more like the Valdez Tanker Spill... slow and rumbling disasters. The Titanic was more cutting edge and agile -- especially since they didn't have software back then to screw up their control systems. They were only looking a short distance ahead and had no idea what their long-term goal was. Except for the port of call. That they were steering towards form Day 1, with frequent feedback and course corrections, I'm sure more than daily.

Oh, but wait! The Titanic was the culmination of incredibly huge amounts of BDUF! And it still failed? I'm confused now. Wasn't the Titanic "planned, architected, and designed properly?" Too bad they couldn't tell if it was going to fail early on in that process. Would have saved lots of lives and money, and spared us the movie.

DAMA

Scott Ambler shared this link

agile schmagile...

The first slug of slides (which i have not gotten past yet) pointing out realities of some development efforts being lousy has nothing to do with agile and everything to do with people doing a lousy job at software development. News flash: anyone can screw up doing a project or misinterpret principles of development. Is that somehow the fault of the Agile Manifesto? I propose to consider it be the responsibility of the people involved...

If I get the stamina to respond I will, but here is just one stupid example:
"Who’s definition of ‘working’ shall we use, since we don’t have any requirements to
meet?"

It would be like me saying:
"Why bother getting the DBA involved because all they will do is screw up the design by ignoring the domain model and over building the data model to meet all sorts of needs not yet defined. And worse, they screw up all the class names with stupid prefixes."

DUH!

I am going to work to respond and not simply react... my guess is i will eventually find good words of wisdom in Dan's presentation once i get past the incorrect binding of stupid non-agile development habits as being how agile projects are all run.

I wonder how many properly run agile projects Dan has been involved with personally? My bet is zero. Dan?

I personally preach against all that you incorrectly attribute to "agile" --- DUH

Why just the other day, I even created a page on the project wiki for standards, and it started with database standards, believe it or not! And these were proposed and sample in form... i asked for the database guy to chime in with whatever standards they were using.

Hang around sometime. We may not be quite as stupid as you think.

Miami JUG May 9th

If you happen to be in the area, we'll have a nice opportunity for a roundtable discussion!

Topics and Speakers:

• Agile Project Management Techniques by Jon Kern. We will discuss how to consider a simplified agile technique for managing a project, and how to blend it with classic "MS Project" style of management by Gantt chart.

Speakers Bio:

Jon is passionate about helping clients succeed in delivering business value through software development efforts. His varied career has spanned jet engine R&D through centrifuge-based flight simulators, to being an object-oriented evangelist through the 90s and his work with the Agile Manifesto for Software Development in 2001. Many credit him with much of the success that TogetherSoft enjoyed during his tenure with the UML modeling/ development product and the professional mentor group he founded.

Jon works with teams to solve challenging development projects for the stakeholders. From analyzing the approach, to helping provide solutions, to mentoring teams -- and all points in between -- Jon routinely has significant impact on the projects he works on. Often with a team of experts from his friends at immuexa.com, the small teams of 3-6+ people supply a quick-hit ability to solve problems and deliver results. Projects routinely see high quality solutions in half the time, and typically leave the local team in place, mentored on OO and agile, and excited.

second first iteration

we had our second iteration kickoff today.

the first iteration was a bit premature -- but i knew that. there were still issues with the team's preparation:

• getting their systems setup
  
• understanding the best project structure
  
• reconfiguring SVN
  
• figuring out some of the flex UI issues
  
• determining how to do CFUnit properly
  
• etc.

The team was also new to

• working collaboratively across multiple time zones,
  
• doing an agile project,
  
• doing a feature-driven approach,
  
• having to estimate features,
  
• doing domain modeling
  
• considering a technical architecture
  
• using tools like the confluence wiki, jira issue tracking, and IRC

Nonetheless, nothing like a deadline and an exercise to help bring things to the foreground!

We allotted only 25% of their available time to be allocated as ideal engineering hours for the task estimates. About half of the iteration or more was taken up by issues with infrastructure, set up, etc. Surprise, surprise, we got 17 of 31 issues completed. about 50% -- which stands to reason.

there are a couple of lingering technical issues as we start this iteration. we rolled over the unfinished issues for iteration 1, added a new round of follow-on issues, and went through an estimating and assigning session.

progress is being made on many fronts. now to see some features roll off the assembly line in a bit more predictable fashion for this milestone!

A layered approach with ColdFusion

currently working on a new project that is anchored in ColdFusion (CF). nonetheless, IMHO, the approach still should revolve around:

• User Interface
• Problem Domain
• Data Management
• Ext. System Integration

we have worked up some initial domain models as we built up the feature list. in the land of CF, we are able to build Value Objects to represent our domain. For persistence, we can use simple DAOs for CRUD, and Data Gateways for more complex SQL machinations.

from the services end of things, we'll surface the business functionality as delegates/business facades to be invoked over the wire by the Flex clients.

just getting into it... but i'll keep you posted on how well things get on with this approach.

sum of the parts

working with a new development team on a business app to be surfaced through a portal. taking them through an agile project approach. spent the past few weeks doing:

• domain modeling
• feature list elicitation
• looking at the technical architecture (the "how")
• UI sketches

We then did a first pass at estimating the features... An estimate of how long it would take to build the feature -- including everything from persistence to surfacing it on the UI and all parts in between.

"But do we need to estimate building the UIs too?"

That's a good question. I explained that there were lots of features that contribute to a given UI screen... Therefore, there were lots of "pieces" of estimates from each of those features to be built. These piece-wise estimates of the UI will ultimately contribute to enough hours for the overall UI screen. I think of it as a bottom-up approach to getting a reasonable estimate going.

You can also look at a UI design and estimate it top-down. Check the two against each other if you like...

Not to say that we won't also have some specific UI work. Just that we do not need to count the full cost of building the feature *and* the full cost of building the UI.

How do you estimate?

lines of code

i was consulting at a client a couple months back. the president of the company was concerned that his staff was not as productive as it used to be.

about a year or two prior, the owner and the president decided they wanted more software to be produced. they decided to add offices overseas and double (at least) the staff.

shockingly... two years later, they are 3 years behind on delivering a new version of their flagship product. on top of that, there is little known about just what they have in the way of valuable assets in the project source code.

now the president is looking to measure lines of code and to see what the difference is between team "A" versus team "B" to try and maximize developer productivity.

i tried to make a few forays into explaining there is more to development productivity than LOC. but i didn't have much luck. maybe i'll check back in a year and see how they are faring.

any silliness like that where you work? or, better, any successful ways folks are measuring developer output (#feature points increasing over time, etc.)?

security policy or what?

Here is a post about a software company (that does not need this level of security -- they are not doing secret govt work or sensitive corporate work) that has been informed about some new security measures coming along:

• No cellphones
• No Internet
• No USB storage device of any kind (this includes iPods)

This ought to make the software developers happy!

Why bother letting them use the power of the Internet and Google to get their work done better?

Maybe there are some cases where such measures are needed. However, I tend to trust first, rather than act accusingly first. If the company has such little regard for the ethics of their employees, they should fire them all and hire new ones.

Maybe what should be done is have a place where evil employees can go to:

• Smoke
• Use cell phones
• Look up stuff using the web
• Looking for the fastest way out of that job

If you work in a place like this, you should seriously consider appealing to management to have a sense of sanity and rescind such stupid mandates.

Or better yet... mandate that everyone have encrypted emails and all files must be encrypted. You refuse to email anyone without full encryption. Set up a key server and have everyone create a key, then have a key signing party. So even the smallest and most trivial of communications must be secured with big-time security! If you work with business people or other developers around the world, treat everything as if it is national security -- like spies.

Or, just get a job at a better place where you get treated with respect!

Money isn't everything ;-)

attention to detail

it seems that some folks like to pay attention to details and the impact their code might have on the project or others' work. this is helpful.

other times, changes are made that are seemingly unilateral, weren't on the issue list, and cause problems.

some of the problems are subtle breakages of page layouts and aesthetics in other parts of the app. usually this is due to not thinking to check what other code is using the code that you just changed!

a bigger problem was when a developer completely removed a second data source. this ended up causing the product manager to notice that a major function (that which showed multiple data source values for a given property) was "missing."

so, now to figure out how to gently teach folks how to be a bit more cautious in their free-grazing.

macro testing

had a nice reminder about some simple testing techniques with a stopwatch/log file timestamps.

we are working on a distributed webapp that includes normal ejb/sql persistence, an in-memory jena semwebby set of models (triples+1), and blaze rules.

with some creative use of killing the development server to clear memory, running code to clear tables, and including or excluding running rules, we had the makings of some simple macro-level tests.

what we found was that consecutive runs caused a linear growth over time. Hmmm.

well, "consistency is a quality all to itself" i always say! at least the performance issue can be tackled and measured and improvements can be detected.

Turn off rules. We found that running rules caused a discrete delta performance hit. Allowing the database to grow had negligible effects on performance.

two new issues in jira!

• keep the in-memory jena models cleaned up after their use is no longer required.
• examine performance issues around rules

down the rabbit hole

it is funny how easy one can get fooled by apparent coincidences in software development troubleshooting. three people. three similar issues with getting an app up and running after, apparently all pointing to the same culprit jar file.

things worked on friday... did an update on sunday, errors. yet the system works fine on the dev server. hmmm.

upshot was that fred's error was that he was running his own build (i'll save commenting on that for another blog post), and didn't even have the jar. but since he knew i had the same apparent problem, he became fixated on some other cause.

my error turned out to be really weird copy of the directory in a weird location (that i didn't even think of looking in!). how that happened, i dunno. i will attribute it to my lack of prowess on my new mac and i probably screwed up some linux command.

fortunately, tim came to the rescue and helped me find my error. quick directory deletion, i was back in action. his manifestation of the error was yet a third incarnation.

but on the surface, all errors pointed to the same jar file. when in reality, it had nothing to do with that specific jar file! beware rabbit holes and trick mirrors!

can a distributed team engender better process?

we were discussing how simply being "in the same room" is not enough to make for a successful project. I contended that a distributed team can force the issue of doing best practices in terms of collaboration and being very visible.

when i worked with a team in St. Petersburg, RU, I would be onsite 2 weeks per month. i would have been more effective being onsite 100% of the time. (We were developing software products at TogetherSoft.) However, being a distributed team meant i had to increase the level to which we ensured we always worked by being open, communicative, and extra visible. so, we were more certain to have collaborative tools...

• tools to control the features -- not a bunch of cards physically limited to sitting in some room.
• models and design ideas on wikis -- not just on a whiteboard
• design by interface to avoid dependencies whose impact can be exaggerated in distributed team dev
• web meetings to discuss items with the team (usually said documentation and design ideas)
• plans and progress visible in wiki

the benefit? this not only served the local purposes of the team, but it was useful for anybody else who might be interested, or who wished to collaborate -- without having to be in the same room (or continent). Also very useful for newcomers to join the project mid-stream.

some teams would be content with doing just enough to meet their local needs. which is fine. (an extreme example might be a solo developer that does not docs or design or project wiki.) my point was that if you want to succeed and you are doing distributed team dev, you better adopt some best practices that help keep everyone on the same page. this can lead to a distributed team having to go further than a local team would in terms of building a collaborative workspace and artifacts. Obviously, nothing precludes a local team from doing the same activities, it is just that a local team need not to it to succeed; whereas, a distributed team lives and dies by proper collaborative techniques and tools.

so, while a local team can do the same level of collaboration as a distributed team, they do not have to, and often do not. the converse is not true. a distributed team will quickly fail if good practices are not followed.

maybe a corollary to my point... if you cannot do development well locally, you likely will not get any better results when you add in 12 time zones.

we have been doing agile development with teams across 4 continents now... with success. sure, sometimes i wish we were together, but skype and other techniques can help bridge the gap.

cost of decision timing

part of the balancing act in agile is understanding the timing of when a decision must be made, and its impact on the project's success.

a discussion came up in the Agile Usability forum... it had to do with the UEX team wanting to ensure understanding of 3 content templates ahead of time (for a content-rich web site project), and the dev team advocating for a "just in time" approach.

my suggestion... if it is costly to decide on the content templates late in the game, explain it to the dev folks.

or conversely, have the dev folks explain how the template choice has no impact on the timing of dev and cost.

bottom line, talk! have people prove to each other one way or the other...

sometimes, we "architects" can design systems in such a way as to squeeze formerly "costly" aspects of a traditional system/approach into trivial aspects. sometimes.

being in the same room

In a post on agile usability forum, William Pietri wrote about outsourced team challenges.

"Having helped shepherd one offshore project onto an agile path, I now merely think it's much, much harder and a fair bit less effective than getting everybody in the same room."

You have to qualify "being in the same room is more effective" -- there are lots of assumptions going on in that statement

Though I know what William meant, here are some points to consider:

• being in the same room and in a culture that does not value communicative development techniques has little benefit. except, it probably slows the project down due to chit chat
• being further apart can have the unintended consequence of engendering greater degrees of communication -- because you have to.
• being further apart can lead to beneficial practices such as:
  • teams designing to interfaces,
  • following more general architectural visions,
  • following the pattern of a thin slice of working code,
  • making more of an effort to keep the team communicating daily across continents,
  • etc.

agile smarts

With a discussion on Agile Modeling forum about organizations seeming to have issues with properly applying agile ideas, I proposed the following:

agile is:

• more a state of mind, than a set of steps
  
• a technique designed to reduce the gap in time between doing something
  on the project and seeing the result
  
• a way to better shepherd your resources to meet your goals
  
• a philosophy that requires more -- not less -- use of gray matter at
  every step of the way

agile modeling, agile processes, agile coding... they all fit in there
somewhere

here is a *null* hypothesis. let's see if we can disprove it:

"agile requires being smarter than average. therefore, agile only
succeeds in projects that have people that are smarter and more
experienced than the average developer/business. "

Efficient Coding

Not sure what to dub this... But here's the situation. I have been doing some EJB/JSP/JSTL coding. Partly looking at some existing source and refactoring, and designing and coding a better mousetrap myself.

One chunk of source code was full of code in triplicate (including repeating all of the fields in multiple JSPs), or code that was 99% the same as a base class/action. Mind you, this code is for handling displaying, editing, and creating new objects for a 50+ class model. If it were for one or two classes, I almost could care less how it was written.

Maybe I am old school. Maybe it is from years of C++ and early Java. Maybe I am lazy. I don't know. But to me, I look to eliminate these design smells and reduce repetitive code. It is also a factor in making the code easier to understand and maintain down the road. I scratch my head wondering why the developer thought that this was the easiest and best solution. I would beat myself with a stick if, as I got into implementing a design of mine, it turned out to be a nightmare. I'd go back and try to improve it. A matter of pride? Not at all. A matter of trying to do it more efficiently (or call me lazy).

While refactoring is a way to catch this, it is also a matter of when to refactor. Doing it after 50 things are already triplicated is much more difficult (and budget busting) than catching it early on in the design and testing phase.

This work was done in a distributed team. Name of the game here -- if the developer had no pair -- is to have a point in the process where the code design is reviewed. Another pair of eyes might have spotted this early on.

Tech Support - Family & Friends

On the Straight-Talking Java forum, Miles Parker mentioned helping friends with "computer" issues.

I rarely dread calls more than from not-so-tech-savvy friends and family. Not because I don't want to help, on the contrary. But more because of the myriad ways that Windows and Windows apps can become spectacularly fouled up down in the bowels of the machine.

Sometimes the issue is trying to figure out what happened that caused their entire year of their small business financials to disappear. Coincidentally after Windows did an auto-update -- and I suspect auto re-boot during a long-running backup process. (BTW: data was lost.)

Or how my own out-facing network address made it to the spam blocker list which in turn caused my own outbound email to sporadically fail with strange messages! Thanks girls... glad you found those wonderful trojan viruses that were spamming from one of your machines! I made the firewall and virus protection tighter.

Now on to migrate over a desktop PC to an iMac -- xfer email address book, personal files, etc. Should be loads of fun. After all, I just made a first pass at exporting Address Book from Outlook and it said the Export feature was not installed. Yikes, I'll have to break out the original install disks.

I'll let you know how it goes!

Homage to Momofuku Ando

So, what might you have eaten for lunch? How about during college? Or late-night coding sessions? I still eat Ramen noodles!

Momofuku Ando ring a bell? Didn't with me either. How about "Ramen Noodles?"

At the age of 96, Momofuku died this past Friday (6 Jan 07). And to think, the day before he had Chicken Ramen for lunch with Nissin employees!

So, next time you grab that 3-minute meal while you are waiting for a compile or contemplating that next great solution, say a short prayer of thanks to Momofuku and send best wishes to his family.

Simple is Hard

Three passes at building as simple of a system as we could to allow for a dynamic JSP based on a 2-dimensional menu matrix... Tabs across the top, selecting one and you get various left nav tabs. Select a left nav and you get a list of those objects and the data for the first one in the list. The idea was to maintain the menu system in a database table to allow ease of change. First implementation was very bulky and expensive -- too many files!

Many late nights... 103 deg fever for Lee. After second failed attempt, Lee says "I'm out of ideas."

That scared me... Lee out of ideas? He's one of the best we have! He's one of "our guys." You know the kind. Ask him anything, and he can offer help or a solution or just get it done!

I kept trying to bring the solution back to simplesville. So the next morning I set out with pen and paper and went back to the drawing board. We have a two-pronged object model, with a 1..* between the two top core objects. Then simply other "stuff" hanging off of those root objects via associations. The menu system is merely a way to ease the navigation through the various parts of the system.

Turns out Lee was also mulling it over during the day. At about 4 o'clock, we chatted on the phone. "You go first," he said. So I explained my approach. We were on the same wavelength. At about 8 o'clock, we meet in IRC chat a bit. I go off to do some other parts of the app, Lee starts in on the ideas we had discussed.

A few hours later, he's got it working! One small slice, mind you... But Lee cracked the puzzle! We're using older EJB and struts 1.0, so we were not able to iterate over the bean to automate the JSPs.

Got it down to a pattern for {1 EJB, 1 form bean, one JSP, and one action} per class. And all CRUD functionality is possible.

Third time's a charm, as they say (whoever they are -- let me know if you know).

Roundtrip - 2

Paul posits:

Logically speaking, I can't see why, if any tool were *really* good at forward-engineering, one would ever want to reverse-engineer. The "round-trip" tools I know are there because they do a good job of organising and re-organising the gross structure of the code as models, but can't do anything with the fine detail of the code.

Scott Ambler tossed out the case where folks that are visually oriented would appreciate reverse engineering the code [into models].

I added: one might want to use reverse engineering of
machine-generated source to:

• see what the code looks like in a different view(s).
• see code added manually outside of generator
• check for compliance with intentions of the fwd engineering tool

Paul's reply tends to lead me to think that Paul is making his points based on an assumption that the "(near) perfect" tool to go from some higher form (modeling+?) to source code is in existence. That this tool automagically forward engineers perfect code -- hence why would you need to reverse engineer the code? An analogous situation to source code being forward engineered into assembly code.

I'm no computer scientist, but someone help me out here. I would surmise the leap from source code to assembly code being one of nearly a mathematical proportionality. That is, the source code -- being of a higher order language -- is merely a more compact form of an equivalent expression in assembly. Sort of like multiplication is a shorthand notation for lots of addition statements.

The breakdown occurs as the model-to-source code paradigm is considered. Paul's view may be one of the "Holy Grail" of modeling tools that can capture any type of application and all of its complexity at some level higher than simple source code, yet be able to generate the requisite code to have the application run.

I think we have a long way to go before this tool crosses our computers!

Roundtripping

And I don't mean with LSD... In a post, Juha-Pekka postulates the following:

• "My personal opinion is that roundtripping is a recipe for trouble. "
• "Still I need to ask what is the value of having a rectangle symbol in a digram and an equivalent class in a file."

When I first laid eyes on an ad for Together/C++ in 1994 that promised code and diagram were always in sync, I spent $1200 of my own money and bought it sight unseen.

Not having accurate diagrams that represented the code was a problem... Though I had powerpoint templates for Object Modeling and excel spreadsheet templates for sequence diagrams, these always seemed to get out of sync with the code after a while. This is quite likely worse than having no diagrams at all! (Kind of like a map with an error might be worse than no map at all... depending if the road now goes through a man-made lake.)

Think of round-tripping a class diagram as follows:

• edit the code, the diagram automatically reflects the change.
• edit the diagram and the code is immediately updated.

Sounds like a visual editor to me... always in-sync diagram and code.

Much like we represent bytecode and assembly with higher-order languages (i.e., things like source code), another level up of representation of the same thing can be useful.

At least I found in-sync round-tripping useful for the past dozen years. Others' experience may vary.

Is it Friday?

My friend Danny pointed out two rather humorous links that I am sure you all will find valuable!

• Is It 5:30?
• Is it Friday?

Productivity

A given software problem can be solved in more than one way.

Suffice to say, Developer A might require 5 times as much effort as Developer B. To add to that statistic, the design and code from Developer A might be 5 times the LOC, and much worse quality (maybe 5 times the complexity) -- basically more expensive to maintain. Might as well add one more problem: the code from Developer A exhibited brittleness when attempting to extend the functionality.

However, both A and B products provide the desired functionality.

A project team or a company likely has both A and B type developers, but rarely has the opportunity to compare parallel solutions to the same problem.

How do you go about trying to get the best solution for your money at your company?

Defect Testing

I filled in to teach Professor Meketon's CSC class at Villanova University. We used a book by Ian Sommerville, titled “Software Engineering, 7ed,” and I was given a set of slides to run through on the topic of "Verification & Validation" and "Software Testing."

Maybe someone can help explain what the author means by these definitions, because i am confused a bit:

Defect Testing

• A successful defect test is one which reveals the presence of defects in a system.
• A successful test is a test that makes the system perform incorrectly and so exposes a defect in the system.

I think it may be semantics... I would generally go for tests that expose correct behavior of a system (e.g., a feature is working properly). Even if a test was to ensure that an out-of-bounds condition was properly handled by throwing an exception, that would be a "positive" and not a defect.

I can see the "Acceptance Test" or even "Unit Tests" exposing "defects" while the code is under development. For example, in test-first development, the test will fail the first time you run it by design. Or if you write some code that breaks a unit test... Is this defect testing as the author intends it to be?

I think I need someone to crack me over the head with a rubber mallet so that the author's sense of "Defect Testing" makes sense to me.

'cuz I don't get it.

clarity of purpose

Paul makes a nice comment here regarding whether the "build vs. buy" simple example scales. His example revolves around less clear requirements for the "widget."

"We still don't know what we want, but now we don't know how to get this package to do it for us either."

Of course, one of the problems in this example is not knowing what is needed. My example of the millisecond resolution timer software package was based around very clear requirements and a very clear "domain of requirements" (I just made that up, but it sounds good). That is, not only did the commercial off-the-shelf (COTS) package meet my precise needs, it also supplied other features useful to me. Features like self-calibration for improving accuracy.

When the purpose isn't clear, it can be difficult to find a package that meets your needs (or find too many). In addition, it can also be a case where you end up looking for a way to employ whiz-bang features just because they came in your COTS package.

build or buy?

I was IMing with a friend in the Dominican Republic tonight. He mentioned how great it is with the open source community to be able to get various products to solve problems.

"In the good ol' days," I jokingly opined, "you had to buy everything."

"Or write it yourself," he added.

"Key is to know what to spend time and resources on for a given project," I noted.

Which brings me to a story. Years back, maybe late 80s, early 90s, I did a lot of real-time work on PC programs (in addition to real-time apps on supercomputers). The client I was moonlighting for needed a millisecond resolution timer to record time between image display and test subject making a keystroke. All to support Dr. John D's research into human responses to visual stimuli.

So, I found some great assembly code that I could write to diddle with the 8250 Timer Chip (if my memory serves me correctly -- and if it does, that is scary). Got everything working perfectly, shipped the application, grad students ran all sorts of tests using the software. Perfect.

About two weeks after releasing my application, I spotted a commercially available millisecond resolution timer package from Ryle Design (again, IIRC). Something like $49. I bought it immediately.

I couldn't rip out my custom code fast enough and put in this new package. After all, unlike my timer code, this package:

• Was supported
• Had documentation
• Had more features (useful ones too)
• Would get the benefit if many users forcing improvements, bug fixes
• Would have future enhancements
• Was code I didn't have to maintain!

Think about this the next time you choose to build versus buy -- or in today's modern computing landscape -- use open source...

"test first" the data too!

no, this isn't rocket science... just a gentle reminder about why some practices are valuable.

now i am reminded why i had been pushing a team to get the "golden data" identified for a project 6 months ago. By "Golden Data" I mean that data which has been carefully selected -- or hand crafted -- to reveal elements of the application's functionality. Though the novel technology for this project is a challenge and is by no means trivial, the data itself has been a big struggle as well, mostly due to inherent complexities in gnarly (a technical term ) health care systems' actual databases.

you see, the application involves grabbing data from n-many sources, some of which may hold duplicates of other sources. in addition, there are certain key values that are deemed "sensitive" and must be handled properly so that only authorized folks can see the sensitive information.

so, there is a combination of code and rules and presentation magic, etc, that all have to jive. however, there also must be the proper data with which to reveal the intended logic. the code could be right, the rules could be right, but if the data is not right, nothing shows up. then starts the wild goose chase... is it the code? is it the rules? is it the presentation logic? or is it the data? or all of the above?

next time, i'll be sure to emphasize that we have the precise knowledge of the data side of the equation to prove the existence -- or not -- of a feature.

Note to Self: write the test from the data side -- not just code!

Rest In Peace, Rye

this evening, rye -- one of two guinea pigs -- seemed to be lethargic. we braced the girls for the ultimate likelihood... after all, gp's don't live too long.

i checked on rye earlier tonight. he was no better. i picked him up and cradled him in my arms. i used a dropper and gave him some water since they always drink us out of house and home. i stroked his fur, i looked in his eyes. he was shivering and i figured he wasn't long for this world. i carefully returned him to a nice spot in his cage with some fresh pine shavings. i laid him down and made him comfortable, with his back against a wooden thing we have in the cage. his piggy brother, Poppy, seemed mildly aware of the distress.

just about an hour ago, i checked again. he had passed on to the other side. he is now where the grass is always green, carrots are forever showing up on the carpet of green grass, and a babbling brook brings fresh water by for all time. (well, actually, he is now in an amazon box, wrapped in a little towel, as i figure the girls will want to say goodbye.)

rye, we will miss your little piggy legs elevating your hot dog physique across our kitchen floor. we will miss your incessant audible reminders that your water bottle is empty. and we will miss your squeaks when we peel carrots or chop vegetables.

rest in peace, rye

pre demo day

today it turned out that we were able to get a demo ready for a preview by the business folks. a lot of visual tweaks, but some functionality as well.

the team is

• spread across continents -- which is challenging at times for sure,
• involves 6 contractors, and
• includes 2 developers from client

today things clicked when needed. our tools:

• IRC chat channel - for coordinating and discussion
• IM – for quick non-team chats
• dev server – where app ultimately has to run for the demo
• handy shell scripts – to make it easy to do common tasks
• jira issue tracker – to keep our team on task and to not step on each other’s toes
• cvs – allowed for changes – and backing out changes
• mysql – easy-to-use workhorse dbms
• resin - app server

'Jon, can you change the layout of fields x,y,z on form A to ...' (the application is meta-data driven to allow for dynamic search and report forms.)

okay. I do a cvs update, tap, tap, tap. check-in. over to the terminal window (on a mac) that is shelled into the server, I do an update to get the latest changes; mdinsert to run a bunch of sql scripts to update the database, and back to testing the app.

It was a good day. Client was pleased. Tomorrow some more tweaks, and maybe the president gets to see our progress :-)

acceptance tests

Acceptance tests are a way to ensure the user functionality exists in the product. Enough test cases are put in to not only prove the feature is there, but that the feature is correct and robust.

I first ran across the term "Acceptance Test" as it had to do with testing cruise missile jet engines. If the engines passed the Acceptance Test, then they were literally gaining a stamp of approval. The tests ensured, primarily, that the core "feature" of thrust and specific fuel consumption were met.

These were two "end user" features critical to the success of the mission -- being able to fly far enough and to have enough power to maneuver.

Unit tests, by contrast are more about the sub-components meeting spec. The fuel control needed various temperature and pressure input signals, +/- 5 volts. However, the end user could care less about these low-level details.

So be sure to think about writing those acceptance tests from the end user perspective!

Another benefit is the ability to have automatic traceability between the feature and the code it touches based on the acceptance tests.

knowing when a feature is clear

Here is the way I like to think about a feature:

Small, client-valued bit of functionality. For example, "Hide data from non-admin user view if the data represents a 'sensitive value.'"

If the paragraph of text describing the feature is clear enough:

• the developer can estimate the time required to implement the feature.
• the tester can write an acceptance test to prove the feature exists and is working properly.

If the estimate or the test can't be made due to lack of detail, then ask for more explanation about the feature.

Complexity

The urge to fight complexity is strong in me... though on my way to a simple solution i often wade through quite a bit of self-created muck.

however, for a long-term solution -- be it an application you are building, or a process you are trying to implement -- you should take the time necessary to seek the simpler solution.

remember, (almost) anyone can build a complex solution and over think the problem. it is far more valuable to provide the more cost-effective solution up front that seeks to simplify and provide a high return on investment.

the need for slack

unless you are pulling something heavy, slack is a good thing.

why?

because to try and do something new in your development team will require that you have the freedom and slack time to explore, to learn, to practice.

whether this is for a new process, a new tool, or a new technology... without slack, you are more likely to fail.

talking improves app performance!

so there we were... watching the system running. we were on the phone, both tailing the app server log file (tail -f ... ) to watch some progress via the age-old, tried and true, logging statements.

the app is doing a lot... running through more than one data source, dynamically generating sql based on meta data and an ontology, dumping it to a triplestore, running rules (invoking some gnarly stuff), culminating in a report.

it was streaming past at a good clip, then you could see it bog down at one point where there was an especially nasty bit of stuff going down for accomplishing a bunch of rule stuff. Then it began moving again at a good clip. There was definitely a visible "rhythm" just watching each part of the system crank through the dozen or so "sections."

"I'm pretty sure that is where it is doing the 'saveToStore()' call." I thought it was instrumented with some logging... Hmmm. "Wait a minute" my friend said. "Those rule forms... we don't need to put any of that info into the triplestore since we aren't displaying any of that info. The triplestore is merely for running SPARQL queries against to get the info out to the browser."

"Good point!" I said. So he added one line of code to disable calling saveToStore in the case of no SPARQL being specified. That single change resulted in reducing the time by 70%.

Funny what can happen when a couple of people are almost casually talking through and re-thinking the system behavior. thanks lb

the foggy bits

sometimes, in challenging development projects, some of the solutions are out there... you can see them through the mist, or the rolling fog. you know intuitively what the solution should look like. but the implementation can be tricky.

it is worth pushing onward, through the fog, trying to burn it off.

why? because in all likelihood, it is the hard bits of an innovative project that really provide the value to the client.

Estimate by the Season

Just listened to a wonderful keynote by Timothy Lister at GLSEC conference.. A man after my own heart with his ability to say the things many people don't want to hear about software projects.

This was the first quote I jotted down:
Estimate software projects by the season

It came from his mathemetician father who was reviewing some stats that Tim and Tom DeMarco had gathered on initial estimates for software projects and actual results. His father summarized the granularity of estimates could not be any finer than seasonal!

Brilliant!

can you code this?

I was doing tech support for a good friend... seems an email message in Outlook Express would cause the app to freeze. Move to it, and BAM. So, deleting it outright was out of the question. Even archiving the email would cause it to freeze. Mind you, other messages from this sender were fine.

So I thought of a clever way. I created a rule for the sender and the subject, and selected the action to delete the message. Told the app to execute the rule. Guess what? Yeah, it froze again. With the blank outline of the rule execution progress dialog up in all its blank-stare glory.

My response as a professional software person? Besides embarrassment for our entire industry, I said "I don't think I could code this as a feature if I had to." (Of course, we all know we could do something clever like "If message number 10959 arrives, freeze app's UI message handler."

So, could you code this behavior? Have you ever found yourself expressing such wonderment at amazingly bizarre program behavior?

Mac Conversion

I am enjoying working on my new MacBookPro after nothing but windows machines, a half dozen dell laptops, etc.

What is really cool, besides stuff just working, is the ease with which I can do development work. The Mac OSX makes it seamless to switch back and forth with the linux server. Most useful!

Getting back on the windows laptop is still familiar territory, but I already find my fingers in the wrong places.

Yeah, the Mac's weirdness surrounding Control, Option, and Command modifier keys, could stand to have someone bonked on the head with a rubber mallet .

Deja Vu

A good friend of mine, Chris Lank, from TogetherSoft days, is now the CEO of Ivis, proud creators of xProcess. He's piqued my interest lately (well, not *that* way)!

This looks very much along the lines of a process/workflow dream/vision i have held for years (and implemented independently from them to a beta stage). Ever since working on developing IBM's (as of 1995) next-gen Manufacturing Execution System, ideas have swirled around how to break the log-jam of classic project management solutions for processes that don't lend themselves well to simple Gantt charts.

I'll have to poke around more on their site and talk to my old TS pals that are there. This tutorial looks like a good start...

Great Lakes Software Excellence Conference

The GLSEC is being held in Grand Rapids, MI, on 25 & 26th October.

I am scheduled to deliver a talk on "Pragmatic Agile Model-Driven Architecture."

Carl Erickson figures it just has to be an oxymoron!

Welcome Back

As many of you may have guessed, I decided to go off and do my own thing. Compuware was fun and I really enjoyed the folks I got to work with on a daily basis.

So now i will work to help teams be more pragmatic; consult on architecture, work with project teams, help with implementation, mentor on lightweight development processes, etc. Basically, figure out the next thing i want to be when i grow up ;=) I even have a novel that i want to write!

My blogs from the past year can still be found here

I'm going to resume blogging, thanks for your patience ;=)

-- jon kern

ps: thanks to the little butterfly that kept waiting for my next blog entry