Evaluating Software - Avoiding the Elephant Syndrome

By Dave Kreimer, Principal of Next Step Consulting

You are probably familiar with the parable of the blind men who each assert their limited perception of the qualities of an elephant based on the single body part of the animal that they touch. Generating meaningful qualitative data about a software application can present similar challenges. A current, competitive software product is likely to have such an abundance of features and information that an individual can easily form a limited or inaccurate impression of the product. This article offers four suggestions to avoid the “elephant syndrome” when testing software products.

Be sure your product is reliable and complete enough for testing. To deliver meaningful feedback, participants require a test program that runs well enough to accurately represent its content and navigational conventions. Participants can ignore a fair number of problems commonly caused by Alpha versions of software, especially when they are properly warned. However, it may be difficult to get meaningful feedback from frustrated participants who spend too much time dealing with computer crashes or other technical problems. Alpha version test programs do not need to be complete in terms of content, but should demonstrate at least one level of information completely. For example, earlier this year I successfully tested a title on Home Repair when only the Plumbing chapter was complete. The numerous other incomplete chapters were not functional, but were represented by icons to make participants aware of the product’s scope.

Provide adequate time for participants to explore the program. The research objectives define how much exploration time participants require. Concept testing can be accomplished with quick exposure to the test program using a demonstration monitor in the focus group room. However, a thorough overview of a multimedia title requires that participants spend at least two hours exploring the program. I find participants are best prepared when they test products in their homes prior to the focus groups. Using a computer lab that also has a room suitable for focus groups is another effective method of exposing participants to test software.

Guide participants through all critical elements of the program. To avoid the “elephant syndrome” mentioned at the beginning of this article, provide participants with a worksheet or checklist that ensures a thorough exploration of the product. I often design worksheets that follow the flow of the discussion guide. As we progress through the focus group, I occasionally encourage participants to refer to their recorded comments.

Focus participant feedback on the information that will direct your next steps with the product. Avoid the pitfall of rushing through a two-hour group attempting to cover every feature of the program. Instead, focus your attention on the highest-priority issues. Two hours is usually enough time to explore participants' overall reaction to the user-interface and the information in the program, as well as several other important issues.

The sophistication of the software industry and computer users is growing at an incredibly rapid pace. Numerous clients have recently reported that the value derived from conducting market research on new software products has convinced them that it is a necessity for all future products. Next Step Consulting is committed to applying these principles, and others gleaned from experience, to assist clients in developing successful software products.

© Next Step Consulting