Iterative Design

by Dr. Bob Bailey

May, 2002

Almost ten years ago, Skip Bailey (Bailey, 1993) published the results of his doctorial dissertation, where he had tracked user performance through a series of 3 or 4 iterations. Each of his eight test participants created a software system, and then made improvements to the system over the next several months. Each person made changes to their system after watching (on videotape) three participants individually try to use their system to perform a series of tasks. Over the course of his study, he reported many usability improvements, including an overall 26% improvement in performance time.

This was one of the first reported studies to show convincing evidence that "iterative design" methodologies were valid. If we have usability improvements with each iteration, is this evidence that "usability testing" works? I believe the answer is "Yes." We have many studies showing that each iteration does help to improve the usability of a system—even if the improvements are only modest ones (which is usually the case).

Wei-Siong Tan, Dahai Liu and R. Bishu from the University of Nebraska in Lincoln and A. Muralidhar and J. Meyer from Ajenda Interactive Media in Chicago reported a study that provides another estimate of the percentage improvement in speed with each iteration.

They had 15 representative users complete 5 scenarios while using a preliminary version of a commercial Web site. The scenarios were created to represent tasks that were typical of those encountered in real life, and to reflect the major functionality of the site. The performance test identified a total of 49 usability problems. They attempted to eliminate all the problems "by applying human factors knowledge." After doing their best to eliminate the problems, they then conducted a retest by having 20 representative users complete the same 5 scenarios.

The changes to the Web site resulted in 28% faster average completion times for the 5 scenarios. However, about 45% of the problems "found and fixed" as a result of the first test, showed up in the retest. In addition, they identified 9 new usability problems that were created when fixing previously identified problems. Overall, they reported only a 37% reduction in the number of usability problems.

Using their results, I figured that if they had the same level of improvement with each iteration, it would take at least six tests and 105 participants to detect and correct 95% of their usability problems.

The iteration-related questions that have not yet been answered are:

References

Bailey, G. (1993), Iterative methodology and designer training in human-computer interface design, INTERCHI '93, 198-205.

Tan, W., Liu, D., Bishu, R. R., Muralidhar, A. and Meyer, J. (2001), Design improvements through user testing, Proceedings of the Human Factors and Ergonomics Society 45th Annual Meeting, 1181-1185.