1. People were better able to carry out two tasks simultaneously (repeating spoken words and learning new words) if each task involved a different modality (visual vs. auditory).
2. People were presented with a verbal description of a layout that was sufficiently complex to be unintelligible unless visualized. The description was presented either in auditory form or simultaneously in auditory and written form. The auditory-alone mode resulted in superior performance.
3. Two groups of children were asked to either listen to or read a story. Half of each group were instructed to visualize the story while it was being presented. On a test given after the story, the "visualizers" performed better, but only those that listened to, not those that read the story.

These findings suggest that in certain, complex situations working memory capacity can be "increased" by using two senses rather than one. For example, performance can be substantially degraded when people must attend to multiple sources of information that must be mentally integrated before meaning can be derived. Thus, designers should present information to users in ways that reduces the need for mental integration, and consequently reduces the demands on working memory.

Tindall-Ford, et.al., conducted three experiments using electrical trade apprentices. In the first study, one group learned by using a diagram and separated written text, a second group used a diagram and integrated written text, and a final group used a diagram and auditory instructions. The latter two groups performed reliably better because their working memory resources were not exceeded.

In the second study they evaluated user performance on a complex task when using:

(a) a table and related text, versus
(b) a table with an auditory explanation.

The visual-audio group performed reliably better; again, because of the reduced load on working memory.

In the third study, they had participants either look at diagrams and read instructional materials (visual-only) or look at diagrams while listening to instructional material (audio-visual). They performed two easy tasks and one difficult task. In the easy tasks there were no differences between visual-only and visual-audio. In the difficult task, the participants using two senses (vision and hearing) performed reliably better.

This article makes a strong case for having designers take the time to physically integrate information in computer systems, i.e., put all required information within close proximity. When this is not possible, and when the task is complex, working memory capacity can be extended by presenting information using both visual and auditory modes.

 

Multimedia in Instruction

Williams (1998) reviewed the literature on using multimedia in instruction. He extracted numerous guidelines on the effective use of multimedia after reviewing about 100 literature sources. One of his many discussions was a section on using combined visual and verbal information.

In general, the past research seems to indicate that combining visual and verbal (auditory) information can lead to enhanced comprehension, when compared to their use alone (see the Tindall- Ford, et.al., article discussed above). But designers also should be aware that having both visual and audio modes may result in no performance improvements (if the task is too simple), and may or may not increase user satisfaction.

Some guidelines:

• Past research suggests that visual and narrative information should be presented simultaneously, or the visuals should precede the narrative by no more than seven seconds.
• Both the visual and auditory information should be totally relevant to the task being performed.
• When words are spoken, the content should be simple, and the speed of narration should be about 160 words per minute. The narration should be slowed when used to introduce new ideas or concepts.

Off-screen narration should be used rather than on-screen narration, unless the narrator is a recognized authority on the topic.

 

References

Tindall-Ford, S., Chandler, P. and Sweller, J., When Two Sensory Modes are Better than One, Journal of Experimental Psychology: Applied, 3(4), 257-287 (1997).

Williams, J. R., Guidelines for the Use of Multimedia in Instruction, Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting, 1447-1451 (1998).